egarim / syncframework

SyncFramework

is c# a library that helps you synchronize any type of data, using delta encoding technology

There are currently 2 implementations of the SyncFramework

• SyncFramework for XPO
• SyncFramework for Entity Framework Core

in a nutshell

Data synchronization in EFCore is accomplished by replacing the DbContext's internal service collection with a custom one, containing the SyncFramework services.

The SyncFramework services are registered in the service collection using the extension method, AddEfSynchronization.

SyncFramework replaces the BatchExecutor service with a custom one, capable of intercepting the generation of SQL commands (Deltas), and storing them so they can be executed in a remote database.

Deltas are typically generated for the same database engine in use, but they can also be generated for a different database. For instance, your main database could be MS SQL Server, but your remote database could be PostgreSQL.

If you want to learn more about data synchronization you can checkout the following blog posts:

1. Data synchronization in a few words - https://www.jocheojeda.com/2021/10/10/data-synchronization-in-a-few-words/
2. Parts of a Synchronization Framework - https://www.jocheojeda.com/2021/10/10/parts-of-a-synchronization-framework/
3. Let’s write a Synchronization Framework in C# - https://www.jocheojeda.com/2021/10/11/lets-write-a-synchronization-framework-in-c/
4. Synchronization Framework Base Classes - https://www.jocheojeda.com/2021/10/12/synchronization-framework-base-classes/
5. Planning the first implementation - https://www.jocheojeda.com/2021/10/12/planning-the-first-implementation/
6. Testing the first implementation - https://youtu.be/l2-yPlExSrg
7. Adding network support - https://www.jocheojeda.com/2021/10/17/syncframework-adding-network-support/

Current version 7.0.X

Target Framework net6.0

• EfCore Version 7.0.3
• Postgres Version 7.0.3
• Pomelo Mysql Version 7.0.0
• Sqlite Version 7.0.3
• Sql Server Version 7.0.3

Nugets

• Common base classes and interfaces to implement delta synchronization https://www.nuget.org/packages/BIT.Data.Sync/
• EfCore Implementation https://www.nuget.org/packages/BIT.Data.Sync.EfCore/
• EfCore Implementation for postgres https://www.nuget.org/packages/BIT.Data.Sync.EfCore.Npgsql/
• EfCore Implementation for Mysql/MariaDB using Pomelo provider https://www.nuget.org/packages/BIT.Data.Sync.EfCore.Pomelo.MySql/
• EfCore Implementation for Sqlite https://www.nuget.org/packages/BIT.Data.Sync.EfCore.Sqlite/
• EfCore Implementation for MsSqlServer https://www.nuget.org/packages/BIT.Data.Sync.EfCore.SqlServer/

Playground

Chcke our new playground demo here

https://syncframework.jocheojeda.com/

Getting started

The first step is to install the Nuget package that matches your current database, for example if you are using MsSqlServer you should install the following Nuget package

BIT.Data.Sync.EfCore.SqlServer

If you want to use the SyncFramework with a different database engine, you should install the corresponding Nuget package.

Let's prepare the services needed for database synchronization

Service collection

//The options needed for the DbContext, you can build them in any way you want, in this case we are using Sqlite
DbContextOptionsBuilder OptionsBuilder = new DbContextOptionsBuilder();
OptionsBuilder.UseSqlite(ConnectionString);

//The http client is used to send the deltas to the server
 HttpClient Client = new HttpClient();
 Client.BaseAddress = new Uri("https://ReplaceWithYourServerUrl/");

 //The DeltaGenerator is used to generate the deltas, you can add as many as you want, in this case we are using the Sqlite and SqlServer generators
 List<DeltaGeneratorBase> DeltaGenerators = new List<DeltaGeneratorBase>();
 DeltaGenerators.Add(new SqliteDeltaGenerator());
 DeltaGenerators.Add(new SqlServerDeltaGenerator());
 DeltaGeneratorBase[] additionalDeltaGenerators = DeltaGenerators.ToArray();

//We prepare the service collection that will be used to register the SyncFramework services
ServiceCollection ServiceCollection = new ServiceCollection();
ServiceCollection.AddEfSynchronization((options) =>
{
    string ConnectionString $"Data Source=Deltas.db;";
    //The EfSynchronizationOptions is used to configure the database synchronization
    options.UseSqlite(ConnectionString);// we are going to store the deltas in a Sqlite database
    //it's possible to store deltas any any of the supported database engines, for example you could use MsSqlServer
    //options.UseSqlServer(ConnectionString);
},
//The http client that will send the data to the server
Client, 
//the ID of the delta store on the server side,a sync server can have multiple delta stores
"MemoryDeltaStore1",
//the ID of this client, each client has to have an unique id
Maui",
//Additional delta generators, in this case we are using the Sqlite and SqlServer generators
additionalDeltaGenerators);

//We add the entity framework services, in this case we are using Sqlite
ServiceCollection.AddEntityFrameworkSqlite();

//We build the service provider
var ServiceProvider = ServiceCollection.BuildServiceProvider();

The DbContext

You should use SyncFrameworkDbContext instead of the regular DbContext. SyncFrameworkDbContext is a subclass of the regular DbContext. The only difference is that it implements all the boilerplate code needed for synchronization, so you can use it in the same way.

MyDbContext MyDbContext = new MyDbContext(OptionsBuilder.Options, ServiceProvider);
MyDbContext.Database.EnsureCreated();

If you don't want to use SyncFrameworkDbContext you can extend your own DbContext by implementing ISyncClientNode interface.

public class MyAppDbContext : DbContext, ISyncClientNode

Then, you need to create a new instance of the delta processor and the services provider that we built in previous steps. Let's do that in the constructor of your DbContext.

public MyAppDbContext(DbContextOptions options,IServiceProvider serviceProvider) : base(options)
{
    this.serviceProvider = serviceProvider;
    this.DeltaProcessor = new EFDeltaProcessor(this);
} 

Then we need to replaces the service provider with our own, let's do that by override or extend the configuration method as follows

protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder)
{
    this.Identity = serviceProvider.GetService<ISyncIdentityService>()?.Identity;
    this.DeltaStore = serviceProvider.GetService<IDeltaStore>();
    this.SyncFrameworkClient = serviceProvider.GetService<ISyncFrameworkClient>();
    optionsBuilder.UseInternalServiceProvider(serviceProvider);
}

Our DbContext is able to store and process deltas and push and pull deltas from a SyncServer.

We are done!!!

Fetching pulling and pushing deltas

The operations of fetch, pull and push are generic and they are implemented as extension methods of in the ISyncClientNode interface.

• Fetch:Fetches the deltas from the server and returns them as a list of Delta objects
• Pull: Fetches the deltas from the server, process them by execute the sql operations in your local database
• Push: Pushes the deltas to the server

await MyDbContext.PushAsync();

await MyDbContext.PullAsync();

await MyDbContext.FetchAsync();

Best practices using your DbConext to push and pull deltas

You should use your DbContext withing a using statement, this will ensure that the DbContext is disposed and the connection is closed, also that the http client is not dissposed (this can happend in platforms like MAUI).

using (var context = GetDbContext())
{
  // use your conext here
}

Implementing a SyncServer

The easiest way to implement a server is to add register a instace of SyncServer in the service collection.

public void ConfigureServices(IServiceCollection services)
{
    SyncServerNode syncServerNode = new SyncServerNode(new MemoryDeltaStore(), null, "MemoryDeltaStore1");
    services.AddSingleton<ISyncServer>(new BIT.Data.Sync.Server.SyncServer(syncServerNode));
}

and use that service in your controller

public class SyncController : ControllerBase
{


    private readonly ILogger<SyncController> _logger;
    private readonly ISyncServer _SyncServer;
    protected string GetHeader(string HeaderName)
    {
        Microsoft.Extensions.Primitives.StringValues stringValues = HttpContext.Request.Headers[HeaderName];
        return stringValues;
    }


    public SyncController(ILogger<SyncController> logger, ISyncServer SyncServer)
    {
        _logger = logger;
        _SyncServer = SyncServer;
    }
    [HttpPost(nameof(Push))]
    public virtual async Task Push()
    {

        string NodeId = GetHeader("NodeId");
        var stream = new StreamReader(this.Request.Body);
        var body = await stream.ReadToEndAsync();
        if (string.IsNullOrEmpty(body))
        return;
        using (var ms = new MemoryStream(Encoding.Unicode.GetBytes(body)))
        {
              
            DataContractJsonSerializer deserializer = new DataContractJsonSerializer(typeof(List<Delta>));
            List<Delta> Deltas = (List<Delta>)deserializer.ReadObject(ms);
            await _SyncServer.SaveDeltasAsync(NodeId, Deltas, new CancellationToken());
            var Message = $"Push to node:{NodeId}{Environment.NewLine}Deltas Received:{Deltas.Count}{Environment.NewLine}Identity:{Deltas.FirstOrDefault()?.Identity}";
            _logger.LogInformation(Message);
            Debug.WriteLine(Message);
              
        }
    }
    [HttpGet("Fetch")]
    public async Task<string> Fetch(Guid startindex, string identity = null)
    {
        string NodeId = GetHeader("NodeId");
        var Message = $"Fetch from node:{NodeId}{Environment.NewLine}Start delta index:{startindex}{Environment.NewLine}Client identity:{identity}";
        _logger.LogInformation(Message);
        Debug.WriteLine(Message);
        IEnumerable<IDelta> enumerable;
        if (string.IsNullOrEmpty(identity))
            enumerable = await _SyncServer.GetDeltasAsync(NodeId, startindex, new CancellationToken());
        else
            enumerable = await _SyncServer.GetDeltasFromOtherNodes(NodeId, startindex, identity, new CancellationToken());
        List<Delta> toserialzie = new List<Delta>();
        var knowTypes = new List<Type>() { typeof(DateTimeOffset) };

        foreach (IDelta delta in enumerable)
        {
            toserialzie.Add(new Delta(delta));
        }
        DataContractJsonSerializer js = new DataContractJsonSerializer(typeof(List<Delta>), knowTypes);
        MemoryStream msObj = new MemoryStream();
        js.WriteObject(msObj, toserialzie);
        msObj.Position = 0;
        StreamReader sr = new StreamReader(msObj);
        string jsonDeltas = sr.ReadToEnd();
        return jsonDeltas;

    }

}

Installing project templates

dotnet new -i BIT.Data.Sync.Templates

To create a new SyncServer

dotnet new SyncServer -o MySyncServer

Changelog

You can see the current list of changes here

syncframework's Issues

Implement delta processor for Entity framework Core

A delta processor is needed to fulfil the basic implementation of the SyncFramework

Clean up SyncFrameworkDbContext

This property is not needed anymore so it can be removed from the class and from the configuration

public IModificationCommandToCommandDataService IEFSyncFrameworkService { get; private set; }

Monolithic Client DB with Microservices DB syncing

Dear @egarim,

How can I achieve a single monolithic database syncing with the microservice databases?

The client database has entities that have tables of different databases. The server service has a specific purpose database. For example, User Management Service/API, PoS API, HR API, etc., have databases with a particular purpose. Is there a possibility that I can generate Deltas based on configurations? If you have any Ideas or solutions, please share them.

I appreciate your help
Thank you,
Qamar Abbas

Create SyncFramework Nugets

update the projects to be ready to publish the nuget packages

Allow EfCore Sync providers to turn on/off delta generation by implementing IEnableDeltaTracking

Allow EfCore Sync providers to turn on/off delta generation by implementing IEnableDeltaTracking

Create a WebAssembly playground

Create a WebAssembly playground

Move all extensions to the namespace Microsoft.Extensions.DependencyInjection

Move all extensions to the namespace Microsoft.Extensions.DependencyInjection

Create SynServer Project Template

Create SynServer Project Template for the DotNet CLI and visual studio

Add a method to the sync server to add nodes on runtime

Add a method to the sync server to add nodes on runtime

Create extensions to create a Fake HttpHandler to make possible handle push and pull request without network communication

Create extensions to create a Fake HttpHandler to make possible handle push and pull request without network communication

Add missing constructors to SyncFrameworkDbContext

Add missing constructors to SyncFrameworkDbContext, the should match the constructors of a Regular DbContext

Implement testing project for Entity Framework implementation

Implement testing project for Entity Framework implementation

Implement a DataStore for Entity Framework

A datastore is needed to fulfill the basic implementation of the SyncFramework

Create delta store implementation using XPO

Create delta store implementation using XPO

Create SyncFramework for EfCore Nugets

update the projects to be ready to publish the nuget packages

Encapsulate push and fetch request logic so I can be reused without the SyncControllerBase

Encapsulate push and fetch request logic so I can be reused without the SyncControllerBase

Add an exception when the server node is not found

Add an exception when the server node is not found, currently you can send deltas to the server and if there is no server node that matches, it just ignores the request the correct behavior should be tell the node that the operation was not successful

The nuget for BIT.Data.Sync.EfCore.Npgsql is not being generated

The nuget for BIT.Data.Sync.EfCore.Npgsql is not being generated

DeltaStore cannot be share b/w multiple client nodes

Hello @egarim,

Do we have a possibility to use the same delta store for multiple client databases? It can be used for multiple client databases if we update the EFSyncStatus table with another column called Identity. I think this could fix the issue. I would like your thoughts.

Thank you,
Qamar Abbas

Create a DbContext class that implements the ISyncClientNode

The most common scenario for EFCore is to synchronize DbContext that are connected to different databases, so the DbContext is the perfect class to implement ISyncClientNode

Delta index is not sortable

With the current implementation in some cases deltas can't be sorted in the same way they were generated this leads to sometimes skip some deltas in the sync process

New client node data pull from server

Dear @egarim

I appreciate your hard work. I found it an excellent sync library. I have seen all of your videos regarding the sync framework.
The Unit test is not depicting all the scenarios. Please consider the following procedure.

Let's say I have a client node connected to the server with Identity = store 1. The client syncs all its data to the server. Somehow client one crashed/lost their data, or I want the same data to be synced on 2nd device, or a fresh client with the same Identity want to restore data. It is working fine, but It does not sync its data; it syncs all other data. I have seen the code in ISyncServerNode, which fetches deltas for other Nodes.

public class SyncServerNode : ISyncServerNode
{
   ...
    public virtual Task<IEnumerable<IDelta>> GetDeltasAsync(Guid startindex, string identity, CancellationToken cancellationToken)
    {
        return this.deltaStore?.GetDeltasFromOtherNodes(startindex, identity, cancellationToken);
    }
   ...

Look at this code return this.deltaStore?.GetDeltasFromOtherNodes(startindex, identity, cancellationToken); It fetches data of other nodes. There should be another function in the ISyncServerNode.

public interface ISyncServerNode
{
    string NodeId { get; set; }
    Task SaveDeltasAsync(IEnumerable<IDelta> deltas, CancellationToken cancellationToken);
    Task<IEnumerable<IDelta>> GetDeltasFromOtherNodesAsync(Guid startindex, string identity, CancellationToken cancellationToken);
    Task<IEnumerable<IDelta>> GetDeltasAsync(Guid startindex, CancellationToken cancellationToken);
    Task ProcessDeltasAsync(IEnumerable<IDelta> deltas, CancellationToken cancellationToken);
}

There should be two functions like

Task<IEnumerable<IDelta>> GetDeltasFromOtherNodesAsync(Guid startindex, string identity, CancellationToken cancellationToken);
Task<IEnumerable<IDelta>> GetDeltasAsync(Guid startindex, CancellationToken cancellationToken);

One should be used for the normal operation scenario, and the 2nd should be used in case of a fresh/new/crashed node. Similarly, the SynController Fetch function should also be updated with NodeId and without NodeId or another filter header.

Thank you,
Qamar Abbas