Coder Social home page Coder Social logo

sqlite-sync / sqlite-sync.com Goto Github PK

View Code? Open in Web Editor NEW
321.0 21.0 82.0 144.15 MB

AMPLI-SYNC is a framework for synchronizing data between a Sqlite database and an MS SQL/MySQL/Oracle/PostgreSQL database. With this framework your application can work completely offline (Airplane Mode), then perform an automated Bidirectional Synchronization when an internet connection becomes available.

Home Page: https://ampliapps.com/sqlite-sync/

License: BSD 3-Clause "New" or "Revised" License

JavaScript 8.67% HTML 2.06% Java 37.59% Objective-C 36.72% C# 7.93% TypeScript 5.79% Starlark 0.37% SCSS 0.87% Shell 0.01%

sqlite-sync.com's Introduction

AMPLI-SYNC community edition

For samples for older version 2.x see branch 'clients-for-v2.x'.

Mobile platform specific implementations are under ampli sync-clients folder.

Background

Mobile users now expect desktop quality data driven interactive applications.

Yet as a mobile users, ‘Offline’ or ‘Intermittent connectivity’ is a fact of life.

Mobile networks simply don’t provide that same level of consistency that we get from the desktop. We, as app developers can’t keep building applications with the desktop mentality of assuming we have a permanent, fast and reliable connection.

The time has come for mobile app developers to accept reality.

It’s time to think about designing and building our applications as ‘offline-first’, where the offline mode isn’t simply a fall-back for an error scenario but the main mode of operation, and internet connectivity is established under app control.

There are many ways to build mobile applications and we all have our own favourite tools which we don’t want to give up.

When building data driven occasionally connected mobile applications, the common issue we all face is Bi-Directional Data Synchronisation and when/where to start that sync. Automatic data sync when a connection becomes available can create its own issues. What is needed is a simple and controlled method for performing Bi-Directional Synchronisation.

AMPLI-SYNC service provides you with a possibility of providing your system with full functionality even when not connected to the Internet.

The service supports many different platforms, such as:

  • Android (JAVA)
  • iOS (Objective C)
  • Xamarin
  • .NET (C#)
  • Javascript
  • JQuery
  • Universal Windows Platform
  • React-Native
  • Cordova

Our priority is our customer therefore do not hesitate to get in touch with us under [email protected]. We read every email and every incoming message is given the highest priority.

In a nutshell:

AMPLI-SYNC Offline-first applications, with working offline SQLite storage. Synchronisation with Microsoft SQL or MySQL back-end servers. Automatic mobile schema creation if required. ‘Click-To-Sync’ web service. Various platforms supported. Two subsriptions: Paid and Open-Source

Buy me a coffee or beer: tomek(at)dziemidowicz.cloud

Find out more at: https://ampliapps.com/

ampli-sync documentation

AMPLI-SYNC concept

Solution diagram

  • Devices communicate with server using HTTP/HTTPS protocol.
  • Proxy Load Balancer. If necessary, proxy can redirect request to another instance of SQLite-core.
  • Authorization provider will generate token based on response from authorization provider.

Proxy Balancer flowchart

Authorization Provider flowchart

REST API (server API)

Protocol version

This document describes integration with the REST API 3.x protocol.

Service description

There are two stages to processing a request:

  • Device places an request.
  • SQLite-sync server confirms the request has been processed successfully and sends confirmation with eventual conflict list that need to be resolved.

Synchronization flowchart

Request URL format

Sample REST API call:

https://example.com/API3/__method___

Explanation:
https://example.com/ - adres of REST API service
API3 - version of synchronization
method_ - method/action

API methods

API3 - control method

Method : GET
Path : “/API3”
Produces : TEXT_HTML
Description : control method. Returns “API[v3] ampli-sync is working correctly!” if web service is correctly configured.

InitializeSubscriber - Reinitialize subscriber

Method : GET
Path : “/InitializeSubscriber/{subscriberUUID}”
Produces : TEXT_PLAIN
Description : Reinitialize subscriber, create empty schema on device, prepare master database for new subscriber.

Sync - gets changes for table

Method : GET
Path : /Sync/{subscriberUUID}/{tableName}
Produces : TEXT_PLAIN
Description : Get changed data.
Params:
subscriberUUID - identifier of subscriber. By default device unique ID is used. But we can place there any value (also #user.UUID)
tableName - name of table from database (without schema)
Response:

    <?xml version="1.0" encoding="utf-8"?>
    <records>
      <r a="1">
        <c>2</c>
        <c>Document</c>
        <c>75541</c>
        <c>2014-02-13 00:00:00</c>
        <c>665.000</c>
        <c>2c93d64e-cc72-11e3-87e0-f82fa8e587f9</c>
      </r>
      <r a="2">
        <c>4</c>
        <c>Document 4</c>
        <c>4879</c>
        <c>2014-04-23 13:44:48</c>
        <c>4875.000</c>
        <c>2c93d765-cc72-11e3-87e0-f82fa8e587f9</c>
      </r>
    </records>

- section contains records
- here starts record.
- here record ends
Attribute “a” (action type)
1 - new record
2 - update for record

CommitSync - control method

Method : GET
Path : “/CommitSync/{syncId}”
Produces : TEXT_PLAIN
Description : If device recieved all changes without error this method should be call to tell server that there was no errors during receiving package. Params: syncId - id of data package

Send - control method

Method : POST
Path : “/Send”
Consumes : JSON
Produces : TEXT_PLAIN
Description : Send changes from device to master database.

For sample data format with changes see XML code sample.

AddTable - control method

Method : GET
Path : “/AddTable/{tableName}”
Produces : TEXT_PLAIN
Description : Add table to synchronization.

RemoveTable - control method

Method : GET
Path: “/RemoveTable/{tableName}”
Produces : TEXT_PLAIN
Description : Remove table from synchronization.

Conflict Resolution

Update procedure

When the user first starts the client application, will be forced to go online and do an initial sync with the master DB, which sends a schema used to create the local database and its tables. After that, the user can work offline.
When user is in older version will be forced to make update of schema. All updates will be sent to client and apply locally. After successfully update client will send unsync data to master database.
A schema update usually means an update of the application is also needed, since the application will need different SQL code to deal with the new schema.
The master DB will never receive changes from clients with the old schema, since a client always pulls before pushing changes - and in the pull it would have received and applied the new schema.

Data filtering

Goal : send to device only documents that are main.
We will going to use the device with subscriber id 1.
In the database we have a table where we are storing documents [dbo].[Documents] , user data [dbo].[Users] and table with relations between documents and users [dbo].[UserDocuments].

[dbo].[Documents] structure:

[dbo].[Users] structure:

[dbo].[UserDocuments] structure:

Please notice, that in table dbo.User we have a column usrSubscriberId - it tells exactly which user uses particular subscriber id. Based on this column we are able to select which user id is used while the device is undergoing the synchronization process.
The next step is to prepare view:

SQL code:

    SELECT dbo.Documents.RowId, dbo.MergeSubscribers.SubscriberId AS pdaIdent 
    FROM dbo.Documents 
    INNER JOIN dbo.UserDocuments ON dbo.Documents.docId = dbo.UserDocuments.usdDocId 
    INNER JOIN dbo.Users ON dbo.UserDocuments.usdUsrId = dbo.Users.usrId 
    INNER JOIN dbo.MergeSubscribers ON dbo.Users.usrSubscriberId = dbo.MergeSubscribers.SubscriberId

The view needs to return two things:

  1. RowId of filtered table (dbo.Documents is this scenario) subscriberId aliased as ‘pdaIdent’
  2. You can add extra condition in this view - it’s up to you.

Next step, we need to tell synchronization to use that filter:

    update [MergeTablesToSync] set TableFilter='vwMerge_Documents' where TableName='Documents' and TableSchema='dbo'

where vwMerge_Documents is the name of your view.

Installation

Server Prerequisites

To make ampli-sync server work you need:

  • Apache Tomcat 8.
  • Java
  • Linux/Windows environment.

Manual

Steps needed to install AMPLI-SYNC manually on Ubuntu.

  1. Install Tomcat on Ubuntu:
    https://www.digitalocean.com/community/tutorials/how-to-install-apache-tomcat-8-on-ubuntu-16-04

  2. Create new user sudo adduser amplisync

    System will ask you for password for newly created user.

  3. Add user to group ‘tomcat’ sudo usermod -a -G tomcat amplisync

  4. Chang in web.xml path variable to \home/sqlitesync/demo

  5. Install new application in Tomcat. Start with switching to amplisync user. su amplisync

  6. Create new folder /home/sqlitesync/demo

  7. Upload new service amplisync-demo to Tomcat. You can do that using Tomcat application manager, or you can put WAR file in Tomcat webapps folder. Name of your WAR file is app name in Tomcat environment. Remember to not place spaces and special chars in name of your WAR file.

  8. Restart Tomcat: service tomcat restart

    Now you can access your installation using link: http://your_ip:8080/amplisync-app-name/API3

  9. Setup permissions: chown -R sqlitesync:tomcat /home/amplisync/demo/

Docker

Configuring AMPLI-SYNC service

First you need to adjust website configuration file (web.xml), then you need to change main configuration file (sync.properties).
Service configuration (web.xml)
Go to your_webapps_folder/SqliteSync/WEB-INF/web.xml and open for edit. Navigate to section:

    <env-entry>
    <env-entry-name>working-dir</env-entry-name>
    <env-entry-type>java.lang.String</env-entry-type>
    <env-entry-value>/your/working/dir/sqlite-sync/</env-entry-value>
    </env-entry>

change env-entry-value key and point to working dir where ampli-sync server will store log files, temporary files and configuration. Create manually a subfolder named config. Create a text file sync.properties in folder config. The path should look like this:

\working_dir\config\sync.properties

IMPORTANT Restart service after changing web.xml. Make sure that Tomcat has read/write access to working dir.
Sample configurations for MySQL server

DB_ENGINE = mysql
DBURL = jdbc:mysql://server:3306/dbname?rewriteBatchedStatements=true
DBUSER = user
DBPASS = pass
DBDRIVER = com.mysql.cj.jdbc.Driver
DATE_FORMAT = yyyy-MM-dd HH:mm:ss
TIMESTAMP_FORMAT = yyyy-MM-dd HH:mm:ssZ
HISTORY_DAYS = 7
LOG_LEVEL = 4

If you faced a timezone error after configuration in `sync.properties` add at the end of connection string:

&useUnicode=true&useJDBCCompliantTimezoneShift=true&useLegacyDatetimeCode=false&serverTimezone=UTC

Keys explanation
DB_ENGINE - type of database engine. Available options:

  • mysql
  • mssql
  • postgresql
  • oracle

LOG_LEVEL - defain details level for log
> 4: TRACE, DEBUG, INFO, WARN; (default)
> 3: DEBUG, INFO, WARN;
> 2: INFO,WARN;
> 1: WARN;
> 0 - disable logs
DATE_FORMAT, TIMESTAMP_FORMAT - set format of date
default format: yyyy-MM-dd HH:mm:ss
HISTORY_DAYS - How long files with sync data will be kept
default value: 7
When you use MySQL database DO NOT remove from the end of the connection string:

?rewriteBatchedStatements=true

Supported databases

AMPLI-SYNC supports those databases:

  • MySQL
  • Microsoft SQL Server (2005 and newer)
  • Oracle
  • PostgreSQL

Supported columns data types

ampli-sync uses own conversion table to match column data types when schema from master database is converted to sqlite database.

  • blob
  • longblob
  • varbinary
  • binary
  • image
  • mediumblob
  • varbinarymax
  • byte[]
  • longtext
  • varchar
  • nvarchar
  • char
  • varcharmax
  • enum
  • mediumtext
  • text
  • string
  • geography
  • geometry
  • hierarchyid
  • nchar
  • ntext
  • nvarcharmax
  • userdefineddatatype
  • userdefinedtabletype
  • userdefinedtype
  • variant
  • xml
  • tinytext
  • set
  • time
  • timestamp
  • year
  • datetime
  • uniqueidentifier
  • datetime2
  • date
  • mediumint
  • bit
  • tinyint
  • smallint
  • bigint
  • int
  • boolean
  • byte
  • long
  • int64
  • serial
  • int32
  • smalldatetime
  • double
  • float
  • numeric
  • decimal
  • real
  • money

Primary Key requirements

Single and mupltiple columns are supported as primary key.
When column is AUTO_INCREMENT/SERIAL, identity pool management is handled by AMPLI_SYNC. It means when you insert a new record onto the device, the PK will be automatically changed for the first value available for device.

sqlite-sync.com's People

Contributors

dependabot[bot] avatar k-gardocki-amplifier avatar rafalgolubowicz avatar sqlite-sync avatar tdziemidowicz avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

sqlite-sync.com's Issues

Improve Documentation

With the descriped setup tomcat is not able to write to amplisync home.
So no logs and syncData is created.

DotNET C# demo not working

The Demo seems to be only a fragment. Three tables should be sync, but they are never added to sync. As a consequence the program runs in ArgumentNullException
Bildschirmfoto 2019-11-14 um 11 52 43

where is the SQL script

after hours trying to make it work with postgreSQL i did, and now i notice that there is no sql file to create the sync database ...

Recommended java memory settings

With an out-of-the box install of Java and Tomcat, the logs typically end up reporting: (Using SqliteSync-3.2.16)

javax.servlet.ServletException: org.glassfish.jersey.server.ContainerException: java.lang.OutOfMemoryError: GC overhead limit exceeded

  1. Is there any guidance and to-what and where a setting change to give the VM more memory can be made?
  2. Would that need to scale depending on the number of users / tables synced etc, e.g. nnMB per X users?
  3. Could there be something internally not freeing its memory such that this error is seen after a period of time, e.g. a few days?

Sync issue from local to server database.

I have successfully sync the data and get all the data in the local SQLite database but when I add a new record in local and try to sync on the server at that time, the new data is not reflating in the server database. Its creating one .dat file and the data is present in it in XML format.

I am using ionic 4 to send the data on the server database.

So can you please help to solve this issue?

API[3.2.15] SQLite-Sync.COM is working correctly! Error creating database connection.

I'm receiving this error when access: localhost:8080/sync/API3.

My web.xml:

working-dir
java.lang.String
C:\Users\Luiza\sync</env-entry-value>

My sync.properties file(located at 'C:\Users\Luiza\sync\config\sync.properties') :

DB_ENGINE = postgres
DBURL = jdbc:postgresql://localhost:56582/sync
DBUSER = postgres
DBPASS = postgres
DBDRIVER = org.postgresql.Driver
DATE_FORMAT = yyyy-MM-dd HH:mm:ss
HISTORY_DAYS = 7
LOG_LEVEL = 4

Android Demo does not work

The application could not be installed: INSTALL_FAILED_NO_MATCHING_ABIS
Installation failed due to: 'null'

For the later sdk versions of 27 or higher

Constraint Violation

ERROR PushInsertRecords() constraint violation PRIMARY KEY 'PK_XXX'. cannot insert the duplicated key in the object 'dbo.XXX'.

This make no sense because is not a new record. Its the same record the App sent to DB.

In the next syncronization get this message

how to run the .net C# demo on local computer?

I've installed the sqlite-sync on my local computer ,which runs windows with apache,mysql and tomcat.
when I access http://127.0.0.1:8080/SqliteSync/API3 from chrome,it works correctly as display:
API[3.2.7] SQLite-Sync.COM is working correctly! Database connected!

I changed the C# demo code, set wsUrl = "http://127.0.0.1:8080/SqliteSync/API3";
when I click 'Reintialize database' or 'Send and Recieve' button, I always get an error.

the response is :Error creating new subscriber for UUID 1

I open logo file,
There is an error: Table 'testdb.mergesubscribers' doesn't exist. testdb is my database name.

Question is:
how to create mergesubscribers??
Isn't it automatic?

Getting Internal Server Error from service, when updating from client to server

This is my request body, when I call send method, It gives me a internal server error

Request:
{
"subscriber": "1",
"content": "<SyncData xmlns="urn:sync-schema"><tab n="MergeIdentity"><tab n="reportsnew">",
"version": "3"
}

Response:
<!doctype html>

<title>HTTP Status 500 – Internal Server Error</title> <style type="text/css"> h1 { font-family: Tahoma, Arial, sans-serif; color: white; background-color: #525D76; font-size: 22px; }
	h2 {
		font-family: Tahoma, Arial, sans-serif;
		color: white;
		background-color: #525D76;
		font-size: 16px;
	}

	h3 {
		font-family: Tahoma, Arial, sans-serif;
		color: white;
		background-color: #525D76;
		font-size: 14px;
	}

	body {
		font-family: Tahoma, Arial, sans-serif;
		color: black;
		background-color: white;
	}

	b {
		font-family: Tahoma, Arial, sans-serif;
		color: white;
		background-color: #525D76;
	}

	p {
		font-family: Tahoma, Arial, sans-serif;
		background: white;
		color: black;
		font-size: 12px;
	}

	a {
		color: black;
	}

	a.name {
		color: black;
	}

	.line {
		height: 1px;
		background-color: #525D76;
		border: none;
	}
</style>

HTTP Status 500 – Internal Server Error


Type Exception Report

Message java.lang.NullPointerException

Description The server encountered an unexpected condition that prevented it from fulfilling the request.

Exception

javax.servlet.ServletException: java.lang.NullPointerException
	org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:489)
	org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:427)
	org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:388)
	org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:341)
	org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:228)
	org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)

Root Cause

java.lang.NullPointerException
	java.text.SimpleDateFormat.<init>(Unknown Source)
	java.text.SimpleDateFormat.<init>(Unknown Source)
	SQLiteSyncCore.SyncServer.Synchronization.SyncService.ParseStatmentParameter(SyncService.java:1307)
	SQLiteSyncCore.SyncServer.Synchronization.SyncService.PushUpdateRecords(SyncService.java:1157)
	SQLiteSyncCore.SyncServer.Synchronization.SyncService.PushTableData(SyncService.java:1007)
	SQLiteSyncCore.SyncServer.Synchronization.SyncService.CommitChangesToDb(SyncService.java:978)
	SQLiteSyncCore.SyncServer.Synchronization.SyncService.ReceiveData(SyncService.java:840)
	com.sqlitesync.ws.SyncAPI3.RecieveChanges(SyncAPI3.java:91)
	sun.reflect.GeneratedMethodAccessor40.invoke(Unknown Source)
	sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	java.lang.reflect.Method.invoke(Unknown Source)
	org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81)
	org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144)
	org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161)
	org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$VoidOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:143)
	org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99)
	org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389)
	org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347)
	org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102)
	org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326)
	org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
	org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
	org.glassfish.jersey.internal.Errors.process(Errors.java:315)
	org.glassfish.jersey.internal.Errors.process(Errors.java:297)
	org.glassfish.jersey.internal.Errors.process(Errors.java:267)
	org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317)
	org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305)
	org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154)
	org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:473)
	org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:427)
	org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:388)
	org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:341)
	org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:228)
	org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)

Note The full stack trace of the root cause is available in the server logs.


Apache Tomcat/8.5.46

Can this be installed with npm?

I want to sync data between an offline windows app (electron with angular) where data is stored on a local sqlite database (not in the browser) to a sql server db. Can this be done using ampli-sync?

how can i install it? (development on windows)

HTML5 demo not working

I tried the demo HTML5/SQLiteSyncDEMO HTML+jQuery/index.htm on MacOS 10.14.6 with Safari and Firefox and Chrome and on iPhone with MobileSafari. The results are disappointing:

  • Safari, Firefox and MobileSafari return an error on window.openDatabase (index.htm:130)
  • Chrome can cope with window.openDatabase (index.htm:130), but:
    a) The test() function (index.htm:104) yields a 404 error
    b) Reinitialize() button gives
Starting synchronization

Connecting to server...

Connected to server...

Creating object 00001 MergeDelete drop...
Creating object 00002 MergeDelete create...
Creating object 00003 MergeIdentity drop...
Creating object 00004 MergeIdentity...
Creating object 00005 MergeIdentity_Index...
Creating object 00006 Documents drop...
Creating object 00007 Documents...
Creating object 00008 Documents_MergeInsert...
Creating object 00009 Documents_MergeUpdate...
Creating object 00010 Documents_MergeDelete...
Creating object 00011 _Documents_PRIMARY...
Creating object 00012 _Documents_IX_Documents_AuthorId...
Creating object 00013 _Documents_IX_MERGE__Documents_RowId...
Creating object 00014 Documents_MergeUpdate_Index...
Synchronization completed

c) Synchronize() is blocked by CORS policy. Debugger output:

<b>Starting sending data</b>
2jquery-2.0.2.min.js:6 OPTIONS http://macmini:8080/SqliteSync-3.2.14/API3/Send 500
send @ jquery-2.0.2.min.js:6
ajax @ jquery-2.0.2.min.js:6
sqlitesync_SyncSendToServer @ sqlitesync.js:436
(anonymous) @ sqlitesync.js:426
SQLStatement (async)
sqlitesync_SyncSendTableDelete @ sqlitesync.js:411
(anonymous) @ sqlitesync.js:326
SQLStatement (async)
(anonymous) @ sqlitesync.js:300
SQLStatement (async)
(anonymous) @ sqlitesync.js:278
SQLStatement (async)
(anonymous) @ sqlitesync.js:261
SQLTransaction (async)
sqlitesync_SyncSendTable @ sqlitesync.js:260
(anonymous) @ sqlitesync.js:324
SQLStatement (async)
(anonymous) @ sqlitesync.js:300
SQLStatement (async)
(anonymous) @ sqlitesync.js:278
SQLStatement (async)
(anonymous) @ sqlitesync.js:261
SQLTransaction (async)
sqlitesync_SyncSendTable @ sqlitesync.js:260
(anonymous) @ sqlitesync.js:335
SQLStatement (async)
(anonymous) @ sqlitesync.js:261
SQLTransaction (async)
sqlitesync_SyncSendTable @ sqlitesync.js:260
sqlitesync_SyncSendData @ sqlitesync.js:255
(anonymous) @ sqlitesync.js:47
SQLTransaction (async)
sqlitesync_SyncSendAndReceive @ sqlitesync.js:22
doSync @ index.htm:75
onclick @ index.htm:28
index.htm:1 Access to XMLHttpRequest at 'http://macmini:8080/SqliteSync-3.2.14/API3/Send' from origin 'http://luzi2' has been blocked by CORS policy: Response to preflight request doesn't pass access control check: No 'Access-Control-Allow-Origin' header is present on the requested resource.

Sending data failed

The content object I send to Server:

"<?xml version=\"1.0\" encoding=\"utf-8\"?><SyncData xmlns=\"urn:sync-schema\"><tab n=\"MergeIdentity\"><ins></ins><upd></upd></tab><tab n=\"person\"><ins><r><PersonID><![CDATA[6]]></PersonID><LastName><![CDATA[g]]></LastName><FirstName><![CDATA[b]]></FirstName><RowId><![CDATA[null]]></RowId></r></ins><upd></upd></tab><delete></delete></SyncData>"

subscriber id: 642661417

The log I found in 'sync.log':

2020-10-19 14:05:40,948 INFO Receiving data from subscriber 642661417
2020-10-19 14:05:40,971 ERROR CommitChangesToDb() ParseError at [row,col]:[1,1]
Message: Content is not allowed in prolog.
2020-10-19 14:05:40,975 INFO Finished receiving data from subscriber 642661417
2020-10-19 14:05:41,235 INFO Getting changes for subscriber 1 and table MergeIdentity
2020-10-19 14:05:41,247 TRACE select distinct * from ( select  tb.*,1 as MergeContent_TableId,   1 as MergeContent_SubscriberId,  tb.RowId as MergeContent_RowId,  tb.RowVer as MergeContent_RowVer,  t.ChangeDate as MergeContent_ChangeDate, -1 as MergeContent_Action,   null as MergeContent_SyncId   from `MergeIdentity` tb left join `MergeIdentity` vw on tb.RowId=vw.RowId left join `MergeContent_MergeIdentity` t on vw.RowId=t.RowId and t.SubscriberId=1 where (t.RowVer<>tb.RowVer or t.RowId is null) and vw.SubscriberId=1  union all select  t.*,1 as MergeContent_TableId,   1 as MergeContent_SubscriberId,  m.RowId as MergeContent_RowId,  m.RowVer as MergeContent_RowVer,  m.ChangeDate as MergeContent_ChangeDate,   3 as MergeContent_Action,   null as MergeContent_SyncId   from `MergeContent_MergeIdentity` m left join `MergeIdentity` vw on m.RowId=vw.RowId and m.SubscriberId=1 and vw.SubscriberId=1  left join `MergeIdentity` t on vw.RowId=t.RowId  where vw.RowId is null  and m.SubscriberId=1  ) as mergeVw
2020-10-19 14:05:41,259 TRACE [ {
  "TableName" : "",
  "Records" : "",
  "QueryInsert" : "",
  "QueryUpdate" : "",
  "QueryDelete" : "",
  "TriggerInsert" : "",
  "TriggerUpdate" : "",
  "TriggerDelete" : "",
  "TriggerInsertDrop" : "",
  "TriggerUpdateDrop" : "",
  "TriggerDeleteDrop" : "",
  "SyncId" : -1,
  "SQLiteSyncVersion" : ""
} ]
2020-10-19 14:05:41,369 INFO Getting changes for subscriber 1 and table person
2020-10-19 14:05:41,381 TRACE select distinct * from ( select  tb.*,2 as MergeContent_TableId,   1 as MergeContent_SubscriberId,  tb.RowId as MergeContent_RowId,  tb.RowVer as MergeContent_RowVer,  t.ChangeDate as MergeContent_ChangeDate, -1 as MergeContent_Action,   null as MergeContent_SyncId   from `person` tb left join `person` vw on tb.RowId=vw.RowId left join `MergeContent_person` t on vw.RowId=t.RowId and t.SubscriberId=1 where (t.RowVer<>tb.RowVer or t.RowId is null)   union all select  t.*,2 as MergeContent_TableId,   1 as MergeContent_SubscriberId,  m.RowId as MergeContent_RowId,  m.RowVer as MergeContent_RowVer,  m.ChangeDate as MergeContent_ChangeDate,   3 as MergeContent_Action,   null as MergeContent_SyncId   from `MergeContent_person` m left join `person` vw on m.RowId=vw.RowId and m.SubscriberId=1   left join `person` t on vw.RowId=t.RowId  where vw.RowId is null  and m.SubscriberId=1  ) as mergeVw
2020-10-19 14:05:41,392 TRACE [ {
  "TableName" : "person",
  "Records" : "<?xml version=\"1.0\" encoding=\"utf-8\"?><records><r a=\"1\"><c>1</c><c>q</c><c>y</c><c>2543d43a-11b3-11eb-819d-309c23bf0c2c</c></r><r a=\"1\"><c>2</c><c>w</c><c>i</c><c>2543dec8-11b3-11eb-819d-309c23bf0c2c</c></r><r a=\"1\"><c>3</c><c>e</c><c>p</c><c>2543dfb2-11b3-11eb-819d-309c23bf0c2c</c></r><r a=\"1\"><c>4</c><c>giao</c><c>1</c><c>2543e093-11b3-11eb-819d-309c23bf0c2c</c></r></records>",
  "QueryInsert" : "INSERT INTO person ([PersonID],[LastName],[FirstName],[RowId]) VALUES (?,?,?,?);",
  "QueryUpdate" : "UPDATE person set [LastName]=?,[FirstName]=?,[RowId]=? where PersonID=?;",
  "QueryDelete" : "DELETE FROM person where RowId=",
  "TriggerInsert" : "",
  "TriggerUpdate" : " CREATE TRIGGER IF NOT EXISTS \"trMergeUpdate_person\"     AFTER UPDATE OF [LastName],[FirstName]    ON \"person\"  BEGIN  \tUPDATE \"person\" SET MergeUpdate = 1 WHERE RowId = old.RowId and RowId<>'';  END; ",
  "TriggerDelete" : " CREATE TRIGGER IF NOT EXISTS \"trMergeDelete_person\"     AFTER DELETE     ON \"person\"   BEGIN \t  \tINSERT INTO MergeDelete values (2,  old.RowId);  END; ",
  "TriggerInsertDrop" : "DROP TRIGGER IF EXISTS \"trMergeInsert_person\"",
  "TriggerUpdateDrop" : "DROP TRIGGER IF EXISTS \"trMergeUpdate_person\"",
  "TriggerDeleteDrop" : "DROP TRIGGER IF EXISTS \"trMergeDelete_person\"",
  "SyncId" : 21,
  "SQLiteSyncVersion" : "3.2.16"
} ]

Please help!

oracle version

We ordered oracle version of sqlite-sync, but it has the MySQL version of the script "ADD_TABLE" named ORACLE_ADD_TABLE.sql. Where can we get the proper version of this script?

Identity insert error

Hi, I'm getting this error when I try to insert in the master db a new record that has an Identity primary key:

ERROR PushInsertRecords() Cannot insert explicit value for identity column in table 'Table' when IDENTITY_INSERT is set to OFF.

is an special configuration required??

Thanks

SQL Server scripts

Hi,

does anyone know where to find the sql scripts for SQWL Server to setup the host database? In resources folder only mysql scripts are available.

Regards,
Ingo

Do it supports react native?

Hello,

Do it support react native, I am creating one app in react native framework and i want it to be working with syncing process. Please let me know.

Bad query when initializing a suscriber

Hello,

First thank you for that tool that is so useful.

I tried it on versions 3.2.7 and 3.2.15.

I got an issue initializing a suscriber: the returned query contains an error.

In the mergeupdate trigger creation, there is :
"CREATE TRIGGER IF NOT EXISTS "trMergeUpdate_" AFTER UPDATE OF"

The "OF" create an error when the query is executed on the local database.
I fixed it removing the word after get the response from the server.

some errors about ReinitializeDatabase (C#)

when I ReinitializeDatabase , the framework start to create table .
but some times, it doesn't add [RowID] column to the client.

as I know, if a table whithout RowID then can't sync.

I've created 2 tables on the server, test and test2.
when I click ReinitializeDatabase button
test2 has no [Rowid] column,but test has!

I add the exist table to sync(call AddTable API) always get this error.

here is the ReinitializeDatabase response content:

we can see about the create table test2 code, it really has no [RowID]!

there is a 'rowid' in the test table:

"00014 test" : "CREATE TABLE "test" ([id] INTEGER ,[name] TEXT DEFAULT ('NULL') ,[age] INTEGER DEFAULT (NULL) ,[birthday] DATETIME ,[RowId] TEXT DEFAULT ('NULL') ,"MergeUpdate" INTEGER NOT NULL DEFAULT (0) );",

but,there is not a 'rowid' in the test2 table.

"00007 test2" : "CREATE TABLE "test2" ([id] INTEGER ,[country] TEXT DEFAULT ('NULL') ,[city] TEXT DEFAULT ('NULL') ,[code] INTEGER DEFAULT (NULL) ,"MergeUpdate" INTEGER NOT NULL DEFAULT (0) );",

Xamarine.Forms example

Hey, Is there any working example for Xamarine.Forms and MySQL.

I am working on a project to sync data between MySQL database hosted on a web server and SQLite in Xamarin.Forms app. I need some references to start with. A working example will be appreciated.

Kindly let me know.

Regards

Error while syncing databases!

Im getting this error message in the mobile App
"Error while syncing databases!"

In my case I have 3 main table to sync.

Product-->always syncing from DB to App
Customer-->always syncing from DB to App
Orders<--just App creates new records and send to DB

Sometimes for instance Product's price changes. Some internal process update the table Product.

Next time the App sync get this error message:
"Error while syncing databases!"

Sync is not finished.
Sqlite-sync sometimes crashes.
If we restart maybe the next sync will work.

Limit amount data to sync

Hi, i want to synchronize in two nodes a table with a amount of 400000 rows. When i try it, the system never come back the JSON because the size of the data to send. There is not problem when y synchronize less info.
So, i want to know if there is a way to limit the amount of data of the JSON. The only table i want to sync have the next columns:

  • Subscriber_id
  • Date
  • Name
  • RowVer
  • RowId

I use MySQL like a primary node and SQLiteSync 3.2.3

Thanks

SyncService.java - "EnumarateChanges" method not using "tableFilter" parameter

Hi. Thank you for this interesting proyect. Right now I'm interested in making "filtering" work. Is there something wrong with the code version of SyncService.java? In particular in the implementation of method "EnumarateChanges"? I see from the code that the parameter "tableFilter" is never used inside the method. How can I used the VIEW I created in MySQL to filter the data generated?

Thanks...

Luis Fernando

Problem during Send: can't write new records on the remote db

Hi, I'm using the Ionic client provided in this repo.

As a database I'm usng SQL SERVER 2016.

The connection to the server works fine, and I'm able to add a table, initialise the client's database and hit the Sync endpoint to get data from the remote db.

Then I create some record on the client, and when I hit the Send endpoint, it looks like the data is not being sent.
I mean, I do send the XML payload and everything, and get a 204 response, it just doesn't recognise my local changes and thus the remote db doesn't receive the new data. I mean, in the XML payload, there's no trace of my local changes.

In the attempt to fix the issue, I added an auto-increment key to my table in the remote db.
At this point, when I hit the Sync endpoint, I get Error while syncing databases!.
I debugged the issue, and it's actually a column index out of range from SqLite.

I when to the bottom of it, this is what I found: basically, if my table has 4 columns, the Sync endpoint sends me 4 + 1 columns, where the extra one is RowId.

I'm not sure, at this point, whether it's a client or server issue. I'm just stuck and I don't know what to do.

Maybe someone has a clue?

Thanks in advance

[EDIT]
I've tried with the basic HTML5 and the C# client examples.
In both cases I can create new records locally but they are not sent to the server during the Send.
If I manually change the value of the MergeUpdate field to 1, it sends the new rows but they are not saved in the remote DB. I'm using the code provided by the examples, but 3 examples that behave the same makes me think I'm doing something wrong...

Bad query initializing a suscriber

Hello,

I meet this concern again, when the names of the tables are well informed. After several tests I realized that it happened on the tables of joins with only two fields which form a primary key.

I note that the creation of update trigger on a table is performed only on non-key fields. There are none on these tables, which must be the source of the error.

Log line triggering the error:

"00139 PE_ENTRETIEN_ETAT_MergeUpdate": "CREATE TRIGGER IF NOT EXISTS " trMergeUpdate_PE_ENTRETIEN_ETAT \ "AFTER UPDATE OF ON " PE_ENTRETIEN_ETAT \ "BEGIN \ tUPDATE " PE_ENTRETIEN_ETAT = "old Row"; END; ",

The table creation script :
"CREATE TABLE [dbo].[PE_ENTRETIEN_ETAT](
[CLE_ENTRETIEN] varchar NOT NULL,
[CLE_ETAT] varchar NOT NULL,
[RowId] [uniqueidentifier] NULL DEFAULT (newid()),
[RowVer] [int] NOT NULL DEFAULT ((1))
) ON [PRIMARY]"

I am using version 3.2.15 with Microsoft SQL Server.

Do you have a solution?

Version 3.2.3 cannot connect to MSSQL Server

Hi,
I'm using the sqlite-sync in our project since 2018 with success.

Recently you released the 3.2.3 version, I tried to install but I cannot connect to MSSQL Server. the config file i s the same used for the 3.1.9 version:

DB_ENGINE = mssql
DBURL = jdbc:sqlserver://crt-sql01.crottiant.local:1433;databaseName=CrottiDB
DBUSER = crotti
DBPASS = xxxxxxx
DBDRIVER = com.microsoft.sqlserver.jdbc.SQLServerDriver
DATE_FORMAT = yyyy-MM-dd HH:mm:ss
HISTORY_DAYS = 30
LOG_LEVEL = 3

The config file change with 3.2.3 version?

Thanks.
G. Raccuia

Empty values are inserted as null strings

Hello,

I'm developing an Android app synced with a MSSQL Server.
After adding tables and syncing, i'm encountering an issue because there is no null values in my SQLite database, all has been inserted as null strings. It happens with TEXT columns, but also INT, that causes conversion errors in my app.

Is there a setting to change that?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.