zzt93 / syncer Goto Github PK

View Code? Open in Web Editor NEW

69.0 5.0 30.0 1.81 MB

Sync & Manipulate data from MySQL/MongoDB to Elasticsearch/MySQL/Kafka Endpoint

License: BSD 3-Clause "New" or "Revised" License

Java 94.64% Shell 4.68% Python 0.61% Dockerfile 0.07%

mysql binlog elasticsearch syncer mongodb filter kafka oplog-tailing

syncer's People

Contributors

Stargazers

Watchers

syncer's Issues

How to debug `consumer.filter` code?

Use newest code config
Use intellij create new Debug Configuration and config like following.

Use itellij breakpoint to debug

maven package not found

Describe the bug
maven version is invalid

Environtment setup

am using ubuntu 18~

java -version
openjdk version "1.8.0_191"
OpenJDK Runtime Environment (build 1.8.0_191-8u191-b12-2ubuntu0.18.10.1-b12)
OpenJDK 64-Bit Server VM (build 25.191-b12, mixed mode)

mvn -version
Apache Maven 3.5.4

when i try to run
java -server -XX:+UseG1GC -jar ./syncer-core/target/syncer-core-1.0-SNAPSHOT.jar --port=40000 --host_address=192.168.55.101 --config=syncer-config.yml --producerConfig=sample-producer.yml --comsumerConfig=my-consumer.yml

path is correct

i got this error
``

019-02-15 16:36:21,820 ERROR [] [syncer@ACTIVE_PROFILE_IS_UNDEFINED@HOST_ADDRESS_IS_UNDEFINED] --- [syncer-shutdown-starter] com.github.zzt93.syncer.ShutDownCenter   : [Shutting down] Init
org.yaml.snakeyaml.constructor.ConstructorException: Cannot create property=syncer for JavaBean=com.github.zzt93.syncer.config.syncer.SyncerConfig@60c6f5b
 in 'string', line 1, column 1:
    syncer:
    ^
Unable to find property 'syncer' on class: com.github.zzt93.syncer.config.syncer.SyncerConfig
 in 'string', line 2, column 3:
      ack:
      ^

	at org.yaml.snakeyaml.constructor.Constructor$ConstructMapping.constructJavaBean2ndStep(Constructor.java:268)
	at org.yaml.snakeyaml.constructor.Constructor$ConstructMapping.construct(Constructor.java:149)
	at org.yaml.snakeyaml.constructor.Constructor$ConstructYamlObject.construct(Constructor.java:308)
	at org.yaml.snakeyaml.constructor.BaseConstructor.constructObjectNoCheck(BaseConstructor.java:207)
	at org.yaml.snakeyaml.constructor.BaseConstructor.constructObject(BaseConstructor.java:196)
	at org.yaml.snakeyaml.constructor.BaseConstructor.constructDocument(BaseConstructor.java:161)
	at org.yaml.snakeyaml.constructor.BaseConstructor.getSingleData(BaseConstructor.java:147)
	at org.yaml.snakeyaml.Yaml.loadFromReader(Yaml.java:524)
	at org.yaml.snakeyaml.Yaml.loadAs(Yaml.java:501)
	at com.github.zzt93.syncer.config.YamlEnvironmentPostProcessor.initConfig(YamlEnvironmentPostProcessor.java:129)
	at com.github.zzt93.syncer.config.YamlEnvironmentPostProcessor.processEnvironment(YamlEnvironmentPostProcessor.java:52)
	at com.github.zzt93.syncer.SyncerApplication.main(SyncerApplication.java:41)
Caused by: org.yaml.snakeyaml.error.YAMLException: Unable to find property 'syncer' on class: com.github.zzt93.syncer.config.syncer.SyncerConfig
	at org.yaml.snakeyaml.introspector.PropertyUtils.getProperty(PropertyUtils.java:189)
	at org.yaml.snakeyaml.introspector.PropertyUtils.getProperty(PropertyUtils.java:178)
	at com.github.zzt93.syncer.config.YamlEnvironmentPostProcessor$1.getProperty(YamlEnvironmentPostProcessor.java:144)
	at org.yaml.snakeyaml.TypeDescription.discoverProperty(TypeDescription.java:240)
	at org.yaml.snakeyaml.TypeDescription.getProperty(TypeDescription.java:251)
	at org.yaml.snakeyaml.constructor.Constructor$ConstructMapping.constructJavaBean2ndStep(Constructor.java:210)
	... 11 common frames omitted
2019-02-15 16:45:47,122 ERROR [] [syncer@ACTIVE_PROFILE_IS_UNDEFINED@HOST_ADDRESS_IS_UNDEFINED] --- [syncer-shutdown-starter] com.github.zzt93.syncer.ShutDownCenter   : [Shutting down] Init
org.yaml.snakeyaml.constructor.ConstructorException: Cannot create property=syncer for JavaBean=com.github.zzt93.syncer.config.syncer.SyncerConfig@60c6f5b
 in 'string', line 1, column 1:
    syncer:
    ^
Unable to find property 'syncer' on class: com.github.zzt93.syncer.config.syncer.SyncerConfig
 in 'string', line 2, column 3:
      ack:
      ^

	at org.yaml.snakeyaml.constructor.Constructor$ConstructMapping.constructJavaBean2ndStep(Constructor.java:268)
	at org.yaml.snakeyaml.constructor.Constructor$ConstructMapping.construct(Constructor.java:149)
	at org.yaml.snakeyaml.constructor.Constructor$ConstructYamlObject.construct(Constructor.java:308)
	at org.yaml.snakeyaml.constructor.BaseConstructor.constructObjectNoCheck(BaseConstructor.java:207)
	at org.yaml.snakeyaml.constructor.BaseConstructor.constructObject(BaseConstructor.java:196)
	at org.yaml.snakeyaml.constructor.BaseConstructor.constructDocument(BaseConstructor.java:161)
	at org.yaml.snakeyaml.constructor.BaseConstructor.getSingleData(BaseConstructor.java:147)
	at org.yaml.snakeyaml.Yaml.loadFromReader(Yaml.java:524)
	at org.yaml.snakeyaml.Yaml.loadAs(Yaml.java:501)
	at com.github.zzt93.syncer.config.YamlEnvironmentPostProcessor.initConfig(YamlEnvironmentPostProcessor.java:129)
	at com.github.zzt93.syncer.config.YamlEnvironmentPostProcessor.processEnvironment(YamlEnvironmentPostProcessor.java:52)
	at com.github.zzt93.syncer.SyncerApplication.main(SyncerApplication.java:41)
Caused by: org.yaml.snakeyaml.error.YAMLException: Unable to find property 'syncer' on class: com.github.zzt93.syncer.config.syncer.SyncerConfig
	at org.yaml.snakeyaml.introspector.PropertyUtils.getProperty(PropertyUtils.java:189)
	at org.yaml.snakeyaml.introspector.PropertyUtils.getProperty(PropertyUtils.java:178)
	at com.github.zzt93.syncer.config.YamlEnvironmentPostProcessor$1.getProperty(YamlEnvironmentPostProcessor.java:144)
	at org.yaml.snakeyaml.TypeDescription.discoverProperty(TypeDescription.java:240)
	at org.yaml.snakeyaml.TypeDescription.getProperty(TypeDescription.java:251)
	at org.yaml.snakeyaml.constructor.Constructor$ConstructMapping.constructJavaBean2ndStep(Constructor.java:210)
	... 11 common frames omitted
2019-02-15 16:47:42,894 ERROR [] [syncer@ACTIVE_PROFILE_IS_UNDEFINED@HOST_ADDRESS_IS_UNDEFINED] --- [syncer-shutdown-starter] com.github.zzt93.syncer.ShutDownCenter   : [Shutting down] Init
org.yaml.snakeyaml.constructor.ConstructorException: Cannot create property=syncer for JavaBean=com.github.zzt93.syncer.config.syncer.SyncerConfig@60c6f5b
 in 'string', line 1, column 1:
    syncer:
    ^
Unable to find property 'syncer' on class: com.github.zzt93.syncer.config.syncer.SyncerConfig
 in 'string', line 2, column 3:
      ack:
      ^

	at org.yaml.snakeyaml.constructor.Constructor$ConstructMapping.constructJavaBean2ndStep(Constructor.java:268)
	at org.yaml.snakeyaml.constructor.Constructor$ConstructMapping.construct(Constructor.java:149)
	at org.yaml.snakeyaml.constructor.Constructor$ConstructYamlObject.construct(Constructor.java:308)
	at org.yaml.snakeyaml.constructor.BaseConstructor.constructObjectNoCheck(BaseConstructor.java:207)
	at org.yaml.snakeyaml.constructor.BaseConstructor.constructObject(BaseConstructor.java:196)
	at org.yaml.snakeyaml.constructor.BaseConstructor.constructDocument(BaseConstructor.java:161)
	at org.yaml.snakeyaml.constructor.BaseConstructor.getSingleData(BaseConstructor.java:147)
	at org.yaml.snakeyaml.Yaml.loadFromReader(Yaml.java:524)
	at org.yaml.snakeyaml.Yaml.loadAs(Yaml.java:501)
	at com.github.zzt93.syncer.config.YamlEnvironmentPostProcessor.initConfig(YamlEnvironmentPostProcessor.java:129)
	at com.github.zzt93.syncer.config.YamlEnvironmentPostProcessor.processEnvironment(YamlEnvironmentPostProcessor.java:52)
	at com.github.zzt93.syncer.SyncerApplication.main(SyncerApplication.java:41)
Caused by: org.yaml.snakeyaml.error.YAMLException: Unable to find property 'syncer' on class: com.github.zzt93.syncer.config.syncer.SyncerConfig
	at org.yaml.snakeyaml.introspector.PropertyUtils.getProperty(PropertyUtils.java:189)
	at org.yaml.snakeyaml.introspector.PropertyUtils.getProperty(PropertyUtils.java:178)
	at com.github.zzt93.syncer.config.YamlEnvironmentPostProcessor$1.getProperty(YamlEnvironmentPostProcessor.java:144)
	at org.yaml.snakeyaml.TypeDescription.discoverProperty(TypeDescription.java:240)
	at org.yaml.snakeyaml.TypeDescription.getProperty(TypeDescription.java:251)
	at org.yaml.snakeyaml.constructor.Constructor$ConstructMapping.constructJavaBean2ndStep(Constructor.java:210)
	... 11 common frames omitted

Test framework: performance test

Describe the solution you'd like
Make loadToMysql async, then count from target mysql/es every 5 seconds or so to find sync speed.

Additional context
Upgrade DataGenerator or add a script to support generate foreign key table csv.

What is coldstart.class used for?

hey, I've been reading your project code recently, it's awesome work and I don't know the usage of cold start, so cloud you please explain it? It would be very helpful for me， Thanks

Not follow rotate event & no interested event may cause data loss

Precondtion: Not follow rotate event & no interested event for a specific consumer

Next: This consumer will not update binlog position to follo. If the syncer stopped, then some interested event is added and old binlog is removed. We may just connect to latest binlog position and lost some data.

Solution: listen to rotate event.

impl.md The doc address cannot be accessed

These hyperlinks cannot be accessed

ProduerStarter: too many master source will not run at the same time

ProduerStarter: too many master source will not be able to run at the same time, i.e. some master source will blocked.

    service = Executors
        .newFixedThreadPool(syncerConfigInput.getWorker(),
            new NamedThreadFactory("syncer-producer"));
   // ...
    service.submit(masterConnector);

Bug List

BatchBuffer flush same time
BatchBuffer should be thread local
ES output, insert by query will not retry
ES/Mysql output retry logic opt: if output fail too long, all filter threads will be blocked in retry this output channel; memory will be consumed too much.

TODO list

TODO

Cold-start (ETL) opt
- hold buffer full handle
- hold change after now
Test Framework more
- join
- performance test
Add /stat, /input endpoint for syncer
Timezone config
Convert MySQL integer as byte array
- add auto conversion with meta info fetched: unsigned int: long, unsigned long: byte array
Dependency module: not package & load if not use mongo sync etc.
- SLF4j
- Nginx
IncludeBefore & IncludeUpdated config?
rpm and dpkg
Sync check: query input & output for comparing
- Implement by a special SyncData?
- Should has a http endpoint to invoke it.
Warning if multiple schema.table has different rows
A better serializer than json, which lost the type info: PB?~~Avro?~~
Test redis output & nested sql
Config thread for each consumer
Support set parent of ES
Row image format support?
- Add must appeared field restriction -- now only primary key
- Opt: keep only changed field in update event & primary key in delete event -- include must appear field
Batch module & failure module is coupled with channel module
- Filter chain?
- Failure module as last channel?
MDC.put eventId is necessary??

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.