lalitheswar / storagetapper

StorageTapper is a scalable realtime MySQL change data streaming and transformation service

StorageTapper

StorageTapper is a scalable realtime MySQL change data streaming and transformation service.

Service reads data from MySQL, transforms it into an Avro schema serialized format, and publishes these events to Kafka. Consumers can then consume these events directly from Kafka.

• Features
• Limitations
• Build & Install
• Configuration

Features

• Producing events to Kafka
• Automatic work distribution between instances
• Avro output format
• JSON output format
• HTTP endpoints to control
  • Tables to be ingested
  • Output schema
  • Database address resolver
• Snapshot is taken from slave to reduce load on master
• Binlogs streaming from master for better SLA
• Throttling when taking snapshot

Limitations

• Tables must have a primary key

Build & Install

Get build dependencies

go get github.com/Masterminds/glide
go get github.com/alecthomas/gometalinter

It's recommended to have local MySQL and Kafka installed, so as many tests depend on them. Tweak development config to correspond your setup. WARNING: Tests run dangerous commands like RESET MASTER and DROP DATABASE, so for tests don't use any MySQL instances with precious data.

Debian & Ubuntu

cd storagetapper
make deb && dpkg -i ../storagetapper_1.0_amd64.deb

Others

cd storagetapper
make test && make install

Configuration

Storagetapper loads configuration from the following files and location in the given order:

    /etc/storagetapper/base.yaml
    /etc/storagetapper/production.yaml
    $(HOME)/base.yaml
    $(HOME)/production.yaml
    $(STORAGETAPPER_CONFIG_DIR)/base.yaml
    $(STORAGETAPPER_CONFIG_DIR)/production.yaml

Available options described in Options section

License

This software is licensed under the MIT License.