zendesk / maxwell

Maxwell's daemon, a mysql-to-json kafka producer

This is Maxwell's daemon, a change data capture application that reads MySQL binlogs and writes data changes as JSON to Kafka, Kinesis, and other streaming platforms.

↓ Download | ⚝ Source / Community | ☝ Getting Started | ☷ Reference

What's it for?

• ETL of all sorts
• maintaining an audit log of all changes to your database
• cache building/expiring
• search indexing
• inter-service communication

It goes like this:

  mysql> update `test`.`maxwell` set mycol = 55, daemon = 'Stanislaw Lem';
  maxwell -> kafka: 
  {
    "database": "test",
    "table": "maxwell",
    "type": "insert",
    "ts": 1449786310,
    "data": { "id":1, "daemon": "Stanislaw Lem", "mycol": 55 },
    "old": { "mycol":, 23, "daemon": "what once was" }
  }

maxwell's Issues

Limit maxwell producer to specific SQL databases and tables?

Thanks for making a great tool. I believe this comes 99% of the way to solving my use case. But I need to limit the maxwell producer to only certain SQL databases and tables on the server. In other words, changes to some SQL databases and tables cause maxwell to emit a JSON object, while changes in other databases and tables do not. Is this possible? Thanks!

Missing support for 'BIT' datatype

Probably somewhat unusual, but there's no BIT datatype support:

https://dev.mysql.com/doc/refman/5.6/en/bit-type.html

00:13:15,085 INFO  Maxwell - Maxwell is capturing initial schema
00:13:15,086 DEBUG SchemaStore - Capturing schema
java.lang.IllegalArgumentException: unsupported column type bit
       at com.zendesk.maxwell.schema.columndef.ColumnDef.build(ColumnDef.java:83)

com.zendesk.maxwell.schema.ddl.MaxwellSQLSyntaxError: SELECT

I had this error occur today.

line 4:0 no viable alternative at input 'SELECT'
line 5:3 token recognition error at: '*\r'
10:58:19,973 ERROR MysqlParserListener - (create_table (create_table_preamble CREATE TEMPORARY TABLE (table_name (name (id 172898_16841_transmem)))) SELECT t . FROM map . transmem AS t WHERE t . deleted = 'N' AND IFNULL ( t . trnnumrev , 0 ) = 0 AND t . locked = 'Y' AND t . lockedby = 'AUTOSTL' AND t . allocated = 'N' AND t . special = 'N' AND t . invnum_d = 172898 AND t . imnum = 16841)
10:58:19,974 ERROR SchemaChange - Error parsing SQL: 'CREATE
TEMPORARY
TABLE 172898_16841_transmem
SELECT
 t.*
FROM map.transmem AS t
WHERE t.deleted = 'N'
AND IFNULL(t.trnnumrev, 0) = 0
AND t.locked = 'Y'
AND t.lockedby = 'AUTOSTL'
AND t.allocated = 'N'
AND t.special = 'N'
AND t.invnum_d = 172898
AND t.imnum = 16841'
com.zendesk.maxwell.schema.ddl.MaxwellSQLSyntaxError: SELECT
    at com.zendesk.maxwell.schema.ddl.MysqlParserListener.visitErrorNode(MysqlParserListener.java:85)
    at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:41)
    at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
    at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
    at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
    at com.zendesk.maxwell.schema.ddl.SchemaChange.parse(SchemaChange.java:67)
    at com.zendesk.maxwell.MaxwellReplicator.processQueryEvent(MaxwellReplicator.java:274)
    at com.zendesk.maxwell.MaxwellReplicator.getRow(MaxwellReplicator.java:254)
    at com.zendesk.maxwell.MaxwellReplicator.work(MaxwellReplicator.java:87)
    at com.zendesk.maxwell.RunLoopProcess.runLoop(RunLoopProcess.java:31)
    at com.zendesk.maxwell.Maxwell.run(Maxwell.java:89)
    at com.zendesk.maxwell.Maxwell.main(Maxwell.java:95)
10:58:19,975 INFO  SchemaPosition - Storing final position: BinlogPosition[awmstvx00441-bin.000001:79187451]

maxwell CDC with Apache NiFi

I am using maxwell CDC solution along with Apache NiFi and it is working great:
https://github.com/xmlking/nifi-examples/tree/master/oltp-cdc-olap

How i can know the amount of data that maxwell hasn't send yet

Hi
i am trying to transfer data from mysql to kafka by using maxwell . the day before yesterday ,about 09:30,I construct three million rows in one table in mysql database.and the maxwell began to load data to kafka. but at now the maxwell is still display message like bellow
11:41:36,914 DEBUG MaxwellKafkaProducer - {"database":"emon","table":"msql","type":"insert","data":{"EDITION_NAME":"a","OBJECT_NAME":"ob","GENERATED":"a","OBJECT_TYPE":"OO","OBJECT_ID":813790,"CREATED":"2015-08-28","STATUS":"a","OWNER":"测试","SECONDARY":"a","SUBOBJECT_NAME":"o","TEMPORARY":"a","TIMESTAMP":"1","DATA_OBJECT_ID":813790,"NAMESPACE":1,"LAST_DDL_TIME":"2015-08-28"},"ts":1441165290}
11:41:36,914 DEBUG MaxwellKafkaProducer - BinlogPosition[master.000007:1013840679]
11:41:36,914 DEBUG MaxwellKafkaProducer -
11:41:37,219 DEBUG SchemaPosition - Writing binlog position to maxwell.positions: BinlogPosition[master.000007:1013840679]

how i can know the amount of the data that hasn't been transferred by maxwell.

Crash if column collation specified in CREATE statement

Maxwell crashes if collation is specified in a CREATE statement.

MySQL version 5.1.73
Maxwell version 0.10.0

15:14:12,892 DEBUG SchemaPosition - Writing binlog position to maxwell.positions: BinlogPosition[mysql-binlog.000001:438782]
15:16:15,900 DEBUG SchemaChange - SQL_PARSE <- "CREATE TABLE `maxwell_collate_test` (
    `id` INT(10) NOT NULL AUTO_INCREMENT,
    `parent_id` INT(10) NULL DEFAULT NULL,
    `model` VARCHAR(255) NULL DEFAULT NULL COLLATE 'utf8_unicode_ci',
    `foreign_key` INT(10) NULL DEFAULT NULL,
    PRIMARY KEY (`id`),
    INDEX `parent` (`parent_id`),
    INDEX `foreign_key` (`foreign_key`),
    INDEX `model` (`model`)
)
COLLATE='utf8_unicode_ci'
ENGINE=InnoDB
ROW_FORMAT=COMPACT
AUTO_INCREMENT=371"
line 4:40 extraneous input 'COLLATE' expecting {',', ')', AUTO_INCREMENT, COLUMN_FORMAT, COMMENT, DEFAULT, NOT, NULL, ON, PRIMARY, STORAGE, UNIQUE}
line 4:48 missing '=' at ''utf8_unicode_ci''
15:16:15,940 ERROR MysqlParserListener - (create_specifications ( (create_specification (column_definition (name (id `id`)) (data_type (signed_type INT (length ( 10 )))) (column_options (nullability NOT NULL)) (column_options AUTO_INCREMENT))) , (create_specification (column_definition (name (id `parent_id`)) (data_type (signed_type INT (length ( 10 )))) (column_options (nullability NULL)) (column_options (default_value DEFAULT NULL)))) , (create_specification (column_definition (name (id `model`)) (data_type (string_type VARCHAR (length ( 255 )))) (column_options (nullability NULL)) (column_options (default_value DEFAULT NULL)))) <missing ')'>)
15:16:15,940 ERROR SchemaChange - Error parsing SQL: 'CREATE TABLE `maxwell_collate_test` (
    `id` INT(10) NOT NULL AUTO_INCREMENT,
    `parent_id` INT(10) NULL DEFAULT NULL,
    `model` VARCHAR(255) NULL DEFAULT NULL COLLATE 'utf8_unicode_ci',
    `foreign_key` INT(10) NULL DEFAULT NULL,
    PRIMARY KEY (`id`),
    INDEX `parent` (`parent_id`),
    INDEX `foreign_key` (`foreign_key`),
    INDEX `model` (`model`)
)
COLLATE='utf8_unicode_ci'
ENGINE=InnoDB
ROW_FORMAT=COMPACT
AUTO_INCREMENT=371'
com.zendesk.maxwell.schema.ddl.MaxwellSQLSyntaxError: <missing ')'>
    at com.zendesk.maxwell.schema.ddl.MysqlParserListener.visitErrorNode(MysqlParserListener.java:85)
    at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:41)
    at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
    at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
    at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
    at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
    at com.zendesk.maxwell.schema.ddl.SchemaChange.parse(SchemaChange.java:67)
    at com.zendesk.maxwell.MaxwellParser.processQueryEvent(MaxwellParser.java:208)
    at com.zendesk.maxwell.MaxwellParser.getEvent(MaxwellParser.java:194)
    at com.zendesk.maxwell.MaxwellParser.getEvent(MaxwellParser.java:200)
    at com.zendesk.maxwell.MaxwellParser.run(MaxwellParser.java:100)
    at com.zendesk.maxwell.Maxwell.run(Maxwell.java:69)
    at com.zendesk.maxwell.Maxwell.main(Maxwell.java:75)
2015-09-10 15:16:15,981 FATAL Unable to register shutdown hook because JVM is shutting down.

Support for binlog_row_image = minimal

We turned on row-based replication in our mysql database but we also changed the binlog_row_image to be minimal rather than full. When we did this, Maxwell was logging the following message: "binlog_row_image must be FULL"
From our understanding minimal row image will log only changed columns, and columns needed to identify rows which should be enough for our purposes. This will avoid performance issues. Is there any plan to support MINIMAL binlog_row_image?

Resetting AUTO_INCREMENT in ALTER statement causes crash

Updating a table's AUTO_INCREMENT value causes maxwell to crash when it encounters the statement in the bin log.

Maxwell 0.11.2
MySQL 5.1.

Statement to reproduce

ALTER TABLE test_table AUTO_INCREMENT = 1

11:01:31,725 DEBUG SchemaChange - SQL_PARSE <- "ALTER TABLE test_table AUTO_INCREMENT = 1"
line 1:41 no viable alternative at input 'AUTO_INCREMENT'
11:01:31,751 ERROR MysqlParserListener - (alter_specification AUTO_INCREMENT = 1)
11:01:31,751 ERROR SchemaChange - Error parsing SQL: 'ALTER TABLE test_table AUTO_INCREMENT = 1'
com.zendesk.maxwell.schema.ddl.MaxwellSQLSyntaxError: AUTO_INCREMENT
    at com.zendesk.maxwell.schema.ddl.MysqlParserListener.visitErrorNode(MysqlParserListener.java:85)
    at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:41)
    at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
    at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
    at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
    at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
    at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
    at com.zendesk.maxwell.schema.ddl.SchemaChange.parse(SchemaChange.java:67)
    at com.zendesk.maxwell.MaxwellParser.processQueryEvent(MaxwellParser.java:284)
    at com.zendesk.maxwell.MaxwellParser.getEvent(MaxwellParser.java:251)
    at com.zendesk.maxwell.MaxwellParser.run(MaxwellParser.java:93)
    at com.zendesk.maxwell.Maxwell.run(Maxwell.java:73)
    at com.zendesk.maxwell.Maxwell.main(Maxwell.java:79)

I assume Maxwell should be safe to ignore this, or does it track the auto increment value?

kafka to hbase or hdfs example

Do you have example or reference code to read this events from kafka and persists on hbase or hdfs?

Quickstart instructions lead to error

After following instructions to the letter in the quickstart, I get the following error when I run the bin/maxwell command:

Slave can not handle replication events with the checksum that master is configured to log..

After googling the error, one of the solutions was to turn of binlog_checksum, which I did. Once I did that, I get this:

Could not find first log file name in binary log index file.

I'm using MySQL 5.6 on Ubuntu. What could be going wrong?

Tests fail - boot_isolated_mysql_server missing

mvn clean install
DEBUG MysqlIsolatedServer - Booting server: /Users/agaton/maxwell/src/test/mysql_isolated_server/bin/boot_isolated_mysql_server -- --binlog_format=row --innodb_flush_log_at_trx_commit=1 --server_id=123123
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.008 sec <<< FAILURE!
com.zendesk.maxwell.SchemaCaptureTest Time elapsed: 0.007 sec <<< ERROR!
java.io.IOException: Cannot run program "/Users/agaton/maxwell/src/test/mysql_isolated_server/bin/boot_isolated_mysql_server": error=2, No such file or directory

BINARY / VARBINARY support

Arose when testing #94.

Any love for BINARY and VARBINARY columns? I naively presume they could just be treated the same as BLOB and its variants?

16:03:46,359 DEBUG SchemaChange - SQL_PARSE <- "ALTER TABLE `maxwell_blob_test`
    CHANGE COLUMN `blob` `blob` BINARY(50) NOT NULL AFTER `id`"
java.lang.IllegalArgumentException: unsupported column type binary
    at com.zendesk.maxwell.schema.columndef.ColumnDef.build(ColumnDef.java:81)
    at com.zendesk.maxwell.schema.ddl.MysqlParserListener.exitColumn_definition(MysqlParserListener.java:273)
    at com.zendesk.maxwell.schema.ddl.mysqlParser$Column_definitionContext.exitRule(mysqlParser.java:4370)
    at org.antlr.v4.runtime.tree.ParseTreeWalker.exitRule(ParseTreeWalker.java:71)
    at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:54)
    at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
    at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
    at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
    at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
    at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
    at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
    at com.zendesk.maxwell.schema.ddl.SchemaChange.parse(SchemaChange.java:67)
    at com.zendesk.maxwell.MaxwellParser.processQueryEvent(MaxwellParser.java:208)
    at com.zendesk.maxwell.MaxwellParser.getEvent(MaxwellParser.java:194)
    at com.zendesk.maxwell.MaxwellParser.getEvent(MaxwellParser.java:200)
    at com.zendesk.maxwell.MaxwellParser.run(MaxwellParser.java:100)
    at com.zendesk.maxwell.Maxwell.run(Maxwell.java:69)
    at com.zendesk.maxwell.Maxwell.main(Maxwell.java:75)
16:03:46,362 DEBUG SchemaPosition - Writing binlog position to maxwell.positions: BinlogPosition[mysql-binlog.000003:648414]

when install maxwell 0.11.1 it raise error gzip: stdin: unexpected end of file

Hi
i am installing maxwell 0.11.1 using command as below:
curl -sLo - https://github.com/zendesk/maxwell/releases/download/v0.11.1/maxwell-0.11.1.tar.gz
| tar zxvf -
then it raise error :
gzip: stdin: unexpected end of file
tar: Child returned status 1
tar: Error is not recoverable: exiting now

Feature Request: Maxwell log show date as well as time

the current log has this format

09:01:07,003 DEBUG SchemaChange - ...

it would be nice to have the date included with the time so that you can tell if maxwell is still running correctly and responsively.

Backtracking master recovery

From the docs...

Currently Maxwell is not very smart about master recovery or detecting a promoted slave; if it determines that the server_id has changed between runs, Maxwell will simply delete its old schema cache and binlog position and start again. We plan on improving this situation in 0.12.

From the changelog I guess this hasn't changed yet. Are there any new targets for this? I presume it's still on the roadmap?

Great job!

Hi!

I wanted to say fantastic work on this project. I've been working on identical functionality for about 6 months and was just about to open-source my code, but looks like you guys got there first! shakes fist at the sky From what I've seen so far, Maxwell seems farther along than mine, so I'm not sure what I'll do with mine now.

I have a bunch of design questions. Can I get your email address so we can talk more about it? Or, if you'd rather discuss here on Github, that works too, but it might get pretty long. My email is [email protected]. Looking forward to talking to you.

-James Cheng

Un-Graceful exit when encountering TRUNCATE statements

I issued a truncate table statement and Maxwell didn't like it. To the extent that it crashed, and failed every time I tried to run it. I had to use the purge binary logs before now() and delete from maxwell.positions commands to get it running again. Here is the log:

TRUNCATE table error:

23:12:55,915 DEBUG SchemaChange - SQL_PARSE <- "truncate table max_test3"
line 1:0 no viable alternative at input 'truncate'
23:12:55,941 ERROR MysqlParserListener - (statement truncate table max_test3)
23:12:55,942 ERROR SchemaChange - Error parsing SQL: 'truncate table max_test3'
com.zendesk.maxwell.schema.ddl.MaxwellSQLSyntaxError: truncate
at com.zendesk.maxwell.schema.ddl.MysqlParserListener.visitErrorNode(MysqlParserListener.java:85)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:41)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at com.zendesk.maxwell.schema.ddl.SchemaChange.parse(SchemaChange.java:65)
at com.zendesk.maxwell.MaxwellParser.processQueryEvent(MaxwellParser.java:206)
at com.zendesk.maxwell.MaxwellParser.getEvent(MaxwellParser.java:192)
at com.zendesk.maxwell.MaxwellParser.getEvent(MaxwellParser.java:198)
at com.zendesk.maxwell.MaxwellParser.run(MaxwellParser.java:98)
at com.zendesk.maxwell.Maxwell.run(Maxwell.java:69)
at com.zendesk.maxwell.Maxwell.main(Maxwell.java:75)
23:12:55,953 DEBUG SchemaPosition - Writing binlog position to maxwell.positions: BinlogPosition[mysql-logs.000003:6424]

Maxwell does not recover after truncate statement:

bin/maxwell --user='root' --password='root' --host='127.0.0.1' --producer=stdout
Picked up JAVA_TOOL_OPTIONS: -javaagent:/usr/share/java/jayatanaag.jar 
23:17:15,992 INFO  Maxwell - Maxwell is booting, starting at BinlogPosition[mysql-logs.000003:6424]
23:17:15,993 DEBUG SchemaStore - looking to restore schema at target position BinlogPosition[mysql-logs.000003:6424]
23:17:15,995 INFO  SchemaStore - Restoring schema id 6 (last modified at BinlogPosition[mysql-logs.000003:330])
23:17:16,057 INFO  TransportImpl - connecting to host: 127.0.0.1, port: 3306
23:17:16,088 INFO  TransportImpl - connected to host: 127.0.0.1, port: 3306, context: AbstractTransport.Context[threadId=18,scramble=EDcI$uB*Gi[m4V5E>E}h,protocolVersion=10,serverHost=127.0.0.1,serverPort=3306,serverStatus=2,serverCollation=8,serverVersion=5.6.24-0ubuntu2-log,serverCapabilities=63487]
23:17:16,088 INFO  AuthenticatorImpl - start to login, user: root, host: 127.0.0.1, port: 3306
23:17:16,093 INFO  AuthenticatorImpl - login successfully, user: root, detail: OKPacket[packetMarker=0,affectedRows=0,insertId=0,serverStatus=2,warningCount=0,message=<null>]
23:17:16,226 DEBUG SchemaChange - SQL_PARSE <- "truncate table max_test3"
line 1:0 no viable alternative at input 'truncate'
23:17:16,274 ERROR MysqlParserListener - (statement truncate table max_test3)
23:17:16,283 ERROR SchemaChange - Error parsing SQL: 'truncate table max_test3'
com.zendesk.maxwell.schema.ddl.MaxwellSQLSyntaxError: truncate
at com.zendesk.maxwell.schema.ddl.MysqlParserListener.visitErrorNode(MysqlParserListener.java:85)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:41)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at com.zendesk.maxwell.schema.ddl.SchemaChange.parse(SchemaChange.java:65)
at com.zendesk.maxwell.MaxwellParser.processQueryEvent(MaxwellParser.java:206)
at com.zendesk.maxwell.MaxwellParser.getEvent(MaxwellParser.java:192)
at com.zendesk.maxwell.MaxwellParser.getEvent(MaxwellParser.java:198)
at com.zendesk.maxwell.MaxwellParser.run(MaxwellParser.java:98)
at com.zendesk.maxwell.Maxwell.run(Maxwell.java:69)
at com.zendesk.maxwell.Maxwell.main(Maxwell.java:75)

Please add support for TRUNCATE statements, or at least ignore them (maybe not a great idea :)). But this kind of failure is no good. Thanks!

Tests hang when running MysqlIsolatedServer on Ubuntu 14 & 15

running the tests using mvn install results in the following behavior:

Running com.zendesk.maxwell.SchemaStoreTest
DEBUG MysqlIsolatedServer - booting onetimeserver: ./src/test/onetimeserver --mysql-version=5.5 --log-bin=master --binlog_format=row --innodb_flush_log_at_trx_commit=1 --server_id=123123

which then hangs forever, even though a ps aux | grep mysql shows the wrapper, onetimeserver-go and mysqld processes running along famously.

Parse table names that look like floating point numbers

Parse error showed up in my log

2015-12-14T13:40:57,653 ERROR com.zendesk.maxwell.schema.ddl.MysqlParserListener: (alter_specification .20151214 _TSP_16819__CGPos_0_C7D1A82D5B6243259A60D3CAD54A4B61 ADD KEY `TRNNUM` ( `trnnum`)
2015-12-14T13:40:57,653 ERROR com.zendesk.maxwell.schema.ddl.SchemaChange: Error parsing SQL: 'ALTER TABLE cgt.20151214_TSP_16819__CGPos_0_C7D1A82D5B6243259A60D3CAD54A4B61 asis-uat3.hq.local

Terminates after MaxwelSQLSyntaxError

Running on maxwell-0.14.4

08:28:30,736 DEBUG SchemaChange - SQL_PARSE <- "CREATE TABLE 
                     `mvlogs` (table_schema varchar(50), 
                             table_name varchar(50), 
                             mvlog_name varchar(50),
                             active_flag boolean default true,
                             primary key(table_schema,table_name),
                             unique key(mvlog_name)
                        ) ENGINE=INNODB DEFAULT CHARSET=utf8"
line 5:57 no viable alternative at input 'true'
08:28:30,743 ERROR MysqlParserListener - (default_value default true)
08:28:30,743 ERROR SchemaChange - Error parsing SQL: 'CREATE TABLE 
                     `mvlogs` (table_schema varchar(50), 
                             table_name varchar(50), 
                             mvlog_name varchar(50),
                             active_flag boolean default true,
                             primary key(table_schema,table_name),
                             unique key(mvlog_name)
                        ) ENGINE=INNODB DEFAULT CHARSET=utf8'
com.zendesk.maxwell.schema.ddl.MaxwellSQLSyntaxError: true
    at com.zendesk.maxwell.schema.ddl.MysqlParserListener.visitErrorNode(MysqlParserListener.java:85)
    at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:41)
    at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
    at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
    at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
    at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
    at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
    at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
    at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
    at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
    at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
    at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
    at com.zendesk.maxwell.schema.ddl.SchemaChange.parse(SchemaChange.java:67)
    at com.zendesk.maxwell.MaxwellReplicator.processQueryEvent(MaxwellReplicator.java:273)
    at com.zendesk.maxwell.MaxwellReplicator.getEvent(MaxwellReplicator.java:253)
    at com.zendesk.maxwell.MaxwellReplicator.work(MaxwellReplicator.java:89)
    at com.zendesk.maxwell.RunLoopProcess.runLoop(RunLoopProcess.java:31)
    at com.zendesk.maxwell.Maxwell.run(Maxwell.java:89)
    at com.zendesk.maxwell.Maxwell.main(Maxwell.java:95)
08:28:30,744 INFO  SchemaPosition - Storing final position: BinlogPosition[awmsphx90022-bin.000003:237982669]
08:28:30,745 DEBUG SchemaPosition - Writing binlog position to maxwell.positions: BinlogPosition[awmsphx90022-bin.000003:237982669]

when loading chinese from mysql .maxwell display messy code

Hi
I encounter a problem when using maxwell to load change data from mysql . I insert a chinese in a table and it display normol in mysql but messy code in maxwell
in mysql:
mysql> insert into a values("哈哈");
Query OK, 1 row affected (0.00 sec)

mysql> select * from a;
+------+
| col |
+------+
| 测试 |
| 哈哈 |
+------+
2 rows in set (0.00 sec)

mysql> exit
Bye
[root@localhost maxwell-0.9.0]# echo $LANG
en_US.UTF-8

in maxwell:
16:47:18,504 DEBUG SchemaPosition - Writing binlog position to maxwell.positions: BinlogPosition[master.000002:1832]
16:48:45,903 DEBUG NetworkClient - Initiating connection to node 1 at 10.89.131.15:9092.
16:48:45,907 DEBUG NetworkClient - Completed connection to node 1
16:48:45,918 DEBUG MaxwellKafkaProducer - -> key:{"database":"test","table":"a","pk._uuid":"6fcd9885-d3a3-4d32-a0dc-064a119e8429"}, partition:1, offset:86427
16:48:45,918 DEBUG MaxwellKafkaProducer - {"database":"test","table":"a","type":"insert","data":{"col":"¹þ¹þ"},"ts":1440406125}
16:48:45,919 DEBUG MaxwellKafkaProducer - BinlogPosition[master.000002:7471]

it dispay "¹þ¹þ".

CREATE SCHEMA causes MaxwellSQLSyntaxError

Running Maxwell 0.14.5

15:19:53,024 DEBUG SchemaChange - SQL_PARSE <- "CREATE SCHEMA product_service_test"
line 1:7 no viable alternative at input 'CREATESCHEMA'
15:19:53,026 ERROR MysqlParserListener - (statement CREATE SCHEMA product_service_test)
15:19:53,026 ERROR SchemaChange - Error parsing SQL: 'CREATE SCHEMA product_service_test'
com.zendesk.maxwell.schema.ddl.MaxwellSQLSyntaxError: CREATE
    at com.zendesk.maxwell.schema.ddl.MysqlParserListener.visitErrorNode(MysqlParserListener.java:85)
    at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:41)
    at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
    at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
    at com.zendesk.maxwell.schema.ddl.SchemaChange.parse(SchemaChange.java:67)
    at com.zendesk.maxwell.MaxwellReplicator.processQueryEvent(MaxwellReplicator.java:273)
    at com.zendesk.maxwell.MaxwellReplicator.getEvent(MaxwellReplicator.java:253)
    at com.zendesk.maxwell.MaxwellReplicator.work(MaxwellReplicator.java:89)
    at com.zendesk.maxwell.RunLoopProcess.runLoop(RunLoopProcess.java:31)
    at com.zendesk.maxwell.Maxwell.run(Maxwell.java:89)
    at com.zendesk.maxwell.Maxwell.main(Maxwell.java:95)
15:19:53,028 INFO  SchemaPosition - Storing final position: null

Support parsing CHECK( ... ) in CREATE TABLE

15:59:29,552 ERROR MysqlParserListener - (data_type ( valid IN ( 0)
15:59:29,553 ERROR SchemaChange - Error parsing SQL: 'CREATE TABLE xxx (
id INTEGER NOT NULL AUTO_INCREMENT,
...,
valid BOOL,
...,
PRIMARY KEY (id),
CHECK (valid IN (0, 1))
)'
com.zendesk.maxwell.schema.ddl.MaxwellSQLSyntaxError: (
at com.zendesk.maxwell.schema.ddl.MysqlParserListener.visitErrorNode(MysqlParserListener.java:88)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:41)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at com.zendesk.maxwell.schema.ddl.SchemaChange.parse(SchemaChange.java:74)
at com.zendesk.maxwell.MaxwellReplicator.processQueryEvent(MaxwellReplicator.java:288)
at com.zendesk.maxwell.MaxwellReplicator.getRow(MaxwellReplicator.java:268)
at com.zendesk.maxwell.MaxwellReplicator.work(MaxwellReplicator.java:89)
at com.zendesk.maxwell.RunLoopProcess.runLoop(RunLoopProcess.java:31)
at com.zendesk.maxwell.Maxwell.run(Maxwell.java:91)
at com.zendesk.maxwell.Maxwell.main(Maxwell.java:97)
15:59:29,562 INFO SchemaPosition - Storing final position: null

Documentation on Kafka key is incorrect

The website http://maxwells-daemon.io/kafka/ says the format is something like

db:test_db/tbl:test_tbl/id:93/id_2:17910090

But in practice, it appears the key is actually json:

{"database":"test","table":"users","pk.userId":4}

This output is from my test database.

Maxwell needs to parse SQL inside /*! */ "comments"

I have been seeing this error while evaluating Maxwell in a Fedora VM.

09:25:53,554 DEBUG SchemaChange - SQL_PARSE <- "CREATE DATABASE /*! IF NOT EXISTS */ error"
09:25:53,556 DEBUG SchemaChange - SQL_PARSE ->   (parse (statement (create_database CREATE DATABASE (name (id error)))) <EOF>)
com.zendesk.maxwell.schema.ddl.SchemaSyncError: Unexpectedly asked to create existing database error
    at com.zendesk.maxwell.schema.ddl.DatabaseCreate.apply(DatabaseCreate.java:25)
    at com.zendesk.maxwell.MaxwellReplicator.processQueryEvent(MaxwellReplicator.java:280)
    at com.zendesk.maxwell.MaxwellReplicator.getEvent(MaxwellReplicator.java:253)
    at com.zendesk.maxwell.MaxwellReplicator.work(MaxwellReplicator.java:89)
    at com.zendesk.maxwell.RunLoopProcess.runLoop(RunLoopProcess.java:31)
    at com.zendesk.maxwell.Maxwell.run(Maxwell.java:89)
    at com.zendesk.maxwell.Maxwell.main(Maxwell.java:95)
09:25:53,558 INFO  SchemaPosition - Storing final position: null

A similar error occurs at the table level.

10:24:04,086 DEBUG SchemaChange - SQL_PARSE <- "create table /*! if not exists */ `_work`.`cdctest3` (
  `id` int (10) unsigned not null auto_increment,
  primary key (`id`)
)"
10:24:04,090 DEBUG SchemaChange - SQL_PARSE ->   (parse (statement (create_table (create_table_preamble create table (table_name (db_name (name (id `_work`))) . (name (id `cdctest3`)))) (create_specifications ( (create_specification (column_definition (name (id `id`)) (data_type (signed_type int (length ( 10 )) (int_flags unsigned) (column_options (nullability not null)) (column_options auto_increment))))) , (create_specification (index_definition (index_type_pk primary key (index_column_list ( (index_columns (index_column (name (id `id`)))) ))))) )))) <EOF>)
com.zendesk.maxwell.schema.ddl.SchemaSyncError: Unexpectedly asked to create existing table cdctest3
    at com.zendesk.maxwell.schema.ddl.TableCreate.apply(TableCreate.java:45)
    at com.zendesk.maxwell.MaxwellReplicator.processQueryEvent(MaxwellReplicator.java:280)
    at com.zendesk.maxwell.MaxwellReplicator.getEvent(MaxwellReplicator.java:253)
    at com.zendesk.maxwell.MaxwellReplicator.work(MaxwellReplicator.java:89)
    at com.zendesk.maxwell.RunLoopProcess.runLoop(RunLoopProcess.java:31)
    at com.zendesk.maxwell.Maxwell.run(Maxwell.java:89)
    at com.zendesk.maxwell.Maxwell.main(Maxwell.java:95)
10:24:04,188 INFO  SchemaPosition - Storing final position: BinlogPosition[awmsphx90022-bin.000003:203414608]
10:24:04,189 DEBUG SchemaPosition - Writing binlog position to maxwell.positions: BinlogPosition[awmsphx90022-bin.000003:203414608]

Output some form of the DDL in kafka

Hi Guys,

When I test the maxwell, I find it only can send the DML to kafka, but DDL, like create,drop,alter.
Do I miss something? or it can't support this feature?

And I also can't make sure which database the table is created in from DEBUG log.
e.g.
17:47:13,088 DEBUG SchemaChange - SQL_PARSE <- "create table t (id int auto_increment primary key,name varchar(10))"

Un-Graceful exit when table with 'real' datatype is created

Similar to #64, Maxwell chokes when it encounters a field with real type.

Maxwell error with real datatype:

22:39:47,906 DEBUG SchemaChange - SQL_PARSE <- "create table max_test2 (col1 int, col2 tinyint, col3 bigint, col4 decimal, col5 float, col6 real)"
java.lang.IllegalArgumentException: unsupported column type real
at com.zendesk.maxwell.schema.columndef.ColumnDef.build(ColumnDef.java:73)
at com.zendesk.maxwell.schema.ddl.MysqlParserListener.exitColumn_definition(MysqlParserListener.java:273)
at com.zendesk.maxwell.schema.ddl.mysqlParser$Column_definitionContext.exitRule(mysqlParser.java:4362)
at org.antlr.v4.runtime.tree.ParseTreeWalker.exitRule(ParseTreeWalker.java:71)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:54)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at com.zendesk.maxwell.schema.ddl.SchemaChange.parse(SchemaChange.java:65)
at com.zendesk.maxwell.MaxwellParser.processQueryEvent(MaxwellParser.java:206)
at com.zendesk.maxwell.MaxwellParser.getEvent(MaxwellParser.java:192)
at com.zendesk.maxwell.MaxwellParser.getEvent(MaxwellParser.java:198)
at com.zendesk.maxwell.MaxwellParser.run(MaxwellParser.java:98)
at com.zendesk.maxwell.Maxwell.run(Maxwell.java:69)
at com.zendesk.maxwell.Maxwell.main(Maxwell.java:75)
2015-06-18 05:39:47,952 FATAL Unable to register shutdown hook because JVM is shutting down.

After error with real, subsequent runs fail:

bin/maxwell --user='root' --password='root' --host='127.0.0.1' --producer=stdout
Picked up JAVA_TOOL_OPTIONS: -javaagent:/usr/share/java/jayatanaag.jar 
22:47:29,260 INFO  Maxwell - Maxwell is booting, starting at BinlogPosition[mysql-logs.000002:11877]
22:47:29,262 DEBUG SchemaStore - looking to restore schema at target position BinlogPosition[mysql-logs.000002:11877]
22:47:29,263 INFO  SchemaStore - Restoring schema id 5 (last modified at BinlogPosition[mysql-logs.000002:9053])
22:47:29,305 INFO  TransportImpl - connecting to host: 127.0.0.1, port: 3306
22:47:29,323 INFO  TransportImpl - connected to host: 127.0.0.1, port: 3306, context: AbstractTransport.Context[threadId=28,scramble=\\9IAJ3[TpYMS<o="0kK,protocolVersion=10,serverHost=127.0.0.1,serverPort=3306,serverStatus=2,serverCollation=8,serverVersion=5.6.24-0ubuntu2-log,serverCapabilities=63487]
22:47:29,324 INFO  AuthenticatorImpl - start to login, user: root, host: 127.0.0.1, port: 3306
22:47:29,332 INFO  AuthenticatorImpl - login successfully, user: root, detail: OKPacket[packetMarker=0,affectedRows=0,insertId=0,serverStatus=2,warningCount=0,message=<null>]
22:47:29,550 DEBUG SchemaChange - SQL_PARSE <- "create table max_test2 (col1 int, col2 tinyint, col3 bigint, col4 decimal, col5 float, col6 real)"
java.lang.IllegalArgumentException: unsupported column type real
at com.zendesk.maxwell.schema.columndef.ColumnDef.build(ColumnDef.java:73)
at com.zendesk.maxwell.schema.ddl.MysqlParserListener.exitColumn_definition(MysqlParserListener.java:273)
at com.zendesk.maxwell.schema.ddl.mysqlParser$Column_definitionContext.exitRule(mysqlParser.java:4362)
at org.antlr.v4.runtime.tree.ParseTreeWalker.exitRule(ParseTreeWalker.java:71)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:54)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at com.zendesk.maxwell.schema.ddl.SchemaChange.parse(SchemaChange.java:65)
at com.zendesk.maxwell.MaxwellParser.processQueryEvent(MaxwellParser.java:206)
at com.zendesk.maxwell.MaxwellParser.getEvent(MaxwellParser.java:192)
at com.zendesk.maxwell.MaxwellParser.getEvent(MaxwellParser.java:198)
at com.zendesk.maxwell.MaxwellParser.run(MaxwellParser.java:98)
at com.zendesk.maxwell.Maxwell.run(Maxwell.java:69)
at com.zendesk.maxwell.Maxwell.main(Maxwell.java:75)

Add command-line flag to reset master position

I am testing Maxwell.
I am getting following error when started maxwell today. is there a way to fix this if my binary log files are rotated/deleted?

12:04:14,048 INFO  AuthenticatorImpl - start to login, user: maxwell, host: localhost, port: 3306
12:04:14,051 INFO  AuthenticatorImpl - login successfully, user: maxwell, detail: OKPacket[packetMarker=0,affectedRows=0,insertId=0,serverStatus=2,warningCount=0,message=<null>]
com.google.code.or.net.TransportException: Could not find first log file name in binary log index file
        at com.google.code.or.OpenReplicator.dumpBinlog(OpenReplicator.java:288)
        at com.google.code.or.OpenReplicator.start(OpenReplicator.java:104)
        at com.zendesk.maxwell.MaxwellReplicator.beforeStart(MaxwellReplicator.java:83)
        at com.zendesk.maxwell.RunLoopProcess.runLoop(RunLoopProcess.java:25)
        at com.zendesk.maxwell.Maxwell.run(Maxwell.java:89)
        at com.zendesk.maxwell.Maxwell.main(Maxwell.java:95)
12:04:14,079 INFO  SchemaPosition - Storing final position: null

Warning on tables non-included tables

One of my environments is reporting this WARN (a lot!).

14:31:07,477 WARN  MaxwellReplicator - Unhandled QueryEvent inside transaction: QueryEvent[header=BinlogEventV4HeaderImpl[timestamp=1449710974000,eventType=2,serverId=441,eventLength=279,nextPosition=219695949,flags=0,timestampOfReceipt=1449718267477],threadId=4,elapsedTime=0,databaseNameLength=6,errorCode=0,statusVariablesLength=35,statusVariables=[QFlags2Code[flags=0], QSQLModeCode[sqlMode=2097152], QCatalogNzCode[catalogName=std], QCharsetCode[characterSetClient=8,collationConnection=8,collationServer=8], QUpdatedDBNames[accessedDbCount=1,accessedDbs=[Lcom.google.code.or.common.glossary.column.StringColumn;@2da0770]],databaseName=merlyn,sql=INSERT INTO merlyn.system_variables(project, var_name, var_value) VALUES('xxxxx', 'yyyyy', 'zzzzz') ON DUPLICATE KEY UPDATE var_value= 'zzzzz']

My config file is setup to include specify databases (include_dbs) and tables (include_tables). Both merlyn and system_variables are not included.

Is there any reason that Maxwell needs to care about tables and databases that I'm not tracking events for?

And secondly, if I got this WARN on a table that I was tracking, what does it actually mean? have I missed a change? What is causing this WARN?

Bootstrap initial database into kafka

Can we use this to do a Mysql db dump to a kafka topic.

Allow disabling of xid/commit fields to avoid disk buffering

I have seen an issue where if I update a large amount of records, ~400,000+ rows with a single update query, a memory limit is reached and a spill over to disk, sometimes never finishing or recovering.

Is there a way to avoid such an issue? Or a way around it? Increasing some kind of memory limit?

This will also happen if an alter statement, i.e adding a new column with a new default value, occurs on a relatively large table..~1,000,000 rows.

No ObjectCodec defined for the generator, ... (type passed java.sql.Time)

I'm experimenting a little with maxwell, following the getting started guide and appear to have hit an issue while processing an event generated by the following row:

mysql> select * from events where event_id = 5;
+----------+--------------+---------------------+----------------------------------+-----------+---------+----------------------------------+--------------+
| event_id | business_ref | when                | what_ref                         | what_type | action  | who_ref                          | deleted_name |
+----------+--------------+---------------------+----------------------------------+-----------+---------+----------------------------------+--------------+
|        5 |            1 | 2015-11-25 23:59:00 | 785d226973bf4eae9bfc903f0b9f2033 | Task      | updated | d4ed50fe84ba473fb2690b149e7f2933 | NULL         |
+----------+--------------+---------------------+----------------------------------+-----------+---------+----------------------------------+--------------+`

From the stack trace it appears that maxwell has trouble with the when column, which is of type datetime:

08:54:57,083 ERROR MaxwellReplicator - Got an unexpected row-event: WriteRowsEvent[header=BinlogEventV4HeaderImpl[timestamp=1448495940000,eventType=23,serverId=1,eventLength=290,nextPosition=2705491,flags=0,timestampOfReceipt=1448528097073],tableId=332,reserved=1,columnCount=10,usedColumns=1111111111,rows=[Row[columns=[560, 1, Wed Nov 25 23:59:00 UTC 2015, 785d226973bf4eae9bfc903f0b9f2033, Task, d4ed50fe84ba473fb2690b149e7f2933, webapp, null, BlobColumn[value=[B@7c417213], b4a678826df0408d869346ff67275627]]]]
java.lang.IllegalStateException: No ObjectCodec defined for the generator, can only serialize simple wrapper types (type passed java.sql.Time)
    at com.fasterxml.jackson.core.JsonGenerator._writeSimpleObject(JsonGenerator.java:1725)
    at com.fasterxml.jackson.core.base.GeneratorBase.writeObject(GeneratorBase.java:327)
    at com.fasterxml.jackson.core.JsonGenerator.writeObjectField(JsonGenerator.java:1415)
    at com.zendesk.maxwell.RowMap.toJSON(RowMap.java:91)
    at com.zendesk.maxwell.MaxwellAbstractRowsEvent.toJSONStrings(MaxwellAbstractRowsEvent.java:257)
    at com.zendesk.maxwell.producer.MaxwellKafkaProducer.push(MaxwellKafkaProducer.java:86)
    at com.zendesk.maxwell.MaxwellReplicator.work(MaxwellReplicator.java:97)
    at com.zendesk.maxwell.RunLoopProcess.runLoop(RunLoopProcess.java:31)
    at com.zendesk.maxwell.Maxwell.run(Maxwell.java:89)
    at com.zendesk.maxwell.Maxwell.main(Maxwell.java:95)
08:54:57,147 INFO  SchemaPosition - Storing final position: null

Have I recognised the problem correctly, or is it likely that the problem lies elsewhere?

---

Version info:
MySQL: 5.5.44
Maxwell: 0.14.5
Java: 1.8.0_66
OS: OS X 10.11.1

Maxwell shutdown on 'bool' and 'boolean'

Same as #64, #65 and #66, Maxwell can't handle 'bool' and (just to be completely sure) 'boolean'. This occurs at the create table stage. To get Maxwell to run again, I had to purge binary logs before now() and delete from maxwell.positions.

BOOL:

bin/maxwell --user='root' --password='root' --host='127.0.0.1' --producer=stdout
Picked up JAVA_TOOL_OPTIONS: -javaagent:/usr/share/java/jayatanaag.jar 
10:40:30,183 INFO  Maxwell - Maxwell is capturing initial schema
10:40:30,185 DEBUG SchemaStore - Capturing schema
10:40:30,217 DEBUG SchemaStore - Finished capturing schema
10:40:30,294 INFO  TransportImpl - connecting to host: 127.0.0.1, port: 3306
10:40:30,343 INFO  TransportImpl - connected to host: 127.0.0.1, port: 3306, context: AbstractTransport.Context[threadId=31,scramble=}_EvG<6<gt16X;]DyW(],protocolVersion=10,serverHost=127.0.0.1,serverPort=3306,serverStatus=2,serverCollation=8,serverVersion=5.6.24-0ubuntu2-log,serverCapabilities=63487]
10:40:30,344 INFO  AuthenticatorImpl - start to login, user: root, host: 127.0.0.1, port: 3306
10:40:30,348 INFO  AuthenticatorImpl - login successfully, user: root, detail: OKPacket[packetMarker=0,affectedRows=0,insertId=0,serverStatus=2,warningCount=0,message=<null>]
10:40:31,175 DEBUG SchemaPosition - Writing binlog position to maxwell.positions: BinlogPosition[mysql-logs.000003:12471]
10:55:36,558 DEBUG SchemaChange - SQL_PARSE <- "create table mmf_test(col1 smallint, col2 int, col3 bigint,  col4 float, col5 date, col6 datetime, col7 decimal(10,2), col8 varchar(500), col9 text, col10 bool)"
line 1:155 no viable alternative at input 'bool'
10:55:36,590 ERROR MysqlParserListener - (data_type bool)
10:55:36,590 ERROR SchemaChange - Error parsing SQL: 'create table mmf_test(col1 smallint, col2 int, col3 bigint,  col4 float, col5 date, col6 datetime, col7 decimal(10,2), col8 varchar(500), col9 text, col10 bool)'
com.zendesk.maxwell.schema.ddl.MaxwellSQLSyntaxError: bool
    at com.zendesk.maxwell.schema.ddl.MysqlParserListener.visitErrorNode(MysqlParserListener.java:85)
    at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:41)
    at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
    at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
    at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
    at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
    at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
    at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
    at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
    at com.zendesk.maxwell.schema.ddl.SchemaChange.parse(SchemaChange.java:65)
    at com.zendesk.maxwell.MaxwellParser.processQueryEvent(MaxwellParser.java:206)
    at com.zendesk.maxwell.MaxwellParser.getEvent(MaxwellParser.java:192)
    at com.zendesk.maxwell.MaxwellParser.getEvent(MaxwellParser.java:198)
    at com.zendesk.maxwell.MaxwellParser.run(MaxwellParser.java:98)
    at com.zendesk.maxwell.Maxwell.run(Maxwell.java:69)
    at com.zendesk.maxwell.Maxwell.main(Maxwell.java:75)
2015-06-18 17:55:36,635 FATAL Unable to register shutdown hook because JVM is shutting down.

BOOLEAN:

bin/maxwell --user='root' --password='root' --host='127.0.0.1' --producer=stdout
Picked up JAVA_TOOL_OPTIONS: -javaagent:/usr/share/java/jayatanaag.jar 
10:58:46,071 INFO  Maxwell - Maxwell is capturing initial schema
10:58:46,073 DEBUG SchemaStore - Capturing schema
10:58:46,111 DEBUG SchemaStore - Finished capturing schema
10:58:47,020 INFO  TransportImpl - connecting to host: 127.0.0.1, port: 3306
10:58:47,044 INFO  TransportImpl - connected to host: 127.0.0.1, port: 3306, context: AbstractTransport.Context[threadId=35,scramble=pWD+Jt,w6D5z*_~!&7#J,protocolVersion=10,serverHost=127.0.0.1,serverPort=3306,serverStatus=2,serverCollation=8,serverVersion=5.6.24-0ubuntu2-log,serverCapabilities=63487]
10:58:47,045 INFO  AuthenticatorImpl - start to login, user: root, host: 127.0.0.1, port: 3306
10:58:47,054 INFO  AuthenticatorImpl - login successfully, user: root, detail: OKPacket[packetMarker=0,affectedRows=0,insertId=0,serverStatus=2,warningCount=0,message=<null>]
10:58:47,063 DEBUG SchemaPosition - Writing binlog position to maxwell.positions: BinlogPosition[mysql-logs.000003:18202]
10:58:55,665 DEBUG SchemaChange - SQL_PARSE <- "create table mmf_test(col1 smallint, col2 int, col3 bigint,  col4 float, col5 date, col6 datetime, col7 decimal(10,2), col8 varchar(500), col9 text, col10 boolean)"
line 1:155 no viable alternative at input 'boolean'
10:58:55,712 ERROR MysqlParserListener - (data_type boolean)
10:58:55,712 ERROR SchemaChange - Error parsing SQL: 'create table mmf_test(col1 smallint, col2 int, col3 bigint,  col4 float, col5 date, col6 datetime, col7 decimal(10,2), col8 varchar(500), col9 text, col10 boolean)'
com.zendesk.maxwell.schema.ddl.MaxwellSQLSyntaxError: boolean
    at com.zendesk.maxwell.schema.ddl.MysqlParserListener.visitErrorNode(MysqlParserListener.java:85)
    at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:41)
    at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
    at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
    at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
    at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
    at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
    at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
    at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
    at com.zendesk.maxwell.schema.ddl.SchemaChange.parse(SchemaChange.java:65)
    at com.zendesk.maxwell.MaxwellParser.processQueryEvent(MaxwellParser.java:206)
    at com.zendesk.maxwell.MaxwellParser.getEvent(MaxwellParser.java:192)
    at com.zendesk.maxwell.MaxwellParser.getEvent(MaxwellParser.java:198)
    at com.zendesk.maxwell.MaxwellParser.run(MaxwellParser.java:98)
    at com.zendesk.maxwell.Maxwell.run(Maxwell.java:69)
    at com.zendesk.maxwell.Maxwell.main(Maxwell.java:75)

Maxwell is using JDK7 and not support JDK 6 ?

Maxwell is using JDK 7 and not support JDK 6 ?

I'm reading maxwell and found something maybe could rewrite

such as :

        InputStream schemaSQL = SchemaStore.class.getResourceAsStream(
                "/sql/maxwell_schema.sql");
        BufferedReader r = new BufferedReader(new InputStreamReader(schemaSQL));
        String sql = "", line;

        LOGGER.info("Creating maxwell database");
        while ((line = r.readLine()) != null) {
            sql += line + "\n";
        }

In JDK 7 Files.readAllLines(path, cs) is looks better :)

Maxwell reference to Databus

I enjoyed the introductory blog post for Maxwell at:

https://developer.zendesk.com/blog/introducing-maxwell-a-mysql-to-kafka-binlog-processor

This looks like a great project!

I just wanted to respond to one comment:

"We're fairly sure LinkedIn's engineers, while writing DataBus solved this problem by patching mysql to output full schema information into its binlogs. They never seemed to release the patch, though."

Actually, we didn't make any such patch. The use case for Databus consuming from MySQL (as opposed to Oracle) is limited to Espresso - LinkedIn's NoSQL store. The Open Replicator based binlog to Databus event translator has access to the Espresso schema registry that contains all necessary information about the MySQL columnar data.

Espresso stores and retrieves Avro documents via a REST API. The URL path for a document is defined by a table schema that defines the columns that compose the primary key in MySQL. All other columns are common to all tables in the system. The actual documents are stored in avro/binary as a longblob field so there is no further specialization of the MySQL tables based on the layout of the documents stored. When processing a row event from the binlog, one only needs the table name from the corresponding TableMapEvent to fetch (and cache) the corresponding table schema from a registry to obtain all necessary information about the row.

Tom Quiggle,
https://www.linkedin.com/in/tquiggle

GTID support

Is there a planned release for supporting replication based on the GTID, as opposed to the position, which I believe is the implementation that is currently existing?

Tests unable to start onetimeserver on windows

If you're on Windows, and do mvn test, even under cygwin, you get the error

java.io.IOException: Cannot run program "C:\.....\maxwell/src/test/onetimeserver": CreateProcess error=193, %1 is not a valid Win32 application
        at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
        at com.zendesk.maxwell.MysqlIsolatedServer.boot(MysqlIsolatedServer.java:34)
        at com.zendesk.maxwell.AbstractMaxwellTest.setUpBeforeClass(AbstractMaxwellTest.java:29)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:483)
        at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
        at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
....

Not sure what can be done about this.

Maxwell shutdown on encountering 'year' datatype

Similar to #64 and #65, maxwell barfs and fails when it encounters a year datatype.

bin/maxwell --user='root' --password='root' --host='127.0.0.1' --producer=stdout
Picked up JAVA_TOOL_OPTIONS: -javaagent:/usr/share/java/jayatanaag.jar 
23:29:28,708 INFO  Maxwell - Maxwell is booting, starting at BinlogPosition[mysql-logs.000003:6982]
23:29:28,710 DEBUG SchemaStore - looking to restore schema at target position BinlogPosition[mysql-logs.000003:6982]
23:29:28,711 INFO  SchemaStore - Restoring schema id 7 (last modified at BinlogPosition[mysql-logs.000003:6982])
23:29:28,753 INFO  TransportImpl - connecting to host: 127.0.0.1, port: 3306
23:29:28,785 INFO  TransportImpl - connected to host: 127.0.0.1, port: 3306, context: AbstractTransport.Context[threadId=25,scramble=@/f8$H"l|J=4*t,{KCN?,protocolVersion=10,serverHost=127.0.0.1,serverPort=3306,serverStatus=2,serverCollation=8,serverVersion=5.6.24-0ubuntu2-log,serverCapabilities=63487]
23:29:28,785 INFO  AuthenticatorImpl - start to login, user: root, host: 127.0.0.1, port: 3306
23:29:28,792 INFO  AuthenticatorImpl - login successfully, user: root, detail: OKPacket[packetMarker=0,affectedRows=0,insertId=0,serverStatus=2,warningCount=0,message=<null>]
23:29:28,974 DEBUG SchemaChange - SQL_PARSE <- "create table max_test4(dt datetime, dt_y year(4))"
line 1:45 mismatched input '(' expecting {',', ')', AUTO_INCREMENT, COLUMN_FORMAT, COMMENT, DEFAULT, NOT, NULL, ON, PRIMARY, STORAGE, UNIQUE}
line 1:48 extraneous input ')' expecting {<EOF>, AUTO_INCREMENT, AVG_ROW_LENGTH, CHARACTER, CHARSET, CHECKSUM, COLLATE, COMMENT, CONNECTION, DATA, DEFAULT, DELAY_KEY_WRITE, ENGINE, INDEX, INSERT_METHOD, KEY_BLOCK_SIZE, MAX_ROWS, MIN_ROWS, PACK_KEYS, PASSWORD, ROW_FORMAT, TABLESPACE, UNION}
23:29:29,065 ERROR MysqlParserListener - (column_definition (name (id dt_y)) (data_type (generic_type year)) ( 4)
23:29:29,065 ERROR SchemaChange - Error parsing SQL: 'create table max_test4(dt datetime, dt_y year(4))'
com.zendesk.maxwell.schema.ddl.MaxwellSQLSyntaxError: (
at com.zendesk.maxwell.schema.ddl.MysqlParserListener.visitErrorNode(MysqlParserListener.java:85)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:41)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at com.zendesk.maxwell.schema.ddl.SchemaChange.parse(SchemaChange.java:65)
at com.zendesk.maxwell.MaxwellParser.processQueryEvent(MaxwellParser.java:206)
at com.zendesk.maxwell.MaxwellParser.getEvent(MaxwellParser.java:192)
at com.zendesk.maxwell.MaxwellParser.getEvent(MaxwellParser.java:198)
at com.zendesk.maxwell.MaxwellParser.run(MaxwellParser.java:98)
at com.zendesk.maxwell.Maxwell.run(Maxwell.java:69)
at com.zendesk.maxwell.Maxwell.main(Maxwell.java:75)

And just like #64 and #65, re-running Maxwell does not work and produces the same text as above and exits. I will need to purge and delete again for this to work.

Blob / binary columns

I realise from the docs that binary column support is experimental. I'd like to use maxwell with an existing database which has some blob columns but maxwell crashes whenever a row with a blob column is modified.

MySQL 5.1.59
Maxwell 0.11.0-RC1

Steps to reproduce:

CREATE TABLE `maxwell_blob_test` (
    `id` INT UNSIGNED NOT NULL AUTO_INCREMENT,
    `blob` BLOB NOT NULL,
    PRIMARY KEY (`id`)
)
COLLATE='utf8_unicode_ci'
ENGINE=MyISAM
;

INSERT INTO `sellstuf1`.`maxwell_blob_test` (`blob`) VALUES (0x22);

Maxwell log

16:36:04,286 DEBUG SchemaPosition - Writing binlog position to maxwell.positions: BinlogPosition[mysql-binlog.000002:65790]
16:42:58,092 DEBUG SchemaChange - SQL_PARSE <- "CREATE TABLE `maxwell_blob_test` (
    `id` INT UNSIGNED NOT NULL AUTO_INCREMENT,
    `blob` BLOB NOT NULL,
    PRIMARY KEY (`id`)
)
COLLATE='utf8_unicode_ci'
ENGINE=MyISAM"
16:42:58,127 DEBUG SchemaChange - SQL_PARSE ->   (parse (statement (create_table (create_table_preamble CREATE TABLE (table_name (name (id `maxwell_blob_test`)))) (create_specifications ( (create_specification (column_definition (name (id `id`)) (data_type (signed_type INT (int_flags UNSIGNED))) (column_options (nullability NOT NULL)) (column_options AUTO_INCREMENT))) , (create_specification (column_definition (name (id `blob`)) (data_type (generic_type BLOB)) (column_options (nullability NOT NULL)))) , (create_specification (index_definition (index_type_pk PRIMARY KEY (index_column_list ( (index_columns (index_column (name (id `id`)))) ))))) )) (table_creation_options (creation_collation COLLATE = (string 'utf8_unicode_ci'))) (table_creation_options (creation_engine ENGINE = (string MyISAM))))) <EOF>)
16:42:58,129 INFO  MaxwellParser - storing schema @BinlogPosition[mysql-binlog.000002:124061] after applying "CREATE T ENGINE=MyISAM"unicode_ci'LL, NULL AUTO_INCREMENT,
16:42:58,252 DEBUG SchemaPosition - syncing binlog position: BinlogPosition[mysql-binlog.000002:124061]
16:42:58,254 DEBUG SchemaPosition - Writing binlog position to maxwell.positions: BinlogPosition[mysql-binlog.000002:124061]
java.lang.NullPointerException: charset
    at java.lang.String.<init>(String.java:449)
    at java.lang.String.<init>(String.java:503)
    at com.zendesk.maxwell.schema.columndef.StringColumnDef.asJSON(StringColumnDef.java:54)
    at com.zendesk.maxwell.MaxwellAbstractRowsEvent.jsonMaps(MaxwellAbstractRowsEvent.java:229)
    at com.zendesk.maxwell.MaxwellAbstractRowsEvent.toJSONStrings(MaxwellAbstractRowsEvent.java:251)
    at com.zendesk.maxwell.StdoutProducer.push(StdoutProducer.java:12)
    at com.zendesk.maxwell.MaxwellParser.run(MaxwellParser.java:111)
    at com.zendesk.maxwell.Maxwell.run(Maxwell.java:69)
    at com.zendesk.maxwell.Maxwell.main(Maxwell.java:75)
2015-09-10 16:43:43,071 FATAL Unable to register shutdown hook because JVM is shutting down.

What is the license on this project?

How is Maxwell licensed? Apache License, BSD, GPL, etc.

Emitting diffs in addition to the full row.

It would be useful to support a configuration that emits a diff record, only including the columns that have changed.

The old column data appears to be available in update events. I'm interested to know if this is something that has been considered? And if so is there a roadmap or design?

Must Create Maxwell Database ?

Hi ~

I have read Introducing Maxwell, a mysql-to-kafka binlog processor found maxwell is a wonderful component to sync MySQL Binary Log into Kafka and I want to replace our origin python scripts .

And now , I have a question:

maxwell will create a maxwell database on each MySQL instance which it reading ?

Maybe could store this metadata in a MySQL instance such as hive

Assertion Failed

Updated to 0.16.1 this morning and one of my environments reported this.

08:12:25,823 ERROR AbstractBinlogParser - failed to parse binlog
java.lang.RuntimeException: assertion failed!  We left -339 unconsumed bytes in the buffer for event: BinlogEventV4HeaderImpl[timestamp=1449632908000,eventType=31,serverId=11543,eventLength=102,nextPosition=917023964,flags=0,timestampOfReceipt=1450041145808]  asis-regressionprod.hq.local
        at com.google.code.or.binlog.impl.AbstractBinlogParser$Task.run(AbstractBinlogParser.java:263) [open-replicator-1.3.2.jar:?]    asis-regressionprod.hq.local
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_40]   asis-regressionprod.hq.local
        at com.google.code.or.binlog.impl.ReplicationBasedBinlogParser.doParse(ReplicationBasedBinlogParser.java:110) ~[open-replicator-1.3.2.jar:?]    asis-regressionprod.hq.local
        at com.google.code.or.net.impl.EventInputStream.finishEvent(EventInputStream.java:86) ~[open-replicator-1.3.2.jar:?]

better starter script

checking if there is any interest to implement maxwell starter script to support start, restart, stop, status etc in the background mode. something similar to :

https://github.com/apache/nifi/blob/b809031195405c04ec7a950c293482225f64e8dc/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-resources/src/main/resources/bin/nifi.sh

maxwell down because of the creating table operation of mysql database

Hi:
I created a table in mysql. then maxwell down .i don't know how to make it works normal.and the error log as below:
10:37:55,677 DEBUG SchemaChange - SQL_PARSE <- "CREATE TABLE emon.chk ( group_name VARCHAR(8) NOT NULL, group_key NUMERIC(19) NOT NULL, seqno NUMERIC(10), rba NUMERIC(19) NOT NULL, audit_ts VARCHAR(29), create_ts DATETIME NOT NULL, last_update_ts DATETIME NOT NULL, current_dir VARCHAR(255) NOT NULL, PRIMARY KEY (group_name, group_key))"
line 1:78 mismatched input ')' expecting ','
line 1:80 mismatched input 'NOT' expecting {, AUTO_INCREMENT, AVG_ROW_LENGTH, CHARACTER, CHARSET, CHECKSUM, COLLATE, COMMENT, CONNECTION, DATA, DEFAULT, DELAY_KEY_WRITE, ENGINE, INDEX, INSERT_METHOD, KEY_BLOCK_SIZE, MAX_ROWS, MIN_ROWS, PACK_KEYS, PASSWORD, ROW_FORMAT, TABLESPACE, UNION}
java.lang.IllegalArgumentException: unsupported column type numeric
at com.zendesk.maxwell.schema.columndef.ColumnDef.build(ColumnDef.java:80)
at com.zendesk.maxwell.schema.ddl.MysqlParserListener.exitColumn_definition(MysqlParserListener.java:273)
at com.zendesk.maxwell.schema.ddl.mysqlParser$Column_definitionContext.exitRule(mysqlParser.java:4370)
at org.antlr.v4.runtime.tree.ParseTreeWalker.exitRule(ParseTreeWalker.java:71)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:54)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at com.zendesk.maxwell.schema.ddl.SchemaChange.parse(SchemaChange.java:67)
at com.zendesk.maxwell.MaxwellParser.processQueryEvent(MaxwellParser.java:206)
at com.zendesk.maxwell.MaxwellParser.getEvent(MaxwellParser.java:192)
at com.zendesk.maxwell.MaxwellParser.getEvent(MaxwellParser.java:198)
at com.zendesk.maxwell.MaxwellParser.run(MaxwellParser.java:98)
at com.zendesk.maxwell.Maxwell.run(Maxwell.java:69)
at com.zendesk.maxwell.Maxwell.main(Maxwell.java:75)

undead maxwell processes

Saw an open-replicator thread die without maxwell restarting it:

06,364 ERROR AbstractBinlogParser - failed to parse binlog
 Connection reset
    at java.net.SocketInputStream.read(SocketInputStream.java:196) ~[?:1.7.0_79]
    at java.net.SocketInputStream.read(SocketInputStream.java:122) ~[?:1.7.0_79]
    at com.google.code.or.io.util.ActiveBufferedInputStream.run(ActiveBufferedInputStream.java:85) ~[open-replicator-1.3.0.jar:?]
    at java.lang.Thread.run(Thread.java:745) [?:1.7.0_79]

Lingering mysql test instances

After running maxwell tests on a VM with Ubuntu 12.04.5 LTS.

(2.1.6) vagrant: /harmony/code/maxwell (master)$ ps aux | grep 'mysql.*isolated'
vagrant  20030  1.3  1.3 911812 53924 pts/10   Sl   17:19   0:02 /usr/sbin/mysqld --no-defaults --default-storage-engine=innodb --datadir=/tmp/isolated20150610-19935-1lyvqt4/mysqld --pid-file=/tmp/isolated20150610-19935-1lyvqt4/mysqld.pid --port=32255 --socket=/tmp/isolated20150610-19935-1lyvqt4/mysqld/mysql.sock --log-bin --log-slave-updates --binlog_format=row
vagrant  20342  1.3  1.2 846016 51628 pts/10   Sl   17:19   0:02 /usr/sbin/mysqld --no-defaults --default-storage-engine=innodb --datadir=/tmp/isolated20150610-20229-mxy1pv/mysqld --pid-file=/tmp/isolated20150610-20229-mxy1pv/mysqld.pid --port=34500 --socket=/tmp/isolated20150610-20229-mxy1pv/mysqld/mysql.sock --log-bin --log-slave-updates --binlog_format=row
vagrant  20522  1.1  1.2 846016 50364 pts/10   Sl   17:20   0:02 /usr/sbin/mysqld --no-defaults --default-storage-engine=innodb --datadir=/tmp/isolated20150610-20427-dkka81/mysqld --pid-file=/tmp/isolated20150610-20427-dkka81/mysqld.pid --port=46633 --socket=/tmp/isolated20150610-20427-dkka81/mysqld/mysql.sock --log-bin --log-slave-updates --binlog_format=row
vagrant  20679  1.1  1.2 846016 49968 pts/10   Sl   17:20   0:01 /usr/sbin/mysqld --no-defaults --default-storage-engine=innodb --datadir=/tmp/isolated20150610-20584-1yi3ria/mysqld --pid-file=/tmp/isolated20150610-20584-1yi3ria/mysqld.pid --port=21353 --socket=/tmp/isolated20150610-20584-1yi3ria/mysqld/mysql.sock --log-bin --log-slave-updates --binlog_format=row
vagrant  20822  0.0  0.0   9620   960 pts/10   S+   17:22   0:00 grep --color=auto mysql.*isolated

Maxwel Crashing when Altering table with ADD INDEX specified by Order

It looks like Maxwell is crashing when I ALTER a table with an ADD INDEX clause specified with an ORDER.

Ex:

ALTER TABLE test
ADD COLUMN test_col ENUM('Y', 'N') NULL DEFAULT 'N',
ADD INDEX test_col (test_col_index ASC);

Results in an error stack like the below:
com.zendesk.maxwell.schema.ddl.MaxwellSQLSyntaxError: ASC
at com.zendesk.maxwell.schema.ddl.MysqlParserListener.visitErrorNode(MysqlParserListener.java:85)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:41)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at com.zendesk.maxwell.schema.ddl.SchemaChange.parse(SchemaChange.java:67)
at com.zendesk.maxwell.MaxwellReplicator.processQueryEvent(MaxwellReplicator.java:273)
at com.zendesk.maxwell.MaxwellReplicator.getEvent(MaxwellReplicator.java:253)
at com.zendesk.maxwell.MaxwellReplicator.work(MaxwellReplicator.java:89)
at com.zendesk.maxwell.RunLoopProcess.runLoop(RunLoopProcess.java:31)
at com.zendesk.maxwell.Maxwell.run(Maxwell.java:89)
at com.zendesk.maxwell.Maxwell.main(Maxwell.java:95)
16:59:24,374 INFO SchemaPosition - Storing final position

Unexpectedly asked to create existing database zendesk_development

After bootstrapping the docker images I can no longer start the maxwell image which throws this error:

13:06:09,458 DEBUG SchemaChange - SQL_PARSE ->   (parse (statement (create_database CREATE DATABASE (name (id `zendesk_development`)))) <EOF>)
com.zendesk.maxwell.schema.ddl.SchemaSyncError: Unexpectedly asked to create existing database zendesk_development
    at com.zendesk.maxwell.schema.ddl.DatabaseCreate.apply(DatabaseCreate.java:25)
    at com.zendesk.maxwell.MaxwellReplicator.processQueryEvent(MaxwellReplicator.java:279)
    at com.zendesk.maxwell.MaxwellReplicator.getEvent(MaxwellReplicator.java:253)
    at com.zendesk.maxwell.MaxwellReplicator.work(MaxwellReplicator.java:89)
    at com.zendesk.maxwell.RunLoopProcess.runLoop(RunLoopProcess.java:27)
    at com.zendesk.maxwell.Maxwell.run(Maxwell.java:82)
    at com.zendesk.maxwell.Maxwell.main(Maxwell.java:88)
13:06:09,460 INFO  SchemaPosition - Storing final position: null

Both @rbarfoed and I have been experiencing this error.

How to reproduce:

1. run ./bin/onboard from the docker-images folder
2. zdi maxwell start

Maxwell database becomes large if you make lots of table creates

In our app, there are a selection of tables (one per day), and they get created every minute or so. We have several servers, and they each create these tables using "CREATE ... IF NOT EXISTS". Normally this is fine, as it will do nothing in mysql. These DDL lines still get replicated, and maxwell parses them, and adds the 'new' table to the schema each time it is seen.

> select count(*) from tables;
+----------+
| count(*) |
+----------+
|   386694 |
+----------+

> select count(distinct name) from tables;
+----------------------+
| count(distinct name) |
+----------------------+
|                  824 |
+----------------------+

This is after running Maxwell for less than an hour.