zendesk / maxwell Goto Github PK
View Code? Open in Web Editor NEWMaxwell's daemon, a mysql-to-json kafka producer
Home Page: https://maxwells-daemon.io/
License: Other
Maxwell's daemon, a mysql-to-json kafka producer
Home Page: https://maxwells-daemon.io/
License: Other
Updated to 0.16.1 this morning and one of my environments reported this.
08:12:25,823 ERROR AbstractBinlogParser - failed to parse binlog
java.lang.RuntimeException: assertion failed! We left -339 unconsumed bytes in the buffer for event: BinlogEventV4HeaderImpl[timestamp=1449632908000,eventType=31,serverId=11543,eventLength=102,nextPosition=917023964,flags=0,timestampOfReceipt=1450041145808] asis-regressionprod.hq.local
at com.google.code.or.binlog.impl.AbstractBinlogParser$Task.run(AbstractBinlogParser.java:263) [open-replicator-1.3.2.jar:?] asis-regressionprod.hq.local
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_40] asis-regressionprod.hq.local
at com.google.code.or.binlog.impl.ReplicationBasedBinlogParser.doParse(ReplicationBasedBinlogParser.java:110) ~[open-replicator-1.3.2.jar:?] asis-regressionprod.hq.local
at com.google.code.or.net.impl.EventInputStream.finishEvent(EventInputStream.java:86) ~[open-replicator-1.3.2.jar:?]
After running maxwell tests on a VM with Ubuntu 12.04.5 LTS.
(2.1.6) vagrant: /harmony/code/maxwell (master)$ ps aux | grep 'mysql.*isolated'
vagrant 20030 1.3 1.3 911812 53924 pts/10 Sl 17:19 0:02 /usr/sbin/mysqld --no-defaults --default-storage-engine=innodb --datadir=/tmp/isolated20150610-19935-1lyvqt4/mysqld --pid-file=/tmp/isolated20150610-19935-1lyvqt4/mysqld.pid --port=32255 --socket=/tmp/isolated20150610-19935-1lyvqt4/mysqld/mysql.sock --log-bin --log-slave-updates --binlog_format=row
vagrant 20342 1.3 1.2 846016 51628 pts/10 Sl 17:19 0:02 /usr/sbin/mysqld --no-defaults --default-storage-engine=innodb --datadir=/tmp/isolated20150610-20229-mxy1pv/mysqld --pid-file=/tmp/isolated20150610-20229-mxy1pv/mysqld.pid --port=34500 --socket=/tmp/isolated20150610-20229-mxy1pv/mysqld/mysql.sock --log-bin --log-slave-updates --binlog_format=row
vagrant 20522 1.1 1.2 846016 50364 pts/10 Sl 17:20 0:02 /usr/sbin/mysqld --no-defaults --default-storage-engine=innodb --datadir=/tmp/isolated20150610-20427-dkka81/mysqld --pid-file=/tmp/isolated20150610-20427-dkka81/mysqld.pid --port=46633 --socket=/tmp/isolated20150610-20427-dkka81/mysqld/mysql.sock --log-bin --log-slave-updates --binlog_format=row
vagrant 20679 1.1 1.2 846016 49968 pts/10 Sl 17:20 0:01 /usr/sbin/mysqld --no-defaults --default-storage-engine=innodb --datadir=/tmp/isolated20150610-20584-1yi3ria/mysqld --pid-file=/tmp/isolated20150610-20584-1yi3ria/mysqld.pid --port=21353 --socket=/tmp/isolated20150610-20584-1yi3ria/mysqld/mysql.sock --log-bin --log-slave-updates --binlog_format=row
vagrant 20822 0.0 0.0 9620 960 pts/10 S+ 17:22 0:00 grep --color=auto mysql.*isolated
Parse error showed up in my log
2015-12-14T13:40:57,653 ERROR com.zendesk.maxwell.schema.ddl.MysqlParserListener: (alter_specification .20151214 _TSP_16819__CGPos_0_C7D1A82D5B6243259A60D3CAD54A4B61 ADD KEY `TRNNUM` ( `trnnum`)
2015-12-14T13:40:57,653 ERROR com.zendesk.maxwell.schema.ddl.SchemaChange: Error parsing SQL: 'ALTER TABLE cgt.20151214_TSP_16819__CGPos_0_C7D1A82D5B6243259A60D3CAD54A4B61 asis-uat3.hq.local
Hi
i am installing maxwell 0.11.1 using command as below:
curl -sLo - https://github.com/zendesk/maxwell/releases/download/v0.11.1/maxwell-0.11.1.tar.gz
| tar zxvf -
then it raise error :
gzip: stdin: unexpected end of file
tar: Child returned status 1
tar: Error is not recoverable: exiting now
Hi ~
I have read Introducing Maxwell, a mysql-to-kafka binlog processor found maxwell is a wonderful component to sync MySQL Binary Log into Kafka and I want to replace our origin python scripts .
And now , I have a question:
maxwell will create a maxwell database on each MySQL instance which it reading ?
Maybe could store this metadata in a MySQL instance such as hive
I'm experimenting a little with maxwell, following the getting started guide and appear to have hit an issue while processing an event generated by the following row:
mysql> select * from events where event_id = 5;
+----------+--------------+---------------------+----------------------------------+-----------+---------+----------------------------------+--------------+
| event_id | business_ref | when | what_ref | what_type | action | who_ref | deleted_name |
+----------+--------------+---------------------+----------------------------------+-----------+---------+----------------------------------+--------------+
| 5 | 1 | 2015-11-25 23:59:00 | 785d226973bf4eae9bfc903f0b9f2033 | Task | updated | d4ed50fe84ba473fb2690b149e7f2933 | NULL |
+----------+--------------+---------------------+----------------------------------+-----------+---------+----------------------------------+--------------+`
From the stack trace it appears that maxwell has trouble with the when
column, which is of type datetime
:
08:54:57,083 ERROR MaxwellReplicator - Got an unexpected row-event: WriteRowsEvent[header=BinlogEventV4HeaderImpl[timestamp=1448495940000,eventType=23,serverId=1,eventLength=290,nextPosition=2705491,flags=0,timestampOfReceipt=1448528097073],tableId=332,reserved=1,columnCount=10,usedColumns=1111111111,rows=[Row[columns=[560, 1, Wed Nov 25 23:59:00 UTC 2015, 785d226973bf4eae9bfc903f0b9f2033, Task, d4ed50fe84ba473fb2690b149e7f2933, webapp, null, BlobColumn[value=[B@7c417213], b4a678826df0408d869346ff67275627]]]]
java.lang.IllegalStateException: No ObjectCodec defined for the generator, can only serialize simple wrapper types (type passed java.sql.Time)
at com.fasterxml.jackson.core.JsonGenerator._writeSimpleObject(JsonGenerator.java:1725)
at com.fasterxml.jackson.core.base.GeneratorBase.writeObject(GeneratorBase.java:327)
at com.fasterxml.jackson.core.JsonGenerator.writeObjectField(JsonGenerator.java:1415)
at com.zendesk.maxwell.RowMap.toJSON(RowMap.java:91)
at com.zendesk.maxwell.MaxwellAbstractRowsEvent.toJSONStrings(MaxwellAbstractRowsEvent.java:257)
at com.zendesk.maxwell.producer.MaxwellKafkaProducer.push(MaxwellKafkaProducer.java:86)
at com.zendesk.maxwell.MaxwellReplicator.work(MaxwellReplicator.java:97)
at com.zendesk.maxwell.RunLoopProcess.runLoop(RunLoopProcess.java:31)
at com.zendesk.maxwell.Maxwell.run(Maxwell.java:89)
at com.zendesk.maxwell.Maxwell.main(Maxwell.java:95)
08:54:57,147 INFO SchemaPosition - Storing final position: null
Have I recognised the problem correctly, or is it likely that the problem lies elsewhere?
Version info:
MySQL: 5.5.44
Maxwell: 0.14.5
Java: 1.8.0_66
OS: OS X 10.11.1
Hi
I encounter a problem when using maxwell to load change data from mysql . I insert a chinese in a table and it display normol in mysql but messy code in maxwell
in mysql:
mysql> insert into a values("哈哈");
Query OK, 1 row affected (0.00 sec)
mysql> select * from a;
+------+
| col |
+------+
| 测试 |
| 哈哈 |
+------+
2 rows in set (0.00 sec)
mysql> exit
Bye
[root@localhost maxwell-0.9.0]# echo $LANG
en_US.UTF-8
in maxwell:
16:47:18,504 DEBUG SchemaPosition - Writing binlog position to maxwell.positions: BinlogPosition[master.000002:1832]
16:48:45,903 DEBUG NetworkClient - Initiating connection to node 1 at 10.89.131.15:9092.
16:48:45,907 DEBUG NetworkClient - Completed connection to node 1
16:48:45,918 DEBUG MaxwellKafkaProducer - -> key:{"database":"test","table":"a","pk._uuid":"6fcd9885-d3a3-4d32-a0dc-064a119e8429"}, partition:1, offset:86427
16:48:45,918 DEBUG MaxwellKafkaProducer - {"database":"test","table":"a","type":"insert","data":{"col":"¹þ¹þ"},"ts":1440406125}
16:48:45,919 DEBUG MaxwellKafkaProducer - BinlogPosition[master.000002:7471]
it dispay "¹þ¹þ".
Same as #64, #65 and #66, Maxwell can't handle 'bool' and (just to be completely sure) 'boolean'. This occurs at the create table
stage. To get Maxwell to run again, I had to purge binary logs before now()
and delete from maxwell.positions
.
BOOL:
bin/maxwell --user='root' --password='root' --host='127.0.0.1' --producer=stdout
Picked up JAVA_TOOL_OPTIONS: -javaagent:/usr/share/java/jayatanaag.jar
10:40:30,183 INFO Maxwell - Maxwell is capturing initial schema
10:40:30,185 DEBUG SchemaStore - Capturing schema
10:40:30,217 DEBUG SchemaStore - Finished capturing schema
10:40:30,294 INFO TransportImpl - connecting to host: 127.0.0.1, port: 3306
10:40:30,343 INFO TransportImpl - connected to host: 127.0.0.1, port: 3306, context: AbstractTransport.Context[threadId=31,scramble=}_EvG<6<gt16X;]DyW(],protocolVersion=10,serverHost=127.0.0.1,serverPort=3306,serverStatus=2,serverCollation=8,serverVersion=5.6.24-0ubuntu2-log,serverCapabilities=63487]
10:40:30,344 INFO AuthenticatorImpl - start to login, user: root, host: 127.0.0.1, port: 3306
10:40:30,348 INFO AuthenticatorImpl - login successfully, user: root, detail: OKPacket[packetMarker=0,affectedRows=0,insertId=0,serverStatus=2,warningCount=0,message=<null>]
10:40:31,175 DEBUG SchemaPosition - Writing binlog position to maxwell.positions: BinlogPosition[mysql-logs.000003:12471]
10:55:36,558 DEBUG SchemaChange - SQL_PARSE <- "create table mmf_test(col1 smallint, col2 int, col3 bigint, col4 float, col5 date, col6 datetime, col7 decimal(10,2), col8 varchar(500), col9 text, col10 bool)"
line 1:155 no viable alternative at input 'bool'
10:55:36,590 ERROR MysqlParserListener - (data_type bool)
10:55:36,590 ERROR SchemaChange - Error parsing SQL: 'create table mmf_test(col1 smallint, col2 int, col3 bigint, col4 float, col5 date, col6 datetime, col7 decimal(10,2), col8 varchar(500), col9 text, col10 bool)'
com.zendesk.maxwell.schema.ddl.MaxwellSQLSyntaxError: bool
at com.zendesk.maxwell.schema.ddl.MysqlParserListener.visitErrorNode(MysqlParserListener.java:85)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:41)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at com.zendesk.maxwell.schema.ddl.SchemaChange.parse(SchemaChange.java:65)
at com.zendesk.maxwell.MaxwellParser.processQueryEvent(MaxwellParser.java:206)
at com.zendesk.maxwell.MaxwellParser.getEvent(MaxwellParser.java:192)
at com.zendesk.maxwell.MaxwellParser.getEvent(MaxwellParser.java:198)
at com.zendesk.maxwell.MaxwellParser.run(MaxwellParser.java:98)
at com.zendesk.maxwell.Maxwell.run(Maxwell.java:69)
at com.zendesk.maxwell.Maxwell.main(Maxwell.java:75)
2015-06-18 17:55:36,635 FATAL Unable to register shutdown hook because JVM is shutting down.
BOOLEAN:
bin/maxwell --user='root' --password='root' --host='127.0.0.1' --producer=stdout
Picked up JAVA_TOOL_OPTIONS: -javaagent:/usr/share/java/jayatanaag.jar
10:58:46,071 INFO Maxwell - Maxwell is capturing initial schema
10:58:46,073 DEBUG SchemaStore - Capturing schema
10:58:46,111 DEBUG SchemaStore - Finished capturing schema
10:58:47,020 INFO TransportImpl - connecting to host: 127.0.0.1, port: 3306
10:58:47,044 INFO TransportImpl - connected to host: 127.0.0.1, port: 3306, context: AbstractTransport.Context[threadId=35,scramble=pWD+Jt,w6D5z*_~!&7#J,protocolVersion=10,serverHost=127.0.0.1,serverPort=3306,serverStatus=2,serverCollation=8,serverVersion=5.6.24-0ubuntu2-log,serverCapabilities=63487]
10:58:47,045 INFO AuthenticatorImpl - start to login, user: root, host: 127.0.0.1, port: 3306
10:58:47,054 INFO AuthenticatorImpl - login successfully, user: root, detail: OKPacket[packetMarker=0,affectedRows=0,insertId=0,serverStatus=2,warningCount=0,message=<null>]
10:58:47,063 DEBUG SchemaPosition - Writing binlog position to maxwell.positions: BinlogPosition[mysql-logs.000003:18202]
10:58:55,665 DEBUG SchemaChange - SQL_PARSE <- "create table mmf_test(col1 smallint, col2 int, col3 bigint, col4 float, col5 date, col6 datetime, col7 decimal(10,2), col8 varchar(500), col9 text, col10 boolean)"
line 1:155 no viable alternative at input 'boolean'
10:58:55,712 ERROR MysqlParserListener - (data_type boolean)
10:58:55,712 ERROR SchemaChange - Error parsing SQL: 'create table mmf_test(col1 smallint, col2 int, col3 bigint, col4 float, col5 date, col6 datetime, col7 decimal(10,2), col8 varchar(500), col9 text, col10 boolean)'
com.zendesk.maxwell.schema.ddl.MaxwellSQLSyntaxError: boolean
at com.zendesk.maxwell.schema.ddl.MysqlParserListener.visitErrorNode(MysqlParserListener.java:85)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:41)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at com.zendesk.maxwell.schema.ddl.SchemaChange.parse(SchemaChange.java:65)
at com.zendesk.maxwell.MaxwellParser.processQueryEvent(MaxwellParser.java:206)
at com.zendesk.maxwell.MaxwellParser.getEvent(MaxwellParser.java:192)
at com.zendesk.maxwell.MaxwellParser.getEvent(MaxwellParser.java:198)
at com.zendesk.maxwell.MaxwellParser.run(MaxwellParser.java:98)
at com.zendesk.maxwell.Maxwell.run(Maxwell.java:69)
at com.zendesk.maxwell.Maxwell.main(Maxwell.java:75)
checking if there is any interest to implement maxwell
starter script to support start, restart, stop, status etc in the background mode. something similar to :
I enjoyed the introductory blog post for Maxwell at:
https://developer.zendesk.com/blog/introducing-maxwell-a-mysql-to-kafka-binlog-processor
This looks like a great project!
I just wanted to respond to one comment:
"We're fairly sure LinkedIn's engineers, while writing DataBus solved this problem by patching mysql to output full schema information into its binlogs. They never seemed to release the patch, though."
Actually, we didn't make any such patch. The use case for Databus consuming from MySQL (as opposed to Oracle) is limited to Espresso - LinkedIn's NoSQL store. The Open Replicator based binlog to Databus event translator has access to the Espresso schema registry that contains all necessary information about the MySQL columnar data.
Espresso stores and retrieves Avro documents via a REST API. The URL path for a document is defined by a table schema that defines the columns that compose the primary key in MySQL. All other columns are common to all tables in the system. The actual documents are stored in avro/binary as a longblob field so there is no further specialization of the MySQL tables based on the layout of the documents stored. When processing a row event from the binlog, one only needs the table name from the corresponding TableMapEvent to fetch (and cache) the corresponding table schema from a registry to obtain all necessary information about the row.
Tom Quiggle,
https://www.linkedin.com/in/tquiggle
Hi!
I wanted to say fantastic work on this project. I've been working on identical functionality for about 6 months and was just about to open-source my code, but looks like you guys got there first! shakes fist at the sky From what I've seen so far, Maxwell seems farther along than mine, so I'm not sure what I'll do with mine now.
I have a bunch of design questions. Can I get your email address so we can talk more about it? Or, if you'd rather discuss here on Github, that works too, but it might get pretty long. My email is [email protected]. Looking forward to talking to you.
-James Cheng
Do you have example or reference code to read this events from kafka and persists on hbase or hdfs?
Probably somewhat unusual, but there's no BIT datatype support:
https://dev.mysql.com/doc/refman/5.6/en/bit-type.html
00:13:15,085 INFO Maxwell - Maxwell is capturing initial schema
00:13:15,086 DEBUG SchemaStore - Capturing schema
java.lang.IllegalArgumentException: unsupported column type bit
at com.zendesk.maxwell.schema.columndef.ColumnDef.build(ColumnDef.java:83)
mvn clean install
DEBUG MysqlIsolatedServer - Booting server: /Users/agaton/maxwell/src/test/mysql_isolated_server/bin/boot_isolated_mysql_server -- --binlog_format=row --innodb_flush_log_at_trx_commit=1 --server_id=123123
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.008 sec <<< FAILURE!
com.zendesk.maxwell.SchemaCaptureTest Time elapsed: 0.007 sec <<< ERROR!
java.io.IOException: Cannot run program "/Users/agaton/maxwell/src/test/mysql_isolated_server/bin/boot_isolated_mysql_server": error=2, No such file or directory
I have been seeing this error while evaluating Maxwell in a Fedora VM.
09:25:53,554 DEBUG SchemaChange - SQL_PARSE <- "CREATE DATABASE /*! IF NOT EXISTS */ error"
09:25:53,556 DEBUG SchemaChange - SQL_PARSE -> (parse (statement (create_database CREATE DATABASE (name (id error)))) <EOF>)
com.zendesk.maxwell.schema.ddl.SchemaSyncError: Unexpectedly asked to create existing database error
at com.zendesk.maxwell.schema.ddl.DatabaseCreate.apply(DatabaseCreate.java:25)
at com.zendesk.maxwell.MaxwellReplicator.processQueryEvent(MaxwellReplicator.java:280)
at com.zendesk.maxwell.MaxwellReplicator.getEvent(MaxwellReplicator.java:253)
at com.zendesk.maxwell.MaxwellReplicator.work(MaxwellReplicator.java:89)
at com.zendesk.maxwell.RunLoopProcess.runLoop(RunLoopProcess.java:31)
at com.zendesk.maxwell.Maxwell.run(Maxwell.java:89)
at com.zendesk.maxwell.Maxwell.main(Maxwell.java:95)
09:25:53,558 INFO SchemaPosition - Storing final position: null
A similar error occurs at the table level.
10:24:04,086 DEBUG SchemaChange - SQL_PARSE <- "create table /*! if not exists */ `_work`.`cdctest3` (
`id` int (10) unsigned not null auto_increment,
primary key (`id`)
)"
10:24:04,090 DEBUG SchemaChange - SQL_PARSE -> (parse (statement (create_table (create_table_preamble create table (table_name (db_name (name (id `_work`))) . (name (id `cdctest3`)))) (create_specifications ( (create_specification (column_definition (name (id `id`)) (data_type (signed_type int (length ( 10 )) (int_flags unsigned) (column_options (nullability not null)) (column_options auto_increment))))) , (create_specification (index_definition (index_type_pk primary key (index_column_list ( (index_columns (index_column (name (id `id`)))) ))))) )))) <EOF>)
com.zendesk.maxwell.schema.ddl.SchemaSyncError: Unexpectedly asked to create existing table cdctest3
at com.zendesk.maxwell.schema.ddl.TableCreate.apply(TableCreate.java:45)
at com.zendesk.maxwell.MaxwellReplicator.processQueryEvent(MaxwellReplicator.java:280)
at com.zendesk.maxwell.MaxwellReplicator.getEvent(MaxwellReplicator.java:253)
at com.zendesk.maxwell.MaxwellReplicator.work(MaxwellReplicator.java:89)
at com.zendesk.maxwell.RunLoopProcess.runLoop(RunLoopProcess.java:31)
at com.zendesk.maxwell.Maxwell.run(Maxwell.java:89)
at com.zendesk.maxwell.Maxwell.main(Maxwell.java:95)
10:24:04,188 INFO SchemaPosition - Storing final position: BinlogPosition[awmsphx90022-bin.000003:203414608]
10:24:04,189 DEBUG SchemaPosition - Writing binlog position to maxwell.positions: BinlogPosition[awmsphx90022-bin.000003:203414608]
It looks like Maxwell is crashing when I ALTER a table with an ADD INDEX clause specified with an ORDER.
Ex:
ALTER TABLE test
ADD COLUMN test_col
ENUM('Y', 'N') NULL DEFAULT 'N',
ADD INDEX test_col
(test_col_index
ASC);
Results in an error stack like the below:
com.zendesk.maxwell.schema.ddl.MaxwellSQLSyntaxError: ASC
at com.zendesk.maxwell.schema.ddl.MysqlParserListener.visitErrorNode(MysqlParserListener.java:85)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:41)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at com.zendesk.maxwell.schema.ddl.SchemaChange.parse(SchemaChange.java:67)
at com.zendesk.maxwell.MaxwellReplicator.processQueryEvent(MaxwellReplicator.java:273)
at com.zendesk.maxwell.MaxwellReplicator.getEvent(MaxwellReplicator.java:253)
at com.zendesk.maxwell.MaxwellReplicator.work(MaxwellReplicator.java:89)
at com.zendesk.maxwell.RunLoopProcess.runLoop(RunLoopProcess.java:31)
at com.zendesk.maxwell.Maxwell.run(Maxwell.java:89)
at com.zendesk.maxwell.Maxwell.main(Maxwell.java:95)
16:59:24,374 INFO SchemaPosition - Storing final position
Running Maxwell 0.14.5
15:19:53,024 DEBUG SchemaChange - SQL_PARSE <- "CREATE SCHEMA product_service_test"
line 1:7 no viable alternative at input 'CREATESCHEMA'
15:19:53,026 ERROR MysqlParserListener - (statement CREATE SCHEMA product_service_test)
15:19:53,026 ERROR SchemaChange - Error parsing SQL: 'CREATE SCHEMA product_service_test'
com.zendesk.maxwell.schema.ddl.MaxwellSQLSyntaxError: CREATE
at com.zendesk.maxwell.schema.ddl.MysqlParserListener.visitErrorNode(MysqlParserListener.java:85)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:41)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at com.zendesk.maxwell.schema.ddl.SchemaChange.parse(SchemaChange.java:67)
at com.zendesk.maxwell.MaxwellReplicator.processQueryEvent(MaxwellReplicator.java:273)
at com.zendesk.maxwell.MaxwellReplicator.getEvent(MaxwellReplicator.java:253)
at com.zendesk.maxwell.MaxwellReplicator.work(MaxwellReplicator.java:89)
at com.zendesk.maxwell.RunLoopProcess.runLoop(RunLoopProcess.java:31)
at com.zendesk.maxwell.Maxwell.run(Maxwell.java:89)
at com.zendesk.maxwell.Maxwell.main(Maxwell.java:95)
15:19:53,028 INFO SchemaPosition - Storing final position: null
Maxwell crashes if collation is specified in a CREATE statement.
MySQL version 5.1.73
Maxwell version 0.10.0
15:14:12,892 DEBUG SchemaPosition - Writing binlog position to maxwell.positions: BinlogPosition[mysql-binlog.000001:438782]
15:16:15,900 DEBUG SchemaChange - SQL_PARSE <- "CREATE TABLE `maxwell_collate_test` (
`id` INT(10) NOT NULL AUTO_INCREMENT,
`parent_id` INT(10) NULL DEFAULT NULL,
`model` VARCHAR(255) NULL DEFAULT NULL COLLATE 'utf8_unicode_ci',
`foreign_key` INT(10) NULL DEFAULT NULL,
PRIMARY KEY (`id`),
INDEX `parent` (`parent_id`),
INDEX `foreign_key` (`foreign_key`),
INDEX `model` (`model`)
)
COLLATE='utf8_unicode_ci'
ENGINE=InnoDB
ROW_FORMAT=COMPACT
AUTO_INCREMENT=371"
line 4:40 extraneous input 'COLLATE' expecting {',', ')', AUTO_INCREMENT, COLUMN_FORMAT, COMMENT, DEFAULT, NOT, NULL, ON, PRIMARY, STORAGE, UNIQUE}
line 4:48 missing '=' at ''utf8_unicode_ci''
15:16:15,940 ERROR MysqlParserListener - (create_specifications ( (create_specification (column_definition (name (id `id`)) (data_type (signed_type INT (length ( 10 )))) (column_options (nullability NOT NULL)) (column_options AUTO_INCREMENT))) , (create_specification (column_definition (name (id `parent_id`)) (data_type (signed_type INT (length ( 10 )))) (column_options (nullability NULL)) (column_options (default_value DEFAULT NULL)))) , (create_specification (column_definition (name (id `model`)) (data_type (string_type VARCHAR (length ( 255 )))) (column_options (nullability NULL)) (column_options (default_value DEFAULT NULL)))) <missing ')'>)
15:16:15,940 ERROR SchemaChange - Error parsing SQL: 'CREATE TABLE `maxwell_collate_test` (
`id` INT(10) NOT NULL AUTO_INCREMENT,
`parent_id` INT(10) NULL DEFAULT NULL,
`model` VARCHAR(255) NULL DEFAULT NULL COLLATE 'utf8_unicode_ci',
`foreign_key` INT(10) NULL DEFAULT NULL,
PRIMARY KEY (`id`),
INDEX `parent` (`parent_id`),
INDEX `foreign_key` (`foreign_key`),
INDEX `model` (`model`)
)
COLLATE='utf8_unicode_ci'
ENGINE=InnoDB
ROW_FORMAT=COMPACT
AUTO_INCREMENT=371'
com.zendesk.maxwell.schema.ddl.MaxwellSQLSyntaxError: <missing ')'>
at com.zendesk.maxwell.schema.ddl.MysqlParserListener.visitErrorNode(MysqlParserListener.java:85)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:41)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at com.zendesk.maxwell.schema.ddl.SchemaChange.parse(SchemaChange.java:67)
at com.zendesk.maxwell.MaxwellParser.processQueryEvent(MaxwellParser.java:208)
at com.zendesk.maxwell.MaxwellParser.getEvent(MaxwellParser.java:194)
at com.zendesk.maxwell.MaxwellParser.getEvent(MaxwellParser.java:200)
at com.zendesk.maxwell.MaxwellParser.run(MaxwellParser.java:100)
at com.zendesk.maxwell.Maxwell.run(Maxwell.java:69)
at com.zendesk.maxwell.Maxwell.main(Maxwell.java:75)
2015-09-10 15:16:15,981 FATAL Unable to register shutdown hook because JVM is shutting down.
From the docs...
Currently Maxwell is not very smart about master recovery or detecting a promoted slave; if it determines that the server_id has changed between runs, Maxwell will simply delete its old schema cache and binlog position and start again. We plan on improving this situation in 0.12.
From the changelog I guess this hasn't changed yet. Are there any new targets for this? I presume it's still on the roadmap?
Hi:
I created a table in mysql. then maxwell down .i don't know how to make it works normal.and the error log as below:
10:37:55,677 DEBUG SchemaChange - SQL_PARSE <- "CREATE TABLE emon.chk ( group_name VARCHAR(8) NOT NULL, group_key NUMERIC(19) NOT NULL, seqno NUMERIC(10), rba NUMERIC(19) NOT NULL, audit_ts VARCHAR(29), create_ts DATETIME NOT NULL, last_update_ts DATETIME NOT NULL, current_dir VARCHAR(255) NOT NULL, PRIMARY KEY (group_name, group_key))"
line 1:78 mismatched input ')' expecting ','
line 1:80 mismatched input 'NOT' expecting {, AUTO_INCREMENT, AVG_ROW_LENGTH, CHARACTER, CHARSET, CHECKSUM, COLLATE, COMMENT, CONNECTION, DATA, DEFAULT, DELAY_KEY_WRITE, ENGINE, INDEX, INSERT_METHOD, KEY_BLOCK_SIZE, MAX_ROWS, MIN_ROWS, PACK_KEYS, PASSWORD, ROW_FORMAT, TABLESPACE, UNION}
java.lang.IllegalArgumentException: unsupported column type numeric
at com.zendesk.maxwell.schema.columndef.ColumnDef.build(ColumnDef.java:80)
at com.zendesk.maxwell.schema.ddl.MysqlParserListener.exitColumn_definition(MysqlParserListener.java:273)
at com.zendesk.maxwell.schema.ddl.mysqlParser$Column_definitionContext.exitRule(mysqlParser.java:4370)
at org.antlr.v4.runtime.tree.ParseTreeWalker.exitRule(ParseTreeWalker.java:71)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:54)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at com.zendesk.maxwell.schema.ddl.SchemaChange.parse(SchemaChange.java:67)
at com.zendesk.maxwell.MaxwellParser.processQueryEvent(MaxwellParser.java:206)
at com.zendesk.maxwell.MaxwellParser.getEvent(MaxwellParser.java:192)
at com.zendesk.maxwell.MaxwellParser.getEvent(MaxwellParser.java:198)
at com.zendesk.maxwell.MaxwellParser.run(MaxwellParser.java:98)
at com.zendesk.maxwell.Maxwell.run(Maxwell.java:69)
at com.zendesk.maxwell.Maxwell.main(Maxwell.java:75)
I am testing Maxwell.
I am getting following error when started maxwell today. is there a way to fix this if my binary log files are rotated/deleted?
12:04:14,048 INFO AuthenticatorImpl - start to login, user: maxwell, host: localhost, port: 3306
12:04:14,051 INFO AuthenticatorImpl - login successfully, user: maxwell, detail: OKPacket[packetMarker=0,affectedRows=0,insertId=0,serverStatus=2,warningCount=0,message=<null>]
com.google.code.or.net.TransportException: Could not find first log file name in binary log index file
at com.google.code.or.OpenReplicator.dumpBinlog(OpenReplicator.java:288)
at com.google.code.or.OpenReplicator.start(OpenReplicator.java:104)
at com.zendesk.maxwell.MaxwellReplicator.beforeStart(MaxwellReplicator.java:83)
at com.zendesk.maxwell.RunLoopProcess.runLoop(RunLoopProcess.java:25)
at com.zendesk.maxwell.Maxwell.run(Maxwell.java:89)
at com.zendesk.maxwell.Maxwell.main(Maxwell.java:95)
12:04:14,079 INFO SchemaPosition - Storing final position: null
Can we use this to do a Mysql db dump to a kafka topic.
I am using maxwell CDC solution along with Apache NiFi and it is working great:
https://github.com/xmlking/nifi-examples/tree/master/oltp-cdc-olap
After following instructions to the letter in the quickstart, I get the following error when I run the bin/maxwell command:
Slave can not handle replication events with the checksum that master is configured to log..
After googling the error, one of the solutions was to turn of binlog_checksum, which I did. Once I did that, I get this:
Could not find first log file name in binary log index file.
I'm using MySQL 5.6 on Ubuntu. What could be going wrong?
How is Maxwell licensed? Apache License, BSD, GPL, etc.
I have seen an issue where if I update a large amount of records, ~400,000+ rows with a single update query, a memory limit is reached and a spill over to disk, sometimes never finishing or recovering.
Is there a way to avoid such an issue? Or a way around it? Increasing some kind of memory limit?
This will also happen if an alter statement, i.e adding a new column with a new default value, occurs on a relatively large table..~1,000,000 rows.
Arose when testing #94.
Any love for BINARY
and VARBINARY
columns? I naively presume they could just be treated the same as BLOB
and its variants?
16:03:46,359 DEBUG SchemaChange - SQL_PARSE <- "ALTER TABLE `maxwell_blob_test`
CHANGE COLUMN `blob` `blob` BINARY(50) NOT NULL AFTER `id`"
java.lang.IllegalArgumentException: unsupported column type binary
at com.zendesk.maxwell.schema.columndef.ColumnDef.build(ColumnDef.java:81)
at com.zendesk.maxwell.schema.ddl.MysqlParserListener.exitColumn_definition(MysqlParserListener.java:273)
at com.zendesk.maxwell.schema.ddl.mysqlParser$Column_definitionContext.exitRule(mysqlParser.java:4370)
at org.antlr.v4.runtime.tree.ParseTreeWalker.exitRule(ParseTreeWalker.java:71)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:54)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at com.zendesk.maxwell.schema.ddl.SchemaChange.parse(SchemaChange.java:67)
at com.zendesk.maxwell.MaxwellParser.processQueryEvent(MaxwellParser.java:208)
at com.zendesk.maxwell.MaxwellParser.getEvent(MaxwellParser.java:194)
at com.zendesk.maxwell.MaxwellParser.getEvent(MaxwellParser.java:200)
at com.zendesk.maxwell.MaxwellParser.run(MaxwellParser.java:100)
at com.zendesk.maxwell.Maxwell.run(Maxwell.java:69)
at com.zendesk.maxwell.Maxwell.main(Maxwell.java:75)
16:03:46,362 DEBUG SchemaPosition - Writing binlog position to maxwell.positions: BinlogPosition[mysql-binlog.000003:648414]
We turned on row-based replication in our mysql database but we also changed the binlog_row_image to be minimal rather than full. When we did this, Maxwell was logging the following message: "binlog_row_image must be FULL"
From our understanding minimal row image will log only changed columns, and columns needed to identify rows which should be enough for our purposes. This will avoid performance issues. Is there any plan to support MINIMAL binlog_row_image?
Hi Guys,
When I test the maxwell, I find it only can send the DML to kafka, but DDL, like create,drop,alter.
Do I miss something? or it can't support this feature?
And I also can't make sure which database the table is created in from DEBUG log.
e.g.
17:47:13,088 DEBUG SchemaChange - SQL_PARSE <- "create table t (id int auto_increment primary key,name varchar(10))"
One of my environments is reporting this WARN
(a lot!).
14:31:07,477 WARN MaxwellReplicator - Unhandled QueryEvent inside transaction: QueryEvent[header=BinlogEventV4HeaderImpl[timestamp=1449710974000,eventType=2,serverId=441,eventLength=279,nextPosition=219695949,flags=0,timestampOfReceipt=1449718267477],threadId=4,elapsedTime=0,databaseNameLength=6,errorCode=0,statusVariablesLength=35,statusVariables=[QFlags2Code[flags=0], QSQLModeCode[sqlMode=2097152], QCatalogNzCode[catalogName=std], QCharsetCode[characterSetClient=8,collationConnection=8,collationServer=8], QUpdatedDBNames[accessedDbCount=1,accessedDbs=[Lcom.google.code.or.common.glossary.column.StringColumn;@2da0770]],databaseName=merlyn,sql=INSERT INTO merlyn.system_variables(project, var_name, var_value) VALUES('xxxxx', 'yyyyy', 'zzzzz') ON DUPLICATE KEY UPDATE var_value= 'zzzzz']
My config file is setup to include specify databases (include_dbs
) and tables (include_tables
). Both merlyn
and system_variables
are not included.
Is there any reason that Maxwell needs to care about tables and databases that I'm not tracking events for?
And secondly, if I got this WARN
on a table that I was tracking, what does it actually mean? have I missed a change? What is causing this WARN
?
Saw an open-replicator thread die without maxwell restarting it:
06,364 ERROR AbstractBinlogParser - failed to parse binlog
Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:196) ~[?:1.7.0_79]
at java.net.SocketInputStream.read(SocketInputStream.java:122) ~[?:1.7.0_79]
at com.google.code.or.io.util.ActiveBufferedInputStream.run(ActiveBufferedInputStream.java:85) ~[open-replicator-1.3.0.jar:?]
at java.lang.Thread.run(Thread.java:745) [?:1.7.0_79]
15:59:29,552 ERROR MysqlParserListener - (data_type ( valid IN ( 0)
15:59:29,553 ERROR SchemaChange - Error parsing SQL: 'CREATE TABLE xxx
(
id INTEGER NOT NULL AUTO_INCREMENT,
...,
valid BOOL,
...,
PRIMARY KEY (id),
CHECK (valid IN (0, 1))
)'
com.zendesk.maxwell.schema.ddl.MaxwellSQLSyntaxError: (
at com.zendesk.maxwell.schema.ddl.MysqlParserListener.visitErrorNode(MysqlParserListener.java:88)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:41)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at com.zendesk.maxwell.schema.ddl.SchemaChange.parse(SchemaChange.java:74)
at com.zendesk.maxwell.MaxwellReplicator.processQueryEvent(MaxwellReplicator.java:288)
at com.zendesk.maxwell.MaxwellReplicator.getRow(MaxwellReplicator.java:268)
at com.zendesk.maxwell.MaxwellReplicator.work(MaxwellReplicator.java:89)
at com.zendesk.maxwell.RunLoopProcess.runLoop(RunLoopProcess.java:31)
at com.zendesk.maxwell.Maxwell.run(Maxwell.java:91)
at com.zendesk.maxwell.Maxwell.main(Maxwell.java:97)
15:59:29,562 INFO SchemaPosition - Storing final position: null
It would be useful to support a configuration that emits a diff record, only including the columns that have changed.
The old column data appears to be available in update events. I'm interested to know if this is something that has been considered? And if so is there a roadmap or design?
If you're on Windows, and do mvn test, even under cygwin, you get the error
java.io.IOException: Cannot run program "C:\.....\maxwell/src/test/onetimeserver": CreateProcess error=193, %1 is not a valid Win32 application at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048) at com.zendesk.maxwell.MysqlIsolatedServer.boot(MysqlIsolatedServer.java:34) at com.zendesk.maxwell.AbstractMaxwellTest.setUpBeforeClass(AbstractMaxwellTest.java:29) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:483) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) ....
Not sure what can be done about this.
Similar to #64 and #65, maxwell barfs and fails when it encounters a year datatype.
bin/maxwell --user='root' --password='root' --host='127.0.0.1' --producer=stdout
Picked up JAVA_TOOL_OPTIONS: -javaagent:/usr/share/java/jayatanaag.jar
23:29:28,708 INFO Maxwell - Maxwell is booting, starting at BinlogPosition[mysql-logs.000003:6982]
23:29:28,710 DEBUG SchemaStore - looking to restore schema at target position BinlogPosition[mysql-logs.000003:6982]
23:29:28,711 INFO SchemaStore - Restoring schema id 7 (last modified at BinlogPosition[mysql-logs.000003:6982])
23:29:28,753 INFO TransportImpl - connecting to host: 127.0.0.1, port: 3306
23:29:28,785 INFO TransportImpl - connected to host: 127.0.0.1, port: 3306, context: AbstractTransport.Context[threadId=25,scramble=@/f8$H"l|J=4*t,{KCN?,protocolVersion=10,serverHost=127.0.0.1,serverPort=3306,serverStatus=2,serverCollation=8,serverVersion=5.6.24-0ubuntu2-log,serverCapabilities=63487]
23:29:28,785 INFO AuthenticatorImpl - start to login, user: root, host: 127.0.0.1, port: 3306
23:29:28,792 INFO AuthenticatorImpl - login successfully, user: root, detail: OKPacket[packetMarker=0,affectedRows=0,insertId=0,serverStatus=2,warningCount=0,message=<null>]
23:29:28,974 DEBUG SchemaChange - SQL_PARSE <- "create table max_test4(dt datetime, dt_y year(4))"
line 1:45 mismatched input '(' expecting {',', ')', AUTO_INCREMENT, COLUMN_FORMAT, COMMENT, DEFAULT, NOT, NULL, ON, PRIMARY, STORAGE, UNIQUE}
line 1:48 extraneous input ')' expecting {<EOF>, AUTO_INCREMENT, AVG_ROW_LENGTH, CHARACTER, CHARSET, CHECKSUM, COLLATE, COMMENT, CONNECTION, DATA, DEFAULT, DELAY_KEY_WRITE, ENGINE, INDEX, INSERT_METHOD, KEY_BLOCK_SIZE, MAX_ROWS, MIN_ROWS, PACK_KEYS, PASSWORD, ROW_FORMAT, TABLESPACE, UNION}
23:29:29,065 ERROR MysqlParserListener - (column_definition (name (id dt_y)) (data_type (generic_type year)) ( 4)
23:29:29,065 ERROR SchemaChange - Error parsing SQL: 'create table max_test4(dt datetime, dt_y year(4))'
com.zendesk.maxwell.schema.ddl.MaxwellSQLSyntaxError: (
at com.zendesk.maxwell.schema.ddl.MysqlParserListener.visitErrorNode(MysqlParserListener.java:85)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:41)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at com.zendesk.maxwell.schema.ddl.SchemaChange.parse(SchemaChange.java:65)
at com.zendesk.maxwell.MaxwellParser.processQueryEvent(MaxwellParser.java:206)
at com.zendesk.maxwell.MaxwellParser.getEvent(MaxwellParser.java:192)
at com.zendesk.maxwell.MaxwellParser.getEvent(MaxwellParser.java:198)
at com.zendesk.maxwell.MaxwellParser.run(MaxwellParser.java:98)
at com.zendesk.maxwell.Maxwell.run(Maxwell.java:69)
at com.zendesk.maxwell.Maxwell.main(Maxwell.java:75)
And just like #64 and #65, re-running Maxwell does not work and produces the same text as above and exits. I will need to purge
and delete
again for this to work.
Is there a planned release for supporting replication based on the GTID, as opposed to the position, which I believe is the implementation that is currently existing?
I issued a truncate table
statement and Maxwell didn't like it. To the extent that it crashed, and failed every time I tried to run it. I had to use the purge binary logs before now()
and delete from maxwell.positions
commands to get it running again. Here is the log:
TRUNCATE table error:
23:12:55,915 DEBUG SchemaChange - SQL_PARSE <- "truncate table max_test3"
line 1:0 no viable alternative at input 'truncate'
23:12:55,941 ERROR MysqlParserListener - (statement truncate table max_test3)
23:12:55,942 ERROR SchemaChange - Error parsing SQL: 'truncate table max_test3'
com.zendesk.maxwell.schema.ddl.MaxwellSQLSyntaxError: truncate
at com.zendesk.maxwell.schema.ddl.MysqlParserListener.visitErrorNode(MysqlParserListener.java:85)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:41)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at com.zendesk.maxwell.schema.ddl.SchemaChange.parse(SchemaChange.java:65)
at com.zendesk.maxwell.MaxwellParser.processQueryEvent(MaxwellParser.java:206)
at com.zendesk.maxwell.MaxwellParser.getEvent(MaxwellParser.java:192)
at com.zendesk.maxwell.MaxwellParser.getEvent(MaxwellParser.java:198)
at com.zendesk.maxwell.MaxwellParser.run(MaxwellParser.java:98)
at com.zendesk.maxwell.Maxwell.run(Maxwell.java:69)
at com.zendesk.maxwell.Maxwell.main(Maxwell.java:75)
23:12:55,953 DEBUG SchemaPosition - Writing binlog position to maxwell.positions: BinlogPosition[mysql-logs.000003:6424]
Maxwell does not recover after truncate statement:
bin/maxwell --user='root' --password='root' --host='127.0.0.1' --producer=stdout
Picked up JAVA_TOOL_OPTIONS: -javaagent:/usr/share/java/jayatanaag.jar
23:17:15,992 INFO Maxwell - Maxwell is booting, starting at BinlogPosition[mysql-logs.000003:6424]
23:17:15,993 DEBUG SchemaStore - looking to restore schema at target position BinlogPosition[mysql-logs.000003:6424]
23:17:15,995 INFO SchemaStore - Restoring schema id 6 (last modified at BinlogPosition[mysql-logs.000003:330])
23:17:16,057 INFO TransportImpl - connecting to host: 127.0.0.1, port: 3306
23:17:16,088 INFO TransportImpl - connected to host: 127.0.0.1, port: 3306, context: AbstractTransport.Context[threadId=18,scramble=EDcI$uB*Gi[m4V5E>E}h,protocolVersion=10,serverHost=127.0.0.1,serverPort=3306,serverStatus=2,serverCollation=8,serverVersion=5.6.24-0ubuntu2-log,serverCapabilities=63487]
23:17:16,088 INFO AuthenticatorImpl - start to login, user: root, host: 127.0.0.1, port: 3306
23:17:16,093 INFO AuthenticatorImpl - login successfully, user: root, detail: OKPacket[packetMarker=0,affectedRows=0,insertId=0,serverStatus=2,warningCount=0,message=<null>]
23:17:16,226 DEBUG SchemaChange - SQL_PARSE <- "truncate table max_test3"
line 1:0 no viable alternative at input 'truncate'
23:17:16,274 ERROR MysqlParserListener - (statement truncate table max_test3)
23:17:16,283 ERROR SchemaChange - Error parsing SQL: 'truncate table max_test3'
com.zendesk.maxwell.schema.ddl.MaxwellSQLSyntaxError: truncate
at com.zendesk.maxwell.schema.ddl.MysqlParserListener.visitErrorNode(MysqlParserListener.java:85)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:41)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at com.zendesk.maxwell.schema.ddl.SchemaChange.parse(SchemaChange.java:65)
at com.zendesk.maxwell.MaxwellParser.processQueryEvent(MaxwellParser.java:206)
at com.zendesk.maxwell.MaxwellParser.getEvent(MaxwellParser.java:192)
at com.zendesk.maxwell.MaxwellParser.getEvent(MaxwellParser.java:198)
at com.zendesk.maxwell.MaxwellParser.run(MaxwellParser.java:98)
at com.zendesk.maxwell.Maxwell.run(Maxwell.java:69)
at com.zendesk.maxwell.Maxwell.main(Maxwell.java:75)
Please add support for TRUNCATE statements, or at least ignore them (maybe not a great idea :)). But this kind of failure is no good. Thanks!
The website http://maxwells-daemon.io/kafka/ says the format is something like
db:test_db/tbl:test_tbl/id:93/id_2:17910090
But in practice, it appears the key is actually json:
{"database":"test","table":"users","pk.userId":4}
This output is from my test database.
Maxwell is using JDK 7
and not support JDK 6
?
I'm reading maxwell and found something maybe could rewrite
such as :
InputStream schemaSQL = SchemaStore.class.getResourceAsStream(
"/sql/maxwell_schema.sql");
BufferedReader r = new BufferedReader(new InputStreamReader(schemaSQL));
String sql = "", line;
LOGGER.info("Creating maxwell database");
while ((line = r.readLine()) != null) {
sql += line + "\n";
}
In JDK 7 Files.readAllLines(path, cs) is looks better :)
Hi
i am trying to transfer data from mysql to kafka by using maxwell . the day before yesterday ,about 09:30,I construct three million rows in one table in mysql database.and the maxwell began to load data to kafka. but at now the maxwell is still display message like bellow
11:41:36,914 DEBUG MaxwellKafkaProducer - {"database":"emon","table":"msql","type":"insert","data":{"EDITION_NAME":"a","OBJECT_NAME":"ob","GENERATED":"a","OBJECT_TYPE":"OO","OBJECT_ID":813790,"CREATED":"2015-08-28","STATUS":"a","OWNER":"测试","SECONDARY":"a","SUBOBJECT_NAME":"o","TEMPORARY":"a","TIMESTAMP":"1","DATA_OBJECT_ID":813790,"NAMESPACE":1,"LAST_DDL_TIME":"2015-08-28"},"ts":1441165290}
11:41:36,914 DEBUG MaxwellKafkaProducer - BinlogPosition[master.000007:1013840679]
11:41:36,914 DEBUG MaxwellKafkaProducer -
11:41:37,219 DEBUG SchemaPosition - Writing binlog position to maxwell.positions: BinlogPosition[master.000007:1013840679]
how i can know the amount of the data that hasn't been transferred by maxwell.
Similar to #64, Maxwell chokes when it encounters a field with real
type.
Maxwell error with real datatype:
22:39:47,906 DEBUG SchemaChange - SQL_PARSE <- "create table max_test2 (col1 int, col2 tinyint, col3 bigint, col4 decimal, col5 float, col6 real)"
java.lang.IllegalArgumentException: unsupported column type real
at com.zendesk.maxwell.schema.columndef.ColumnDef.build(ColumnDef.java:73)
at com.zendesk.maxwell.schema.ddl.MysqlParserListener.exitColumn_definition(MysqlParserListener.java:273)
at com.zendesk.maxwell.schema.ddl.mysqlParser$Column_definitionContext.exitRule(mysqlParser.java:4362)
at org.antlr.v4.runtime.tree.ParseTreeWalker.exitRule(ParseTreeWalker.java:71)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:54)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at com.zendesk.maxwell.schema.ddl.SchemaChange.parse(SchemaChange.java:65)
at com.zendesk.maxwell.MaxwellParser.processQueryEvent(MaxwellParser.java:206)
at com.zendesk.maxwell.MaxwellParser.getEvent(MaxwellParser.java:192)
at com.zendesk.maxwell.MaxwellParser.getEvent(MaxwellParser.java:198)
at com.zendesk.maxwell.MaxwellParser.run(MaxwellParser.java:98)
at com.zendesk.maxwell.Maxwell.run(Maxwell.java:69)
at com.zendesk.maxwell.Maxwell.main(Maxwell.java:75)
2015-06-18 05:39:47,952 FATAL Unable to register shutdown hook because JVM is shutting down.
After error with real, subsequent runs fail:
bin/maxwell --user='root' --password='root' --host='127.0.0.1' --producer=stdout
Picked up JAVA_TOOL_OPTIONS: -javaagent:/usr/share/java/jayatanaag.jar
22:47:29,260 INFO Maxwell - Maxwell is booting, starting at BinlogPosition[mysql-logs.000002:11877]
22:47:29,262 DEBUG SchemaStore - looking to restore schema at target position BinlogPosition[mysql-logs.000002:11877]
22:47:29,263 INFO SchemaStore - Restoring schema id 5 (last modified at BinlogPosition[mysql-logs.000002:9053])
22:47:29,305 INFO TransportImpl - connecting to host: 127.0.0.1, port: 3306
22:47:29,323 INFO TransportImpl - connected to host: 127.0.0.1, port: 3306, context: AbstractTransport.Context[threadId=28,scramble=\\9IAJ3[TpYMS<o="0kK,protocolVersion=10,serverHost=127.0.0.1,serverPort=3306,serverStatus=2,serverCollation=8,serverVersion=5.6.24-0ubuntu2-log,serverCapabilities=63487]
22:47:29,324 INFO AuthenticatorImpl - start to login, user: root, host: 127.0.0.1, port: 3306
22:47:29,332 INFO AuthenticatorImpl - login successfully, user: root, detail: OKPacket[packetMarker=0,affectedRows=0,insertId=0,serverStatus=2,warningCount=0,message=<null>]
22:47:29,550 DEBUG SchemaChange - SQL_PARSE <- "create table max_test2 (col1 int, col2 tinyint, col3 bigint, col4 decimal, col5 float, col6 real)"
java.lang.IllegalArgumentException: unsupported column type real
at com.zendesk.maxwell.schema.columndef.ColumnDef.build(ColumnDef.java:73)
at com.zendesk.maxwell.schema.ddl.MysqlParserListener.exitColumn_definition(MysqlParserListener.java:273)
at com.zendesk.maxwell.schema.ddl.mysqlParser$Column_definitionContext.exitRule(mysqlParser.java:4362)
at org.antlr.v4.runtime.tree.ParseTreeWalker.exitRule(ParseTreeWalker.java:71)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:54)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at com.zendesk.maxwell.schema.ddl.SchemaChange.parse(SchemaChange.java:65)
at com.zendesk.maxwell.MaxwellParser.processQueryEvent(MaxwellParser.java:206)
at com.zendesk.maxwell.MaxwellParser.getEvent(MaxwellParser.java:192)
at com.zendesk.maxwell.MaxwellParser.getEvent(MaxwellParser.java:198)
at com.zendesk.maxwell.MaxwellParser.run(MaxwellParser.java:98)
at com.zendesk.maxwell.Maxwell.run(Maxwell.java:69)
at com.zendesk.maxwell.Maxwell.main(Maxwell.java:75)
Thanks for making a great tool. I believe this comes 99% of the way to solving my use case. But I need to limit the maxwell producer to only certain SQL databases and tables on the server. In other words, changes to some SQL databases and tables cause maxwell to emit a JSON object, while changes in other databases and tables do not. Is this possible? Thanks!
I had this error occur today.
line 4:0 no viable alternative at input 'SELECT'
line 5:3 token recognition error at: '*\r'
10:58:19,973 ERROR MysqlParserListener - (create_table (create_table_preamble CREATE TEMPORARY TABLE (table_name (name (id 172898_16841_transmem)))) SELECT t . FROM map . transmem AS t WHERE t . deleted = 'N' AND IFNULL ( t . trnnumrev , 0 ) = 0 AND t . locked = 'Y' AND t . lockedby = 'AUTOSTL' AND t . allocated = 'N' AND t . special = 'N' AND t . invnum_d = 172898 AND t . imnum = 16841)
10:58:19,974 ERROR SchemaChange - Error parsing SQL: 'CREATE
TEMPORARY
TABLE 172898_16841_transmem
SELECT
t.*
FROM map.transmem AS t
WHERE t.deleted = 'N'
AND IFNULL(t.trnnumrev, 0) = 0
AND t.locked = 'Y'
AND t.lockedby = 'AUTOSTL'
AND t.allocated = 'N'
AND t.special = 'N'
AND t.invnum_d = 172898
AND t.imnum = 16841'
com.zendesk.maxwell.schema.ddl.MaxwellSQLSyntaxError: SELECT
at com.zendesk.maxwell.schema.ddl.MysqlParserListener.visitErrorNode(MysqlParserListener.java:85)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:41)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at com.zendesk.maxwell.schema.ddl.SchemaChange.parse(SchemaChange.java:67)
at com.zendesk.maxwell.MaxwellReplicator.processQueryEvent(MaxwellReplicator.java:274)
at com.zendesk.maxwell.MaxwellReplicator.getRow(MaxwellReplicator.java:254)
at com.zendesk.maxwell.MaxwellReplicator.work(MaxwellReplicator.java:87)
at com.zendesk.maxwell.RunLoopProcess.runLoop(RunLoopProcess.java:31)
at com.zendesk.maxwell.Maxwell.run(Maxwell.java:89)
at com.zendesk.maxwell.Maxwell.main(Maxwell.java:95)
10:58:19,975 INFO SchemaPosition - Storing final position: BinlogPosition[awmstvx00441-bin.000001:79187451]
I realise from the docs that binary column support is experimental. I'd like to use maxwell with an existing database which has some blob columns but maxwell crashes whenever a row with a blob
column is modified.
MySQL 5.1.59
Maxwell 0.11.0-RC1
Steps to reproduce:
CREATE TABLE `maxwell_blob_test` (
`id` INT UNSIGNED NOT NULL AUTO_INCREMENT,
`blob` BLOB NOT NULL,
PRIMARY KEY (`id`)
)
COLLATE='utf8_unicode_ci'
ENGINE=MyISAM
;
INSERT INTO `sellstuf1`.`maxwell_blob_test` (`blob`) VALUES (0x22);
Maxwell log
16:36:04,286 DEBUG SchemaPosition - Writing binlog position to maxwell.positions: BinlogPosition[mysql-binlog.000002:65790]
16:42:58,092 DEBUG SchemaChange - SQL_PARSE <- "CREATE TABLE `maxwell_blob_test` (
`id` INT UNSIGNED NOT NULL AUTO_INCREMENT,
`blob` BLOB NOT NULL,
PRIMARY KEY (`id`)
)
COLLATE='utf8_unicode_ci'
ENGINE=MyISAM"
16:42:58,127 DEBUG SchemaChange - SQL_PARSE -> (parse (statement (create_table (create_table_preamble CREATE TABLE (table_name (name (id `maxwell_blob_test`)))) (create_specifications ( (create_specification (column_definition (name (id `id`)) (data_type (signed_type INT (int_flags UNSIGNED))) (column_options (nullability NOT NULL)) (column_options AUTO_INCREMENT))) , (create_specification (column_definition (name (id `blob`)) (data_type (generic_type BLOB)) (column_options (nullability NOT NULL)))) , (create_specification (index_definition (index_type_pk PRIMARY KEY (index_column_list ( (index_columns (index_column (name (id `id`)))) ))))) )) (table_creation_options (creation_collation COLLATE = (string 'utf8_unicode_ci'))) (table_creation_options (creation_engine ENGINE = (string MyISAM))))) <EOF>)
16:42:58,129 INFO MaxwellParser - storing schema @BinlogPosition[mysql-binlog.000002:124061] after applying "CREATE T ENGINE=MyISAM"unicode_ci'LL, NULL AUTO_INCREMENT,
16:42:58,252 DEBUG SchemaPosition - syncing binlog position: BinlogPosition[mysql-binlog.000002:124061]
16:42:58,254 DEBUG SchemaPosition - Writing binlog position to maxwell.positions: BinlogPosition[mysql-binlog.000002:124061]
java.lang.NullPointerException: charset
at java.lang.String.<init>(String.java:449)
at java.lang.String.<init>(String.java:503)
at com.zendesk.maxwell.schema.columndef.StringColumnDef.asJSON(StringColumnDef.java:54)
at com.zendesk.maxwell.MaxwellAbstractRowsEvent.jsonMaps(MaxwellAbstractRowsEvent.java:229)
at com.zendesk.maxwell.MaxwellAbstractRowsEvent.toJSONStrings(MaxwellAbstractRowsEvent.java:251)
at com.zendesk.maxwell.StdoutProducer.push(StdoutProducer.java:12)
at com.zendesk.maxwell.MaxwellParser.run(MaxwellParser.java:111)
at com.zendesk.maxwell.Maxwell.run(Maxwell.java:69)
at com.zendesk.maxwell.Maxwell.main(Maxwell.java:75)
2015-09-10 16:43:43,071 FATAL Unable to register shutdown hook because JVM is shutting down.
In our app, there are a selection of tables (one per day), and they get created every minute or so. We have several servers, and they each create these tables using "CREATE ... IF NOT EXISTS". Normally this is fine, as it will do nothing in mysql. These DDL lines still get replicated, and maxwell parses them, and adds the 'new' table to the schema each time it is seen.
> select count(*) from tables; +----------+ | count(*) | +----------+ | 386694 | +----------+ > select count(distinct name) from tables; +----------------------+ | count(distinct name) | +----------------------+ | 824 | +----------------------+
This is after running Maxwell for less than an hour.
Running on maxwell-0.14.4
08:28:30,736 DEBUG SchemaChange - SQL_PARSE <- "CREATE TABLE
`mvlogs` (table_schema varchar(50),
table_name varchar(50),
mvlog_name varchar(50),
active_flag boolean default true,
primary key(table_schema,table_name),
unique key(mvlog_name)
) ENGINE=INNODB DEFAULT CHARSET=utf8"
line 5:57 no viable alternative at input 'true'
08:28:30,743 ERROR MysqlParserListener - (default_value default true)
08:28:30,743 ERROR SchemaChange - Error parsing SQL: 'CREATE TABLE
`mvlogs` (table_schema varchar(50),
table_name varchar(50),
mvlog_name varchar(50),
active_flag boolean default true,
primary key(table_schema,table_name),
unique key(mvlog_name)
) ENGINE=INNODB DEFAULT CHARSET=utf8'
com.zendesk.maxwell.schema.ddl.MaxwellSQLSyntaxError: true
at com.zendesk.maxwell.schema.ddl.MysqlParserListener.visitErrorNode(MysqlParserListener.java:85)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:41)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at com.zendesk.maxwell.schema.ddl.SchemaChange.parse(SchemaChange.java:67)
at com.zendesk.maxwell.MaxwellReplicator.processQueryEvent(MaxwellReplicator.java:273)
at com.zendesk.maxwell.MaxwellReplicator.getEvent(MaxwellReplicator.java:253)
at com.zendesk.maxwell.MaxwellReplicator.work(MaxwellReplicator.java:89)
at com.zendesk.maxwell.RunLoopProcess.runLoop(RunLoopProcess.java:31)
at com.zendesk.maxwell.Maxwell.run(Maxwell.java:89)
at com.zendesk.maxwell.Maxwell.main(Maxwell.java:95)
08:28:30,744 INFO SchemaPosition - Storing final position: BinlogPosition[awmsphx90022-bin.000003:237982669]
08:28:30,745 DEBUG SchemaPosition - Writing binlog position to maxwell.positions: BinlogPosition[awmsphx90022-bin.000003:237982669]
the current log has this format
09:01:07,003 DEBUG SchemaChange - ...
it would be nice to have the date included with the time so that you can tell if maxwell is still running correctly and responsively.
After bootstrapping the docker images I can no longer start the maxwell image which throws this error:
13:06:09,458 DEBUG SchemaChange - SQL_PARSE -> (parse (statement (create_database CREATE DATABASE (name (id `zendesk_development`)))) <EOF>)
com.zendesk.maxwell.schema.ddl.SchemaSyncError: Unexpectedly asked to create existing database zendesk_development
at com.zendesk.maxwell.schema.ddl.DatabaseCreate.apply(DatabaseCreate.java:25)
at com.zendesk.maxwell.MaxwellReplicator.processQueryEvent(MaxwellReplicator.java:279)
at com.zendesk.maxwell.MaxwellReplicator.getEvent(MaxwellReplicator.java:253)
at com.zendesk.maxwell.MaxwellReplicator.work(MaxwellReplicator.java:89)
at com.zendesk.maxwell.RunLoopProcess.runLoop(RunLoopProcess.java:27)
at com.zendesk.maxwell.Maxwell.run(Maxwell.java:82)
at com.zendesk.maxwell.Maxwell.main(Maxwell.java:88)
13:06:09,460 INFO SchemaPosition - Storing final position: null
Both @rbarfoed and I have been experiencing this error.
How to reproduce:
./bin/onboard
from the docker-images folderzdi maxwell start
Updating a table's AUTO_INCREMENT value causes maxwell to crash when it encounters the statement in the bin log.
Maxwell 0.11.2
MySQL 5.1.
Statement to reproduce
ALTER TABLE test_table AUTO_INCREMENT = 1
11:01:31,725 DEBUG SchemaChange - SQL_PARSE <- "ALTER TABLE test_table AUTO_INCREMENT = 1"
line 1:41 no viable alternative at input 'AUTO_INCREMENT'
11:01:31,751 ERROR MysqlParserListener - (alter_specification AUTO_INCREMENT = 1)
11:01:31,751 ERROR SchemaChange - Error parsing SQL: 'ALTER TABLE test_table AUTO_INCREMENT = 1'
com.zendesk.maxwell.schema.ddl.MaxwellSQLSyntaxError: AUTO_INCREMENT
at com.zendesk.maxwell.schema.ddl.MysqlParserListener.visitErrorNode(MysqlParserListener.java:85)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:41)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at org.antlr.v4.runtime.tree.ParseTreeWalker.walk(ParseTreeWalker.java:52)
at com.zendesk.maxwell.schema.ddl.SchemaChange.parse(SchemaChange.java:67)
at com.zendesk.maxwell.MaxwellParser.processQueryEvent(MaxwellParser.java:284)
at com.zendesk.maxwell.MaxwellParser.getEvent(MaxwellParser.java:251)
at com.zendesk.maxwell.MaxwellParser.run(MaxwellParser.java:93)
at com.zendesk.maxwell.Maxwell.run(Maxwell.java:73)
at com.zendesk.maxwell.Maxwell.main(Maxwell.java:79)
I assume Maxwell should be safe to ignore this, or does it track the auto increment value?
running the tests using mvn install
results in the following behavior:
Running com.zendesk.maxwell.SchemaStoreTest
DEBUG MysqlIsolatedServer - booting onetimeserver: ./src/test/onetimeserver --mysql-version=5.5 --log-bin=master --binlog_format=row --innodb_flush_log_at_trx_commit=1 --server_id=123123
which then hangs forever, even though a ps aux | grep mysql
shows the wrapper, onetimeserver-go and mysqld processes running along famously.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.