Quickstart instructions lead to error about maxwell

[osheroff]

hmmm. I haven't tested yet with mysql 5.6, but I've seen "Could not find first log file..." when the master is reset underneath maxwell:

what's the output of both:

show master status

and

select * from maxwell.positions

?

[akshayi1]

show master status shows that there is a binlog with the position intact.

+-------------------+----------+--------------+------------------+-------------------+
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set |
+-------------------+----------+--------------+------------------+-------------------+
| mysql-logs.000002 | 120 | | | |
+-------------------+----------+--------------+------------------+-------------------+

select * from maxwell.positions

| server_id | binlog_file | binlog_position |
+-----------+---------------+-----------------+
| 1 | master.000002 | 2219 |

[osheroff]

nope, the master position pointer is back behind where maxwell thinks it should be. try doing "DELETE from maxwell.positions" and restarting maxwell.

[akshayi1]

My my.cnf file is as follows:

[mysqld]

server-id = 1
log-bin = /tmp/mysql-logs
binlog_format = row
binlog_checksum = NONE

Does it need to be a faithful replication of the quickstart guide for it to work?

Also, I ran the GRANT commands but with user root, and am running the maxwell command with the user as root as well. When I ran it after issuing the delete command, maxwell ran with no issues, but when I tried to insert a row into the maxwell.test table (a table I created), the row inserted successfully, but nothing came up on maxwell's end. Is something fundamentally wrong with the setup?

[osheroff]

maxwell should be able to run as root just fine, but by default it doesn't output any modifications that happen inside the "maxwell" database. try inserting a row into a different database/table. I'm assuming you're running with the "stdout" producer?

btw, you're the first non-zendesk user, so you're hitting lots of issues :)

[akshayi1]

Wow! I'm honored...? :) I began looking into this when we wanted to replicate MySQL and found mypipe, but wanted the logs in JSON. Maxwell does seem to be working, but I can't figure out what the pattern is. So, I created a new database, m_test, and a new table create table max_test (i_id int, ver double, name varchar(12));. AFTER these two steps, I ran maxwell, and the following log was produced:

bin/maxwell --user='root' --password='root' --host='127.0.0.1' --producer=stdout
Picked up JAVA_TOOL_OPTIONS: -javaagent:/usr/share/java/jayatanaag.jar 
16:15:51,508 INFO  Maxwell - Maxwell is booting, starting at BinlogPosition[mysql-logs.000002:327]
16:15:51,510 DEBUG SchemaStore - looking to restore schema at target position BinlogPosition[mysql-logs.000002:327]
16:15:51,511 INFO  SchemaStore - Restoring schema id 2 (last modified at BinlogPosition[mysql-logs.000002:327])
16:15:51,561 INFO  TransportImpl - connecting to host: 127.0.0.1, port: 3306
16:15:51,610 INFO  TransportImpl - connected to host: 127.0.0.1, port: 3306, context: AbstractTransport.Context[threadId=12,scramble=fARYH=7=l(~hkb)BPNt!,protocolVersion=10,serverHost=127.0.0.1,serverPort=3306,serverStatus=2,serverCollation=8,serverVersion=5.6.24-0ubuntu2-log,serverCapabilities=63487]
16:15:51,610 INFO  AuthenticatorImpl - start to login, user: root, host: 127.0.0.1, port: 3306
16:15:51,622 INFO  AuthenticatorImpl - login successfully, user: root, detail: OKPacket[packetMarker=0,affectedRows=0,insertId=0,serverStatus=2,warningCount=0,message=<null>]
16:15:51,793 DEBUG SchemaChange - SQL_PARSE <- "create database m_test"
16:15:51,837 DEBUG SchemaChange - SQL_PARSE ->   (parse (statement (create_database create database (name (id m_test)))) <EOF>)
16:15:51,837 INFO  MaxwellParser - storing schema @BinlogPosition[mysql-logs.000002:2898] after applying "create database m_test"
16:15:51,861 DEBUG SchemaPosition - syncing binlog position: BinlogPosition[mysql-logs.000002:2898]
16:15:51,862 DEBUG SchemaPosition - Writing binlog position to maxwell.positions: BinlogPosition[mysql-logs.000002:2898]
16:15:51,913 DEBUG SchemaChange - SQL_PARSE <- "create table max_test (i_id int, ver double, name varchar(12))"
16:15:51,926 DEBUG SchemaChange - SQL_PARSE ->   (parse (statement (create_table (create_table_preamble create table (table_name (name (id max_test)))) (create_specifications ( (create_specification (column_definition (name (id i_id)) (data_type (signed_type int)))) , (create_specification (column_definition (name (id ver)) (data_type (signed_type double)))) , (create_specification (column_definition (name (id name)) (data_type (string_type varchar (length ( 12 )))))) )))) <EOF>)
16:15:51,929 INFO  MaxwellParser - storing schema @BinlogPosition[mysql-logs.000002:3034] after applying "create table max_test (i_id int, ver double, name varchar(12))"
16:15:51,959 DEBUG SchemaPosition - syncing binlog position: BinlogPosition[mysql-logs.000002:3034]
16:15:51,959 DEBUG SchemaPosition - Writing binlog position to maxwell.positions: BinlogPosition[mysql-logs.000002:3034]

I then inserted a row: insert into max_test values (1,0.1,'max'); to see if it would spew it out, but nothing. So I quit Maxwell, and ran it again, thinking that it would show up the next time around, just like the database and table creation statements it showed the last time. But nothing. I did the same insert again, and an update statement, but I'm getting radio silence.

bin/maxwell --user='root' --password='root' --host='127.0.0.1' --producer=stdout
Picked up JAVA_TOOL_OPTIONS: -javaagent:/usr/share/java/jayatanaag.jar 
16:18:12,063 INFO  Maxwell - Maxwell is booting, starting at BinlogPosition[mysql-logs.000002:3034]
16:18:12,065 DEBUG SchemaStore - looking to restore schema at target position BinlogPosition[mysql-logs.000002:3034]
16:18:12,066 INFO  SchemaStore - Restoring schema id 4 (last modified at BinlogPosition[mysql-logs.000002:3034])
16:18:12,103 INFO  TransportImpl - connecting to host: 127.0.0.1, port: 3306
16:18:12,137 INFO  TransportImpl - connected to host: 127.0.0.1, port: 3306, context: AbstractTransport.Context[threadId=16,scramble=>TlEPC@Uk=,a*Hke6}76,protocolVersion=10,serverHost=127.0.0.1,serverPort=3306,serverStatus=2,serverCollation=8,serverVersion=5.6.24-0ubuntu2-log,serverCapabilities=63487]
16:18:12,137 INFO  AuthenticatorImpl - start to login, user: root, host: 127.0.0.1, port: 3306
16:18:12,158 INFO  AuthenticatorImpl - login successfully, user: root, detail: OKPacket[packetMarker=0,affectedRows=0,insertId=0,serverStatus=2,warningCount=0,message=<null>]

Is this normal? I guess I'm having difficulty in understanding exactly what I'm supposed to be seeing. Also, those create statements don't look like JSON. Is that normal too?

[osheroff]

let me test under 5.6 and see if there's something there that's different. From the logs it certainly looks like maxwell is working (it's processing the schema from the replication stream just fine), but you should also be seeing JSON from your inserts.

[akshayi1]

Sounds good. Thanks! 😄

Are the create statements supposed to be the way they are appearing, or should that be JSON (but is not showing up as that) too?

[osheroff]

nah, they're supposed to be that way. maxwell needs to consume the CREATE/ALTER statements for its own use, but it doesn't output them onto the kafka (or stdout) bus. So those are just log lines

[akshayi1]

I see. Okay. Thanks for the awesome support so far!

[osheroff]

yeah, I can reproduce this easily: 5.6 doesn't seem to work at all! stay tuned..

[akshayi1]

Yay?

[akshayi1]

I have a few issues to report with the 5.6 support. What is the best way to do this?

[osheroff]

go ahead & create issues.

[akshayi1]

Latest version works. Will close this now.