Comments (15)
hmmm. I haven't tested yet with mysql 5.6, but I've seen "Could not find first log file..." when the master is reset underneath maxwell:
what's the output of both:
show master status
and
select * from maxwell.positions
?
from maxwell.
show master status
shows that there is a binlog with the position intact.
+-------------------+----------+--------------+------------------+-------------------+
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set |
+-------------------+----------+--------------+------------------+-------------------+
| mysql-logs.000002 | 120 | | | |
+-------------------+----------+--------------+------------------+-------------------+
select * from maxwell.positions
| server_id | binlog_file | binlog_position |
+-----------+---------------+-----------------+
| 1 | master.000002 | 2219 |
from maxwell.
nope, the master position pointer is back behind where maxwell thinks it should be. try doing "DELETE from maxwell.positions" and restarting maxwell.
from maxwell.
My my.cnf
file is as follows:
[mysqld]
server-id = 1
log-bin = /tmp/mysql-logs
binlog_format = row
binlog_checksum = NONE
Does it need to be a faithful replication of the quickstart guide for it to work?
Also, I ran the GRANT commands but with user root, and am running the maxwell command with the user as root as well. When I ran it after issuing the delete command, maxwell ran with no issues, but when I tried to insert a row into the maxwell.test table (a table I created), the row inserted successfully, but nothing came up on maxwell's end. Is something fundamentally wrong with the setup?
from maxwell.
maxwell should be able to run as root just fine, but by default it doesn't output any modifications that happen inside the "maxwell" database. try inserting a row into a different database/table. I'm assuming you're running with the "stdout" producer?
btw, you're the first non-zendesk user, so you're hitting lots of issues :)
from maxwell.
Wow! I'm honored...? :) I began looking into this when we wanted to replicate MySQL and found mypipe, but wanted the logs in JSON. Maxwell does seem to be working, but I can't figure out what the pattern is. So, I created a new database, m_test, and a new table create table max_test (i_id int, ver double, name varchar(12));
. AFTER these two steps, I ran maxwell, and the following log was produced:
bin/maxwell --user='root' --password='root' --host='127.0.0.1' --producer=stdout
Picked up JAVA_TOOL_OPTIONS: -javaagent:/usr/share/java/jayatanaag.jar
16:15:51,508 INFO Maxwell - Maxwell is booting, starting at BinlogPosition[mysql-logs.000002:327]
16:15:51,510 DEBUG SchemaStore - looking to restore schema at target position BinlogPosition[mysql-logs.000002:327]
16:15:51,511 INFO SchemaStore - Restoring schema id 2 (last modified at BinlogPosition[mysql-logs.000002:327])
16:15:51,561 INFO TransportImpl - connecting to host: 127.0.0.1, port: 3306
16:15:51,610 INFO TransportImpl - connected to host: 127.0.0.1, port: 3306, context: AbstractTransport.Context[threadId=12,scramble=fARYH=7=l(~hkb)BPNt!,protocolVersion=10,serverHost=127.0.0.1,serverPort=3306,serverStatus=2,serverCollation=8,serverVersion=5.6.24-0ubuntu2-log,serverCapabilities=63487]
16:15:51,610 INFO AuthenticatorImpl - start to login, user: root, host: 127.0.0.1, port: 3306
16:15:51,622 INFO AuthenticatorImpl - login successfully, user: root, detail: OKPacket[packetMarker=0,affectedRows=0,insertId=0,serverStatus=2,warningCount=0,message=<null>]
16:15:51,793 DEBUG SchemaChange - SQL_PARSE <- "create database m_test"
16:15:51,837 DEBUG SchemaChange - SQL_PARSE -> (parse (statement (create_database create database (name (id m_test)))) <EOF>)
16:15:51,837 INFO MaxwellParser - storing schema @BinlogPosition[mysql-logs.000002:2898] after applying "create database m_test"
16:15:51,861 DEBUG SchemaPosition - syncing binlog position: BinlogPosition[mysql-logs.000002:2898]
16:15:51,862 DEBUG SchemaPosition - Writing binlog position to maxwell.positions: BinlogPosition[mysql-logs.000002:2898]
16:15:51,913 DEBUG SchemaChange - SQL_PARSE <- "create table max_test (i_id int, ver double, name varchar(12))"
16:15:51,926 DEBUG SchemaChange - SQL_PARSE -> (parse (statement (create_table (create_table_preamble create table (table_name (name (id max_test)))) (create_specifications ( (create_specification (column_definition (name (id i_id)) (data_type (signed_type int)))) , (create_specification (column_definition (name (id ver)) (data_type (signed_type double)))) , (create_specification (column_definition (name (id name)) (data_type (string_type varchar (length ( 12 )))))) )))) <EOF>)
16:15:51,929 INFO MaxwellParser - storing schema @BinlogPosition[mysql-logs.000002:3034] after applying "create table max_test (i_id int, ver double, name varchar(12))"
16:15:51,959 DEBUG SchemaPosition - syncing binlog position: BinlogPosition[mysql-logs.000002:3034]
16:15:51,959 DEBUG SchemaPosition - Writing binlog position to maxwell.positions: BinlogPosition[mysql-logs.000002:3034]
I then inserted a row: insert into max_test values (1,0.1,'max');
to see if it would spew it out, but nothing. So I quit Maxwell, and ran it again, thinking that it would show up the next time around, just like the database and table creation statements it showed the last time. But nothing. I did the same insert again, and an update statement, but I'm getting radio silence.
bin/maxwell --user='root' --password='root' --host='127.0.0.1' --producer=stdout
Picked up JAVA_TOOL_OPTIONS: -javaagent:/usr/share/java/jayatanaag.jar
16:18:12,063 INFO Maxwell - Maxwell is booting, starting at BinlogPosition[mysql-logs.000002:3034]
16:18:12,065 DEBUG SchemaStore - looking to restore schema at target position BinlogPosition[mysql-logs.000002:3034]
16:18:12,066 INFO SchemaStore - Restoring schema id 4 (last modified at BinlogPosition[mysql-logs.000002:3034])
16:18:12,103 INFO TransportImpl - connecting to host: 127.0.0.1, port: 3306
16:18:12,137 INFO TransportImpl - connected to host: 127.0.0.1, port: 3306, context: AbstractTransport.Context[threadId=16,scramble=>TlEPC@Uk=,a*Hke6}76,protocolVersion=10,serverHost=127.0.0.1,serverPort=3306,serverStatus=2,serverCollation=8,serverVersion=5.6.24-0ubuntu2-log,serverCapabilities=63487]
16:18:12,137 INFO AuthenticatorImpl - start to login, user: root, host: 127.0.0.1, port: 3306
16:18:12,158 INFO AuthenticatorImpl - login successfully, user: root, detail: OKPacket[packetMarker=0,affectedRows=0,insertId=0,serverStatus=2,warningCount=0,message=<null>]
Is this normal? I guess I'm having difficulty in understanding exactly what I'm supposed to be seeing. Also, those create statements don't look like JSON. Is that normal too?
from maxwell.
let me test under 5.6 and see if there's something there that's different. From the logs it certainly looks like maxwell is working (it's processing the schema from the replication stream just fine), but you should also be seeing JSON from your inserts.
from maxwell.
Sounds good. Thanks! 😄
Are the create statements supposed to be the way they are appearing, or should that be JSON (but is not showing up as that) too?
from maxwell.
nah, they're supposed to be that way. maxwell needs to consume the CREATE/ALTER statements for its own use, but it doesn't output them onto the kafka (or stdout) bus. So those are just log lines
from maxwell.
I see. Okay. Thanks for the awesome support so far!
from maxwell.
yeah, I can reproduce this easily: 5.6 doesn't seem to work at all! stay tuned..
from maxwell.
Yay?
from maxwell.
I have a few issues to report with the 5.6 support. What is the best way to do this?
from maxwell.
go ahead & create issues.
from maxwell.
Latest version works. Will close this now.
from maxwell.
Related Issues (20)
- when db failover, and set Read_only=true, what exception maxwell will throw? HOT 1
- parse error on algorithm=NOCOPY
- Reduce memory overhead on startup (OOM) HOT 9
- Allow persistent variables in Javascript filters HOT 4
- User sessions
- Failed to deserialize data of EventHeaderV4 HOT 4
- NOT FOUND mysqlBaseListener class HOT 1
- Maxwell Sending final heartbeat: 1706848518690
- row_query_max_length - variable type error
- Request to fix CVEs found in Maxwell v1.41.0 HOT 1
- Need help using javascript filter
- maxwell uses utc to convert timestamp, but not converting datetime. How can we convert datetime by utc as well?
- maxwell 1.13.2 夜里突然无响应,binlog不能产生kafka消息了 HOT 2
- I use maxwell to synchronize MySQL data to Redis, but there is a delay of nearly 30 minutes. How can I improve Maxwell's performance HOT 3
- crash in alters
- parse error on WAIT=30
- Couldn't find chained schema HOT 2
- mysql-binlog-connector-java: Deadlock when connection try to disconnect
- Error: No matches for kafka version: 2.7.0
- Maxwell v1.41.2 which based on openjdk:11-jkd-slim is vulnerable and outdated
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from maxwell.