codership / mysql-wsrep Goto Github PK

wsrep API patch for MySQL server

License: Other

Shell 0.98% CMake 0.90% C 31.14% C++ 57.98% Perl 2.89% Objective-C 0.49% Pascal 0.14% Python 0.03% DTrace 0.03% Makefile 0.09% Java 4.45% HTML 0.05% GAP 0.08% Awk 0.01% Scilab 0.01% JavaScript 0.01% Batchfile 0.01% Yacc 0.67% Lex 0.02% Assembly 0.03%

mysql-wsrep's People

Contributors

Stargazers

Watchers

mysql-wsrep's Issues

Assertion `thd->wsrep_exec_mode == LOCAL_STATE' failed in wsrep_to_isolation_begin() with MyISAM Stored Procedure and wsrep-replicate-myisam = ON

mysqld: /home/philips/git/mysql-wsrep/sql/wsrep_mysqld.cc:1424: int wsrep_to_isolation_begin(THD*, char*, char*, const TABLE_LIST*): Assertion `thd->wsrep_exec_mode == LOCAL_STATE' failed.

#7  0x0000003e9042da82 in __assert_fail () from /lib64/libc.so.6
#8  0x000000000063eee7 in wsrep_to_isolation_begin (thd=0x298f880, db_=0x0, table_=0x0, table_list=0x7f7c50004d38) at /home/philips/git/mysql-wsrep/sql/wsrep_mysqld.cc:1424
#9  0x0000000000789067 in open_tables (thd=0x298f880, start=0x7f7c94065e50, counter=0x2991c88, flags=0, prelocking_strategy=0x7f7c94065e90) at /home/philips/git/mysql-wsrep/sql/sql_base.cc:5225
#10 0x0000000000789d75 in open_normal_and_derived_tables (thd=0x298f880, tables=0x7f7c50004d38, flags=0) at /home/philips/git/mysql-wsrep/sql/sql_base.cc:5775
#11 0x00000000007d0266 in mysql_insert (thd=0x298f880, table_list=0x7f7c50004d38, fields=..., values_list=..., update_fields=..., update_values=..., duplic=DUP_ERROR, ignore=false) at /home/philips/git/mysql-wsrep/sql/sql_insert.cc:737
#12 0x00000000007f3632 in mysql_execute_command (thd=0x298f880) at /home/philips/git/mysql-wsrep/sql/sql_parse.cc:3774
#13 0x00000000007fca8f in mysql_parse (thd=0x298f880, rawbuf=0x7f7c50004c50 "INSERT INTO t1 VALUES (1)", length=25, parser_state=0x7f7c940676e0) at /home/philips/git/mysql-wsrep/sql/sql_parse.cc:6815
#14 0x00000000007fc050 in wsrep_mysql_parse (thd=0x298f880, rawbuf=0x7f7c50004c50 "INSERT INTO t1 VALUES (1)", length=25, parser_state=0x7f7c940676e0) at /home/philips/git/mysql-wsrep/sql/sql_parse.cc:6595
#15 0x00000000007edc9d in dispatch_command (command=COM_QUERY, thd=0x298f880, packet=0x2a03921 "INSERT INTO t1 VALUES (1)", packet_length=25) at /home/philips/git/mysql-wsrep/sql/sql_parse.cc:1525
#16 0x00000000007ec75d in do_command (thd=0x298f880) at /home/philips/git/mysql-wsrep/sql/sql_parse.cc:1114
#17 0x00000000007b5978 in do_handle_one_connection (thd_arg=0x298f880) at /home/philips/git/mysql-wsrep/sql/sql_connect.cc:1007
#18 0x00000000007b548d in handle_one_connection (arg=0x298f880) at /home/philips/git/mysql-wsrep/sql/sql_connect.cc:916
#19 0x0000000000af746a in pfs_spawn_thread (arg=0x2897cf0) at /home/philips/git/mysql-wsrep/storage/perfschema/pfs.cc:1860
#20 0x0000003e90c0753a in start_thread () from /lib64/libpthread.so.0
#21 0x0000003e905045fd in clone () from /lib64/libc.so.6

Test case, put in mysql-test/suite/galera/t:

--source include/galera_cluster.inc
--source include/have_innodb.inc

SET GLOBAL wsrep_replicate_myisam = TRUE;

CREATE TABLE t1 (f1 INTEGER) ENGINE=MyISAM;
CREATE TABLE t2 (f1 INTEGER) ENGINE=MyISAM;
CREATE TRIGGER tr1 AFTER INSERT ON t1 FOR EACH ROW INSERT INTO t2 VALUES (NEW.f1);

INSERT INTO t1 VALUES (1);

InnoDB assertion around sync_thread_levels with --wsrep_log_conflicts=ON

When attempting to run a simple test with --wsrep_log_conflicts=ON, the slaveasserted as follows:

2014-12-15 11:41:28 6517 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
InnoDB: sync levels should be > 298 but a level is 297
Mutex created at trx0trx.cc 108
InnoDB: Locked mutex: addr 0x7fd4580117f0 thread 140550279898880 file /home/philips/git/mysql-wsrep/storage/innobase/lock/lock0lock.cc line 2500
InnoDB: sync_thread_levels_g(array, 298) does not hold!
2014-12-15 11:41:29 7fd46981a700  InnoDB: Assertion failure in thread 140550279898880 in file sync0sync.cc line 1172

#6  0x0000000000cee91c in sync_thread_add_level (latch=0x29c0c48, level=298, relock=0) at /home/philips/git/mysql-wsrep/storage/innobase/sync/sync0sync.cc:1172
#7  0x0000000000cedb83 in mutex_set_debug_info (mutex=0x29c0c48, file_name=0x10429c0 "/home/philips/git/mysql-wsrep/storage/innobase/lock/lock0lock.cc", line=1650) at /home/philips/git/mysql-wsrep/storage/innobase/sync/sync0sync.cc:615
#8  0x0000000000be395a in mutex_enter_func (mutex=0x29c0c48, file_name=0x10429c0 "/home/philips/git/mysql-wsrep/storage/innobase/lock/lock0lock.cc", line=1650) at /home/philips/git/mysql-wsrep/storage/innobase/include/sync0sync.ic:226
#9  0x0000000000be39e5 in pfs_mutex_enter_func (mutex=0x29c0c48, file_name=0x10429c0 "/home/philips/git/mysql-wsrep/storage/innobase/lock/lock0lock.cc", line=1650) at /home/philips/git/mysql-wsrep/storage/innobase/include/sync0sync.ic:257
#10 0x0000000000beb1c3 in wsrep_kill_victim (trx=0x7fd4580117e8, lock=0x7fd438079780) at /home/philips/git/mysql-wsrep/storage/innobase/lock/lock0lock.cc:1650
#11 0x0000000000beb41f in lock_rec_other_has_conflicting (mode=1027, block=0x7fd46859b040, heap_no=2, trx=0x7fd4580117e8) at /home/philips/git/mysql-wsrep/storage/innobase/lock/lock0lock.cc:1714
#12 0x0000000000bed243 in lock_rec_lock_slow (impl=0, mode=1027, block=0x7fd46859b040, heap_no=2, index=0x7fd45803d258, thr=0x7fd4580104b0) at /home/philips/git/mysql-wsrep/storage/innobase/lock/lock0lock.cc:2508
#13 0x0000000000bed574 in lock_rec_lock (impl=0, mode=1027, block=0x7fd46859b040, heap_no=2, index=0x7fd45803d258, thr=0x7fd4580104b0) at /home/philips/git/mysql-wsrep/storage/innobase/lock/lock0lock.cc:2592
#14 0x0000000000bf7656 in lock_clust_rec_read_check_and_lock (flags=0, block=0x7fd46859b040, rec=0x7fd46886407e "Ñ‚ÐµÐºÑ\302\201Ñ‚", index=0x7fd45803d258, offsets=0x7fd4698167a0, mode=LOCK_X, gap_mode=1024, thr=0x7fd4580104b0) at /home/philips/git/mysql-wsrep/storage/innobase/lock/lock0lock.cc:6774
#15 0x0000000000cba9a5 in sel_set_rec_lock (block=0x7fd46859b040, rec=0x7fd46886407e "Ñ‚ÐµÐºÑ\302\201Ñ‚", index=0x7fd45803d258, offsets=0x7fd4698167a0, mode=3, type=1024, thr=0x7fd4580104b0) at /home/philips/git/mysql-wsrep/storage/innobase/row/row0sel.cc:1011
#16 0x0000000000cc10df in row_search_for_mysql (buf=0x7fd458042070 "\026", mode=2, prebuilt=0x7fd45800f878, match_mode=1, direction=0) at /home/philips/git/mysql-wsrep/storage/innobase/row/row0sel.cc:4499
#17 0x0000000000b9bf75 in ha_innobase::index_read (this=0x7fd45803a0f0, buf=0x7fd458042070 "\026", key_ptr=0x7fd458043210 "\026", key_len=767, find_flag=HA_READ_KEY_EXACT) at /home/philips/git/mysql-wsrep/storage/innobase/handler/ha_innodb.cc:8218
#18 0x0000000000b9cf6c in ha_innobase::rnd_pos (this=0x7fd45803a0f0, buf=0x7fd458042070 "\026", pos=0x7fd458043210 "\026") at /home/philips/git/mysql-wsrep/storage/innobase/handler/ha_innodb.cc:8720
#19 0x0000000000658c04 in handler::ha_rnd_pos (this=0x7fd45803a0f0, buf=0x7fd458042070 "\026", pos=0x7fd458043210 "\026") at /home/philips/git/mysql-wsrep/sql/handler.cc:2786
#20 0x0000000000663f5b in handler::rnd_pos_by_record (this=0x7fd45803a0f0, record=0x7fd458042070 "\026") at /home/philips/git/mysql-wsrep/sql/handler.h:2325
#21 0x0000000000a2423b in Rows_log_event::do_index_scan_and_update (this=0x7fd458045380, rli=0x7fd45800bc20) at /home/philips/git/mysql-wsrep/sql/log_event.cc:10540
#22 0x0000000000a2640d in Rows_log_event::do_apply_event (this=0x7fd458045380, rli=0x7fd45800bc20) at /home/philips/git/mysql-wsrep/sql/log_event.cc:11362
#23 0x0000000000a11ad5 in Log_event::apply_event (this=0x7fd458045380, rli=0x7fd45800bc20) at /home/philips/git/mysql-wsrep/sql/log_event.cc:3112
#24 0x0000000000649d45 in wsrep_apply_events (thd=0x7fd4580009a0, events_buf=0x7fd46c9848bb, buf_len=0) at /home/philips/git/mysql-wsrep/sql/wsrep_applier.cc:151
#25 0x000000000064a32d in wsrep_apply_cb (ctx=0x7fd4580009a0, buf=0x7fd46c9848bb, buf_len=199, flags=1, meta=0x7fd469818b80) at /home/philips/git/mysql-wsrep/sql/wsrep_applier.cc:238
#26 0x00007fd474b5ecd9 in galera::TrxHandle::apply (this=this@entry=0x7fd458004680, recv_ctx=recv_ctx@entry=0x7fd4580009a0, apply_cb=apply_cb@entry=0x64a141 <wsrep_apply_cb(void*, void const*, unsigned long, unsigned int, wsrep_trx_meta const*)>, meta=...) at galera/src/trx_handle.cpp:315
#27 0x00007fd474b9f43f in apply_trx_ws (recv_ctx=recv_ctx@entry=0x7fd4580009a0, apply_cb=0x64a141 <wsrep_apply_cb(void*, void const*, unsigned long, unsigned int, wsrep_trx_meta const*)>, commit_cb=0x64a6fe <wsrep_commit_cb(void*, unsigned int, wsrep_trx_meta const*, bool*, bool)>, trx=..., meta=...) at galera/src/replicator_smm.cpp:37
#28 0x00007fd474ba22a3 in galera::ReplicatorSMM::apply_trx (this=this@entry=0x281b9c0, recv_ctx=recv_ctx@entry=0x7fd4580009a0, trx=trx@entry=0x7fd458004680) at galera/src/replicator_smm.cpp:433
#29 0x00007fd474ba583e in galera::ReplicatorSMM::process_trx (this=0x281b9c0, recv_ctx=0x7fd4580009a0, trx=0x7fd458004680) at galera/src/replicator_smm.cpp:1223
#30 0x00007fd474b80b19 in galera::GcsActionSource::dispatch (this=this@entry=0x281c048, recv_ctx=recv_ctx@entry=0x7fd4580009a0, act=..., exit_loop=@0x7fd46981949c: false) at galera/src/gcs_action_source.cpp:116
#31 0x00007fd474b81b82 in galera::GcsActionSource::process (this=0x281c048, recv_ctx=0x7fd4580009a0, exit_loop=@0x7fd46981949c: false) at galera/src/gcs_action_source.cpp:181
#32 0x00007fd474ba5ed3 in galera::ReplicatorSMM::async_recv (this=0x281b9c0, recv_ctx=0x7fd4580009a0) at galera/src/replicator_smm.cpp:355
#33 0x000000000064bc24 in wsrep_replication_process (thd=0x7fd4580009a0) at /home/philips/git/mysql-wsrep/sql/wsrep_thd.cc:309
#34 0x000000000062a99a in start_wsrep_THD (arg=0x64bb6c <wsrep_replication_process(THD*)>) at /home/philips/git/mysql-wsrep/sql/mysqld.cc:5376
#35 0x0000003e90c0753a in start_thread () from /lib64/libpthread.so.0
#36 0x0000003e905045fd in clone () from /lib64/libc.so.6

Galera server does not restart properly if killed

The galera_kill_nochanges test fails in Jenkins once every 20 runs or so with the following message:

galera.galera_kill_nochanges             [ fail ]
        Test ended at 2014-12-05 06:17:14

CURRENT_TEST: galera.galera_kill_nochanges
mysqltest: In included file "./include/wait_until_connected_again.inc": 
included from ./include/wait_until_connected_again.inc at line 22:
included from ./include/start_mysqld.inc at line 11:
At line 22: Server failed to restart

The result from queries just before the failure was:
CREATE TABLE t1 (f1 INTEGER) ENGINE=InnoDB;
INSERT INTO t1 VALUES (1);

Turns out that the cluster is stuck in some sort of a loop, with the restarted node showing the following log entries repeatedly:

2014-12-05 07:16:29 32150 [Note] WSREP: view(view_id(NON_PRIM,ee819a89,10) memb {
        ee819a89,0
} joined {
} left {
} partitioned {
})
2014-12-05 07:16:29 32150 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
2014-12-05 07:16:29 32150 [Note] WSREP: Flow-control interval: [16, 16]
2014-12-05 07:16:29 32150 [Note] WSREP: Received NON-PRIMARY.
2014-12-05 07:16:29 32150 [Note] WSREP: New cluster view: global state: :-1, view# -1: non-Primary, number of nodes: 1, my index: 0, protocol version -1
2014-12-05 07:16:29 32150 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2014-12-05 07:16:29 32150 [Warning] WSREP: no nodes coming from prim view, prim not possible
2014-12-05 07:16:29 32150 [Note] WSREP: view(view_id(NON_PRIM,ee819a89,11) memb {
        ee819a89,0
} joined {
} left {
} partitioned {
})
...

The line is repeated nineteen thousand times in the course of 1 minute, before MTR gives up.

InnoDB: Failing assertion: !srv_read_only_mode with server restart

Several InnoDB tests which perform server restart assert as follows:

2014-11-13 13:58:27 7f02b638b700  InnoDB: Assertion failure in thread 139649623832320 in file mtr0mtr.cc line 323
InnoDB: Failing assertion: !srv_read_only_mode

#3  <signal handler called>
#4  0x0000003e90434e27 in raise () from /lib64/libc.so.6
#5  0x0000003e90436a8a in abort () from /lib64/libc.so.6
#6  0x0000000000c14c2b in mtr_commit (mtr=0x7f02b6389090) at /home/philips/git/mysql-wsrep/storage/innobase/mtr/mtr0mtr.cc:323
#7  0x0000000000bac767 in innobase_wsrep_set_checkpoint (hton=0x1d3f400, xid=0x7f02b6389b40) at /home/philips/git/mysql-wsrep/storage/innobase/handler/ha_innodb.cc:16968
#8  0x000000000063a63c in set_SE_checkpoint (unused=0x0, plugin=0x7f02b6389ab8, arg=0x7f02b6389b40) at /home/philips/git/mysql-wsrep/sql/wsrep_mysqld.cc:165
#9  0x000000000080765c in plugin_foreach_with_mask (thd=0x0, func=0x63a4f6 <set_SE_checkpoint(THD*, plugin_ref, void*)>, type=1, state_mask=4294967287, arg=0x7f02b6389b40) at /home/philips/git/mysql-wsrep/sql/sql_plugin.cc:2162
#10 0x000000000063a671 in wsrep_set_SE_checkpoint (xid=0x7f02b6389b40) at /home/philips/git/mysql-wsrep/sql/wsrep_mysqld.cc:172
#11 0x000000000063aeac in wsrep_view_handler_cb (app_ctx=0x3e90225148, recv_ctx=0x7f02a40009a0, view=0x7f02a400efb0, state=0x0, state_len=0, sst_req=0x7f02b6389fe0, sst_req_len=0x7f02b6389fe8) at /home/philips/git/mysql-wsrep/sql/wsrep_mysqld.cc:346
#12 0x00007f02c1713750 in galera::ReplicatorSMM::process_conf_change (this=0x1db55a0, recv_ctx=0xb252, view_info=..., repl_proto=-1, next_state=404, seqno_l=0) at galera/src/replicator_smm.cpp:1385
#13 0x00007f02c16ecccc in galera::GcsActionSource::dispatch (this=this@entry=0x1db5c28, recv_ctx=recv_ctx@entry=0x7f02a40009a0, act=..., exit_loop=@0x7f02b638a49c: false) at galera/src/gcs_action_source.cpp:138
#14 0x00007f02c16ee772 in galera::GcsActionSource::process (this=0x1db5c28, recv_ctx=0x7f02a40009a0, exit_loop=@0x7f02b638a49c: false) at galera/src/gcs_action_source.cpp:180
#15 0x00007f02c1712de3 in galera::ReplicatorSMM::async_recv (this=0x1db55a0, recv_ctx=0x7f02a40009a0) at galera/src/replicator_smm.cpp:354
#16 0x000000000064b89e in wsrep_replication_process (thd=0x7f02a40009a0) at /home/philips/git/mysql-wsrep/sql/wsrep_thd.cc:309
#17 0x000000000062a8da in start_wsrep_THD (arg=0x64b7e6 <wsrep_replication_process(THD*)>) at /home/philips/git/mysql-wsrep/sql/mysqld.cc:5376
#18 0x0000003e90c0753a in start_thread () from /lib64/libpthread.so.0
#19 0x0000003e905045fd in clone () from /lib64/libc.so.6

To reproduce:

./mysql-test-run.pl --suites=innodb --mysqld=--performance-schema=OFF --mysqld=--wsrep-cluster-address="gcomm://" --mysqld=--wsrep-provider=/home/philips/git/galera/libgalera_smm.so --force --retry=0  innodb_bug-13628249 innodb-wl6445-1

Add socat as a dependency for mysql-wsrep package

xtrabackup SST requires socat

InnoDB: Failing assertion: xid_seqno > trx_sys_cur_xid_seqno in trx_sys_update_wsrep_checkpoint with mysqldump IST

If IST is forced by temporarily setting wsrep_provider to "none" and then setting it back to its original value, the node asserts as follows:

2014-12-17 16:30:13 58763 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 5)
2014-12-17 16:30:13 58763 [Note] WSREP: Requesting state transfer: success, donor: 0
2014-12-17 16:30:13 7f7e383fc700  InnoDB: Assertion failure in thread 140180086310656 in file trx0sys.cc line 353
InnoDB: Failing assertion: xid_seqno > trx_sys_cur_xid_seqno

#5  0x0000003e90436a8a in abort () from /lib64/libc.so.6
#6  0x0000000000d0d1d9 in trx_sys_update_wsrep_checkpoint (xid=0x7f7e383f9c30, sys_header=0x7f7e38dd4026 "", mtr=0x7f7e383f91a0) at /home/philips/git/mysql-wsrep/storage/innobase/trx/trx0sys.cc:353
#7  0x0000000000bacd98 in innobase_wsrep_set_checkpoint (hton=0x2535ca0, xid=0x7f7e383f9c30) at /home/philips/git/mysql-wsrep/storage/innobase/handler/ha_innodb.cc:16988
#8  0x000000000063a72c in set_SE_checkpoint (unused=0x0, plugin=0x7f7e383f9bc8, arg=0x7f7e383f9c30) at /home/philips/git/mysql-wsrep/sql/wsrep_mysqld.cc:165
#9  0x0000000000807a24 in plugin_foreach_with_mask (thd=0x0, func=0x63a5e6 <set_SE_checkpoint(THD*, plugin_ref, void*)>, type=1, state_mask=4294967287, arg=0x7f7e383f9c30) at /home/philips/git/mysql-wsrep/sql/sql_plugin.cc:2162
#10 0x000000000063a761 in wsrep_set_SE_checkpoint (xid=0x7f7e383f9c30) at /home/philips/git/mysql-wsrep/sql/wsrep_mysqld.cc:172
#11 0x0000000000646a8f in wsrep_set_local_position (value=0x7f7de4005080 "b8c6df20-85f0-11e4-aabd-d2bdcbd94cf4:4") at /home/philips/git/mysql-wsrep/sql/wsrep_var.cc:158
#12 0x0000000000646b15 in wsrep_start_position_update (self=0x18377c0 <Sys_wsrep_start_position>, thd=0x27267d0, type=OPT_GLOBAL) at /home/philips/git/mysql-wsrep/sql/wsrep_var.cc:166
#13 0x000000000074326a in sys_var::update (this=0x18377c0 <Sys_wsrep_start_position>, thd=0x27267d0, var=0x7f7de4005458) at /home/philips/git/mysql-wsrep/sql/set_var.cc:194
#14 0x0000000000744312 in set_var::update (this=0x7f7de4005458, thd=0x27267d0) at /home/philips/git/mysql-wsrep/sql/set_var.cc:670
#15 0x0000000000743eea in sql_set_variables (thd=0x27267d0, var_list=0x27297c8) at /home/philips/git/mysql-wsrep/sql/set_var.cc:573
#16 0x00000000007f4aca in mysql_execute_command (thd=0x27267d0) at /home/philips/git/mysql-wsrep/sql/sql_parse.cc:4075
#17 0x00000000007fcd81 in mysql_parse (thd=0x27267d0, rawbuf=0x7f7de40051f0 "SET GLOBAL wsrep_start_position='b8c6df20-85f0-11e4-aabd-d2bdcbd94cf4:4'", length=72, parser_state=0x7f7e383fb6e0) at /home/philips/git/mysql-wsrep/sql/sql_parse.cc:6815
#18 0x00000000007fc342 in wsrep_mysql_parse (thd=0x27267d0, rawbuf=0x7f7de40051f0 "SET GLOBAL wsrep_start_position='b8c6df20-85f0-11e4-aabd-d2bdcbd94cf4:4'", length=72, parser_state=0x7f7e383fb6e0) at /home/philips/git/mysql-wsrep/sql/sql_parse.cc:6595
#19 0x00000000007edf8f in dispatch_command (command=COM_QUERY, thd=0x27267d0, packet=0x2690e11 "SET GLOBAL wsrep_start_position='b8c6df20-85f0-11e4-aabd-d2bdcbd94cf4:4'", packet_length=72) at /home/philips/git/mysql-wsrep/sql/sql_parse.cc:1525
#20 0x00000000007eca4f in do_command (thd=0x27267d0) at /home/philips/git/mysql-wsrep/sql/sql_parse.cc:1114
#21 0x00000000007b5c6a in do_handle_one_connection (thd_arg=0x27267d0) at /home/philips/git/mysql-wsrep/sql/sql_connect.cc:1007
#22 0x00000000007b577f in handle_one_connection (arg=0x27267d0) at /home/philips/git/mysql-wsrep/sql/sql_connect.cc:916
#23 0x0000000000af77b6 in pfs_spawn_thread (arg=0x2685c30) at /home/philips/git/mysql-wsrep/storage/perfschema/pfs.cc:1860
#24 0x0000003e90c0753a in start_thread () from /lib64/libpthread.so.0
#25 0x0000003e905045fd in clone () from /lib64/libc.so.6

[FR] Detach I_S queries from wsrep_ready

Allowing queries to I_S tables even when wsrep_ready is OFF (like SHOW & SET commands)
will open up a great deal of possibility for monitoring Galera cluster.

performance schema assert due to autocommit retrying

Some test scenarios, using performance schema and autocommit retrying (wsrep_retry_autocommit > 0), lead to assert:

mysqld: /home/seppo/work/wsrep/wsrep-git-features/storage/perfschema/pfs.cc:4531: PSI_statement_locker* refine_statement_v1(PSI_statement_locker*, PSI_statement_key): Assertion `klass->is_mutable()' failed.
22:08:37 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary

This seems to be due to missing performance schema related initialization when query retrying happens.

Stored Procedures with MyISAM are not replicated correctly under wsrep_replicate_myisam = TRUE

If a stored procedure includes MyISAM tables, the updates are not replicated on the slave . Instead, the slave reports the following:

2014-11-13 16:19:36 13936 [ERROR] Slave SQL: Error 'Unknown column 'i' in 'field list'' on query. Default database: 'test'. Query: 'INSERT INTO t1 VALUES (i)', Error_code: 1054

Can it be a missing or malformed TABLE_MAP_EVENT?

Test case:

--source include/galera_cluster.inc
--source include/have_innodb.inc

--let $wsrep_replicate_myisam_orig = `SELECT @@wsrep_replicate_myisam`
SET GLOBAL wsrep_replicate_myisam = TRUE;

--connection node_1
CREATE TABLE t1 (f1 INTEGER PRIMARY KEY) ENGINE=MyISAM;
CREATE TABLE t2 (f1 INTEGER PRIMARY KEY) ENGINE=InnoDB;
delimiter |;
CREATE PROCEDURE p1(IN i INT)
BEGIN
        INSERT INTO t1 VALUES (i);
        INSERT INTO t2 VALUES (i);
END|
delimiter ;|

# --connection node_2
# Force causal read, so that CALL p1(1) succeeds
# SELECT 1 FROM DUAL;

# --connection node_1

CALL p1(1);
SELECT COUNT(*) = 1 FROM t1;
SELECT COUNT(*) = 1 FROM t2;

--connection node_2
SELECT COUNT(*) = 1 FROM t1;
SELECT COUNT(*) = 1 FROM t2;

DROP PROCEDURE p1;
DROP TABLE t1;
DROP TABLE t2;

--connection node_1
--eval SET GLOBAL wsrep_replicate_myisam = $wsrep_replicate_myisam_orig

main.file_contents MTR test fails in mysql-wsrep

The main.file_contents MTR test seems to be expecting a tree cloned with bzr and we now use git.

I have added this test to the t/disabled.def list.

Transaction receives deadlock error twice in row

A transaction will receive a deadlock error not only on COMMIT, but also on a follow-up ROLLBACK statement.

To reproduce, run the following test:

--source include/galera_cluster.inc
--source include/have_innodb.inc

#
# Deadlock
#

--connection node_1
CREATE TABLE t1 (f1 INTEGER PRIMARY KEY) ENGINE=InnoDB;

SET AUTOCOMMIT = OFF;
START TRANSACTION;
INSERT INTO t1 VALUES (1), (2);

--connection node_2
SET AUTOCOMMIT = OFF;
START TRANSACTION;
INSERT INTO t1 VALUES (2), (1);

--connection node_1
COMMIT;

--connection node_2
--error ER_LOCK_DEADLOCK
COMMIT;

--error 0
ROLLBACK;

DROP TABLE t1;

In a loop as follows:

 perl mysql-test-run.pl --suite galera test_name_goes_here --repeat=1000 --no-check-testcases --no-warnings

After a few dozen runs, it will fail as follows:

mysqltest: At line 28: query 'ROLLBACK' failed with wrong errno 1213: 'Deadlock found when trying to get lock; try restarting transaction', instead of 0...

In other words, the ROLLBACK got an error when it should have succeeded.

obs compatible spec file

Use support-file/mysql.spec.sh as a base.

CREATE DATABASE position not synced into innodb header on slaves

To reproduce, nodes 1, 2 and 3.

Node1:

mysql> create database db;
Query OK, 1 row affected (0.02 sec)

Status for node1 
WSREP_LAST_COMMITTED           30602
Status for node2 
WSREP_LAST_COMMITTED           30602
Status for node3 
WSREP_LAST_COMMITTED           30602

However, when restarting the cluster nodes 2 and 3 recover to earlier position:

2014-12-15 23:30:37 INFO Recovering node1 position
2014-12-15 23:30:39 INFO Reading recovery result
2014-12-15 23:30:40 INFO Start position 0768e6ee-849f-11e4-86ef-b716923bcbd6:30602
2014-12-15 23:30:40 INFO Starting node node1
2014-12-15 23:30:40 INFO Recovering node2 position
2014-12-15 23:30:42 INFO Reading recovery result
2014-12-15 23:30:43 INFO Start position 0768e6ee-849f-11e4-86ef-b716923bcbd6:30601
2014-12-15 23:30:43 INFO Starting node node2
2014-12-15 23:30:45 INFO Recovering node3 position
2014-12-15 23:30:47 INFO Reading recovery result
2014-12-15 23:30:47 INFO Start position 0768e6ee-849f-11e4-86ef-b716923bcbd6:30601
2014-12-15 23:30:47 INFO Starting node node3

Similar behavior was observed with DROP DATABASE/CREATE TABLE...

CREATE + DROP causes Assertion `key_count > 0' failed in galera::Certification::do_test_v3()

A simple CREATE + DROP causes the following assertion:

mysqld: galera/src/certification.cpp:590: galera::Certification::TestResult galera::Certification::do_test_v3(galera::TrxHandle*, bool): Assertion `key_count > 0' failed.

#7  0x0000003e9042da82 in __assert_fail () from /lib64/libc.so.6
#8  0x00007f3af278daea in galera::Certification::do_test_v3 (this=this@entry=0x2a00f58, trx=trx@entry=0x7f3ab401de30, store_keys=store_keys@entry=true) at galera/src/certification.cpp:590
#9  0x00007f3af278fb03 in galera::Certification::do_test (this=0x2a00f58, trx=trx@entry=0x7f3ab401de30, store_keys=<optimized out>) at galera/src/certification.cpp:747
#10 0x00007f3af278fe29 in galera::Certification::test (this=<optimized out>, trx=0x7f3ab401de30, bval=<optimized out>) at galera/src/certification.cpp:916
#11 0x00007f3af278ff45 in galera::Certification::append_trx (this=this@entry=0x2a00f58, trx=trx@entry=0x7f3ab401de30) at galera/src/certification.cpp:1024
#12 0x00007f3af27ce199 in galera::ReplicatorSMM::cert (this=0x2a005a0, trx=0x7f3ab401de30) at galera/src/replicator_smm.cpp:1689
#13 0x00007f3af27c4633 in galera::ReplicatorSMM::cert_and_catch (this=this@entry=0x2a005a0, trx=trx@entry=0x7f3ab401de30) at galera/src/replicator_smm.cpp:1774
#14 0x00007f3af27c51f7 in galera::ReplicatorSMM::to_isolation_begin (this=0x2a005a0, trx=0x7f3ab401de30, meta=<optimized out>) at galera/src/replicator_smm.cpp:1010
#15 0x00007f3af27dbccb in galera_to_execute_start (gh=<optimized out>, conn_id=7, keys=<optimized out>, keys_num=<optimized out>, data=0x7f3ae8095970, count=1, meta=0x2c579a0) at galera/src/wsrep_provider.cpp:729
#16 0x000000000063e142 in wsrep_TOI_begin (thd=0x2c558f0, db_=0x0, table_=0x0, table_list=0x7f3ab404d6a8) at /home/philips/git/mysql-wsrep/sql/wsrep_mysqld.cc:1269
#17 0x000000000063f067 in wsrep_to_isolation_begin (thd=0x2c558f0, db_=0x0, table_=0x0, table_list=0x7f3ab404d6a8) at /home/philips/git/mysql-wsrep/sql/wsrep_mysqld.cc:1443
#18 0x00000000007f42a4 in mysql_execute_command (thd=0x2c558f0) at /home/philips/git/mysql-wsrep/sql/sql_parse.cc:3997
#19 0x000000000099953e in sp_instr_stmt::exec_core (this=0x7f3ab404dc20, thd=0x2c558f0, nextp=0x7f3ae80970b8) at /home/philips/git/mysql-wsrep/sql/sp_instr.cc:901
#20 0x00000000009983a0 in sp_lex_instr::reset_lex_and_exec_core (this=0x7f3ab404dc20, thd=0x2c558f0, nextp=0x7f3ae80970b8, open_tables=false) at /home/philips/git/mysql-wsrep/sql/sp_instr.cc:390
#21 0x0000000000998cae in sp_lex_instr::validate_lex_and_execute_core (this=0x7f3ab404dc20, thd=0x2c558f0, nextp=0x7f3ae80970b8, open_tables=false) at /home/philips/git/mysql-wsrep/sql/sp_instr.cc:638
#22 0x00000000009991cd in sp_instr_stmt::execute (this=0x7f3ab404dc20, thd=0x2c558f0, nextp=0x7f3ae80970b8) at /home/philips/git/mysql-wsrep/sql/sp_instr.cc:808
#23 0x000000000074df46 in sp_head::execute (this=0x7f3ab40916f0, thd=0x2c558f0, merge_da_on_success=true) at /home/philips/git/mysql-wsrep/sql/sp_head.cc:645
#24 0x000000000074fc6a in sp_head::execute_procedure (this=0x7f3ab40916f0, thd=0x2c558f0, args=0x2c588a0) at /home/philips/git/mysql-wsrep/sql/sp_head.cc:1315
#25 0x00000000007f79d9 in mysql_execute_command (thd=0x2c558f0) at /home/philips/git/mysql-wsrep/sql/sql_parse.cc:4987
#26 0x00000000007fc9b9 in mysql_parse (thd=0x2c558f0, rawbuf=0x7f3ab4004c40 "call mtr.check_warnings(@result)", length=32, parser_state=0x7f3ae8098730) at /home/philips/git/mysql-wsrep/sql/sql_parse.cc:6815
#27 0x00000000007fbf7a in wsrep_mysql_parse (thd=0x2c558f0, rawbuf=0x7f3ab4004c40 "call mtr.check_warnings(@result)", length=32, parser_state=0x7f3ae8098730) at /home/philips/git/mysql-wsrep/sql/sql_parse.cc:6595
#28 0x00000000007edbc7 in dispatch_command (command=COM_QUERY, thd=0x2c558f0, packet=0x2d1e6b1 "call mtr.check_warnings(@result)", packet_length=32) at /home/philips/git/mysql-wsrep/sql/sql_parse.cc:1525
#29 0x00000000007ec687 in do_command (thd=0x2c558f0) at /home/philips/git/mysql-wsrep/sql/sql_parse.cc:1114
#30 0x00000000007b58a2 in do_handle_one_connection (thd_arg=0x2c558f0) at /home/philips/git/mysql-wsrep/sql/sql_connect.cc:1007
#31 0x00000000007b53b7 in handle_one_connection (arg=0x2c558f0) at /home/philips/git/mysql-wsrep/sql/sql_connect.cc:916
#32 0x0000003e90c0753a in start_thread () from /lib64/libpthread.so.0
#33 0x0000003e905045fd in clone () from /lib64/libc.so.6

Test case:

--source include/have_innodb.inc

CREATE TABLE t1 (f1 INTEGER) ENGINE = INNODB;
DROP TABLE t1;

Run like this:

./mysql-test-run.pl --suites=innodb --mysqld=--performance-schema=OFF --mysqld=--wsrep-cluster-address="gcomm://" --mysqld=--wsrep-provider=/home/philips/git/galera/libgalera_smm.so test_name_goes_here

wsrep_on is still available as GLOBAL variable, leading to possible data diversion between the nodes

Create a new table
(root@localhost:am) [test]>show create table tbtest1\G
*************************** 1. row ***************************
Table: tbtest1
Create Table: CREATE TABLE tbtest1 (
auAPP1nc bigint(11) NOT NULL AUTO_INCREMENT,
a int(11) NOT NULL,
uuid char(36) COLLATE utf8_bin NOT NULL,
b varchar(100) COLLATE utf8_bin NOT NULL,
c char(200) COLLATE utf8_bin NOT NULL,
counter bigint(20) DEFAULT NULL,
time timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
partitionid int(11) NOT NULL DEFAULT '0',
date date NOT NULL,
strrecordtype char(3) COLLATE utf8_bin DEFAULT NULL,
PRIMARY KEY (auAPP1nc,partitionid),
KEY IDX_a (a),
KEY IDX_uuid (uuid)
) ENGINE=InnoDB AUTO_INCREMENT=482 DEFAULT CHARSET=utf8 COLLATE=utf8_bin
1 row in set (0.00 sec)

set
SET GLOBAL wsrep_on=off;

run something like this:

! /bin/bash --

i=1

while :
do
echo "$i "
mysql -ustress -ptool -h192.168.0.35 -P 3307 -e "SET @hh=@@hostname;insert into test.tbtest1 (a,uuid,b,c,strrecordtype,date,partitionid) values($i,UUID(),@hh,'a','APP1',now(),RAND()*100)";
i=$((i + 1))
if [ $i -eq 1000000 ]
then
break
fi
sleep 0.5;
done
against the HAProxy (using 3 nodes all of them in the active pool)

similar but change the APP1 to APP2
and point directly to the node where you have issue the SET GLOBAL wsrep_on=off

! /bin/bash --

i=1

while :
do
echo "$i "
mysql -ustress -ptool -h192.168.0.35 -P 3307 -e "SET @hh=@@hostname;insert into test.tbtest1 (a,uuid,b,c,strrecordtype,date,partitionid) values($i,UUID(),@hh,'a','APP2',now(),RAND()*100)";
i=$((i + 1))
if [ $i -eq 1000000 ]
then
break
fi
sleep 0.5;
done

Enjoy the mess

Solution
Can you please set the wsrep_on as SESSION and not as GLOBAL?
static Sys_var_mybool Sys_wsrep_on (
"wsrep_on", "To enable wsrep replication ",
SESSION_VAR(wsrep_on), <----------------------- This allow global
CMD_LINE(OPT_ARG), DEFAULT(TRUE),
NO_MUTEX_GUARD, NOT_IN_BINLOG, ON_CHECK(0),
ON_UPDATE(wsrep_on_update));

Change to SESSION_ONLY

DDL is logged twice on slave

DDL statements are logged on the slave. While this can be considered a useful feature, the log output looks as follows:

                    2 Query     CREATE TABLE t1 (f1 INTEGER PRIMARY KEY) ENGINE=InnoDB
                    2 Query     CREATE TABLE t1 (f1 INTEGER PRIMARY KEY) ENGINE=InnoDB
                    2 Query     BEGIN
                    2 Query     BEGIN

So the DDL is logged twice, followed by two BEGIN statements.

Furthermore, with logging to CSV table, the output on the slave is as follows:

2015-01-21 15:15:36     [] @  []        1       2       Query   CREATE TABLE t1 (f1 INTEGER PRIMARY KEY) ENGINE=InnoDB
2015-01-21 15:15:36     [] @  []        1       2       Query   CREATE TABLE t1 (f1 INTEGER PRIMARY KEY) ENGINE=InnoDB
2015-01-21 15:15:36     [] @  []        1       2       Query   BEGIN
2015-01-21 15:15:36     [] @  []        1       2       Query   BEGIN

The username and hostname are missing from the output. Contrast with the log on the master:

2015-01-21 15:15:36     root[root] @ localhost [127.0.0.1]      7       1       Query   CREATE TABLE t1 (f1 INTEGER PRIMARY KEY) ENGINE=InnoDB

"this member has applied more events than the primary component.Data loss is possible. Aborting." with mysqldump SST

If a node performs rsync SST , another node in the cluster reports the following:

2014-12-16 16:46:17 32450 [ERROR] WSREP: gcs/src/gcs_group.cpp:void group_post_state_exchange(gcs_group_t*)():319: Reversing history: 0 -> 0, this member has applied 139683099214865 more events than the primary component.Data loss is possible. Aborting.

Fedora 20,21: mysql_install_db does not run due to missing Perl module

Starting the server on Fedora 21 fails because of missing mysql.user table. I think this table is created using the mysql_install_db utility, however it fails due to missing perl module:

[root@localhost galera]# mysql_install_db
FATAL ERROR: please install the following Perl modules before executing /bin/mysql_install_db:
Data::Dumper

The package where this module resides is named perl-Data-Dumper

Don't replicate MyISAM DDLs when wsrep_replicate_myisam is OFF

Percona issue: https://bugs.launchpad.net/codership-mysql/+bug/1402338

merge debian packaging scripts

Merge from 5.6-debian

mysql sysv init script does not stop mysqld on opensuse 13.1, 13.2

How to reproduce:

# /etc/init.d/mysql bootstrap
Bootstrapping the clusterStarting MySQL........                      done
# /etc/init.d/mysql stop
redirecting to systemctl stop mysql.service
# ps ax |grep mysql
 7282 pts/0    S      0:01 /bin/sh /usr/bin/mysqld_safe --datadir=/var/lib/mysql --pid-file=/var/lib/mysql/mysqld.pid --wsrep-new-cluster
 7567 pts/0    Sl     0:01 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --user=mysql --wsrep-new-cluster --log-error=/var/lib/mysql/linux-xhyy.err --pid-file=/var/lib/mysql/mysqld.pid --wsrep_start_position=7ca19594-966d-11e4-a84b-27c2e1051be8:0

Log file does not show any indication about shutdown in progress:

Version: '5.6.21'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MySQL Community Server (GPL), wsrep_25.6
2015-01-07 13:48:02 7567 [Note] WSREP: inited wsrep sidno 1
2015-01-07 13:48:02 7567 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2015-01-07 13:48:02 7567 [Note] WSREP: REPL Protocols: 7 (3, 2)
2015-01-07 13:48:02 7567 [Note] WSREP: Service thread queue flushed.
2015-01-07 13:48:02 7567 [Note] WSREP: Assign initial position for certification: 0, protocol version: 3
2015-01-07 13:48:02 7567 [Note] WSREP: Service thread queue flushed.
2015-01-07 13:48:02 7567 [Note] WSREP: Synchronized with group, ready for connections
2015-01-07 13:48:02 7567 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.

Import 5.5 branch from launchpad

CREATE INDEX InnoDB: Assertion failure in file lock0lock.cc line 6384 - Failing assertion: lock_table_has(trx, index->table, LOCK_IX)

Reported in PXC: https://bugs.launchpad.net/codership-mysql/+bug/1282707

opt_trace.general_no_prot_none and opt_trace.general_no_prot_all fail in mysql-wsrep

For some reason those two optimizer trace test report that the row_length for a temporary table is different in mysql-wsrep:

CURRENT_TEST: opt_trace.general_no_prot_all
--- /var/lib/jenkins/workspace/mtr-galera-mysql-5.6/mysql-wsrep-5.6/mysql-test/suite/opt_trace/r/general_no_prot_all.result 2014-10-27 19:38:08.335348683 +0300
+++ /var/lib/jenkins/workspace/mtr-galera-mysql-5.6/mysql-wsrep-5.6/mysql-test/var/1/log/general_no_prot_all.reject 2014-10-27 21:42:26.351219673 +0300
@@ -11593,11 +11593,11 @@
     {
       "creating_tmp_table": {
         "tmp_table_info": {
-          "row_length": 3268,
+          "row_length": 6340,
           "key_length": 0,
           "unique_constraint": false,
           "location": "memory (heap)",
-          "row_limit_estimate": 320
+          "row_limit_estimate": 165
         } /* tmp_table_info */
       } /* creating_tmp_table */
     },

I am opening this bug so that I can reference its number in the suite/opt_trace/t/disabled.def file.

/! WSREP_DESYNC / does not work

Both MariaDB and Percona documentation have the following:

wsrep_desync

Description: When set to ON (OFF is default), the node will be desynced from the cluster. The same effect can be achieved with a /*! WSREP_DESYNC */ query comment.

However, using /*! WSREP_DESYNC */ produces a syntax error:

mysql> CREATE TABLE t1 (f1 integer) /*! WSREP_DESYNC */;
ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'WSREP_DESYNC */' at line 1

Grepping the mysql-wsrep and mariadb-server codebase reveals no handling for /*! WSREP_DESYNC */

wsrep_scripts reference qpress and pv utilities, not available by default

The wsrep_* SST scripts reference qpress and pv utilities. Unfortunately those do not seem to be installable via yum install . rpmfind reports qpress packages for OpenSuse, but not for Fedora/CentOS.

Should we / can we make the galera RPMs dependent on those utilities?

SLES 12: garbd fails to install

On SLES, garbd fails to install:

linux-xi5a:/home/galera # zypper install galera-3
Loading repository data...
Reading installed packages...
Resolving package dependencies...

The following 4 NEW packages are going to be installed:
  galera-3 libboost_program_options1_54_0 liblua5_2 nmap

4 new packages to install.
Overall download size: 5.0 MiB. Already cached: 0 B  After the operation, additional 21.5 MiB will be used.
Continue? [y/n/? shows all options] (y): y
Retrieving package libboost_program_options1_54_0-1.54.0-10.5.x86_64                                                                                             (1/4), 125.6 KiB (456.7 KiB unpacked)
Failed to mount cd:///?devices=/dev/disk/by-id/ata-QEMU_DVD-ROM_QM00003 on /var/adm/mount/AP_0xkTqbSL: Mounting media failed (mount: no medium found on /dev/sr0)

Please insert medium [SLES12-12-0] #1 and type 'y' to continue or 'n' to cancel the operation. [yes/no] (no): n
Problem occured during or after installation or removal of packages:
Installation aborted by user

Please see the above error message for a hint.

The problem seems to be specific to libboost_program_options . The other libboost_ are already installed as part of the minimal install.

gvwstate.dat is removed when in SST by wsrep_sst_xtrabackup-v2.sh

Refs codership/galera#130

Assertion `!flags.finalized' failed in binlog_cache_data::finalize causes dozens of InnoDB tests to fail

Running various InnoDB tests causes the following assertion:

mysqld: /home/philips/git/mysql-wsrep/sql/binlog.cc:1205: int binlog_cache_data::finalize(THD*, Log_event*): Assertion `!flags.finalized' failed.

#7  0x0000003e9042da82 in __assert_fail () from /lib64/libc.so.6
#8  0x0000000000a107d0 in binlog_cache_data::finalize (this=0x7fce8c0551d0, thd=0x2603820, end_event=0x7fcec03b8300) at /home/philips/git/mysql-wsrep/sql/binlog.cc:1205
#9  0x0000000000a0f833 in binlog_stmt_cache_data::finalize (this=0x7fce8c0551d0, thd=0x2603820) at /home/philips/git/mysql-wsrep/sql/binlog.cc:623
#10 0x0000000000a1d159 in MYSQL_BIN_LOG::commit (this=0x17f6e00 <mysql_bin_log>, thd=0x2603820, all=false) at /home/philips/git/mysql-wsrep/sql/binlog.cc:6476
#11 0x0000000000a2297e in wsrep_thd_binlog_commit (thd=0x2603820, all=false) at /home/philips/git/mysql-wsrep/sql/binlog.cc:9243
#12 0x000000000071d6a5 in TC_LOG_DUMMY::commit (this=0x17d24e8 <tc_log_dummy>, thd=0x2603820, all=false) at /home/philips/git/mysql-wsrep/sql/log.h:124
#13 0x000000000089be7e in trans_commit_stmt (thd=0x2603820) at /home/philips/git/mysql-wsrep/sql/transaction.cc:486
#14 0x00000000007e0511 in mysql_execute_command (thd=0x2603820) at /home/philips/git/mysql-wsrep/sql/sql_parse.cc:5444
#15 0x00000000007e39df in mysql_parse (thd=0x2603820, rawbuf=0x7fce8c004c30 "call mtr.add_suppression(\"InnoDB: Error: trying to open a table, but could not$\")", length=81, parser_state=0x7fcec03bb730) at /home/philips/git/mysql-wsrep/sql/sql_parse.cc:6815
#16 0x00000000007e2fe6 in wsrep_mysql_parse (thd=0x2603820, rawbuf=0x7fce8c004c30 "call mtr.add_suppression(\"InnoDB: Error: trying to open a table, but could not$\")", length=81, parser_state=0x7fcec03bb730) at /home/philips/git/mysql-wsrep/sql/sql_parse.cc:6595
#17 0x00000000007d5372 in dispatch_command (command=COM_QUERY, thd=0x2603820, packet=0x26cc5e1 "call mtr.add_suppression(\"InnoDB: Error: trying to open a table, but could not$\")", packet_length=81) at /home/philips/git/mysql-wsrep/sql/sql_parse.cc:1525
#18 0x00000000007d3ec1 in do_command (thd=0x2603820) at /home/philips/git/mysql-wsrep/sql/sql_parse.cc:1114
#19 0x000000000079fab7 in do_handle_one_connection (thd_arg=0x2603820) at /home/philips/git/mysql-wsrep/sql/sql_connect.cc:1007
#20 0x000000000079f5ed in handle_one_connection (arg=0x2603820) at /home/philips/git/mysql-wsrep/sql/sql_connect.cc:916
#21 0x0000003e90c0753a in start_thread () from /lib64/libpthread.so.0
#22 0x0000003e905045fd in clone () from /lib64/libc.so.6

For example:

./mysql-test-run.pl --suites=innodb --mysqld=--performance-schema=OFF --mysqld=--wsrep-provider=$WSREP_PROVIDER --skip-test-list=suite/innodb/t/galera.skip --no-check-testcases --mysqld=--wsrep-cluster-address="gcomm://" --force innodb.innodb-table-online --retry=0

Delete_rows event failed

Randgen run failed because Delete_rows event could not be applied on slave, command line:

runall-new.pl --queries=10000 --mysqld=--query-cache-type=1 --mysqld=--query-cache-size=1M --seed=random --short_column_names --mysqld=--skip-performance-schema --mysqld=--log-output=none --mysqld=--sql_mode=ONLY_FULL_GROUP_BY --grammar=conf/galera/galera_stress-56.yy --gendata=conf/galera/galera_stress-56.zz --threads=10 --reporter=Shutdown,Backtrace,ErrorLog,ErrorLogAlarm,Deadlock --mysqld=--innodb_flush_method=O_DIRECT --mysqld=--innodb_change_buffering=none --mysqld=--innodb_lock_wait_timeout=50 --mysqld=--lock_wait_timeout=50 --mysqld=--wsrep_retry_autocommit=2 --mysqld=--wsrep_slave_threads=4 --mysqld=--wsrep_causal_reads=OFF --mysqld=--innodb_flush_log_at_trx_commit=0 --mysqld=--transaction-isolation=READ-COMMITTED --mysqld=--innodb_file_per_table=1 --mysqld=--log-bin=mysql-bin --mysqld=--log-slave-updates --mysqld=--binlog_format=ROW --mysqld=--gtid-mode=ON --mysqld=--enforce-gtid-consistency --mysqld=--binlog-row-image=minimal --rows=1000 --galera=ms --mysqld=--wsrep-provider=/home/vagrant/galera/libgalera_smm.so --mtr-build-thread=300 --mask=3890 --duration=60 --basedir1=/home/vagrant/mysql-wsrep/build_debug --gendata=./conf/galera/galera_stress-56.zz --grammar=./conf/galera/galera_stress-56.yy --vardir1=/tmp/randgen/current1_1

Grammar: http://bazaar.launchpad.net/~codership/randgen/randgen_gc/view/head:/conf/galera/galera_stress-56.yy

Slave SQL error:

2015-01-05 06:32:29 19158 [ERROR] Slave SQL: Could not execute Delete_rows event
 on table test.table1000_innodb_int_autoinc; Can't find record in 'table1000_inn
odb_int_autoinc', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the even
t's master log FIRST, end_log_pos 223, Error_code: 1032
2015-01-05 06:32:29 19158 [Warning] WSREP: RBR event 3 Delete_rows apply warning
: 120, 355

The corresponding delete was successful on master:

BEGIN
/*!*/;
# at 239624
#150105  6:32:27 server id 1  end_log_pos 239703 CRC32 0xa5a5e94a       Table_ma
p: `test`.`table1000_innodb_int_autoinc` mapped to number 450
# at 239703
#150105  6:32:27 server id 1  end_log_pos 239743 CRC32 0xc4d8c49a       Delete_r
ows: table id 450 flags: STMT_END_F

BINLOG '
+y+qVBMBAAAATwAAAFeoAwAAAMIBAAAAAAEABHRlc3QAHHRhYmxlMTAwMF9pbm5vZGJfaW50X2F1
dG9pbmMABQMD/v4DBP4M/gweSumlpQ==
+y+qVCABAAAAKAAAAH+oAwAAAMIBAAAAAAEAAgAFAf7XAAAAmsTYxA==
'/*!*/;
### DELETE FROM `test`.`table1000_innodb_int_autoinc`
### WHERE
###   @1=215
# at 239743
#150105  6:32:27 server id 1  end_log_pos 239822 CRC32 0xca091a0d       Table_ma
p: `test`.`table1000_innodb_int_autoinc` mapped to number 450
# at 239822
#150105  6:32:27 server id 1  end_log_pos 239862 CRC32 0x961cf621       Write_ro
ws: table id 450 flags: STMT_END_F

BINLOG '
+y+qVBMBAAAATwAAAM6oAwAAAMIBAAAAAAEABHRlc3QAHHRhYmxlMTAwMF9pbm5vZGJfaW50X2F1
dG9pbmMABQMD/v4DBP4M/gweDRoJyg==
+y+qVB4BAAAAKAAAAPaoAwAAAMIBAAAAAAEAAgAFAf5KAAAAIfYclg==
'/*!*/;
### INSERT INTO `test`.`table1000_innodb_int_autoinc`
### SET
###   @1=74
# at 239862
#150105  6:32:27 server id 1  end_log_pos 239893 CRC32 0x1118cae9       Xid = 35
5
COMMIT/*!*/;

The value to be deleted 215 was present in slave binlog only in the insert to initialize the table:

### INSERT INTO `test`.`table1000_innodb_int_autoinc`
### SET
###   @1=215
###   @2=1637351424
###   @3='jtxzmrrer'
###   @4='s'
###   @5=8

After restarting slave as a standalone server the pk value 255 remained absent from the table:

mysql> select * from table1000_innodb_int_autoinc;
+------------+-----------+------+-----------+-----------+
| pk         | c0        | c3   | c2        | c1        |
+------------+-----------+------+-----------+-----------+
|          4 |       203 | NULL | NULL      | 129499136 |
|        116 |      NULL | NULL | 8         |      NULL |
|        146 |      NULL | NULL | 198574080 |      NULL |
|        149 | 934281216 | NULL | NULL      | 129499136 |
|        194 |      NULL | NULL | 0         |      NULL |
| 2014904320 |      NULL | NULL | w         |      NULL |
| 2147483647 |      NULL | NULL | NULL      |      NULL |
+------------+-----------+------+-----------+-----------+
7 rows in set (0.00 sec)

Crash in LOGGER::error_log_print when printing a WSREP message

The bootstrap non-Galera MTR test crashed as follows:

#3  <signal handler called>
#4  0x00000000026581e0 in ?? ()
#5  0x000000000071b05b in LOGGER::error_log_print (this=0x180c5c0 <logger>, level=INFORMATION_LEVEL, format=0xecd6ac "WSREP: %s", args=0x7fff1aca9968)
    at /var/lib/jenkins/workspace/mtr-galera-mysql-5.6/mysql-wsrep-5.6/sql/log.cc:875
#6  0x000000000071e60b in error_log_print (level=INFORMATION_LEVEL, format=0xecd6ac "WSREP: %s", args=0x7fff1aca9968)
    at /var/lib/jenkins/workspace/mtr-galera-mysql-5.6/mysql-wsrep-5.6/sql/log.cc:2084
#7  0x000000000071f033 in sql_print_information (format=0xecd6ac "WSREP: %s") at /var/lib/jenkins/workspace/mtr-galera-mysql-5.6/mysql-wsrep-5.6/sql/log.cc:2402
#8  0x00000000006193c9 in unireg_abort (exit_code=1) at /var/lib/jenkins/workspace/mtr-galera-mysql-5.6/mysql-wsrep-5.6/sql/mysqld.cc:1851
#9  0x000000000062148d in mysqld_main (argc=17, argv=0x2656110) at /var/lib/jenkins/workspace/mtr-galera-mysql-5.6/mysql-wsrep-5.6/sql/mysqld.cc:6303
#10 0x000000000061574d in main (argc=17, argv=0x7fff1acaa588) at /var/lib/jenkins/workspace/mtr-galera-mysql-5.6/mysql-wsrep-5.6/sql/main.cc:25
(gdb) frame 5
#5  0x000000000071b05b in LOGGER::error_log_print (this=0x180c5c0 <logger>, level=INFORMATION_LEVEL, format=0xecd6ac "WSREP: %s", args=0x7fff1aca9968)
    at /var/lib/jenkins/workspace/mtr-galera-mysql-5.6/mysql-wsrep-5.6/sql/log.cc:875
875         error= (*current_handler++)->log_error(level, format, args) || error;

#8  0x00000000006193c9 in unireg_abort (exit_code=1) at /var/lib/jenkins/workspace/mtr-galera-mysql-5.6/mysql-wsrep-5.6/sql/mysqld.cc:1851
1851        WSREP_INFO("Some threads may fail to exit.");

compiler warning in lock0lock.cc

This line was recently changed by us:

/home/philips/git/mysql-wsrep/storage/innobase/lock/lock0lock.cc: In function ‘ulint lock_rec_has_to_wait(ulint, const trx_t*, ulint, const ib_lock_t*, ulint)’:
/home/philips/git/mysql-wsrep/storage/innobase/lock/lock0lock.cc:1034:17: warning: format ‘%d’ expects argument of type ‘int’, but argument 3 has type ‘ulint {aka long unsigned int}’ [-Wformat=]
      for_locking);

ALTER TABLE ENGINE=InnoDB does not work

ALTER TABLE ENIGINE=InnoDB from MyISAM does not work with Galera. As the original table was never replicated on the slaves, the ALTER on the slaves operates on an empty MyISAM table and produces an empty InnoDB table. Further queries against that table cause the slave applier to fail.

The workaround suggested by Jay is to enable MyISAM replication as well, however this does not help much with MyISAM tables that have been created previously.

I guess the fix would be intercept the InnoDB writes that the ALTER statement produces and replicate them.

Duplicate autoinc values can happen during configuration change

Inserts to a table with autoinc primamry key can result in duplicate key error, if another node joins or drops from the cluster during the insert processing. This issue has been logged earlier in:

https://bugs.launchpad.net/percona-xtradb-cluster/+bug/1366997
https://mariadb.atlassian.net/browse/MDEV-7129

Automatic address guessing fails on fedora 20, centos 7

Log

2015-01-23 11:33:43 795 [ERROR] WSREP: Process completed with error: /sbin/ifconfig | grep -E '^[[:space:]]+inet addr:' | grep -m1 -v 'inet addr:127' | sed 's/:/ /' | awk '{ print $3 }': 2 (No such file or directory)
2015-01-23 11:33:43 795 [Warning] WSREP: Failed to guess base node address. Set it explicitly via wsrep_node_address.

The reason for this is different form of ifconfig output:

# ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.3.76  netmask 255.255.255.0  broadcast 192.168.3.255
        inet6 fe80::97:1fff:fefc:1b80  prefixlen 64  scopeid 0x20<link>
        ether 02:97:1f:fc:1b:80  txqueuelen 1000  (Ethernet)
        RX packets 1170  bytes 101571 (99.1 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 871  bytes 188128 (183.7 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 0  (Local Loopback)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

Address guessing should be changed to use system calls instead of shell utilities for better portability.

InnoDB: ERROR: the age of the last checkpoint is 9433946...

Happens only with 5.5 branch.

Randgen combinations test fails consistently because of innodb error message:

InnoDB: ERROR: the age of the last checkpoint is 9433946,
InnoDB: which exceeds the log group capacity 9433498.
InnoDB: If you are using big BLOB or TEXT rows, you must set the
InnoDB: combined size of log files at least 10 times bigger than the
InnoDB: largest such row.

The failing combination is

perl /home/vagrant/randgen_gc/runall-new.pl --queries=10000 --mysqld=--query-cache-type=1 --mysqld=--query-cache-size=1M    --seed=random    --short_column_names     --mysqld=--skip-performance-schema   --mysqld=--log-output=none --mysqld=--sql_mode=ONLY_FULL_GROUP_BY   --grammar=conf/galera/galera_stress-56.yy --gendata=conf/galera/galera_stress-56.zz --threads=10 --reporter=Shutdown,Backtrace,ErrorLog,ErrorLogAlarm,Deadlock --mysqld=--innodb_flush_method=O_DIRECT --mysqld=--innodb_change_buffering=none --mysqld=--innodb_lock_wait_timeout=50 --mysqld=--lock_wait_timeout=50 --mysqld=--wsrep_retry_autocommit=0 --mysqld=--wsrep_slave_threads=4 --mysqld=--wsrep_causal_reads=ON --mysqld=--innodb_flush_log_at_trx_commit=0 --mysqld=--transaction-isolation=READ-COMMITTED  --mysqld=--log-bin=mysql-bin --mysqld=--log-slave-updates --mysqld=--binlog_format=ROW --rows=1000  --galera=ms  --mysqld=--wsrep-provider=/home/vagrant/galera/libgalera_smm.so  --mtr-build-thread=300 --mask=54193 --duration=60 --basedir1=/home/vagrant/mysql-wsrep/build_debug  --gendata=./conf/galera/galera_stress-56.zz  --grammar=./conf/galera/galera_stress-56.yy  --vardir1=/tmp/randgen/current1_1

SHOW CREATE PROCEDURE and CALL are not covered by causal read

It is possible to create a stored procedure on one node and wsrep_sync_wait will not guarantee that a CALL will succeed on the other node.

Test case:

--source include/galera_cluster.inc
--source include/have_innodb.inc

--connection node_1
CREATE USER 'user1';
CREATE TABLE t1 (f1 INTEGER);

CREATE
DEFINER = 'user1'
PROCEDURE p1 (IN param1 INTEGER, OUT param2 INTEGER, INOUT param3 INTEGER)
COMMENT 'p1_comment'
LANGUAGE SQL
NOT DETERMINISTIC
MODIFIES SQL DATA
SQL SECURITY DEFINER
INSERT INTO t1 VALUES (1);
GRANT EXECUTE ON PROCEDURE p1 TO user1;

CREATE
DEFINER = CURRENT_USER
PROCEDURE p2 (param VARCHAR(100))
DETERMINISTIC
NO SQL
SQL SECURITY INVOKER BEGIN END ;

--connection node_1
SHOW CREATE PROCEDURE p1;

--connection node_2
SHOW CREATE PROCEDURE p1;

--connection node_1
SHOW CREATE PROCEDURE p2;

--connection node_2
SHOW CREATE PROCEDURE p2;

CALL p1(@a, @b, @c);
CALL p2('abc');

--connection node_1
DROP PROCEDURE p1;
DROP PROCEDURE p2;

DROP USER 'user1';
DROP TABLE t1;

Warnings treated as errors causes build to fail

Several warnings are being treated as errors under -DWITH_DEBUG=1 and the build fails:

/home/philips/git/mysql-wsrep-gcov/storage/innobase/lock/lock0lock.cc:1813:1: error: ‘trx_t* lock_rec_other_trx_holds_expl(ulint, trx_id_t, const rec_t*, const buf_block_t*)’ defined but not used [-Werror=unused-function]
 lock_rec_other_trx_holds_expl(

/home/philips/git/mysql-wsrep/storage/innobase/handler/ha_innodb.cc: In member function ‘int ha_innobase::wsrep_append_keys(THD*, bool, const uchar*, const uchar*)’:
/home/philips/git/mysql-wsrep/storage/innobase/handler/ha_innodb.cc:9327:8: warning: unused variable ‘key_info’ [-Wunused-variable]
   KEY *key_info = table->key_info;

Compiling without HAVE_WSREP also produces a warning about an unused variable, but I do not have the text with me at this time.

signals attribute not set when using posix_spawn() to start external processes

More details here: https://bugs.launchpad.net/percona-xtradb-cluster/+bug/1399175

wsrep_sst_xtrabackup-v2 script causes innobackupex to print a false positive stack trace into the log

The wsrep_sst_xtrabackup-v2 script contains the following invocation of innobackupex:

if ${INNOBACKUPEX_BIN} /tmp --help  | grep -q -- '--version-check'; then
    disver="--no-version-check"
fi

The problem is that grep -q causes SIGPIPE to be sent to innobackupex:

  -q, --quiet, --silent
              Quiet; do not write anything to standard output.  Exit immediately with zero status if any match is found, even if an error was  detected.   Also  see  the  -s  or  --no-messages
              option.  (-q is specified by POSIX.)

innobackupex in turn detects the signal and prints the following:

innobackupex: got a fatal error with the following stacktrace: at /usr/bin/innobackupex line 1722.
        main::catch_fatal_signal('PIPE') called at /usr/share/perl5/vendor_perl/Pod/Text.pm line 398
        eval {...} called at /usr/share/perl5/vendor_perl/Pod/Text.pm line 398
        Pod::Text::item('Pod::Usage=HASH(0x24520a0)', 'This option specifies the number of seconds innobackupex wait...') called at /usr/share/perl5/vendor_perl/Pod/Text.pm line 416
        Pod::Text::cmd_para('Pod::Usage=HASH(0x24520a0)', 'HASH(0x265b020)', 'This option specifies the number of seconds innobackupex wait...') called at /usr/share/perl5/vendor_perl/Pod/Text.pm line 209
        Pod::Text::_handle_element_end('Pod::Usage=HASH(0x24520a0)', 'Para') called at /usr/share/perl5/vendor_perl/Pod/Usage.pm line 318
        Pod::Usage::_handle_element_end('Pod::Usage=HASH(0x24520a0)', 'Para') called at /usr/share/perl5/vendor_perl/Pod/Simple/BlackBox.pm line 1506
        Pod::Simple::BlackBox::_traverse_treelet_bit(undef, undef, 'HASH(0x265b020)', 'This option specifies the number of seconds innobackupex wait...') called at /usr/share/perl5/vendor_perl/Pod/Simple/BlackBox.pm line 885
        Pod::Simple::BlackBox::_ponder_paragraph_buffer('Pod::Usage=HASH(0x24520a0)') called at /usr/share/perl5/vendor_perl/Pod/Simple/BlackBox.pm line 240
        Pod::Simple::BlackBox::parse_lines('Pod::Usage=HASH(0x24520a0)', '=item --log-copy-interval\x{a}', '\x{a}', 'This option specifies time interval between checks done by lo...', '\x{a}', '=item --incremental-lsn\x{a}', '\x{a}', 'This option specifies the log sequence number (LSN) to use fo...', '\x{a}', ...) called at /usr/share/perl5/vendor_perl/Pod/Simple.pm line 471
        Pod::Simple::parse_file('Pod::Usage=HASH(0x24520a0)', '/usr/bin/innobackupex') called at /usr/share/perl5/vendor_perl/Pod/Text.pm line 727
        Pod::Text::parse_file('Pod::Usage=HASH(0x24520a0)', '/usr/bin/innobackupex') called at /usr/share/perl5/vendor_perl/Pod/Simple.pm line 512
        Pod::Simple::parse_from_file('Pod::Usage=HASH(0x24520a0)', '/usr/bin/innobackupex', 'GLOB(0x1589798)') called at /usr/share/perl5/vendor_perl/Pod/Text.pm line 696
        Pod::Text::parse_from_file('Pod::Usage=HASH(0x24520a0)', '/usr/bin/innobackupex', 'GLOB(0x1589798)') called at /usr/share/perl5/vendor_perl/Pod/Usage.pm line 161
        Pod::Usage::pod2usage('HASH(0x23f4420)') called at /usr/bin/innobackupex line 1646
        main::usage() called at /usr/bin/innobackupex line 3852
        main::check_args() called at /usr/bin/innobackupex line 1529
innobackupex: Error: Terminated with SIGPIPE at /usr/bin/innobackupex line 1722.

This output ends up in the server log.

/etc/rc.d/init.d/mysql is set to print every command executed by default

/etc/rc.d/init.d/mysql is run under a bash -x shebang, which causes it to be verbose. This is in contrast to init scripts provided by other vendors or products.

I tested on centos 6 and 7

Assertion `key_count > 0' failed in galera::Certification::do_test_v3() with MyISAM and Stored Procedure

All InnoDB tests now assert as follows:

mysqld: galera/src/certification.cpp:590: galera::Certification::TestResult galera::Certification::do_test_v3(galera::TrxHandle*, bool): Assertion `key_count > 0' failed.

#7  0x0000003e9042da82 in __assert_fail () from /lib64/libc.so.6
#8  0x00007f4a3d6e7aea in galera::Certification::do_test_v3 (this=this@entry=0x2603f58, trx=trx@entry=0x7f4a0001e2a0, store_keys=store_keys@entry=true) at galera/src/certification.cpp:590
#9  0x00007f4a3d6e9b03 in galera::Certification::do_test (this=0x2603f58, trx=trx@entry=0x7f4a0001e2a0, store_keys=<optimized out>) at galera/src/certification.cpp:747
#10 0x00007f4a3d6e9e29 in galera::Certification::test (this=<optimized out>, trx=0x7f4a0001e2a0, bval=<optimized out>) at galera/src/certification.cpp:916
#11 0x00007f4a3d6e9f45 in galera::Certification::append_trx (this=this@entry=0x2603f58, trx=trx@entry=0x7f4a0001e2a0) at galera/src/certification.cpp:1024
#12 0x00007f4a3d728199 in galera::ReplicatorSMM::cert (this=0x26035a0, trx=0x7f4a0001e2a0) at galera/src/replicator_smm.cpp:1689
#13 0x00007f4a3d71e633 in galera::ReplicatorSMM::cert_and_catch (this=this@entry=0x26035a0, trx=trx@entry=0x7f4a0001e2a0) at galera/src/replicator_smm.cpp:1774
#14 0x00007f4a3d71f1f7 in galera::ReplicatorSMM::to_isolation_begin (this=0x26035a0, trx=0x7f4a0001e2a0, meta=<optimized out>) at galera/src/replicator_smm.cpp:1010
#15 0x00007f4a3d735ccb in galera_to_execute_start (gh=<optimized out>, conn_id=7, keys=<optimized out>, keys_num=<optimized out>, data=0x7f4a30793970, count=1, meta=0x2856920) at galera/src/wsrep_provider.cpp:729
#16 0x000000000063e142 in wsrep_TOI_begin (thd=0x2854870, db_=0x0, table_=0x0, table_list=0x7f4a00043718) at /home/philips/git/mysql-wsrep/sql/wsrep_mysqld.cc:1269
#17 0x000000000063f067 in wsrep_to_isolation_begin (thd=0x2854870, db_=0x0, table_=0x0, table_list=0x7f4a00043718) at /home/philips/git/mysql-wsrep/sql/wsrep_mysqld.cc:1443
#18 0x00000000007f42a4 in mysql_execute_command (thd=0x2854870) at /home/philips/git/mysql-wsrep/sql/sql_parse.cc:3997
#19 0x000000000099953e in sp_instr_stmt::exec_core (this=0x7f4a00043c90, thd=0x2854870, nextp=0x7f4a307950b8) at /home/philips/git/mysql-wsrep/sql/sp_instr.cc:901
#20 0x00000000009983a0 in sp_lex_instr::reset_lex_and_exec_core (this=0x7f4a00043c90, thd=0x2854870, nextp=0x7f4a307950b8, open_tables=false) at /home/philips/git/mysql-wsrep/sql/sp_instr.cc:390
#21 0x0000000000998cae in sp_lex_instr::validate_lex_and_execute_core (this=0x7f4a00043c90, thd=0x2854870, nextp=0x7f4a307950b8, open_tables=false) at /home/philips/git/mysql-wsrep/sql/sp_instr.cc:638
#22 0x00000000009991cd in sp_instr_stmt::execute (this=0x7f4a00043c90, thd=0x2854870, nextp=0x7f4a307950b8) at /home/philips/git/mysql-wsrep/sql/sp_instr.cc:808
#23 0x000000000074df46 in sp_head::execute (this=0x7f4a000924c0, thd=0x2854870, merge_da_on_success=true) at /home/philips/git/mysql-wsrep/sql/sp_head.cc:645
#24 0x000000000074fc6a in sp_head::execute_procedure (this=0x7f4a000924c0, thd=0x2854870, args=0x2857820) at /home/philips/git/mysql-wsrep/sql/sp_head.cc:1315
#25 0x00000000007f79d9 in mysql_execute_command (thd=0x2854870) at /home/philips/git/mysql-wsrep/sql/sql_parse.cc:4987
#26 0x00000000007fc9b9 in mysql_parse (thd=0x2854870, rawbuf=0x7f4a00081650 "call mtr.check_warnings(@result)", length=32, parser_state=0x7f4a30796730) at /home/philips/git/mysql-wsrep/sql/sql_parse.cc:6815
#27 0x00000000007fbf7a in wsrep_mysql_parse (thd=0x2854870, rawbuf=0x7f4a00081650 "call mtr.check_warnings(@result)", length=32, parser_state=0x7f4a30796730) at /home/philips/git/mysql-wsrep/sql/sql_parse.cc:6595
#28 0x00000000007edbc7 in dispatch_command (command=COM_QUERY, thd=0x2854870, packet=0x291d631 "call mtr.check_warnings(@result)", packet_length=32) at /home/philips/git/mysql-wsrep/sql/sql_parse.cc:1525
#29 0x00000000007ec687 in do_command (thd=0x2854870) at /home/philips/git/mysql-wsrep/sql/sql_parse.cc:1114
#30 0x00000000007b58a2 in do_handle_one_connection (thd_arg=0x2854870) at /home/philips/git/mysql-wsrep/sql/sql_connect.cc:1007
#31 0x00000000007b53b7 in handle_one_connection (arg=0x2854870) at /home/philips/git/mysql-wsrep/sql/sql_connect.cc:916
#32 0x0000003e90c0753a in start_thread () from /lib64/libpthread.so.0
#33 0x0000003e905045fd in clone () from /lib64/libc.so.6

To reproduce:

mysql-test-run.pl --suites=innodb --mysqld=--performance-schema=OFF --mysqld=--wsrep-cluster-address="gcomm://" --mysqld=--wsrep-provider=/home/philips/git/galera/libgalera_smm.so --skip-test-list=suite/innodb/t/galera.skip

SELECT FOR UPDATE sometimes allowed to proceed in the face of a concurrent update

The galera_bf_abort_for_update test performs the following deterministic sequence of events:

On node #2, starts a transaction
On node #1, starts and commits a conflicting transaction
On node #2, issues SELECT FOR UPDATE, expecting to get a deadlock error.

However, this is not always the case and once every several dozen or so runs the SELECT FOR UPDATE succeeds when it should have failed.

The test is fully deterministic and runs under wsrep_sync_wait = 7.

To reproduce:

Remove any line relating to galera_bf_abort_for_update from suite/galera/t/disabled.def
Run the test in a loop:

 perl mysql-test-run.pl --suite galera galera_bf_abort_for_update --big-test --repeat 1000

And you will get:

galera.galera_bf_abort_for_update        [ fail ]
        Test ended at 2014-12-10 02:45:39

CURRENT_TEST: galera.galera_bf_abort_for_update
mysqltest: At line 21: query 'SELECT * FROM t1 FOR UPDATE' succeeded - should have failed with errno 1213...

The result from queries just before the failure was:
CREATE TABLE t1 (f1 INTEGER PRIMARY KEY) ENGINE=InnoDB;
SET AUTOCOMMIT=OFF;
START TRANSACTION;
INSERT INTO t1 VALUES (1);
INSERT INTO t1 VALUES (1);

 - saving '/home/philips/git/mysql-wsrep/mysql-test/var/log/galera.galera_bf_abort_for_update/' to '/home/philips/git/mysql-wsrep/mysql-test/var/log/galera.galera_bf_abort_for_update/'

perfschema.ortho_iter MTR test fails sporadically

The perfschema.ortho_iter test fails on the Jenkins server:

        Test ended at 2014-12-19 07:42:04

CURRENT_TEST: perfschema.ortho_iter
--- /var/lib/jenkins/workspace/mtr-galera-mysql-5.6/mysql-wsrep-5.6/mysql-test/suite/perfschema/r/ortho_iter.result 2014-12-19 06:02:55.589010610 +0300
+++ /var/lib/jenkins/workspace/mtr-galera-mysql-5.6/mysql-wsrep-5.6/mysql-test/var/log/ortho_iter.reject    2014-12-19 08:42:03.845232954 +0300
@@ -129,7 +129,7 @@
 Variable_name  Value
 Performance_schema_accounts_lost   0
 Performance_schema_cond_classes_lost   0
-Performance_schema_cond_instances_lost 0
+Performance_schema_cond_instances_lost 6
 Performance_schema_digest_lost 0
 Performance_schema_file_classes_lost   0
 Performance_schema_file_handles_lost   0

mysqltest: Result content mismatch

 - saving '/var/lib/jenkins/workspace/mtr-galera-mysql-5.6/mysql-wsrep-5.6/mysql-test/var/log/perfschema.ortho_iter/' to '/var/lib/jenkins/workspace/mtr-galera-mysql-5.6/mysql-wsrep-5.6/mysql-test/var/log/perfschema.ortho_iter/'

Retrying test perfschema.ortho_iter, attempt(2/3)...

perfschema.ortho_iter                    [ retry-pass ]   1964

As there is no documentation as to what this value is supposed to mean, I will disable the test for the time being.

innodb_bug13867871 test fails with wsrep loaded

With wsrep loaded, the test fails as follows:

CURRENT_TEST: innodb.innodb_bug13867871
mysqltest: At line 87: query 'CREATE INDEX idx ON child(b, c)' failed with wrong errno 1061: 'Duplicate key name 'idx'', instead of 1205...

In other words, a DDL statement is expected to return ER_LOCK_WAIT_TIMEOUT but it executes instead and returns a different error.

To reproduce:

./mysql-test-run.pl --suites=innodb --mysqld=--performance-schema=OFF --mysqld=--wsrep-provider=$WSREP_PROVIDER --no-check-testcases --mysqld=--wsrep-cluster-address="gcomm://" --force --retry=0 --max-test-fail=1000 innodb.innodb_bug13867871

Innodb hangs in row_mysql_handle_errors in a RQG test

All threads hang like this:

#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x0000000000bea020 in os_cond_wait (cond=0x7f855c045560, fast_mutex=0x7f855c045520) at /home/vagrant/codership-mysql/storage/innobase/os/os0sync.cc:214
#2  0x0000000000bea590 in os_event_wait_low (event=0x7f855c045520, reset_sig_count=105) at /home/vagrant/codership-mysql/storage/innobase/os/os0sync.cc:610
#3  0x0000000000bc7e5d in lock_wait_suspend_thread (thr=0x7f8544041aa8) at /home/vagrant/codership-mysql/storage/innobase/lock/lock0wait.cc:344
#4  0x0000000000c5ebd9 in row_mysql_handle_errors (new_err=0x7f85b0567274, trx=0x7f85600b74b8, thr=0x7f8544041aa8, savept=0x0) at /home/vagrant/codership-mysql/storage/innobase/row/row0mysql.cc:632

#5  0x0000000000c92227 in row_search_for_mysql (buf=0x7f854405ff88 "\361", mode=1, prebuilt=0x7f8544041378, match_mode=0, direction=1)
    at /home/vagrant/codership-mysql/storage/innobase/row/row0sel.cc:5058
#6  0x0000000000b676d1 in ha_innobase::general_fetch (this=0x7f854405fcc0, buf=0x7f854405ff88 "\361", direction=1, match_mode=0)
    at /home/vagrant/codership-mysql/storage/innobase/handler/ha_innodb.cc:8466
---Type <return> to continue, or q <return> to quit---
#7  0x0000000000b67c0b in ha_innobase::rnd_next (this=0x7f854405fcc0, buf=0x7f854405ff88 "\361") at /home/vagrant/codership-mysql/storage/innobase/handler/ha_innodb.cc:8682
#8  0x000000000065b9b8 in handler::ha_rnd_next (this=0x7f854405fcc0, buf=0x7f854405ff88 "\361") at /home/vagrant/codership-mysql/sql/handler.cc:2760
#9  0x00000000009594c4 in find_all_keys (param=0x7f85b0567f80, select=0x7f85600023d0, fs_info=0x7f85b0567f10, buffpek_pointers=0x7f85b05681a0, tempfile=0x7f85b0568080, pq=0x7f85b0567ec0,
    found_rows=0x7f85b05683a0) at /home/vagrant/codership-mysql/sql/filesort.cc:771
#10 0x0000000000957e0a in filesort (thd=0x23bbfc0, table=0x7f85440647d0, filesort=0x7f85b0568420, sort_positions=true, examined_rows=0x7f85b0568398, found_rows=0x7f85b05683a0)
    at /home/vagrant/codership-mysql/sql/filesort.cc:337
#11 0x0000000000897225 in mysql_update (thd=0x23bbfc0, table_list=0x7f8560000b70, fields=..., values=..., conds=0x7f8560001578, order_num=5, order=0x7f85600019a0, limit=8,
    handle_duplicates=DUP_ERROR, ignore=false, found_return=0x7f85b0568a00, updated_return=0x7f85b0568a90) at /home/vagrant/codership-mysql/sql/sql_update.cc:572
#12 0x0000000000802100 in mysql_execute_command (thd=0x23bbfc0) at /home/vagrant/codership-mysql/sql/sql_parse.cc:3626
#13 0x000000000080bd27 in mysql_parse (thd=0x23bbfc0,
    rawbuf=0x7f85600008d0 "UPDATE  `table1000_innodb_int_autoinc` AS X SET `col_int` = 8 WHERE X . `col_char_12_key` BETWEEN 1 AND 7 ORDER BY `col_char_12`,`col_char_12_key`,`col_int`,`col_int_key`,`pk` LIMIT 8", length=183, parser_state=0x7f85b056a290) at /home/vagrant/codership-mysql/sql/sql_parse.cc:6815
#14 0x000000000080b32b in wsrep_mysql_parse (thd=0x23bbfc0,
    rawbuf=0x7f85600008d0 "UPDATE  `table1000_innodb_int_autoinc` AS X SET `col_int` = 8 WHERE X . `col_char_12_key` BETWEEN 1 AND 7 ORDER BY `col_char_12`,`col_char_12_key`,`col_int`,`col_int_key`,`pk` LIMIT 8", length=183, parser_state=0x7f85b056a290) at /home/vagrant/codership-mysql/sql/sql_parse.cc:6595
#15 0x00000000007fcd19 in dispatch_command (command=COM_QUERY, thd=0x23bbfc0,
    packet=0x2498311 "UPDATE  `table1000_innodb_int_autoinc` AS X SET `col_int` = 8 WHERE X . `col_char_12_key` BETWEEN 1 AND 7 ORDER BY `col_char_12`,`col_char_12_key`,`col_int`,`col_int_key`,`pk` LIMIT 8", packet_length=183) at /home/vagrant/codership-mysql/sql/sql_parse.cc:1525
#16 0x00000000007fb813 in do_command (thd=0x23bbfc0) at /home/vagrant/codership-mysql/sql/sql_parse.cc:1114
#17 0x00000000007c2127 in do_handle_one_connection (thd_arg=0x23bbfc0) at /home/vagrant/codership-mysql/sql/sql_connect.cc:1007
#18 0x00000000007c1c2a in handle_one_connection (arg=0x23bbfc0) at /home/vagrant/codership-mysql/sql/sql_connect.cc:916
#19 0x00007f85be007182 in start_thread (arg=0x7f85b056b700) at pthread_create.c:312
#20 0x00007f85bd513fbd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

http://127.0.0.1:8080/job/nightly-builds/Arch=64,GaleraVersion=3.x,MySQLVersion=5.6/615/consoleText

RQG command line:

http://127.0.0.1:8080/job/nightly-builds/Arch=64,GaleraVersion=3.x,MySQLVersion=5.6/615/consoleText

Spurious deadlock error on a DROP TABLE that does not conflict with anything

A DROP TABLE issued at the end of a simple MTR test fails with a deadlock error even though there is no concurrent queries to conflict with.

The log says:

2014-12-15 11:45:59 6967 [Note] WSREP: cluster conflict due to high priority abort for threads:
2014-12-15 11:45:59 6967 [Note] WSREP: Winning thread:
   THD: 2, mode: applier, state: executing, conflict: no conflict, seqno: 286
   SQL: (null)
2014-12-15 11:45:59 6967 [Note] WSREP: Victim thread:
   THD: 318, mode: local, state: idle, conflict: no conflict, seqno: -1
   SQL: (null)
2014-12-15 11:45:59 6967 [Note] WSREP: BF kill (1, seqno: 286), victim: (318) trx: 2947
2014-12-15 11:45:59 6967 [Note] WSREP: Aborting query: void
2014-12-15 11:45:59 6967 [Note] WSREP: kill IDLE for 2947
2014-12-15 11:45:59 6967 [Note] WSREP: enqueuing trx abort for (318)
2014-12-15 11:45:59 6967 [Note] WSREP: signaling aborter
2014-12-15 11:45:59 6967 [Note] WSREP: WSREP rollback thread wakes for signal
2014-12-15 11:45:59 6967 [Note] WSREP: client rollback due to BF abort for (318), query: (null)
2014-12-15 11:45:59 6967 [Note] WSREP: WSREP rollbacker aborted thd: (318 140672329082624)
2014-12-15 11:45:59 6967 [Note] WSREP: Deadlock error for: (null)

To reproduce, run the following test in a loop:

#
# Test non-ascii data in table where the PK is unicode
#

--source include/galera_cluster.inc
--source include/have_innodb.inc

CREATE TABLE t1 (
        f1 VARCHAR(255) PRIMARY KEY
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

INSERT INTO t1 VALUES ('текст');

--connection node_2
SELECT f1 = 'текст' FROM t1;

#
# Provoke a conflict
#

--connection node_1
SET AUTOCOMMIT=OFF;
START TRANSACTION;
UPDATE t1 SET f1 = 'текст2';

--connection node_2
SET AUTOCOMMIT=OFF;
START TRANSACTION;
UPDATE t1 SET f1 = 'текст3';

--connection node_1
COMMIT;

--connection node_2
--error ER_LOCK_DEADLOCK
COMMIT;

SELECT f1 = 'текст2' FROM t1;
SELECT f1 = 'текст2' FROM t1 WHERE f1 = 'текст2';

#
# Provoke a duplicate key error
#

--connection node_2
START TRANSACTION;
INSERT INTO t1 VALUES ('текст4');

--connection node_1
START TRANSACTION;
INSERT INTO t1 VALUES ('текст4');

--connection node_2
COMMIT;

First run it with --record to record a .result file. Then run it in a loop with --repeat=500 --mysqld=--wsrep_debug=ON

MySQL error log is hardly readable b/c of missing CR/NL

When galera starts it gives sometimes the error without CRLF which
makes the error log file very hardly readable:

2015-01-27 20:28:09 0 [Warning] TIMESTAMP with implicit DEFAULT value
is deprecated. Please use --explicit_defaults_for_timestamp server
option (see documentation for more details).
150127 20:28:11 mysqld_safe WSREP: Failed to recover position:
2015-01-27 20:28:09 17479 [Note] Plugin 'FEDERATED' is disabled.
2015-01-27 20:28:09 17479 [Note] InnoDB: Using atomics to ref count
buffer pool pages 2015-01-27 20:28:09 17479 [Note] InnoDB: The InnoDB
memory heap is disabled 2015-01-27 20:28:09 17479 [Note] InnoDB:
Mutexes and rw_locks use GCC atomic builtins 2015-01-27 20:28:09 17479
[Note] InnoDB: Memory barrier is not used 2015-01-27 20:28:09 17479
[Note] InnoDB: Compressed tables use zlib 1.2.3 2015-01-27 20:28:09
17479 [Note] InnoDB: Using Linux native AIO 2015-01-27 20:28:09 17479
[Note] InnoDB: Not using CPU crc32 instructions 2015-01-27 20:28:09
17479 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2015-01-27 20:28:09 17479 [Note] InnoDB: Completed initialization of
buffer pool 2015-01-27 20:28:09 17479 [Note] InnoDB: Highest supported
file format is Barracuda. 2015-01-27 20:28:09 17479 [Note] InnoDB: 128
rollback segment(s) are active. 2015-01-27 20:28:09 17479 [Note]
InnoDB: Waiting for purge to start 2015-01-27 20:28:09 17479 [Note]
InnoDB: 5.6.21 started; log sequence number 1626201 2015-01-27
20:28:09 17479 [Note] Server hostname (bind-address): '*'; port: 3306
2015-01-27 20:28:09 17479 [Note] IPv6 is available. 2015-01-27
20:28:09 17479 [Note] - '::' resolves to '::'; 2015-01-27 20:28:09
17479 [Note] Server socket created on IP: '::'. 2015-01-27 20:28:09
17479 [ERROR] Can't start server : Bind on unix socket: Permission
denied 2015-01-27 20:28:09 17479 [ERROR] Do you already have another
mysqld server running on socket: /var/lib/mysql/mysql.sock ?
2015-01-27 20:28:09 17479 [ERROR] Aborting 2015-01-27 20:28:09 17479
[Note] Binlog end 2015-01-27 20:28:09 17479 [Note] Shutting down
plugin 'partition' 2015-01-27 20:28:09 17479 [Note] Shutting down
plugin 'INNODB_SYS_DATAFILES' 2015-01-27 20:28:09 17479 [Note]
Shutting down plugin 'INNODB_SYS_TABLESPACES' 2015-01-27 20:28:09
17479 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN_COLS' 2015-01-27
20:28:09 17479 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN'
2015-01-27 20:28:09 17479 [Note] Shutting down plugin
'INNODB_SYS_FIELDS' 2015-01-27 20:28:09 17479 [Note] Shutting down
plugin 'INNODB_SYS_COLUMNS' 2015-01-27 20:28:09 17479 [Note] Shutting
down plugin 'INNODB_SYS_INDEXES' 2015-01-27 20:28:09 17479 [Note]
Shutting down plugin 'INNODB_SYS_TABLESTATS' 2015-01-27 20:28:09 17479
[Note] Shutting down plugin 'INNODB_SYS_TABLES' 2015-01-27 20:28:09
17479 [Note] Shutting down plugin 'INNODB_FT_INDEX_TABLE' 2015-01-27
20:28:09 17479 [Note] Shutting down plugin 'INNODB_FT_INDEX_CACHE'
2015-01-27 20:28:09 17479 [Note] Shutting down plugin
'INNODB_FT_CONFIG' 2015-01-27 20:28:09 17479 [Note] Shutting down
plugin 'INNODB_FT_BEING_DELETED' 2015-01-27 20:28:09 17479 [Note]
Shutting down plugin 'INNODB_FT_DELETED' 2015-01-27 20:28:09 17479
[Note] Shutting down plugin 'INNODB_FT_DEFAULT_STOPWORD' 2015-01-27
20:28:09 17479 [Note] Shutting down plugin 'INNODB_METRICS' 2015-01-27
20:28:09 17479 [Note] Shutting down plugin 'INNODB_BUFFER_POOL_STATS'
2015-01-27 20:28:09 17479 [Note] Shutting down plugin
'INNODB_BUFFER_PAGE_LRU' 2015-01-27 20:28:09 17479 [Note] Shutting
down plugin 'INNODB_BUFFER_PAGE' 2015-01-27 20:28:09 17479 [Note]
Shutting down plugin 'INNODB_CMP_PER_INDEX_RESET' 2015-01-27 20:28:09
17479 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX' 2015-01-27
20:28:09 17479 [Note] Shutting down plugin 'INNODB_CMPMEM_RESET'
2015-01-27 20:28:09 17479 [Note] Shutting down plugin 'INNODB_CMPMEM'
2015-01-27 20:28:09 17479 [Note] Shutting down plugin
'INNODB_CMP_RESET' 2015-01-27 20:28:09 17479 [Note] Shutting down
plugin 'INNODB_CMP' 2015-01-27 20:28:09 17479 [Note] Shutting down
plugin 'INNODB_LOCK_WAITS' 2015-01-27 20:28:09 17479 [Note] Shutting
down plugin 'INNODB_LOCKS' 2015-01-27 20:28:09 17479 [Note] Shutting
down plugin 'INNODB_TRX' 2015-01-27 20:28:09 17479 [Note] Shutting
down plugin 'InnoDB' 2015-01-27 20:28:09 17479 [Note] InnoDB: FTS
optimize thread exiting. 2015-01-27 20:28:09 17479 [Note] InnoDB:
Starting shutdown... 2015-01-27 20:28:10 17479 [Note] InnoDB: Shutdown
completed; log sequence number 1626211 2015-01-27 20:28:10 17479
[Note] Shutting down plugin 'ARCHIVE' 2015-01-27 20:28:10 17479 [Note]
Shutting down plugin 'BLACKHOLE' 2015-01-27 20:28:10 17479 [Note]
Shutting down plugin 'PERFORMANCE_SCHEMA' 2015-01-27 20:28:10 17479
[Note] Shutting down plugin 'MyISAM' 2015-01-27 20:28:10 17479 [Note]
Shutting down plugin 'MEMORY' 2015-01-27 20:28:10 17479 [Note]
Shutting down plugin 'MRG_MYISAM' 2015-01-27 20:28:10 17479 [Note]
Shutting down plugin 'CSV' 2015-01-27 20:28:10 17479 [Note] Shutting
down plugin 'sha256_password' 2015-01-27 20:28:10 17479 [Note]
Shutting down plugin 'mysql_old_password' 2015-01-27 20:28:10 17479
[Note] Shutting down plugin 'mysql_native_password' 2015-01-27
20:28:10 17479 [Note] Shutting down plugin 'wsrep' 2015-01-27 20:28:10
17479 [Note] Shutting down plugin 'binlog' 2015-01-27 20:28:10 17479
[Note] /usr/sbin/mysqld: Shutdown complete

How to reproduce:

service mysql stop
chown root: /var/lib/mysql
service mysql start
Starting MySQL... ERROR! The server quit without updating PID file
(/var/lib/mysql/centos7tpl.pid).
vi centos7tpl.err

SLES 12: Failed to start mysql.service: Unit mysql.service failed to load: No such file or directory

On SLES 12, the binaries are all installed but the services are missing:

linux-xi5a:/home/galera # /usr/bin/systemctl | grep -i garbd
linux-xi5a:/home/galera # /usr/bin/systemctl | grep -i mysql

Opensuse 13.1: warning: user mysql does not exist - using root

The test-install.sh script has the following output on OpenSuse 13.1:

warning: user mysql does not exist - using root
warning: group mysql does not exist - using root

codership / mysql-wsrep Goto Github PK

mysql-wsrep's People

Contributors

Stargazers

Watchers

Forkers

mysql-wsrep's Issues

! /bin/bash --

! /bin/bash --

Recommend Projects

Recommend Topics

Recommend Org