twitter-forks / mysql

MySQL fork maintained and used at Twitter

Twitter MySQL 5.5

This is Twitter's MySQL development branch, which is based on MySQL 5.5 as published by Oracle on MySQL on Launchpad.

This repository is published in order to share code and information and is not intended to be used directly outside of Twitter. We provide no guarantees of bug fixes, ongoing maintenance, compatibility, or suitability for any user outside of Twitter.

The original README file provided with the upstream MySQL release can be found at README-MySQL.

Features in Twitter MySQL

Additional status variables

Additional status variables have been added, particularly from the internals of InnoDB. This allows us to monitor our systems more effectively and understand their behavior better when handling production workloads. The variables added are:

• The number of InnoDB files and tablespace files opened, closed, and currently open. This information was previously not exposed by InnoDB.
• The number of deadlocks encountered. This information was previously not exposed by InnoDB.
• The current log sequence number (LSN) as well as the LSN flushed up to and checkpointed up to. This information has previously been available in SHOW ENGINE INNODB STATUS.

Optimization of memory allocation under NUMA

On most recent multi-processor systems, a non-uniform memory access NUMA (NUMA) architecture is in use, which divides the total system memory across multiple NUMA "nodes". When allocating large amounts of memory to InnoDB's buffer pool, as is typical, some inefficiencies as well as serious problems can be encountered. More details about the problems typical in NUMA systems running MySQL can be found in a blog post by Jeremy Cole. The following changes have been made to optimize and improve this:

• An option has been added to forcibly pre-allocate the entire buffer pool during startup. This is primarily intended to force the system to decide which pages to allocate, and on which NUMA node to allocate them. If the buffer pool can't be fully allocated for any reason, InnoDB will abort during startup.
• An option has been added to mysqld_safe to wrap the start of mysqld with numactl --interleave=all to interleave memory allocation between all NUMA nodes available. This ensures that no NUMA node is favored for any allocation, so that memory usage will remain even over time between multiple NUMA nodes.
• An option has been added to mysqld_safe to flush the operating system buffer caches before startup (on Linux only). Linux will normally not evict optional caches to make new allocations unless the system is under memory pressure, which can cause supposedly evenly interleaved memory allocations to still be done unevenly, favoring the node with less data cached before startup. Flushing the caches before startup ensures that no large cache allocations are present in the system before InnoDB allocates its buffer pool.

Server-side query timeout

• Reduced unnecessary work through improved server-side statement timeout support. This allows the server to proactively cancel queries that run longer than a millisecond-granularity timeout.

Buffer pool export and restore by prefetch

• Export and restore InnoDB buffer pool in using a safe and lightweight method. This enables us to build tools to support rolling restarts of our services with minimal pain.

Optimization for solid-state drives (SSDs)

• Optimize MySQL for SSD-based machines, including page-flushing behavior and reduction in writes to disk to improve lifespan.

mysql's Issues

A potential bug of free of stack address

hello, I found a potential bug of free of stack address in source code of mysql, the overview of occurrence is shown as follows.


step1:
in mysql/sql-common/client.c line 3585
defined a local variable tmp_mysql in this line

step2:
in mysql/sql-common/client.c line 3615
in a path derived from step1,there is a call of mysql_close and stack address of tmp_mysql is used as 1st parameter

step3:
in mysql/sql-common/client.c line 3823
the parameter of function mysql_close is used as 1st parameter of function my_free

step4:
in mysql/mysys/my_malloc.c line 134
free of stack address here

A possible use-after-free vulnerability

Hi, there is a potential use-after-free bug, please check and confirm if it's real

In the file mysql/mysys/my_malloc.c and the my_realloc function, the variable oldpoint free at line 100 and return it to the caller. The path conditions of these two branches can be satisfied simultaneously.

if (my_flags & MY_FREE_ON_ERROR) //line 109
      my_free(oldpoint); // line 110

if (my_flags & MY_HOLD_ON_ERROR)//111
      DBUG_RETURN(oldpoint);/112

The function my_free is at line 130 in the same file.

Thank you
Ryan

twitter-mysql-ca173b6 was failed to make with the newest google-breakpad

twitter-mysql-ca173b6 was failed to make with the newest google-breakpad
caused by two change:

1. define of callback function was changed
   in sql/minidump.cc:
   dump_callback(const char *dump_path,
   const char *minidump_id,
   void *,
   bool succeeded)

in client/linux/handler/exception_handler.h

typedef bool (MinidumpCallback)(const MinidumpDescriptor& descriptor,
void context,
bool succeeded);

1. "breakpad/src/common/linux/linux_libc_support.cc" defined a set of functions with the prefix "my_" and they were conflicted with the function of mysql.

enum optimization

hi guys, could anyone help me developing this MDEV (mariadb, maybe usefull for twitter too, since it's a nice feature that optimize many queries using ENUM fields)

https://mariadb.atlassian.net/browse/MDEV-4419

it's a enum optimization without index (but can help index too)
example:
a field with => enum ('a','b') NOT NULL

WHERE enum = 'c' => rewrite to WHERE 0 (impossible where)
WHERE enum != 'c' => rewrite to enum IN ('','a','b') OR WHERE 1 (always true)
WHERE enum!='b' => rewrite to enum IN ('','a') (COND_OK)

check that we can use any operator (LIKE for example) the key is:
read all enum possible values, execute operator for each enum value, if we have all false return it's a impossible where, if we have all true we have a always true, if we don't have all true and all false we can rewrite to IN () operator and use very optimized index queries (using HASH or BTREE for example)

Salvaging an ARCHIVE table with archive_reader

While trying to recover a corrupted ARCHIVE table, my searching lead me here. Posting my findings in case it also helps someone in the future. (Note that archive_reader is not specific to this fork - this fork is just where I was lead.)

Patched code with:

diff --git a/storage/archive/CMakeLists.txt b/storage/archive/CMakeLists.txt
index 78fb95c0d2..799ad44a6f 100644
--- a/storage/archive/CMakeLists.txt
+++ b/storage/archive/CMakeLists.txt
@@ -16,3 +16,5 @@
 SET(ARCHIVE_SOURCES  azio.c ha_archive.cc ha_archive.h)
 MYSQL_ADD_PLUGIN(archive ${ARCHIVE_SOURCES} STORAGE_ENGINE LINK_LIBRARIES ${ZLIB_LIBRARY})
 
+MYSQL_ADD_EXECUTABLE(archive_reader archive_reader.c azio.c)
+TARGET_LINK_LIBRARIES(archive_reader mysys)
diff --git a/storage/archive/archive_reader.c b/storage/archive/archive_reader.c
index 2475418fef..7a3fd792c7 100644
--- a/storage/archive/archive_reader.c
+++ b/storage/archive/archive_reader.c
@@ -39,6 +39,8 @@ static char **default_argv;
 int opt_check, opt_force, opt_quiet, opt_backup= 0, opt_extract_frm;
 int opt_autoincrement;
 
+PSI_file_key arch_key_file_metadata, arch_key_file_data, arch_key_file_frm;
+
 int main(int argc, char *argv[])
 {
   unsigned int ret;

• Compiled with cmake && make archive_reader -j4.
• Shutdown MySQL/MariaDB server (should work with either; I do not think anyone has worked on the ARCHIVE engine for over a decade).
• Backup your mysql data dir in case something goes wrong.
• Rebuild the <table-name>.ARZ with archive_reader --backup path/to/table.ARZ new-file.ARZ.
• Replace <table-name>.ARZ with the new file.
• If <table-name>.ARN exists, delete it.
• Start up the db server.
• Before doing anything else, mysqldump the table.
  • The server thinks the table is valid, so even if corruption was not repaired, you should still be able to dump data up to the point where it had an issue. Can load this dump later, if need be.
• If there were no errors, hopefully everything is good to go.

We found ARCHIVE engine to be 6x smaller than compressed InnoDB (and 10x smaller than uncompressed), so still has value. If you too must stay with ARCHIVE, I recommend FLUSHing the table after completing any major write.

Feature merge request

After the release of twitter mysql on github, I noticed https://github.com/twitter/mysql/wiki/Table-Options -- to have pre-determined initial size for per-table tablespaces;
I have some code which should complement it pretty well.

Mainly the code deals with,
1. Use fallocate in os_file_set_size for initial creation of innodb files.
2. Use #1 for extension of ibdata1 files.
3. Use #1 for ibd files.
4. Ability to set increment size for extension of ibd files through a variable.

It starts from,
http://bazaar.launchpad.net/~raghavendra-prabhu/+junk/mysql-server-fallocate/revision/3547
upto
http://bazaar.launchpad.net/~raghavendra-prabhu/+junk/mysql-server-fallocate/revision/3550

(Commits after that deal with other issues).

Also, the code is merged and up-to-date with latest 5.6 bzr pull, also I have tested it myself and results have been pretty good.

I can send these with git-format-patch + git send-email if you need it.

Why mysqld binary file size auto change

OS: Centos6.5
MySQL 5.6.37
at first use mysql binary tar ball mysqld file size is same。but running with 3 day。mysqld size is change:
3 day ago:

find / -name mysqld | xargs ls -la
-rwxr-xr-x 1 mysql dba   87208139 Jun  3 05:30 /data/install_mysql_5.6/mysql/bin/mysqld
-rwxr-xr-x 1 mysql mysql 87208139 Jun  3 05:30 /media/database/mysql/bin/mysqld
-rwxr-xr-x 1 root  root  87208139 Oct 13 11:51 /usr/bin/mysqld
-rwxr-xr-x 1 root  root  87208139 Oct 13 11:51 /usr/local/mysql/bin/mysqld
-rwxr-xr-x 1 mysql mysql 87208139 Jun  3 05:30 /var/lib/mysql/bin/mysqld

now:

find / -name mysqld | xargs ls -la
-rwxr-xr-x 1 mysql dba   87208139 Jun  3 05:30 /data/install_mysql_5.6/mysql/bin/mysqld
-rwxr-xr-x 1 mysql mysql 87208139 Jun  3 05:30 /media/database/mysql/bin/mysqld
-rwxr-xr-x 1 root  root  87987531 Oct 13 11:51 /usr/bin/mysqld
-rwxr-xr-x 1 root  root  87208139 Oct 13 11:51 /usr/local/mysql/bin/mysqld
-rwxr-xr-x 1 mysql mysql 87208139 Jun  3 05:30 /var/lib/mysql/bin/mysqld

Is this Dead?

Is this fork still alive?

Build is failing from problems with sql yacc

mysql/sql/sql_yacc.cc: In function ‘int MYSQLparse()’:
mysql/sql/sql_yacc.cc:16004:30: error: too few arguments to function ‘int MYSQLlex(void_, void_)’
yychar = yylex (&yylval);

[2019-02-24T17:48:02+00:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process terminated by signal 9 (KILL)

Hi Expert
Iam using simple command to change root password after MySql installation ,But getting the Above error (signal 9 kill)

script "change password" do
interpreter "bash"
user "root"
cwd "/tmp"
code <<-EOH
#MYSQL
root_temp_pass=$(grep 'A temporary password' /mysql/log/mysqld.log |tail -1 |awk '{split($0,a,": "); print a[2]}')

#Login as root change password
mysql -uroot -p"$root_temp_pass" -Be "ALTER USER 'root'@'localhost' IDENTIFIED BY 'Czt732ck#';" --connect-expired-password
EOH
end