dawnbreaks / mysql2redis

A high performance mysql udf to sync the newly modified/inserted data from mysql to redis cache.

mysql2redis

Redis udf to sync newly modified data from mysql to redis cache.

When you use redis to cache data only(not persistent), you will face the issue that how to sync the newly modified data from mysql to the redis cache? This project aims to provide a high performance and stable redis udf to solve this issue.

Dependencies

please download the dependencies below and compile/install it properly :

• apr-1.4.6(http://apr.apache.org/download.cgi)
• apr-util-1.5.2(http://apr.apache.org/download.cgi)
• hiredis(https://github.com/redis/hiredis)
• lib_mysqludf_json(https://github.com/mysqludf/lib_mysqludf_json)

Compile

run make from the src dir directly.

make

Install redis udf

please make sure that the lib_mysqludf_redis_v2.so has been put into the mysql plugin dir. By the way, you can examine where is the mysql plugin dir by run ''' mysql_config --plugindir '''. and then connect to your mysql server, run the following command to install the the redis udf.

DROP FUNCTION IF EXISTS redis_servers_set_v2;
DROP FUNCTION IF EXISTS redis_command_v2;
DROP FUNCTION IF EXISTS free_resources;

CREATE FUNCTION redis_servers_set_v2 RETURNS int SONAME "lib_mysqludf_redis_v2.so";
CREATE FUNCTION redis_command_v2 RETURNS int SONAME "lib_mysqludf_redis_v2.so";
CREATE FUNCTION free_resources RETURNS int SONAME "lib_mysqludf_redis_v2.so";

Test redis udf

connect to your mysql server, run the following command to test the the redis udf.

select redis_command_v2("lpush","crmInboxEvents11",json_object(json_members("op","insert","value","valuettt")));

select redis_servers_set_v2("192.168.0.118",6379);
select redis_command_v2("lpush","crmInboxEvents11",json_object(json_members("op","insert","value","valuettt")));

select redis_command_v2("hset","hkey","hfield",json_object(json_members("op","insert","value","valuettt")));


select free_resources();
select redis_servers_set_v2("192.168.0.118",6379);

What's more

you should create a trigger which will lpush the newly modified data to redis list juste as the following example:

DELIMITER $$
CREATE TRIGGER insert_trigger AFTER INSERT ON email_folder
FOR EACH ROW BEGIN
 SET @ret=
  	 redis_command_v2("lpush","crmInboxEvents",
 						  json_object
                           (
 						    	json_members
 								(
 										"op",
 										"insert",
 										"value",
 										json_object
 										(
 											NEW.Id as "id",NEW.type as "type",
 											NEW.mailserver_id as "mailserverId",NEW.sender as "sender",
 											NEW.sender_name as "senderName",NEW.recevier as "recevier",
 											NEW.replyto as "replyto",NEW.bbemails as "bbemails",
 											NEW.ccemails as "ccemails",NEW.subject as "subject"
                                         )
 									)
 					    	)
 				      );
END$$
DELIMITER ;

Finally

Now all update events for that mysql table has been transfer to a redis queue. You need to start a thread(or process) to handle these update events and update the redis cache properly.

For example, you can start a java process to poll the redis queue , and pop the events, then handle it and update the redis cache properly.

Performance test

In my poor labtop, 10,000 row data were synced into redis in 0.4 second.

======== Please feel free to contact me([email protected]) if you have any questions.

mysql2redis's Issues

How to exe redis_servers_set_v2("127.0.0.1", 6379); function automatic while mysql restart?

every time mysql restart,i have to login mysql client and have manual exe redis_servers_set_v2("127.0.0.1", 6379); function everytime,can you tell me how could i exe this automatic?

the openfile and the sredisContext are not release ?

when the free_resources is called,
I think you want to close the pFile and redisFree sredisContext ?

Working on my own SQL to Redis caching lib, would appreciate feedback.

Check out the unstable branch to see the latest features:
https://github.com/DGKSK8LIFE/redisql

How to use this in my password protected redis

hello,

I used password in redis for security reason so how I connect mysql with redis by using password.
please help...

Thanks

How to check the data in Redis?

Hi,
I'm new to Redis and MySQL. Could you tell me how to check the data entered in Redis from Mysql?

LTRIM command not support?

i use this code, but list is not trimed:

SET v_redis_ret=redis_command_v2('LTRIM', 'UserReadingRecord4', 0, 4);

lpush twice

DELIMITER $$
CREATE TRIGGER user_consult_insert_trigger AFTER INSERT ON user_consult
FOR EACH ROW BEGIN
set @id = redis_servers_set_v2("127.0.0.1", 6369);
set @ret = redis_command_v2("", 0, "lpush",
concat("user-consultation-list-", NEW.user_id),
concat(NEW.id)
);
set @ret = redis_command_v2("", 0, "lpush",
concat("consultant-consultation-list-", NEW.consultant_id),
concat(NEW.id)
);
set @ret = redis_command_v2("", 0, "set",
concat("user_consult-", NEW.id),
json_object(
NEW.id as "id",
NEW.user_id as "user_id",
NEW.consultant_id as "consultant_id",
NEW.service_id as "service_id",
NEW.service_type as "service_type",
NEW.title as "title",
NEW.content as "content",
NEW.ctime as "ctime",
NEW.reply_num as "reply_num"
)
);
END$$
DELIMITER ;

the code above is my trigger.this trigger always insert primary id into user-consultation-list* twice

SOMETHING WRONG WITH THE MAKE COMMAND

base on MySQL-server-5.6.32-1.linux_glibc2.5.i386.rpm , when I make the mysql2redis,the below message showed:

[root@localhost mysql2redis]# make
gcc -Werror -O2 -g /usr/bin/mysql_config --include -I/usr/local/include -I/usr/local/apr/include -I./ -I. -fPIC -shared -rdynamic lib_mysqludf_redis.c -lhiredis -L"/usr/lib64/mysql/plugin" -L/usr/local/apr/lib -lapr-1 -laprutil-1 -o lib_mysqludf_redis_v2.so
cc1: warnings being treated as errors
In file included from lib_mysqludf_redis.h:26,
from lib_mysqludf_redis.c:5:
/usr/include/mysql/m_string.h: In function ‘skip_trailing_space’:
/usr/include/mysql/m_string.h:261: error: cast from pointer to integer of different size
/usr/include/mysql/m_string.h:261: error: cast to pointer from integer of different size
/usr/include/mysql/m_string.h:263: error: cast from pointer to integer of different size
/usr/include/mysql/m_string.h:263: error: cast to pointer from integer of different size
/usr/include/mysql/m_string.h:265: error: cast from pointer to integer of different size
make: *** [compile] Error 1

the function ‘skip_trailing_space’ like below:

static inline const uchar *skip_trailing_space(const uchar *ptr,size_t len)
{
const uchar *end= ptr + len;

if (len > 20)
{
const uchar *end_words= (const uchar *)(intptr)
(((ulonglong)(intptr)end) / SIZEOF_INT * SIZEOF_INT);
const uchar *start_words= (const uchar *)(intptr)
((((ulonglong)(intptr)ptr) + SIZEOF_INT - 1) / SIZEOF_INT * SIZEOF_INT);

DBUG_ASSERT(((ulonglong)(intptr)ptr) >= SIZEOF_INT);
if (end_words > ptr)
{
  while (end > end_words && end[-1] == 0x20)
    end--;
  if (end[-1] == 0x20 && start_words < end_words)
    while (end > start_words && ((unsigned *)end)[-1] == SPACE_INT)
      end -= SIZEOF_INT;
}

}
while (end > ptr && end[-1] == 0x20)
end--;
return (end);
}

THANKS FOR HELP.

when to support more args like redis "hset" command

or is there a plan to support it?

when i use redis_command_v2 in cursor, the data count is wrong

BEGIN
DECLARE done INT DEFAULT FALSE;
DECLARE v_id INT;
DECLARE v_ret INT;
DECLARE cur CURSOR FOR SELECT u.user_id FROM kid_users u WHERE u.latest_login_time>=DATE(DATE_ADD(NOW(),INTERVAL -1 DAY)) AND EXISTS(SELECT * FROM kid_child_info WHERE user_id=u.user_id AND gender IN(0,1)) ORDER BY u.latest_login_time DESC;
DECLARE CONTINUE HANDLER FOR NOT FOUND SET done = TRUE;

OPEN cur;
read_loop: LOOP
	FETCH cur INTO v_id;
	IF done THEN
		LEAVE read_loop;
	END IF;

	SET v_ret=redis_command_v2('LPUSH', 'calcReadingRecommendForYou', CONCAT(v_id));
END LOOP;
CLOSE cur;

END

undefined symbol: info

os : centos-5.6
jemalloc: jemalloc-3.5.1
apr: apr-1.4.6
apr-util: apr-util-1.5.2
hiredis: hiredis-master
#1126 - Can't open shared library 'lib_mysqludf_redis_v2.so' (errno: 0 /opt/mysql-5.5.15/lib/plugin/lib_mysqludf_redis_v2.so: undefined symbol: info)