This isn't really a bug report yet, I'm rather looking for hints and pointers. I have a system where some values transported to/from Redis seem to get corrupted. Sometimes.
Case #1: I have a lua script that does this:
1384505478.269537 [0 5.9.23.176:50292] "EVALSHA" "c9ff551721953469eb600c6a9057b5d6ace939ed" "2" "customer" "186kvcq0a-LOrFzrozoRBmVizb"
1384505478.269571 [0 lua] "get" "t2r:customer:186kvcq0a-LOrFzrozoRBmVizb"
[...]
the script in question essentially does this:
local t2r_key = 't2r:' .. _:customer .. ':' .. _:tid
local ruid = redis.call('get', t2r_key)
local created = false
[...]
return {ruid, created}
the elided code doesn't matter, it's an "if" that is not taken in this case. The lua script basically just performs a single GET here and returns the value along with a boolean (false) flag.
Problem is, sometimes the values returned sometimes make no sense. When the problem occurs, Clojure logging code prints the two received values as something similar to "[ 2". Investigating this closer shows that the "ruid" value is "\x00>\x03\b\n\x00[", while it should be:
redis 127.0.0.1:6379> get t2r:customer:186kvcq0a-LOrFzrozoRBmVizb
"165854"
To give you an idea of "sometimes", in this case it happens in about 1.5% of calls.
Case #2: There is also another problem, which looks related. It is in a completely different section of the code, does not involve lua scripting at all, and does not involve reading anything from redis. This is stats code that only stores values (strings, in this case), which arrive to redis in this form:
1384505479.011301 [0 5.9.23.176:50420] "ZINCRBY" "stats:customer:s:2013-11-15" "1" "\x00>\x01\x00\x03"
The "\x00>\x01\x00\x03" should normally be a human-readable string. My Clojure logs show no such corruption there, it's only visible in redis monitor.
Things of note:
- "\x00>\x01\x00\x03" and "\x00>\x03\b\n\x00[" seem to be special values, I see those two all the time, and haven't seen anything else,
- this only happens sometimes. It is not easily reproducible, although my logs are full of occurences of the problem,
- I'm not saying it's a Carmine bug, but rather looking for pointers at the moment,
- perhaps the contents of the strings will ring a bell?
Software involved: Redis 2.6.16, Clojure 1.5.1, Carmine 1.12.0.
Any ideas?