Weird zookeeper error happening every second or so in the zookeeper logs about redis_failover

[ryanlecompte]

Hello Max! Which version of redis_failover are you using? Do you have more than one redis_node_manager daemon running?

[maxjustus]

Hey! 0.9.7 and I'm running 3 instances of redis_node_manager. One per-server.

[ryanlecompte]

Thanks. It looks like the log message is happening due to the use of an ephemeral lock to ensure only a single redis_node_manager is the 'primary' at any given moment. @slyphon, would you expect this log message to be logged every second or so in the ZooKeeper logs? Is it normal given the way the ephemeral lock is implemented in ZK 1.6?

[ryanlecompte]

@maxjustus, I tracked this down with @slyphon. Each redis_node_manager that's not the current primary will attempt to acquire the exclusive lock every couple seconds. Each time it attempts this and fails, a log entry will be written to the ZK logs. I will need to restructure how we acquire the lock in Redis::Failover::NodeManager#with_lock so that this doesn't happen.

[maxjustus]

Ah okay. Thanks for figuring it out!

[maxjustus]

For the time being I just piped the zookeeper logging output through grep before shipping it to papertrail.

( /usr/local/bin/zkServer start-foreground & echo $! >/var/run/zookeeper.pid ) | grep -v 'Directory not empty for /_zklocking/__redis_failover_nodes_lock' | /usr/bin/logger -t zookeeper &

[ryanlecompte]

I just checked in a fix that should address this issue in master. @maxjustus, feel free to give latest master a shot and see if the errors go away for you. I'll put out a release later tonight if all goes well.

Thanks!

[maxjustus]

Awesome! That fix did the trick. Thanks!

[ryanlecompte]

Sweet! I'll release a new version soon.

[ryanlecompte]

Just released 0.9.7.1 with this fix.