Feature Request: Abstract gathering and notification of servers in client about redis_failover

[ryanlecompte]

This is something that I definitely want to investigate. Over on the sf-ruby mailing list, there is a thread about what happens when there are network partitions, etc. In edge conditions it's possible that two masters could be promoted in flappy IP networks. I think ZooKeeper could address this, although I haven't used it before. I'd really love to eliminate this situation by plugging in ZooKeeper or something similar. What do you think would be involved for this change? Would this make the redis_failover server component obsolete?

[eric]

I would definitely suggest playing around with and deploying zookeeper in production.

I was able to go from 0-to-production in a weekend. I'm only using it for a couple little things that I was previously using redis for, but it makes me feel so much better about life to know that I don't have to worry about a master-slave failover situation for service discovery/configuration data.

I'm a huge fan of zookeeper now.

So... in the simple solution, I don't believe it would obsolete the server component at all — you still need code somewhere to decide who is the master and do the monitoring.

There are a couple cool things that zookeeper could help with, though:

1. Provide a way that you can run multiple server components on different systems and have them decide (by an election in zookeeper) which one should actually do the monitoring. The other servers would just wait until the server component who had won the election died and then one of them would win another election and take over.
2. Embed the server component in the redis client and allow all of the clients to participate in the election above. The benefit of doing this is that it would remove the need to run Yet Another Thing, but the downside is that it would be more prone to problems in a given ruby process messing things up (imagine a 60 second mysql query blocking MRI for that whole time when a failover happens).

I would definitely think option 1 would be a great start.

[ryanlecompte]

Closing this issue for now, since communication between RedisFailover::Client and RedisFailover::NodeManager has been implemented with ZooKeeper and heavy use of ZK watchers. Thanks for the original suggestion - you've been called out in the 'Acknowledgements' section of the main README. :)