If ZK has an old master, redis_failover will use it even though the redis instance does not exist about redis_failover

[ryanlecompte]

Thanks for reporting this, @petelacey. This is the first time I've seen this issue, so my guess is that most folks are using redis_failover in an environment where the ip addresses are more stable than yours. I don't have the time to provide a fix for it now, unfortunately. If you'd like, feel free to provide a fix and test it in your environment, then submit a pull request. Thanks!

[arohter]

We looked/ran into this, and determined that the current behavior is the safe and sane methodology. We run on AWS, so we are familiar with cloudy environments.

1. You ran into this issue because you are running only a single node manager. You need more than one manager to maintain healthy monitoring and quorum in a clustered env. We run a manager on each redis node.

2. Node state stored in zk is considered the word of god; it's the only trusted canonical source of truth, especially when running multiple node managers. From the point of view of a single manager, there's no way to automatically determine whether a failed master connection attempt is due to bad/old config, or simply a network partition. We could incur data loss if we reconfigure the cluster based on just one snapshot, since slaveof commands drop all existing data.

3. You can resolve this cluster "deadlock" by simply triggering a manual failover. We lose some automation by making humans pick the proper master in these edge cases, but downtime is preferable to wholesale data loss.