stuck in "Failed to discover master node" about redis_failover

[ryanlecompte]

I've seen this before when the slave dies and then comes back and rereads its redis.conf file that tells it that it is a slave. This is intended behavior, because the node manager is just being conservative and doesn't want to promote a node to master if it is the only node that it sees.

On Dec 1, 2013, at 8:54 AM, Christian Meyer [email protected] wrote:

We're running a setup of 3 zookeeper nodes and 2 redis nodes. On each of those 2 redis nodes, there's an instance of redis_node_manager running.
If the original redis master instance for any reason disappears (kill, crash, reboot), the other original slave redis node correctly becomes the new master node.
If that instance now, for any reason crashes as well and is not recovered within the configured retries but later, redis_node_manager gets into a dead loop.

2013-12-01 16:36:07 UTC RedisFailover 5425 ERROR: Failed to promote a new master after 3 attempts.
2013-12-01 16:36:13 UTC RedisFailover 5425 INFO: Monitoring redis nodes at (172.16.154.152:6379, 172.16.154.153:6379)
2013-12-01 16:36:13 UTC RedisFailover 5425 INFO: Waiting to become master Node Manager ...
2013-12-01 16:36:13 UTC RedisFailover 5425 INFO: Acquired master Node Manager lock.
2013-12-01 16:36:13 UTC RedisFailover 5425 INFO: Configured node strategy RedisFailover::NodeStrategy::Single
2013-12-01 16:36:13 UTC RedisFailover 5425 INFO: Configured failover strategy RedisFailover::FailoverStrategy::Latency
2013-12-01 16:36:13 UTC RedisFailover 5425 INFO: Required Node Managers to make a decision: 1
2013-12-01 16:36:13 UTC RedisFailover 5425 INFO: Master from existing znode config: none
2013-12-01 16:36:13 UTC RedisFailover 5425 WARN: Failed to discover master node: # In order to ensure a safe startup, redis_failover requires that all redis nodes be accessible, and only a single node indicating that it's the master. In order to fix this, you can perform a manual failover via redis_failover, or manually fix the individual redis servers. This discovery process will retry in 5s.
2013-12-01 16:36:18 UTC RedisFailover 5425 INFO: Master from existing znode config: none
2013-12-01 16:36:18 UTC RedisFailover 5425 WARN: Failed to discover master node: # In order to ensure a safe startup, redis_failover requires that all redis nodes be accessible, and only a single node indicating that it's the master. In order to fix this, you can perform a manual failover via redis_failover, or manually fix the individual redis servers. This discovery process will retry in 5s.
2013-12-01 16:36:23 UTC RedisFailover 5425 INFO: Master from existing znode config: none
2013-12-01 16:36:23 UTC RedisFailover 5425 WARN: Failed to discover master node: # In order to ensure a safe startup, redis_failover requires that all redis nodes be accessible, and only a single node indicating that it's the master. In order to fix this, you can perform a manual failover via redis_failover, or manually fix the individual redis servers. This discovery process will retry in 5s.
...
Is this intended behavior? If the master is not recoverable even announcing the original slave by slaveof no one to master did not resolve that issue.
Only replacing the defected master completely and reassigning the original IP and reconstructing the original master/slave setup did recover the node_manager.
I thought, it should just take one of the known nodes and promote it to master ?

—
Reply to this email directly or view it on GitHub.