Comments (1)
I've seen this before when the slave dies and then comes back and rereads its redis.conf file that tells it that it is a slave. This is intended behavior, because the node manager is just being conservative and doesn't want to promote a node to master if it is the only node that it sees.
On Dec 1, 2013, at 8:54 AM, Christian Meyer [email protected] wrote:
We're running a setup of 3 zookeeper nodes and 2 redis nodes. On each of those 2 redis nodes, there's an instance of redis_node_manager running.
If the original redis master instance for any reason disappears (kill, crash, reboot), the other original slave redis node correctly becomes the new master node.
If that instance now, for any reason crashes as well and is not recovered within the configured retries but later, redis_node_manager gets into a dead loop.2013-12-01 16:36:07 UTC RedisFailover 5425 ERROR: Failed to promote a new master after 3 attempts.
2013-12-01 16:36:13 UTC RedisFailover 5425 INFO: Monitoring redis nodes at (172.16.154.152:6379, 172.16.154.153:6379)
2013-12-01 16:36:13 UTC RedisFailover 5425 INFO: Waiting to become master Node Manager ...
2013-12-01 16:36:13 UTC RedisFailover 5425 INFO: Acquired master Node Manager lock.
2013-12-01 16:36:13 UTC RedisFailover 5425 INFO: Configured node strategy RedisFailover::NodeStrategy::Single
2013-12-01 16:36:13 UTC RedisFailover 5425 INFO: Configured failover strategy RedisFailover::FailoverStrategy::Latency
2013-12-01 16:36:13 UTC RedisFailover 5425 INFO: Required Node Managers to make a decision: 1
2013-12-01 16:36:13 UTC RedisFailover 5425 INFO: Master from existing znode config: none
2013-12-01 16:36:13 UTC RedisFailover 5425 WARN: Failed to discover master node: # In order to ensure a safe startup, redis_failover requires that all redis nodes be accessible, and only a single node indicating that it's the master. In order to fix this, you can perform a manual failover via redis_failover, or manually fix the individual redis servers. This discovery process will retry in 5s.
2013-12-01 16:36:18 UTC RedisFailover 5425 INFO: Master from existing znode config: none
2013-12-01 16:36:18 UTC RedisFailover 5425 WARN: Failed to discover master node: # In order to ensure a safe startup, redis_failover requires that all redis nodes be accessible, and only a single node indicating that it's the master. In order to fix this, you can perform a manual failover via redis_failover, or manually fix the individual redis servers. This discovery process will retry in 5s.
2013-12-01 16:36:23 UTC RedisFailover 5425 INFO: Master from existing znode config: none
2013-12-01 16:36:23 UTC RedisFailover 5425 WARN: Failed to discover master node: # In order to ensure a safe startup, redis_failover requires that all redis nodes be accessible, and only a single node indicating that it's the master. In order to fix this, you can perform a manual failover via redis_failover, or manually fix the individual redis servers. This discovery process will retry in 5s.
...
Is this intended behavior? If the master is not recoverable even announcing the original slave by slaveof no one to master did not resolve that issue.
Only replacing the defected master completely and reassigning the original IP and reconstructing the original master/slave setup did recover the node_manager.
I thought, it should just take one of the known nodes and promote it to master ?—
Reply to this email directly or view it on GitHub.
from redis_failover.
Related Issues (20)
- Full Failure of server HOT 1
- Undefined method 'job' for main object with Resque HOT 4
- Occasionally stale server selection HOT 2
- start-stop-daemon to start redis_failover how to save log
- Add informative procline
- passing in a zk and having it close before redis-failover can trigger an infinite loop
- Prevent reads on slaves HOT 3
- Add subscribing part of Pub/Sub ops to REDIS_READ_OPS
- If ZK has an old master, redis_failover will use it even though the redis instance does not exist HOT 2
- License missing from gemspec
- Shutdown fails with Ruby 2.0 HOT 3
- Error bringing back up failed master HOT 8
- Support Ruby 2.0 HOT 2
- Support Redis 2.8 HOT 1
- Allow Manual Failover in cluster bootstrap
- Redis-failover start up error HOT 1
- Starting redis_node_manager connection timeouts HOT 3
- State of redis_failover HOT 3
- ruby1.9 下启动报错
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from redis_failover.