Comments (8)
That's interesting. It looks like the forked worker parent may also need to call #reopen on the Redis failover client. That's what the error being raised is indicating. Do you have a hook for that where you could call it?
On May 11, 2012, at 4:19 AM, Matt [email protected] wrote:
This is related to #13 but running on OSX to avoid hitting the deadlock.
I see the following stack trace when using the Client#reconnect method in my forked children (resque). I do not see the trace when I use Client#shutdown then manually recreate/reassign all the references to existing client instances. Perhaps it isn't resetting all state internally? The trace happens after my job runs, and from the trace I think it is happening in the worker parent, not the worker child nor the top level resque-pool process.
/Users/mconway/.rvm/gems/ruby-1.9.3-p125-falcon/gems/pry-exception_explorer-0.1.9/lib/pry-exception_explorer/core_ext.rb:46:in
'raise': You tried to use a connection inherited from another process [] You need to call reopen() after forking (Zookeeper::Exceptions::InheritedConnectionError) from /Users/mconway/.rvm/gems/ruby-1.9.3-p125-falcon/gems/zookeeper-1.0.4/ext/zookeeper_base.rb:145:in
block in assert_open'
from /Users/mconway/.rvm/rubies/ruby-1.9.3-p125-falcon/lib/ruby/1.9.1/monitor.rb:211:inmon_synchronize' from /Users/mconway/.rvm/gems/ruby-1.9.3-p125-falcon/gems/zookeeper-1.0.4/ext/zookeeper_base.rb:141:in
assert_open'
from /Users/mconway/.rvm/gems/ruby-1.9.3-p125-falcon/gems/zookeeper-1.0.4/lib/zookeeper/client_methods.rb:238:inassert_open' from /Users/mconway/.rvm/gems/ruby-1.9.3-p125-falcon/gems/zookeeper-1.0.4/lib/zookeeper/client_methods.rb:16:in
get'
from /Users/mconway/.rvm/gems/ruby-1.9.3-p125-falcon/gems/zk-1.4.0/lib/zk/client/base.rb:1040:incall_and_check_rc' from /Users/mconway/.rvm/gems/ruby-1.9.3-p125-falcon/gems/zk-1.4.0/lib/zk/client/base.rb:437:in
block in get'
from /Users/mconway/.rvm/gems/ruby-1.9.3-p125-falcon/gems/zk-1.4.0/lib/zk/event_handler.rb:294:inblock in setup_watcher!' from /Users/mconway/.rvm/gems/ruby-1.9.3-p125-falcon/gems/zk-1.4.0/lib/zk/event_handler.rb:236:in
block in synchronize'
from /Users/mconway/.rvm/rubies/ruby-1.9.3-p125-falcon/lib/ruby/1.9.1/monitor.rb:211:inmon_synchronize' from /Users/mconway/.rvm/gems/ruby-1.9.3-p125-falcon/gems/zk-1.4.0/lib/zk/event_handler.rb:236:in
synchronize'
from /Users/mconway/.rvm/gems/ruby-1.9.3-p125-falcon/gems/zk-1.4.0/lib/zk/event_handler.rb:274:insetup_watcher!' from /Users/mconway/.rvm/gems/ruby-1.9.3-p125-falcon/gems/zk-1.4.0/lib/zk/client/base.rb:1087:in
setup_watcher!'
from /Users/mconway/.rvm/gems/ruby-1.9.3-p125-falcon/gems/zk-1.4.0/lib/zk/client/base.rb:436:inget' from /Users/mconway/.rvm/gems/ruby-1.9.3-p125-falcon/bundler/gems/redis_failover-040d5926f179/lib/redis_failover/client.rb:293:in
fetch_nodes'
from /Users/mconway/.rvm/gems/ruby-1.9.3-p125-falcon/bundler/gems/redis_failover-040d5926f179/lib/redis_failover/client.rb:273:inblock in build_clients' from /Users/mconway/.rvm/rubies/ruby-1.9.3-p125-falcon/lib/ruby/1.9.1/monitor.rb:211:in
mon_synchronize'
from /Users/mconway/.rvm/gems/ruby-1.9.3-p125-falcon/bundler/gems/redis_failover-040d5926f179/lib/redis_failover/client.rb:271:inbuild_clients' from /Users/mconway/.rvm/gems/ruby-1.9.3-p125-falcon/bundler/gems/redis_failover-040d5926f179/lib/redis_failover/client.rb:215:in
dispatch'
from /Users/mconway/.rvm/gems/ruby-1.9.3-p125-falcon/bundler/gems/redis_failover-040d5926f179/lib/redis_failover/client.rb:82:inblock (2 levels) in <class:Client>' from /Users/mconway/.rvm/gems/ruby-1.9.3-p125-falcon/gems/redis-namespace-1.0.3/lib/redis/namespace.rb:213:in
method_missing'
from /Users/mconway/.rvm/gems/ruby-1.9.3-p125-falcon/bundler/gems/resque-fcaf987d0724/lib/resque/worker.rb:430:injob' from /Users/mconway/.rvm/gems/ruby-1.9.3-p125-falcon/bundler/gems/resque-fcaf987d0724/lib/resque/worker.rb:362:in
unregister_worker'
from /Users/mconway/.rvm/gems/ruby-1.9.3-p125-falcon/bundler/gems/resque-fcaf987d0724/lib/resque/worker.rb:145:inensure in work' from /Users/mconway/.rvm/gems/ruby-1.9.3-p125-falcon/bundler/gems/resque-fcaf987d0724/lib/resque/worker.rb:145:in
work'
from /Users/mconway/.rvm/gems/ruby-1.9.3-p125-falcon/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:332:inblock in spawn_worker!' from /Users/mconway/.rvm/gems/ruby-1.9.3-p125-falcon/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:327:in
fork'
from /Users/mconway/.rvm/gems/ruby-1.9.3-p125-falcon/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:327:inspawn_worker!' from /Users/mconway/.rvm/gems/ruby-1.9.3-p125-falcon/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:287:in
block in spawn_missing_workers_for'
from /Users/mconway/.rvm/gems/ruby-1.9.3-p125-falcon/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:287:intimes' from /Users/mconway/.rvm/gems/ruby-1.9.3-p125-falcon/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:287:in
spawn_missing_workers_for'
from /Users/mconway/.rvm/gems/ruby-1.9.3-p125-falcon/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:273:inblock in maintain_worker_count' from /Users/mconway/.rvm/gems/ruby-1.9.3-p125-falcon/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:271:in
each'
from /Users/mconway/.rvm/gems/ruby-1.9.3-p125-falcon/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:271:inmaintain_worker_count' from /Users/mconway/.rvm/gems/ruby-1.9.3-p125-falcon/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:191:in
start'
from /Users/mconway/.rvm/gems/ruby-1.9.3-p125-falcon/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:65:inrun' from /Users/mconway/.rvm/gems/ruby-1.9.3-p125-falcon/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool/tasks.rb:17:in
block (2 levels) in <top (required)>'
from /Users/mconway/.rvm/gems/ruby-1.9.3-p125-falcon/gems/rake-0.9.2.2/lib/rake/task.rb:205:incall' from /Users/mconway/.rvm/gems/ruby-1.9.3-p125-falcon/gems/rake-0.9.2.2/lib/rake/task.rb:205:in
block in execute'
from /Users/mconway/.rvm/gems/ruby-1.9.3-p125-falcon/gems/rake-0.9.2.2/lib/rake/task.rb:200:ineach' from /Users/mconway/.rvm/gems/ruby-1.9.3-p125-falcon/gems/rake-0.9.2.2/lib/rake/task.rb:200:in
execute'
from /Users/mconway/.rvm/gems/ruby-1.9.3-p125-falcon/gems/rake-0.9.2.2/lib/rake/task.rb:158:inblock in invoke_with_call_chain' from /Users/mconway/.rvm/rubies/ruby-1.9.3-p125-falcon/lib/ruby/1.9.1/monitor.rb:211:in
mon_synchronize'
from /Users/mconway/.rvm/gems/ruby-1.9.3-p125-falcon/gems/rake-0.9.2.2/lib/rake/task.rb:151:ininvoke_with_call_chain' from /Users/mconway/.rvm/gems/ruby-1.9.3-p125-falcon/gems/rake-0.9.2.2/lib/rake/task.rb:144:in
invoke'
from /Users/mconway/.rvm/gems/ruby-1.9.3-p125-falcon/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool/cli.rb:110:instart_pool' from /Users/mconway/.rvm/gems/ruby-1.9.3-p125-falcon/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool/cli.rb:16:in
run'
from /Users/mconway/.rvm/gems/ruby-1.9.3-p125-falcon/bundler/gems/resque-pool-774e1e384d5c/bin/resque-pool:5:in<top (required)>' from /Users/mconway/.rvm/gems/ruby-1.9.3-p125-falcon/bin/resque-pool:19:in
load'
from /Users/mconway/.rvm/gems/ruby-1.9.3-p125-falcon/bin/resque-pool:19:in `
Reply to this email directly or view it on GitHub:
#15
from redis_failover.
Given that all else stays the same, and I just switch shutdown+new for reopen, it should behave the same because that was the intended behavior of reopen, no?
I have code like this:
Resque::Pool.after_prefork do |job|
# Reset redis failover clients in worker parent so that they
# don't conflict (deadlock) with the ones in resque-pool daemon
RedisFactory.reconnect
redis_factory_chained_after_fork_hook = Resque.after_fork
Resque.after_fork do |job|
# Reset redis failover clients in worker child so that they
# don't conflict (deadlock) with the ones in worker parent
RedisFactory.reconnect
redis_factory_chained_after_fork_hook.call(job) if redis_factory_chained_after_fork_hook
end
end
and I switch out the RedisFactory.reconnect to have one of two behaviors:
def self.reconnect
logger.debug "RedisFactory.reconnect start"
synchronize do
@@clients.each do |name, client|
client.reopen if client.respond_to?(:reopen)
end
@@clients.clear
end
logger.debug "RedisFactory.reconnect complete"
end
or
def self.reconnect
logger.debug "RedisFactory.reconnect start"
# Disconnect all redis clients (specifically for RedisFailover)
self.shutdown
# Reconnect resque's redis
Resque.redis = RedisFactory.connect(:resque)
# Reconnect Rails.cache redis
Rails.application.config.cache_store = RedisCacheStore.new(RedisFactory.connect(:cache))
silence_warnings { Object.const_set "RAILS_CACHE", ActiveSupport::Cache.lookup_store(Rails.application.config.cache_store) }
logger.debug "RedisFactory.reconnect complete"
end
def self.shutdown
logger.debug "RedisFactory.shutdown start"
synchronize do
@@clients.each do |name, client|
client.shutdown if client.respond_to?(:shutdown)
end
@@clients.clear
end
logger.debug "RedisFactory.shutdown complete"
end
from redis_failover.
Actually yes, I think I see the issue. Instead of closing down the ZK client and recreating it, I should just need to call #reopen on it. It's 4:42AM here so I will try this in a few hours. If you'd like to experiment, though you could fork the project and change #reopen in client.rb to be this:
def reopen
purge_clients
@zk.reopen if @zk
build_clients
end
On May 11, 2012, at 4:31 AM, Matt [email protected] wrote:
Given that all else stays the same, and I just switch shutdown+new for reopen, it should behave the same because that was the intended behavior of reopen, no?
I have code like this:Resque::Pool.after_prefork do |job|
# Reset redis failover clients in worker parent so that they # don't conflict (deadlock) with the ones in resque-pool daemon RedisFactory.reconnect redis_factory_chained_after_fork_hook = Resque.after_fork Resque.after_fork do |job| # Reset redis failover clients in worker child so that they # don't conflict (deadlock) with the ones in worker parent RedisFactory.reconnect redis_factory_chained_after_fork_hook.call(job) if redis_factory_chained_after_fork_hook end
end
and I switch out the RedisFactory.reconnect to have one of two behaviors:
def self.reconnect
logger.debug "RedisFactory.reconnect start"
synchronize do
@@clients.each do |name, client|
client.reopen if client.respond_to?(:reopen)
end
@@clients.clear
end
logger.debug "RedisFactory.reconnect complete"
endor
def self.reconnect
logger.debug "RedisFactory.reconnect start"
# Disconnect all redis clients (specifically for RedisFailover)
self.shutdown# Reconnect resque's redis Resque.redis = RedisFactory.connect(:resque) # Reconnect Rails.cache redis Rails.application.config.cache_store = RedisCacheStore.new(RedisFactory.connect(:cache)) silence_warnings { Object.const_set "RAILS_CACHE", ActiveSupport::Cache.lookup_store(Rails.application.config.cache_store) } logger.debug "RedisFactory.reconnect complete"
end
def self.shutdown
logger.debug "RedisFactory.shutdown start"
synchronize do
@@clients.each do |name, client|
client.shutdown if client.respond_to?(:shutdown)
end
@@clients.clear
end
logger.debug "RedisFactory.shutdown complete"
end
Reply to this email directly or view it on GitHub:
#15 (comment)
from redis_failover.
No worries, go to sleep :)
For when you wake up, the alternate reopen exhibited the same problem
from redis_failover.
Hey @slyphon, this could be related to what you are currently fixing in ZK, right?
On May 11, 2012, at 5:03 AM, Matt [email protected] wrote:
No worries, go to sleep :)
For when you wake up, the alternate reopen exhibited the same problem
Reply to this email directly or view it on GitHub:
#15 (comment)
from redis_failover.
It might be, but if this is truly happening in the parent, then that's a bug in zookeeper, but i'm a little skeptical that's where it's getting triggered. It seems like this might be getting triggered from the paranoid "check the node every N seconds loop"?
Anyway, check the pull request, that's how i would respond to this exception.
from redis_failover.
Okay, I just merged @slyphon's pull request and changed the code to use #reopen. @wr0ngway, can you give this a shot? Thanks!
from redis_failover.
Fixed
from redis_failover.
Related Issues (20)
- Full Failure of server HOT 1
- Undefined method 'job' for main object with Resque HOT 4
- Occasionally stale server selection HOT 2
- start-stop-daemon to start redis_failover how to save log
- Add informative procline
- passing in a zk and having it close before redis-failover can trigger an infinite loop
- Prevent reads on slaves HOT 3
- Add subscribing part of Pub/Sub ops to REDIS_READ_OPS
- If ZK has an old master, redis_failover will use it even though the redis instance does not exist HOT 2
- License missing from gemspec
- stuck in "Failed to discover master node" HOT 1
- Shutdown fails with Ruby 2.0 HOT 3
- Error bringing back up failed master HOT 8
- Support Ruby 2.0 HOT 2
- Support Redis 2.8 HOT 1
- Allow Manual Failover in cluster bootstrap
- Redis-failover start up error HOT 1
- Starting redis_node_manager connection timeouts HOT 3
- State of redis_failover HOT 3
- ruby1.9 下启动报错
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from redis_failover.