Comments (22)
ryanlecompte
commented on August 19, 2024
This is interesting. I don't use resque so I actually haven't run into this specific issue before. I will definitely dig into it though and try to reproduce it. It may be related to forking and the ZK client.
Could please provide me with a simple resque worker sample that demonstrates the problem?
Also, have you been able to successfully use the client from other parts of your code?
On May 10, 2012, at 3:57 AM, Matt [email protected] wrote:
This may not be the fault of redis_failover, but all else being the same, when I have it enabled, my resque workers lock up on start. Here are a couple thread dumps - in the first, the worker doesn't get past changing the Starting in the procline, in the second, it got to the "Waiting" procline, but never picks up jobs. I used gdb.rb to get the traces.
"Starting" procline:
[
'"
[0] [
[ 0] "(eval):1:inblock in require_with_backports'", [ 1] "(eval):1:incollect'",
[ 2] "(eval):1:inrequire_with_backports'", [ 3] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/backports-2.5.3/lib/backports/tools.rb:314:inrequire'",
[ 4] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/backports-2.5.3/lib/backports/tools.rb:314:inrequire_with_backports'", [ 5] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/activesupport-3.2.2/lib/active_support/dependencies.rb:251:inblock in require'",
[ 6] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/activesupport-3.2.2/lib/active_support/dependencies.rb:236:inload_dependency'", [ 7] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/activesupport-3.2.2/lib/active_support/dependencies.rb:251:inrequire'",
[ 8] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/activesupport-3.2.2/lib/active_support/dependencies.rb:359:inrequire_or_load'", [ 9] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/activesupport-3.2.2/lib/active_support/dependencies.rb:502:inload_missing_constant'",
[10] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/activesupport-3.2.2/lib/active_support/dependencies.rb:192:inblock in const_missing'", [11] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/activesupport-3.2.2/lib/active_support/dependencies.rb:190:ineach'",
[12] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/activesupport-3.2.2/lib/active_support/dependencies.rb:190:inconst_missing'", [13] "/mnt/appleton-matt/releases/20120510001131/config/initializers/resque.rb:48:inreserve'",
[14] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-fcaf987d0724/lib/resque/worker.rb:119:inblock in work'", [15] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-fcaf987d0724/lib/resque/worker.rb:116:inloop'",
[16] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-fcaf987d0724/lib/resque/worker.rb:116:inwork'", [17] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:332:inblock in spawn_worker!'",
[18] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:327:infork'", [19] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:327:inspawn_worker!'",
[20] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:287:inblock in spawn_missing_workers_for'", [21] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:287:intimes'",
[22] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:287:inspawn_missing_workers_for'", [23] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:273:inblock in maintain_worker_count'",
[24] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:271:ineach'", [25] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:271:inmaintain_worker_count'",
[26] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:191:instart'", [27] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:65:inrun'",
[28] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool/tasks.rb:17:inblock (2 levels) in <top (required)>'", [29] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:205:incall'",
[30] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:205:inblock in execute'", [31] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:200:ineach'",
[32] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:200:inexecute'", [33] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:158:inblock in invoke_with_call_chain'",
[34] "/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/ruby/1.9.1/monitor.rb:211:inmon_synchronize'", [35] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:151:ininvoke_with_call_chain'",
[36] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:144:ininvoke'", [37] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool/cli.rb:110:instart_pool'",
[38] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool/cli.rb:16:inrun'", [39] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/bin/resque-pool:5:in<top (required)>'",
[40] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bin/resque-pool:19:inload'", [41] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bin/resque-pool:19:in
],
[1] [
[0] "/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/ruby/1.9.1/thread.rb:189:inblock in pop'", [1] "<internal:prelude>:10:insynchronize'",
[2] "/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/ruby/1.9.1/thread.rb:184:inpop'", [3] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zk-1.3.1/lib/zk/threadpool.rb:176:inblock (2 levels) in spawn_threadpool'"
],
[2] [
[0] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zookeeper-1.0.2/ext/c_zookeeper.rb:181:in_iterate_event_delivery'", [1] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zookeeper-1.0.2/ext/c_zookeeper.rb:172:in_event_thread_body'"
],
[3] [
[0] "/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/ruby/1.9.1/thread.rb:189:inblock in pop'", [1] "<internal:prelude>:10:insynchronize'",
[2] "/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/ruby/1.9.1/thread.rb:184:inpop'", [3] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zookeeper-1.0.2/lib/zookeeper/common/queue_with_pipe.rb:32:inpop'",
[4] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zookeeper-1.0.2/lib/zookeeper/common.rb:15:inget_next_event'", [5] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zookeeper-1.0.2/lib/zookeeper/common.rb:79:inblock (2 levels) in setup_dispatch_thread!'"
]
]"Waiting" procilne:
[
'"
[0] [
[ 0] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-fcaf987d0724/lib/resque/worker.rb:140:inblock in work'", [ 1] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-fcaf987d0724/lib/resque/worker.rb:116:inloop'",
[ 2] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-fcaf987d0724/lib/resque/worker.rb:116:inwork'", [ 3] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:332:inblock in spawn_worker!'",
[ 4] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:327:infork'", [ 5] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:327:inspawn_worker!'",
[ 6] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:287:inblock in spawn_missing_workers_for'", [ 7] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:287:intimes'",
[ 8] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:287:inspawn_missing_workers_for'", [ 9] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:273:inblock in maintain_worker_count'",
[10] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:271:ineach'", [11] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:271:inmaintain_worker_count'",
[12] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:191:instart'", [13] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:65:inrun'",
[14] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool/tasks.rb:17:inblock (2 levels) in <top (required)>'", [15] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:205:incall'",
[16] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:205:inblock in execute'", [17] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:200:ineach'",
[18] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:200:inexecute'", [19] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:158:inblock in invoke_with_call_chain'",
[20] "/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/ruby/1.9.1/monitor.rb:211:inmon_synchronize'", [21] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:151:ininvoke_with_call_chain'",
[22] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:144:ininvoke'", [23] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool/cli.rb:110:instart_pool'",
[24] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool/cli.rb:16:inrun'", [25] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/bin/resque-pool:5:in<top (required)>'",
[26] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bin/resque-pool:19:inload'", [27] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bin/resque-pool:19:in
],
[1] [
[ 0] "/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/ruby/1.9.1/monitor.rb:185:inmon_enter'", [ 1] "/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/ruby/1.9.1/monitor.rb:209:inmon_synchronize'",
[ 2] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/log4r-1.1.10/lib/log4r/outputter/outputter.rb:130:insynch'", [ 3] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/log4r-gelf-0.8.2/lib/log4r-gelf/gelf_outputter.rb:85:incanonical_log'",
[ 4] "(eval):3:inerror'", [ 5] "(eval):8:inblock in error'",
[ 6] "(eval):8:ineach'", [ 7] "(eval):8:inerror'",
[ 8] "(eval):9:inerror'", [ 9] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/redis_failover-0.8.2/lib/redis_failover/client.rb:267:inrescue in block in build_clients'",
[10] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/redis_failover-0.8.2/lib/redis_failover/client.rb:256:inblock in build_clients'", [11] "/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/ruby/1.9.1/monitor.rb:211:inmon_synchronize'",
[12] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/redis_failover-0.8.2/lib/redis_failover/client.rb:253:inbuild_clients'", [13] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/redis_failover-0.8.2/lib/redis_failover/client.rb:169:inhandle_zk_event'",
[14] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/redis_failover-0.8.2/lib/redis_failover/client.rb:156:inblock in setup_zk'", [15] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zk-1.3.1/lib/zk/subscription.rb:33:incall'",
[16] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zk-1.3.1/lib/zk/subscription.rb:33:incall'", [17] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zk-1.3.1/lib/zk/event_handler.rb:304:inblock (2 levels) in safe_call'",
[18] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zk-1.3.1/lib/zk/threadpool.rb:178:incall'", [19] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zk-1.3.1/lib/zk/threadpool.rb:178:inblock (2 levels) in spawn_threadpool'"
],
[2] [
[0] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zookeeper-1.0.2/ext/c_zookeeper.rb:181:in_iterate_event_delivery'", [1] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zookeeper-1.0.2/ext/c_zookeeper.rb:172:in_event_thread_body'"
],
[3] [
[ 0] "/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/ruby/1.9.1/logger.rb:568:inblock in write'", [ 1] "/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/ruby/1.9.1/monitor.rb:211:inmon_synchronize'",
[ 2] "/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/ruby/1.9.1/logger.rb:559:inwrite'", [ 3] "/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/ruby/1.9.1/logger.rb:377:inadd'",
[ 4] "/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/ruby/1.9.1/logger.rb:452:inerror'", [ 5] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zk-1.3.1/lib/zk/client/threaded.rb:244:inblock in raw_event_handler'",
[ 6] "/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/ruby/1.9.1/monitor.rb:211:inmon_synchronize'", [ 7] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zk-1.3.1/lib/zk/client/threaded.rb:241:inraw_event_handler'",
[ 8] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zk-1.3.1/lib/zk/event_handler.rb:145:inprocess'", [ 9] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zk-1.3.1/lib/zk/event_handler.rb:273:inblock in watcher_callback'",
[10] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zookeeper-1.0.2/lib/zookeeper/callbacks.rb:19:inblock in create'", [11] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zookeeper-1.0.2/lib/zookeeper/callbacks.rb:26:inblock in initialize'",
[12] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zookeeper-1.0.2/lib/zookeeper/callbacks.rb:32:incall'", [13] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zookeeper-1.0.2/lib/zookeeper/callbacks.rb:32:incall'",
[14] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zk-1.3.1/lib/zk/event_handler.rb:214:inblock (2 levels) in get_default_watcher_block'", [15] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zk-1.3.1/lib/zk/event_handler.rb:213:intap'",
[16] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zk-1.3.1/lib/zk/event_handler.rb:213:inblock in get_default_watcher_block'", [17] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zookeeper-1.0.2/lib/zookeeper/common.rb:155:incall'",
[18] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zookeeper-1.0.2/lib/zookeeper/common.rb:155:indispatch_next_callback'", [19] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zookeeper-1.0.2/lib/zookeeper/common.rb:79:inblock (2 levels) in setup_dispatch_thread!'"
],
[4] [
[ 0] "(eval):1:inblock in serialize_hash'", [ 1] "(eval):1:incollect'",
[ 2] "(eval):1:inserialize_hash'", [ 3] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/gelf-1.3.2/lib/gelf/notifier.rb:230:indeflate'",
[ 4] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/gelf-1.3.2/lib/gelf/notifier.rb:230:inserialize_hash'", [ 5] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/gelf-1.3.2/lib/gelf/notifier.rb:206:indatagrams_from_hash'",
[ 6] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/gelf-1.3.2/lib/gelf/notifier.rb:143:innotify_with_level!'", [ 7] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/gelf-1.3.2/lib/gelf/notifier.rb:118:innotify!'",
[ 8] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/log4r-gelf-0.8.2/lib/log4r-gelf/gelf_outputter.rb:88:inblock in canonical_log'", [ 9] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/log4r-1.1.10/lib/log4r/outputter/outputter.rb:130:inblock in synch'",
[10] "/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/ruby/1.9.1/monitor.rb:211:inmon_synchronize'", [11] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/log4r-1.1.10/lib/log4r/outputter/outputter.rb:130:insynch'",
[12] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/log4r-gelf-0.8.2/lib/log4r-gelf/gelf_outputter.rb:85:incanonical_log'", [13] "(eval):3:indebug'",
[14] "(eval):8:inblock in debug'", [15] "(eval):8:ineach'",
[16] "(eval):8:indebug'", [17] "(eval):9:indebug'",
[18] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/redis_failover-0.8.2/lib/redis_failover/client.rb:287:infetch_nodes'", [19] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/redis_failover-0.8.2/lib/redis_failover/client.rb:257:inblock in build_clients'",
[20] "/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/ruby/1.9.1/monitor.rb:211:inmon_synchronize'", [21] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/redis_failover-0.8.2/lib/redis_failover/client.rb:253:inbuild_clients'",
[22] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/redis_failover-0.8.2/lib/redis_failover/client.rb:169:inhandle_zk_event'", [23] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/redis_failover-0.8.2/lib/redis_failover/client.rb:156:inblock in setup_zk'",
[24] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zk-1.3.1/lib/zk/subscription.rb:33:incall'", [25] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zk-1.3.1/lib/zk/subscription.rb:33:incall'",
[26] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zk-1.3.1/lib/zk/event_handler.rb:304:inblock (2 levels) in safe_call'", [27] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zk-1.3.1/lib/zk/threadpool.rb:178:incall'",
[28] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zk-1.3.1/lib/zk/threadpool.rb:178:inblock (2 levels) in spawn_threadpool'" ], [5] [ [0] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zookeeper-1.0.2/ext/c_zookeeper.rb:181:in_iterate_event_delivery'",
[1] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zookeeper-1.0.2/ext/c_zookeeper.rb:172:in_event_thread_body'" ], [6] [ [ 0] "/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/ruby/1.9.1/monitor.rb:185:inmon_enter'",
[ 1] "/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/ruby/1.9.1/monitor.rb:209:inmon_synchronize'", [ 2] "/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/ruby/1.9.1/logger.rb:559:inwrite'",
[ 3] "/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/ruby/1.9.1/logger.rb:377:inadd'", [ 4] "/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/ruby/1.9.1/logger.rb:452:inerror'",
[ 5] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zk-1.3.1/lib/zk/client/threaded.rb:244:inblock in raw_event_handler'", [ 6] "/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/ruby/1.9.1/monitor.rb:211:inmon_synchronize'",
[ 7] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zk-1.3.1/lib/zk/client/threaded.rb:241:inraw_event_handler'", [ 8] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zk-1.3.1/lib/zk/event_handler.rb:145:inprocess'",
[ 9] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zk-1.3.1/lib/zk/event_handler.rb:273:inblock in watcher_callback'", [10] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zookeeper-1.0.2/lib/zookeeper/callbacks.rb:19:inblock in create'",
[11] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zookeeper-1.0.2/lib/zookeeper/callbacks.rb:26:inblock in initialize'", [12] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zookeeper-1.0.2/lib/zookeeper/callbacks.rb:32:incall'",
[13] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zookeeper-1.0.2/lib/zookeeper/callbacks.rb:32:incall'", [14] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zk-1.3.1/lib/zk/event_handler.rb:214:inblock (2 levels) in get_default_watcher_block'",
[15] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zk-1.3.1/lib/zk/event_handler.rb:213:intap'", [16] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zk-1.3.1/lib/zk/event_handler.rb:213:inblock in get_default_watcher_block'",
[17] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zookeeper-1.0.2/lib/zookeeper/common.rb:155:incall'", [18] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zookeeper-1.0.2/lib/zookeeper/common.rb:155:indispatch_next_callback'",
[19] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zookeeper-1.0.2/lib/zookeeper/common.rb:79:in `block (2 levels) in setup_dispatch_thread!'"
]
]
Reply to this email directly or view it on GitHub:
#13
from redis_failover.
wr0ngway
commented on August 19, 2024
After killing those workers and have a couple more respawn, here is what the dumps look like (both in the "Starting" procline state):
[
[0] [
[ 0] "(eval):1:in `block (2 levels) in spawn_threadpool'",
[ 1] "(eval):1:in `collect'",
[ 2] "(eval):1:in `block in spawn_threadpool'",
[ 3] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zk-1.3.1/lib/zk/threadpool.rb:173:in `new'",
[ 4] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zk-1.3.1/lib/zk/threadpool.rb:173:in `block in spawn_threadpool'",
[ 5] "/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'",
[ 6] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zk-1.3.1/lib/zk/threadpool.rb:171:in `spawn_threadpool'",
[ 7] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zk-1.3.1/lib/zk/threadpool.rb:62:in `block in start!'",
[ 8] "/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'",
[ 9] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zk-1.3.1/lib/zk/threadpool.rb:59:in `start!'",
[10] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zk-1.3.1/lib/zk/threadpool.rb:27:in `initialize'",
[11] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zk-1.3.1/lib/zk/client/threaded.rb:135:in `new'",
[12] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zk-1.3.1/lib/zk/client/threaded.rb:135:in `initialize'",
[13] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zk-1.3.1/lib/zk/client.rb:28:in `new'",
[14] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zk-1.3.1/lib/zk/client.rb:28:in `new'",
[15] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zk-1.3.1/lib/zk.rb:199:in `new'",
[16] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/redis_failover-0.8.2/lib/redis_failover/client.rb:155:in `setup_zk'",
[17] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/redis_failover-0.8.2/lib/redis_failover/client.rb:111:in `initialize'",
[18] "/mnt/appleton-matt/releases/20120510001131/lib/redis_factory.rb:15:in `new'",
[19] "/mnt/appleton-matt/releases/20120510001131/lib/redis_factory.rb:15:in `connect'",
[20] "/mnt/appleton-matt/releases/20120510001131/lib/backupify/redis.rb:7:in `redis'",
[21] "/mnt/appleton-matt/releases/20120510001131/lib/backupify/redis.rb:18:in `maintenance_mode?'",
[22] "/mnt/appleton-matt/releases/20120510001131/config/initializers/resque.rb:48:in `reserve'",
[23] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-fcaf987d0724/lib/resque/worker.rb:119:in `block in work'",
[24] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-fcaf987d0724/lib/resque/worker.rb:116:in `loop'",
[25] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-fcaf987d0724/lib/resque/worker.rb:116:in `work'",
[26] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:332:in `block in spawn_worker!'",
[27] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:327:in `fork'",
[28] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:327:in `spawn_worker!'",
[29] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:287:in `block in spawn_missing_workers_for'",
[30] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:287:in `times'",
[31] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:287:in `spawn_missing_workers_for'",
[32] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:273:in `block in maintain_worker_count'",
[33] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:271:in `each'",
[34] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:271:in `maintain_worker_count'",
[35] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:212:in `block in join'",
[36] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:207:in `loop'",
[37] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:207:in `join'",
[38] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:65:in `run'",
[39] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool/tasks.rb:17:in `block (2 levels) in <top (required)>'",
[40] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:205:in `call'",
[41] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:205:in `block in execute'",
[42] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:200:in `each'",
[43] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:200:in `execute'",
[44] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:158:in `block in invoke_with_call_chain'",
[45] "/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'",
[46] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:151:in `invoke_with_call_chain'",
[47] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:144:in `invoke'",
[48] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool/cli.rb:110:in `start_pool'",
[49] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool/cli.rb:16:in `run'",
[50] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/bin/resque-pool:5:in `<top (required)>'",
[51] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bin/resque-pool:19:in `load'",
[52] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bin/resque-pool:19:in `<main>'"
],
[1] [
[ 0] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zookeeper-1.0.2/ext/zookeeper_base.rb:50:in `block in get'",
[ 1] "/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'",
[ 2] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zookeeper-1.0.2/ext/zookeeper_base.rb:50:in `get'",
[ 3] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zookeeper-1.0.2/lib/zookeeper/client_methods.rb:21:in `get'",
[ 4] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zk-1.3.1/lib/zk/client/base.rb:389:in `block in get'",
[ 5] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zk-1.3.1/lib/zk/event_handler.rb:256:in `block in setup_watcher!'",
[ 6] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zk-1.3.1/lib/zk/event_handler.rb:207:in `block in synchronize'",
[ 7] "/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'",
[ 8] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zk-1.3.1/lib/zk/event_handler.rb:207:in `synchronize'",
[ 9] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zk-1.3.1/lib/zk/event_handler.rb:245:in `setup_watcher!'",
[10] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zk-1.3.1/lib/zk/client/base.rb:915:in `setup_watcher!'",
[11] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zk-1.3.1/lib/zk/client/base.rb:388:in `get'",
[12] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/redis_failover-0.8.2/lib/redis_failover/client.rb:285:in `fetch_nodes'",
[13] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/redis_failover-0.8.2/lib/redis_failover/client.rb:257:in `block in build_clients'",
[14] "/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'",
[15] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/redis_failover-0.8.2/lib/redis_failover/client.rb:253:in `build_clients'",
[16] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/redis_failover-0.8.2/lib/redis_failover/client.rb:169:in `handle_zk_event'",
[17] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/redis_failover-0.8.2/lib/redis_failover/client.rb:156:in `block in setup_zk'",
[18] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zk-1.3.1/lib/zk/subscription.rb:33:in `call'",
[19] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zk-1.3.1/lib/zk/subscription.rb:33:in `call'",
[20] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zk-1.3.1/lib/zk/event_handler.rb:304:in `block (2 levels) in safe_call'",
[21] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zk-1.3.1/lib/zk/threadpool.rb:178:in `call'",
[22] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zk-1.3.1/lib/zk/threadpool.rb:178:in `block (2 levels) in spawn_threadpool'"
],
[2] [
[0] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zookeeper-1.0.2/ext/c_zookeeper.rb:181:in `_iterate_event_delivery'",
[1] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zookeeper-1.0.2/ext/c_zookeeper.rb:172:in `_event_thread_body'"
],
[3] [
[0] "/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/ruby/1.9.1/thread.rb:189:in `block in pop'",
[1] "<internal:prelude>:10:in `synchronize'",
[2] "/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/ruby/1.9.1/thread.rb:184:in `pop'",
[3] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zookeeper-1.0.2/lib/zookeeper/common/queue_with_pipe.rb:32:in `pop'",
[4] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zookeeper-1.0.2/lib/zookeeper/common.rb:15:in `get_next_event'",
[5] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zookeeper-1.0.2/lib/zookeeper/common.rb:79:in `block (2 levels) in setup_dispatch_thread!'"
],
[4] [
[0] "/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/ruby/1.9.1/thread.rb:189:in `block in pop'",
[1] "<internal:prelude>:10:in `synchronize'",
[2] "/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/ruby/1.9.1/thread.rb:184:in `pop'",
[3] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zk-1.3.1/lib/zk/threadpool.rb:176:in `block (2 levels) in spawn_threadpool'"
]
]
[
[0] [
[ 0] "(eval):1:in `block (2 levels) in spawn_threadpool'",
[ 1] "(eval):1:in `collect'",
[ 2] "(eval):1:in `block in spawn_threadpool'",
[ 3] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zk-1.3.1/lib/zk/threadpool.rb:173:in `new'",
[ 4] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zk-1.3.1/lib/zk/threadpool.rb:173:in `block in spawn_threadpool'",
[ 5] "/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'",
[ 6] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zk-1.3.1/lib/zk/threadpool.rb:171:in `spawn_threadpool'",
[ 7] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zk-1.3.1/lib/zk/threadpool.rb:62:in `block in start!'",
[ 8] "/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'",
[ 9] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zk-1.3.1/lib/zk/threadpool.rb:59:in `start!'",
[10] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zk-1.3.1/lib/zk/threadpool.rb:27:in `initialize'",
[11] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zk-1.3.1/lib/zk/client/threaded.rb:135:in `new'",
[12] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zk-1.3.1/lib/zk/client/threaded.rb:135:in `initialize'",
[13] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zk-1.3.1/lib/zk/client.rb:28:in `new'",
[14] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zk-1.3.1/lib/zk/client.rb:28:in `new'",
[15] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zk-1.3.1/lib/zk.rb:199:in `new'",
[16] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/redis_failover-0.8.2/lib/redis_failover/client.rb:155:in `setup_zk'",
[17] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/redis_failover-0.8.2/lib/redis_failover/client.rb:111:in `initialize'",
[18] "/mnt/appleton-matt/releases/20120510001131/lib/redis_factory.rb:15:in `new'",
[19] "/mnt/appleton-matt/releases/20120510001131/lib/redis_factory.rb:15:in `connect'",
[20] "/mnt/appleton-matt/releases/20120510001131/lib/backupify/redis.rb:7:in `redis'",
[21] "/mnt/appleton-matt/releases/20120510001131/lib/backupify/redis.rb:18:in `maintenance_mode?'",
[22] "/mnt/appleton-matt/releases/20120510001131/config/initializers/resque.rb:48:in `reserve'",
[23] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-fcaf987d0724/lib/resque/worker.rb:119:in `block in work'",
[24] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-fcaf987d0724/lib/resque/worker.rb:116:in `loop'",
[25] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-fcaf987d0724/lib/resque/worker.rb:116:in `work'",
[26] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:332:in `block in spawn_worker!'",
[27] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:327:in `fork'",
[28] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:327:in `spawn_worker!'",
[29] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:287:in `block in spawn_missing_workers_for'",
[30] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:287:in `times'",
[31] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:287:in `spawn_missing_workers_for'",
[32] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:273:in `block in maintain_worker_count'",
[33] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:271:in `each'",
[34] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:271:in `maintain_worker_count'",
[35] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:212:in `block in join'",
[36] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:207:in `loop'",
[37] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:207:in `join'",
[38] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:65:in `run'",
[39] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool/tasks.rb:17:in `block (2 levels) in <top (required)>'",
[40] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:205:in `call'",
[41] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:205:in `block in execute'",
[42] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:200:in `each'",
[43] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:200:in `execute'",
[44] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:158:in `block in invoke_with_call_chain'",
[45] "/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'",
[46] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:151:in `invoke_with_call_chain'",
[47] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:144:in `invoke'",
[48] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool/cli.rb:110:in `start_pool'",
[49] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool/cli.rb:16:in `run'",
[50] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/bin/resque-pool:5:in `<top (required)>'",
[51] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bin/resque-pool:19:in `load'",
[52] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bin/resque-pool:19:in `<main>'"
],
[1] [
[0] "/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/ruby/1.9.1/thread.rb:189:in `block in pop'",
[1] "<internal:prelude>:10:in `synchronize'",
[2] "/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/ruby/1.9.1/thread.rb:184:in `pop'",
[3] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zk-1.3.1/lib/zk/threadpool.rb:176:in `block (2 levels) in spawn_threadpool'"
],
[2] [
[0] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zookeeper-1.0.2/ext/c_zookeeper.rb:181:in `_iterate_event_delivery'",
[1] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zookeeper-1.0.2/ext/c_zookeeper.rb:172:in `_event_thread_body'"
],
[3] [
[0] "/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/ruby/1.9.1/thread.rb:189:in `block in pop'",
[1] "<internal:prelude>:10:in `synchronize'",
[2] "/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/ruby/1.9.1/thread.rb:184:in `pop'",
[3] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zookeeper-1.0.2/lib/zookeeper/common/queue_with_pipe.rb:32:in `pop'",
[4] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zookeeper-1.0.2/lib/zookeeper/common.rb:15:in `get_next_event'",
[5] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zookeeper-1.0.2/lib/zookeeper/common.rb:79:in `block (2 levels) in setup_dispatch_thread!'"
],
[4] [
[0] "/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/ruby/1.9.1/thread.rb:189:in `block in pop'",
[1] "<internal:prelude>:10:in `synchronize'",
[2] "/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/ruby/1.9.1/thread.rb:184:in `pop'",
[3] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zk-1.3.1/lib/zk/threadpool.rb:176:in `block (2 levels) in spawn_threadpool'"
]
]
from redis_failover.
wr0ngway
commented on August 19, 2024
Yes, client works fine from other parts of my code. Not sure how to package up resque for you since there are a bunch of moving parts to set this up for our codebase. I was going to extract it for a rubber recipe, and then could put it on a throwaway instance for you, or let you into the instance I am testing on now.
from redis_failover.
ryanlecompte
commented on August 19, 2024
That's good to hear that it's working in other parts of your code base. I suspect this may have something to do specifically with resque and the fact that it forks workers. @slyphon, could you please take a look at these thread dumps? The recent traces seem to indicate something happening in the ZK layer. Do you think this is fixed in the new version of ZK that handles forking better?
On May 10, 2012, at 4:42 AM, Matt [email protected] wrote:
Yes, client works fine from other parts of my code. Not sure how to package up resque for you since there are a bunch of moving parts to set this up for our codebase. I was going to extract it for a rubber recipe, and then could put it on a throwaway instance for you, or let you into the instance I am testing on now.
Reply to this email directly or view it on GitHub:
#13 (comment)
from redis_failover.
slyphon
commented on August 19, 2024
Hey, I'm the ZK maintainer, I just got 1.4.0 released yesterday with the zookeeper-1.0.3 driver, which contains a bunch of fixes for fork behavior. I'm not too familiar with resque specifics, but there may be action we need to take to prevent this, specifically you need to call reopen() on the zk instance immediately after forking.
We should definitely run redis_failover w/ the latest ZK and test it.
from redis_failover.
ryanlecompte
commented on August 19, 2024
Great, thanks Jonathan. I will test it today with latest ZK and a simple resque worker.
On May 10, 2012, at 4:50 AM, Jonathan [email protected] wrote:
Hey, I'm the ZK maintainer, I just got 1.4.0 released yesterday with the zookeeper-1.0.3 driver, which contains a bunch of fixes for fork behavior. I'm not too familiar with resque specifics, but there may be action we need to take to prevent this, specifically you need to call reopen() on the zk instance immediately after forking.
We should definitely run redis_failover w/ the latest ZK and test it.
Reply to this email directly or view it on GitHub:
#13 (comment)
from redis_failover.
wr0ngway
commented on August 19, 2024
I'll try updating zk and zookeeper. For the reopen(), how do I call that (I'll try without it first)? Resque-pool has a post-fork handler I can call arbitrary code in, where I do already create a new connection to redis (effectively RedisFailover.new) just not sure what to call reopen() on,k or if its even necessary if I'm creating a new client. Thanks.
from redis_failover.
slyphon
commented on August 19, 2024
yeah, we definitely need to use zookeeper-1.0.3, there was a shutdown ordering problem that caused a lockup on linux after a fork. Worked great on Mac OS, but apparently on linux's pthread implementation, with the zookeeper c client, after a fork, you have to shut down the existing client completely before trying to start a new one. I now have a very thorough spec for testing forking, event delivery, etc.
from redis_failover.
ryanlecompte
commented on August 19, 2024
Excellent! This sounds like it could be the problem then. Matt, this should be as simple as specifying the version in redis_failover.gemspec to be the one that Jonathan recently released. Would be awesome if you could give that a shot.
Also, you probably don't need to deal with reopen if you are creating a brand new RedisFailover::Client instance since you will get a brand new ZK instance that's just tied to that new client.
On May 10, 2012, at 5:09 AM, Jonathan [email protected] wrote:
yeah, we definitely need to use zookeeper-1.0.3, there was a shutdown ordering problem that caused a lockup on linux after a fork. Worked great on Mac OS, but apparently on linux's pthread implementation, with the zookeeper c client, after a fork, you have to shut down the existing client completely before trying to start a new one. I now have a very thorough spec for testing forking, event delivery, etc.
Reply to this email directly or view it on GitHub:
#13 (comment)
from redis_failover.
slyphon
commented on August 19, 2024
I think you need to call close on the original, though. That was the bug, that the zkc connection that survived the fork was damaged goods, and if you started a new zkc connection before closing the old one, you'd deadlock. I designed the fork recovery code with reopen() in mind (for @eric's use case), so maybe we can do a short code review later to make sure it will work in this case.
from redis_failover.
wr0ngway
commented on August 19, 2024
Ok, using these versions, it still hangs, do I need to explicitly call reopen/close? If so, how?
from redis_failover.
slyphon
commented on August 19, 2024
Ok, looking at RedisFailover::Client, I don't see a way of explicitly calling close, once @ryanlecompte gets online I'll consult with him. Sorry, i'm in NY and I have to hop on the train in a few minutes, or I'd take a whack at this myself.
Also, i'm sorry for all the trouble, handling a fork() turned out to be way harder than it appeared.
from redis_failover.
wr0ngway
commented on August 19, 2024
Never fear, I'm used to the forking pain :)
from redis_failover.
ryanlecompte
commented on August 19, 2024
Okay, I just checked in (master) an update to redis_failover.gemspec to use the new ZK version, along with a new RedisFailover::Client#shutdown method that you should invoke in your after_fork hook, right before you create your new RedisFailover::Client instance. I just did a sanity check locally and it seems to work. Could you give master a shot, @wr0ngway and let us know how it works for you? Thanks!
from redis_failover.
wr0ngway
commented on August 19, 2024
In trying to verify this, I saw the following stack trace - should one be able to reuse a client after shutdown similar to the way a normal redis client reconnects? Looks like the "reconnect_zk" doesn't exist - bad refactoring?
[13] pry(main)> r = RedisFailover::Client.new(:zkservers => "matt:2181")
2012-05-10 15:46:08 UTC RedisFailover 29041 INFO: Purging current redis clients
2012-05-10 15:46:08 UTC RedisFailover 29041 INFO: Building new clients for nodes {:master=>"matt:6379", :slaves=>[], :unavailable=>[]}
=> #<RedisFailover::Client (master: matt:6379, slaves: none)>
[14] pry(main)> r.info['redis_version']
=> "2.4.9"
[15] pry(main)> r.shutdown
2012-05-10 15:46:34 UTC RedisFailover 29041 INFO: Purging current redis clients
=> []
[16] pry(main)> r.info['redis_version']
2012-05-10 15:46:37 UTC RedisFailover 29041 ERROR: Error while handling `info` - #<RedisFailover::NoMasterError: RedisFailover::NoMasterError>
2012-05-10 15:46:37 UTC RedisFailover 29041 ERROR: /mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/pry-exception_explorer-0.1.9/lib/pry-exception_explorer/core_ext.rb:46:in `raise'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/redis_failover-836276df1373/lib/redis_failover/client.rb:243:in `master'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/redis_failover-836276df1373/lib/redis_failover/client.rb:218:in `dispatch'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/redis_failover-836276df1373/lib/redis_failover/client.rb:82:in `block (2 levels) in <class:Client>'
(pry):16:in `<main>'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/pry-0.9.8.4/lib/pry/pry_instance.rb:249:in `eval'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/pry-0.9.8.4/lib/pry/pry_instance.rb:249:in `re'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/pry-0.9.8.4/lib/pry/pry_instance.rb:216:in `rep'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/pry-0.9.8.4/lib/pry/pry_instance.rb:200:in `block (2 levels) in repl'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/pry-0.9.8.4/lib/pry/pry_instance.rb:199:in `loop'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/pry-0.9.8.4/lib/pry/pry_instance.rb:199:in `block in repl'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/pry-0.9.8.4/lib/pry/pry_instance.rb:198:in `catch'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/pry-0.9.8.4/lib/pry/pry_instance.rb:198:in `repl'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/pry-0.9.8.4/lib/pry/pry_class.rb:136:in `start'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/pry-nav-0.2.0/lib/pry-nav/pry_ext.rb:17:in `start'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/railties-3.2.2/lib/rails/commands/console.rb:47:in `start'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/railties-3.2.2/lib/rails/commands/console.rb:8:in `start'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/railties-3.2.2/lib/rails/commands.rb:41:in `<top (required)>'
script/rails:6:in `require'
script/rails:6:in `<main>'
2012-05-10 15:46:37 UTC RedisFailover 29041 ERROR: ZK error while attempting to build clients: #<Zookeeper::Exceptions::NotConnected: Zookeeper::Exceptions::NotConnected>
2012-05-10 15:46:37 UTC RedisFailover 29041 ERROR: /mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/pry-exception_explorer-0.1.9/lib/pry-exception_explorer/core_ext.rb:46:in `raise'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zookeeper-1.0.3/ext/zookeeper_base.rb:145:in `block in assert_open'
/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zookeeper-1.0.3/ext/zookeeper_base.rb:143:in `assert_open'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zookeeper-1.0.3/lib/zookeeper/client_methods.rb:238:in `assert_open'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zookeeper-1.0.3/lib/zookeeper/client_methods.rb:16:in `get'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zk-1.4.0/lib/zk/client/base.rb:1040:in `call_and_check_rc'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zk-1.4.0/lib/zk/client/base.rb:437:in `block in get'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zk-1.4.0/lib/zk/event_handler.rb:294:in `block in setup_watcher!'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zk-1.4.0/lib/zk/event_handler.rb:236:in `block in synchronize'
/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zk-1.4.0/lib/zk/event_handler.rb:236:in `synchronize'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zk-1.4.0/lib/zk/event_handler.rb:274:in `setup_watcher!'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zk-1.4.0/lib/zk/client/base.rb:1087:in `setup_watcher!'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zk-1.4.0/lib/zk/client/base.rb:436:in `get'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/redis_failover-836276df1373/lib/redis_failover/client.rb:295:in `fetch_nodes'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/redis_failover-836276df1373/lib/redis_failover/client.rb:267:in `block in build_clients'
/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/redis_failover-836276df1373/lib/redis_failover/client.rb:263:in `build_clients'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/redis_failover-836276df1373/lib/redis_failover/client.rb:226:in `rescue in dispatch'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/redis_failover-836276df1373/lib/redis_failover/client.rb:212:in `dispatch'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/redis_failover-836276df1373/lib/redis_failover/client.rb:82:in `block (2 levels) in <class:Client>'
(pry):16:in `<main>'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/pry-0.9.8.4/lib/pry/pry_instance.rb:249:in `eval'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/pry-0.9.8.4/lib/pry/pry_instance.rb:249:in `re'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/pry-0.9.8.4/lib/pry/pry_instance.rb:216:in `rep'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/pry-0.9.8.4/lib/pry/pry_instance.rb:200:in `block (2 levels) in repl'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/pry-0.9.8.4/lib/pry/pry_instance.rb:199:in `loop'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/pry-0.9.8.4/lib/pry/pry_instance.rb:199:in `block in repl'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/pry-0.9.8.4/lib/pry/pry_instance.rb:198:in `catch'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/pry-0.9.8.4/lib/pry/pry_instance.rb:198:in `repl'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/pry-0.9.8.4/lib/pry/pry_class.rb:136:in `start'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/pry-nav-0.2.0/lib/pry-nav/pry_ext.rb:17:in `start'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/railties-3.2.2/lib/rails/commands/console.rb:47:in `start'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/railties-3.2.2/lib/rails/commands/console.rb:8:in `start'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/railties-3.2.2/lib/rails/commands.rb:41:in `<top (required)>'
script/rails:6:in `require'
script/rails:6:in `<main>'
NameError: undefined local variable or method `reconnect_zk' for #<RedisFailover::Client (master: none, slaves: none)>
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/redis_failover-836276df1373/lib/redis_failover/client.rb:120:in `method_missing'
[17] pry(main)>
from redis_failover.
slyphon
commented on August 19, 2024
I'm working on a possible fix for the hang in zookeeper. i need to test this with the full stack to make sure it does the right thing, but this should prevent the hang by immediately shutting down the underlying low-level C wrapper after a fork, then the reopen will just repair zookeeper ruby-space state.
from redis_failover.
ryanlecompte
commented on August 19, 2024
Hey Matt,
Are you on IRC (freenode)? If so it would be great if you could join #zk-gem and we could diagnose this a bit more easily. I'll be there in 15 minutes.
Ryan
On May 10, 2012, at 8:49 AM, Matt [email protected] wrote:
In trying to verify this, I saw the following stack trace - should one be able to reuse a client after shutdown similar to the way a normal redis client reconnects? Looks like the "reconnect_zk" doesn't exist - bad refactoring?
[13] pry(main)> r = RedisFailover::Client.new(:zkservers => "matt:2181")
'
2012-05-10 15:46:08 UTC RedisFailover 29041 INFO: Purging current redis clients
2012-05-10 15:46:08 UTC RedisFailover 29041 INFO: Building new clients for nodes {:master=>"matt:6379", :slaves=>[], :unavailable=>[]}
=> #<RedisFailover::Client (master: matt:6379, slaves: none)>
[14] pry(main)> r.info['redis_version']
=> "2.4.9"
[15] pry(main)> r.shutdown
2012-05-10 15:46:34 UTC RedisFailover 29041 INFO: Purging current redis clients
=> []
[16] pry(main)> r.info['redis_version']
2012-05-10 15:46:37 UTC RedisFailover 29041 ERROR: Error while handlinginfo- #<RedisFailover::NoMasterError: RedisFailover::NoMasterError>
2012-05-10 15:46:37 UTC RedisFailover 29041 ERROR: /mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/pry-exception_explorer-0.1.9/lib/pry-exception_explorer/core_ext.rb:46:inraise' /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/redis_failover-836276df1373/lib/redis_failover/client.rb:243:inmaster'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/redis_failover-836276df1373/lib/redis_failover/client.rb:218:indispatch' /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/redis_failover-836276df1373/lib/redis_failover/client.rb:82:inblock (2 levels) in class:Client'
(pry):16:in<main>' /mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/pry-0.9.8.4/lib/pry/pry_instance.rb:249:ineval'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/pry-0.9.8.4/lib/pry/pry_instance.rb:249:inre' /mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/pry-0.9.8.4/lib/pry/pry_instance.rb:216:inrep'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/pry-0.9.8.4/lib/pry/pry_instance.rb:200:inblock (2 levels) in repl' /mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/pry-0.9.8.4/lib/pry/pry_instance.rb:199:inloop'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/pry-0.9.8.4/lib/pry/pry_instance.rb:199:inblock in repl' /mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/pry-0.9.8.4/lib/pry/pry_instance.rb:198:incatch'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/pry-0.9.8.4/lib/pry/pry_instance.rb:198:inrepl' /mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/pry-0.9.8.4/lib/pry/pry_class.rb:136:instart'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/pry-nav-0.2.0/lib/pry-nav/pry_ext.rb:17:instart' /mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/railties-3.2.2/lib/rails/commands/console.rb:47:instart'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/railties-3.2.2/lib/rails/commands/console.rb:8:instart' /mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/railties-3.2.2/lib/rails/commands.rb:41:in<top (required)>'
script/rails:6:inrequire' script/rails:6:in
2012-05-10 15:46:37 UTC RedisFailover 29041 ERROR: ZK error while attempting to build clients: #<Zookeeper::Exceptions::NotConnected: Zookeeper::Exceptions::NotConnected>
2012-05-10 15:46:37 UTC RedisFailover 29041 ERROR: /mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/pry-exception_explorer-0.1.9/lib/pry-exception_explorer/core_ext.rb:46:inraise' /mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zookeeper-1.0.3/ext/zookeeper_base.rb:145:inblock in assert_open'
/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/ruby/1.9.1/monitor.rb:211:inmon_synchronize' /mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zookeeper-1.0.3/ext/zookeeper_base.rb:143:inassert_open'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zookeeper-1.0.3/lib/zookeeper/client_methods.rb:238:inassert_open' /mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zookeeper-1.0.3/lib/zookeeper/client_methods.rb:16:inget'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zk-1.4.0/lib/zk/client/base.rb:1040:incall_and_check_rc' /mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zk-1.4.0/lib/zk/client/base.rb:437:inblock in get'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zk-1.4.0/lib/zk/event_handler.rb:294:inblock in setup_watcher!' /mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zk-1.4.0/lib/zk/event_handler.rb:236:inblock in synchronize'
/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/ruby/1.9.1/monitor.rb:211:inmon_synchronize' /mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zk-1.4.0/lib/zk/event_handler.rb:236:insynchronize'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zk-1.4.0/lib/zk/event_handler.rb:274:insetup_watcher!' /mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zk-1.4.0/lib/zk/client/base.rb:1087:insetup_watcher!'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zk-1.4.0/lib/zk/client/base.rb:436:inget' /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/redis_failover-836276df1373/lib/redis_failover/client.rb:295:infetch_nodes'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/redis_failover-836276df1373/lib/redis_failover/client.rb:267:inblock in build_clients' /usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/ruby/1.9.1/monitor.rb:211:inmon_synchronize'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/redis_failover-836276df1373/lib/redis_failover/client.rb:263:inbuild_clients' /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/redis_failover-836276df1373/lib/redis_failover/client.rb:226:inrescue in dispatch'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/redis_failover-836276df1373/lib/redis_failover/client.rb:212:indispatch' /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/redis_failover-836276df1373/lib/redis_failover/client.rb:82:inblock (2 levels) in class:Client'
(pry):16:in<main>' /mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/pry-0.9.8.4/lib/pry/pry_instance.rb:249:ineval'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/pry-0.9.8.4/lib/pry/pry_instance.rb:249:inre' /mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/pry-0.9.8.4/lib/pry/pry_instance.rb:216:inrep'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/pry-0.9.8.4/lib/pry/pry_instance.rb:200:inblock (2 levels) in repl' /mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/pry-0.9.8.4/lib/pry/pry_instance.rb:199:inloop'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/pry-0.9.8.4/lib/pry/pry_instance.rb:199:inblock in repl' /mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/pry-0.9.8.4/lib/pry/pry_instance.rb:198:incatch'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/pry-0.9.8.4/lib/pry/pry_instance.rb:198:inrepl' /mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/pry-0.9.8.4/lib/pry/pry_class.rb:136:instart'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/pry-nav-0.2.0/lib/pry-nav/pry_ext.rb:17:instart' /mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/railties-3.2.2/lib/rails/commands/console.rb:47:instart'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/railties-3.2.2/lib/rails/commands/console.rb:8:instart' /mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/railties-3.2.2/lib/rails/commands.rb:41:in<top (required)>'
script/rails:6:inrequire' script/rails:6:in'
NameError: undefined local variable or methodreconnect_zk' for #<RedisFailover::Client (master: none, slaves: none)> from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/redis_failover-836276df1373/lib/redis_failover/client.rb:120:inmethod_missing'
[17] pry(main)>
Reply to this email directly or view it on GitHub:
#13 (comment)
from redis_failover.
wr0ngway
commented on August 19, 2024
I'm seeing this SEGV now (right when I shutdown failover within the forked resque worker):
resque-pool-manager[2691]: started manager
resque-pool-manager[2691]: Pool contains worker PIDs: [2802, 2806]
resque-pool-worker[2802]: Starting worker matt.appleton.backupify.com:2802:@matt
RedisFactory.shutdown 2802
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zookeeper-1.0.3/ext/c_zookeeper.rb:103: [BUG] Segmentation fault
ruby 1.9.3p125 (2012-02-16 revision 34643) [x86_64-linux]
-- Control frame information -----------------------------------------------
c:0010 p:---- s:0025 b:0025 l:000024 d:000024 CFUNC :close_handle
c:0009 p:0039 s:0022 b:0022 l:000ce0 d:000e28 BLOCK /mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zookeeper-1.0.3/ext/c_zookeeper.rb:103
c:0008 p:---- s:0020 b:0020 l:000019 d:000019 FINISH
c:0007 p:---- s:0018 b:0018 l:000017 d:000017 CFUNC :call
c:0006 p:0051 s:0015 b:0015 l:000ce0 d:000ce0 METHOD /mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zookeeper-1.0.3/ext/c_zookeeper.rb:108
c:0005 p:0021 s:0011 b:0011 l:001500 d:000010 BLOCK /mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zookeeper-1.0.3/ext/zookeeper_base.rb:168
c:0004 p:0019 s:0009 b:0009 l:000008 d:000008 METHOD /usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/ruby/1.9.1/monitor.rb:211
c:0003 p:0011 s:0006 b:0006 l:001500 d:000005 BLOCK /mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zookeeper-1.0.3/ext/zookeeper_base.rb:166
c:0002 p:---- s:0004 b:0004 l:000003 d:000003 FINISH
c:0001 p:---- s:0002 b:0002 l:000001 d:000001 TOP
-- Ruby level backtrace information ----------------------------------------
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zookeeper-1.0.3/ext/zookeeper_base.rb:166:in `block in close'
/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zookeeper-1.0.3/ext/zookeeper_base.rb:168:in `block (2 levels) in close'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zookeeper-1.0.3/ext/c_zookeeper.rb:108:in `close'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zookeeper-1.0.3/ext/c_zookeeper.rb:108:in `call'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zookeeper-1.0.3/ext/c_zookeeper.rb:103:in `block in close'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zookeeper-1.0.3/ext/c_zookeeper.rb:103:in `close_handle'
-- C level backtrace information -------------------------------------------
/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/libruby.so.1.9(+0x17bea5) [0x7feb606faea5] vm_dump.c:796
/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/libruby.so.1.9(+0x5d09b) [0x7feb605dc09b] error.c:258
/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/libruby.so.1.9(rb_bug+0xb7) [0x7feb605dcef7] error.c:277
/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/libruby.so.1.9(+0x11050f) [0x7feb6068f50f] signal.c:609
/lib/x86_64-linux-gnu/libc.so.6(+0x364c0) [0x7feb601f84c0] array.c:3889
/lib/x86_64-linux-gnu/libpthread.so.0(pthread_join+0x27) [0x7feb5ffae0b7]
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zookeeper-1.0.3/ext/zookeeper_c.so(adaptor_finish+0x7c) [0x7feb5688e30c] src/mt_adaptor.c:219
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zookeeper-1.0.3/ext/zookeeper_c.so(zookeeper_close+0x94) [0x7feb56889a54] src/zookeeper.c:2297
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zookeeper-1.0.3/ext/zookeeper_c.so(+0xdf8f) [0x7feb5687ef8f]
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zookeeper-1.0.3/ext/zookeeper_c.so(+0x10e21) [0x7feb56881e21]
/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/libruby.so.1.9(+0x175601) [0x7feb606f4601] vm_insnhelper.c:404
/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/libruby.so.1.9(+0x16b259) [0x7feb606ea259] insns.def:1015
/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/libruby.so.1.9(+0x17175a) [0x7feb606f075a] vm.c:1220
/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/libruby.so.1.9(+0x172721) [0x7feb606f1721] vm.c:670
/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/libruby.so.1.9(+0x175601) [0x7feb606f4601] vm_insnhelper.c:404
/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/libruby.so.1.9(+0x16b259) [0x7feb606ea259] insns.def:1015
/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/libruby.so.1.9(+0x17175a) [0x7feb606f075a] vm.c:1220
/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/libruby.so.1.9(+0x172721) [0x7feb606f1721] vm.c:670
/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/libruby.so.1.9(+0x181d3e) [0x7feb60700d3e] thread.c:453
/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/libruby.so.1.9(+0x181eaf) [0x7feb60700eaf] thread_pthread.c:653
/lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) [0x7feb5fface9a]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7feb602b44bd] regcomp.c:1900
from redis_failover.
ryanlecompte
commented on August 19, 2024
Can you give latest master a try? I just checked in the fixes around
reconnect_zk (the bad refactor).
So basically in your after_fork, you want to do
RedisFailover::Client#shutdown, and then client =
RedisFailover::Client.new(...)
On Thu, May 10, 2012 at 9:07 AM, Matt Conway <
[email protected]
wrote:
I'm seeing this SEGV now (right when I shutdown failover within the forked
resque worker):resque-pool-manager[2691]: started manager
resque-pool-manager[2691]: Pool contains worker PIDs: [2802, 2806]
resque-pool-worker[2802]: Starting worker
matt.appleton.backupify.com:2802:@matt
RedisFactory.shutdown 2802/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zookeeper-1.0.3/ext/c_zookeeper.rb:103:
[BUG] Segmentation fault
ruby 1.9.3p125 (2012-02-16 revision 34643) [x86_64-linux]-- Control frame information
c:0010 p:---- s:0025 b:0025 l:000024 d:000024 CFUNC :close_handle
c:0009 p:0039 s:0022 b:0022 l:000ce0 d:000e28 BLOCK
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zookeeper-1.0.3/ext/c_zookeeper.rb:103
c:0008 p:---- s:0020 b:0020 l:000019 d:000019 FINISH
c:0007 p:---- s:0018 b:0018 l:000017 d:000017 CFUNC :call
c:0006 p:0051 s:0015 b:0015 l:000ce0 d:000ce0 METHOD
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zookeeper-1.0.3/ext/c_zookeeper.rb:108
c:0005 p:0021 s:0011 b:0011 l:001500 d:000010 BLOCK
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zookeeper-1.0.3/ext/zookeeper_base.rb:168
c:0004 p:0019 s:0009 b:0009 l:000008 d:000008 METHOD
/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/ruby/1.9.1/monitor.rb:211
c:0003 p:0011 s:0006 b:0006 l:001500 d:000005 BLOCK
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zookeeper-1.0.3/ext/zookeeper_base.rb:166
c:0002 p:---- s:0004 b:0004 l:000003 d:000003 FINISH
c:0001 p:---- s:0002 b:0002 l:000001 d:000001 TOP-- Ruby level backtrace information
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zookeeper-1.0.3/ext/zookeeper_base.rb:166:in
`block in close'/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/ruby/1.9.1/monitor.rb:211:in
`mon_synchronize'/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zookeeper-1.0.3/ext/zookeeper_base.rb:168:in
`block (2 levels) in close'/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zookeeper-1.0.3/ext/c_zookeeper.rb:108:in
`close'/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zookeeper-1.0.3/ext/c_zookeeper.rb:108:in
`call'/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zookeeper-1.0.3/ext/c_zookeeper.rb:103:in
`block in close'/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zookeeper-1.0.3/ext/c_zookeeper.rb:103:in
`close_handle'-- C level backtrace information
/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/libruby.so.1.9(+0x17bea5)
[0x7feb606faea5] vm_dump.c:796/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/libruby.so.1.9(+0x5d09b)
[0x7feb605dc09b] error.c:258/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/libruby.so.1.9(rb_bug+0xb7)
[0x7feb605dcef7] error.c:277/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/libruby.so.1.9(+0x11050f)
[0x7feb6068f50f] signal.c:609
/lib/x86_64-linux-gnu/libc.so.6(+0x364c0) [0x7feb601f84c0] array.c:3889
/lib/x86_64-linux-gnu/libpthread.so.0(pthread_join+0x27)
[0x7feb5ffae0b7]/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zookeeper-1.0.3/ext/zookeeper_c.so(adaptor_finish+0x7c)
[0x7feb5688e30c] src/mt_adaptor.c:219/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zookeeper-1.0.3/ext/zookeeper_c.so(zookeeper_close+0x94)
[0x7feb56889a54] src/zookeeper.c:2297/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zookeeper-1.0.3/ext/zookeeper_c.so(+0xdf8f)
[0x7feb5687ef8f]/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zookeeper-1.0.3/ext/zookeeper_c.so(+0x10e21)
[0x7feb56881e21]/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/libruby.so.1.9(+0x175601)
[0x7feb606f4601] vm_insnhelper.c:404/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/libruby.so.1.9(+0x16b259)
[0x7feb606ea259] insns.def:1015/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/libruby.so.1.9(+0x17175a)
[0x7feb606f075a] vm.c:1220/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/libruby.so.1.9(+0x172721)
[0x7feb606f1721] vm.c:670/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/libruby.so.1.9(+0x175601)
[0x7feb606f4601] vm_insnhelper.c:404/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/libruby.so.1.9(+0x16b259)
[0x7feb606ea259] insns.def:1015/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/libruby.so.1.9(+0x17175a)
[0x7feb606f075a] vm.c:1220/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/libruby.so.1.9(+0x172721)
[0x7feb606f1721] vm.c:670/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/libruby.so.1.9(+0x181d3e)
[0x7feb60700d3e] thread.c:453/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/libruby.so.1.9(+0x181eaf)
[0x7feb60700eaf] thread_pthread.c:653
/lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) [0x7feb5fface9a]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7feb602b44bd]
regcomp.c:1900
Reply to this email directly or view it on GitHub:
from redis_failover.
slyphon
commented on August 19, 2024
Ok, did a pretty much full rewrite of the C backend of zookeeper between friday/saturday, tests are passing for zookeeper on all platforms and ruby versions, ZK all tests are passing except the fork test in rubinius (but, really, who cares), so I'll cut a release on monday (5/14) and we'll give it a whirl.
from redis_failover.
ryanlecompte
commented on August 19, 2024
Awesome work! This will be great to test on Monday.
On May 13, 2012, at 8:46 AM, Jonathan [email protected] wrote:
Ok, did a pretty much full rewrite of the C backend of zookeeper between friday/saturday, tests are passing for zookeeper on all platforms and ruby versions, ZK all tests are passing except the fork test in rubinius (but, really, who cares), so I'll cut a release on monday (5/14) and we'll give it a whirl.
Reply to this email directly or view it on GitHub:
#13 (comment)
from redis_failover.
wr0ngway
commented on August 19, 2024
Ok, I tested and it looks like this is fixed. Good job! Now that I can get further, I'm running into a new issue, which I think is a redis_failover problem, will create a new issue.
from redis_failover.
Related Issues (20)
- Full Failure of server HOT 1
- Undefined method 'job' for main object with Resque HOT 4
- Occasionally stale server selection HOT 2
- start-stop-daemon to start redis_failover how to save log
- Add informative procline
- passing in a zk and having it close before redis-failover can trigger an infinite loop
- Prevent reads on slaves HOT 3
- Add subscribing part of Pub/Sub ops to REDIS_READ_OPS
- If ZK has an old master, redis_failover will use it even though the redis instance does not exist HOT 2
- License missing from gemspec
- stuck in "Failed to discover master node" HOT 1
- Shutdown fails with Ruby 2.0 HOT 3
- Error bringing back up failed master HOT 8
- Support Ruby 2.0 HOT 2
- Support Redis 2.8 HOT 1
- Allow Manual Failover in cluster bootstrap
- Redis-failover start up error HOT 1
- Starting redis_node_manager connection timeouts HOT 3
- State of redis_failover HOT 3
- ruby1.9 下启动报错
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from redis_failover.