ryanlecompte / redis_failover Goto Github PK
View Code? Open in Web Editor NEWredis_failover is a ZooKeeper-based automatic master/slave failover solution for Ruby.
Home Page: http://github.com/ryanlecompte/redis_failover
License: MIT License
redis_failover's People
Contributors
Stargazers
Watchers
Forkers
epictetus slyphon eric oleriesenberg rudionrails ippy04 tsilen nogara eldariof nivertech arnabc mid9commander cniclsh ulfmagnetics expectedbehavior chrisroberts amicus thaingo jmolinaparadigma anismiles officedrop backupify crestxu jcrivera yubingzhong jaeho-kim ibank magicbill andrewmacquarrie patricktulskie josexie albsen lapenya sforce100 daihu thomasalxdmy p7s1games baina fubuki solanolabs szaharici tomzhang flowdock miqui reedf louistz deyin edhurtig wyzssw jongillies shrcheng dong ingted boonchu dikoufu tom2jack flying0er husttb sunilpentapati mikeaddison93 dyet92k bearerpipelinetest forky-mcforkface iq-scmredis_failover's Issues
FeatureRequest: Implement redis server pools
In our cluster we have more than one redis cluster - one for app caching, etc, another for resque. I'm about to setup node_manager servers for each of these by having a different znode_path. However, it would be nice if I could just have a single set of node_manager servers and tell it I have multiple disparate redis clusters.
Since RedisFailover::Client takes the znode_path, which I would configure differently for connections to each pool, I think all that is left is to make node_manager be able to take a list of znode_paths and do the right thing...?
Shutdown fails with Ruby 2.0
You can't take a mutex from a signal handler in Ruby 2.0+. Here's what Sidekiq does to get around this limitation:
https://github.com/mperham/sidekiq/blob/master/lib/sidekiq/cli.rb#L73
log writing failed. can't be called from trap context
/Users/mperham/.rbenv/versions/2.0.0-p195/lib/ruby/2.0.0/monitor.rb:185:in `lock': can't be called from trap context (ThreadError)
from /Users/mperham/.rbenv/versions/2.0.0-p195/lib/ruby/2.0.0/monitor.rb:185:in `mon_enter'
from /Users/mperham/.rbenv/versions/2.0.0-p195/lib/ruby/2.0.0/monitor.rb:209:in `mon_synchronize'
from /Users/mperham/.rbenv/versions/2.0.0-p195/lib/ruby/gems/2.0.0/gems/redis_failover-1.0.2/lib/redis_failover/node_manager.rb:99:in `shutdown'
from /Users/mperham/.rbenv/versions/2.0.0-p195/lib/ruby/gems/2.0.0/gems/redis_failover-1.0.2/lib/redis_failover/runner.rb:21:in `block (2 levels) in trap_signals'
Redis-failover start up error
Hi ,
I have followed your installation guidelines and sample daemon execution (redis_node_manager -n localhost:6379,localhost:6380 -z localhost:2181,localhost:2182,localhost:2183). But i got following errors.
server:redis_failover-master suresh$ redis_node_manager -C examples/config.yml
Sat Jan 18 11:40:08 UTC 2014 RedisFailover 42803 INFO: Redis Node Manager v1.0.2 starting (ruby 1.8.7 (2012-02-08 patchlevel 358) [universal-darwin12.0])
/Library/Ruby/Gems/1.8/gems/redis_failover-1.0.2/lib/redis_failover/node_strategy.rb:14:in for': Failed to find node strategy: majority (RuntimeError) from /Library/Ruby/Gems/1.8/gems/redis_failover-1.0.2/lib/redis_failover/node_manager.rb:42:ininitialize'
from /Library/Ruby/Gems/1.8/gems/redis_failover-1.0.2/lib/redis_failover/runner.rb:11:in new' from /Library/Ruby/Gems/1.8/gems/redis_failover-1.0.2/lib/redis_failover/runner.rb:11:inrun'
from /Library/Ruby/Gems/1.8/gems/redis_failover-1.0.2/bin/redis_node_manager:6
from /usr/bin/redis_node_manager:23:in `load'
from /usr/bin/redis_node_manager:23
Client#reopen not working as expected
This is related to #13 but running on OSX to avoid hitting the deadlock.
I see the following stack trace when using the Client#reopen method in my forked children (resque). I do not see the trace when I use Client#shutdown then manually recreate/reassign all the references to existing client instances. Perhaps it isn't resetting all state internally? The trace happens after my job runs, and from the trace I think it is happening in the worker parent, not the worker child nor the top level resque-pool process.
/Users/mconway/.rvm/gems/ruby-1.9.3-p125-falcon/gems/pry-exception_explorer-0.1.9/lib/pry-exception_explorer/core_ext.rb:46:in `raise': You tried to use a connection inherited from another process [] You need to call reopen() after forking (Zookeeper::Exceptions::InheritedConnectionError)
from /Users/mconway/.rvm/gems/ruby-1.9.3-p125-falcon/gems/zookeeper-1.0.4/ext/zookeeper_base.rb:145:in `block in assert_open'
from /Users/mconway/.rvm/rubies/ruby-1.9.3-p125-falcon/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'
from /Users/mconway/.rvm/gems/ruby-1.9.3-p125-falcon/gems/zookeeper-1.0.4/ext/zookeeper_base.rb:141:in `assert_open'
from /Users/mconway/.rvm/gems/ruby-1.9.3-p125-falcon/gems/zookeeper-1.0.4/lib/zookeeper/client_methods.rb:238:in `assert_open'
from /Users/mconway/.rvm/gems/ruby-1.9.3-p125-falcon/gems/zookeeper-1.0.4/lib/zookeeper/client_methods.rb:16:in `get'
from /Users/mconway/.rvm/gems/ruby-1.9.3-p125-falcon/gems/zk-1.4.0/lib/zk/client/base.rb:1040:in `call_and_check_rc'
from /Users/mconway/.rvm/gems/ruby-1.9.3-p125-falcon/gems/zk-1.4.0/lib/zk/client/base.rb:437:in `block in get'
from /Users/mconway/.rvm/gems/ruby-1.9.3-p125-falcon/gems/zk-1.4.0/lib/zk/event_handler.rb:294:in `block in setup_watcher!'
from /Users/mconway/.rvm/gems/ruby-1.9.3-p125-falcon/gems/zk-1.4.0/lib/zk/event_handler.rb:236:in `block in synchronize'
from /Users/mconway/.rvm/rubies/ruby-1.9.3-p125-falcon/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'
from /Users/mconway/.rvm/gems/ruby-1.9.3-p125-falcon/gems/zk-1.4.0/lib/zk/event_handler.rb:236:in `synchronize'
from /Users/mconway/.rvm/gems/ruby-1.9.3-p125-falcon/gems/zk-1.4.0/lib/zk/event_handler.rb:274:in `setup_watcher!'
from /Users/mconway/.rvm/gems/ruby-1.9.3-p125-falcon/gems/zk-1.4.0/lib/zk/client/base.rb:1087:in `setup_watcher!'
from /Users/mconway/.rvm/gems/ruby-1.9.3-p125-falcon/gems/zk-1.4.0/lib/zk/client/base.rb:436:in `get'
from /Users/mconway/.rvm/gems/ruby-1.9.3-p125-falcon/bundler/gems/redis_failover-040d5926f179/lib/redis_failover/client.rb:293:in `fetch_nodes'
from /Users/mconway/.rvm/gems/ruby-1.9.3-p125-falcon/bundler/gems/redis_failover-040d5926f179/lib/redis_failover/client.rb:273:in `block in build_clients'
from /Users/mconway/.rvm/rubies/ruby-1.9.3-p125-falcon/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'
from /Users/mconway/.rvm/gems/ruby-1.9.3-p125-falcon/bundler/gems/redis_failover-040d5926f179/lib/redis_failover/client.rb:271:in `build_clients'
from /Users/mconway/.rvm/gems/ruby-1.9.3-p125-falcon/bundler/gems/redis_failover-040d5926f179/lib/redis_failover/client.rb:215:in `dispatch'
from /Users/mconway/.rvm/gems/ruby-1.9.3-p125-falcon/bundler/gems/redis_failover-040d5926f179/lib/redis_failover/client.rb:82:in `block (2 levels) in <class:Client>'
from /Users/mconway/.rvm/gems/ruby-1.9.3-p125-falcon/gems/redis-namespace-1.0.3/lib/redis/namespace.rb:213:in `method_missing'
from /Users/mconway/.rvm/gems/ruby-1.9.3-p125-falcon/bundler/gems/resque-fcaf987d0724/lib/resque/worker.rb:430:in `job'
from /Users/mconway/.rvm/gems/ruby-1.9.3-p125-falcon/bundler/gems/resque-fcaf987d0724/lib/resque/worker.rb:362:in `unregister_worker'
from /Users/mconway/.rvm/gems/ruby-1.9.3-p125-falcon/bundler/gems/resque-fcaf987d0724/lib/resque/worker.rb:145:in `ensure in work'
from /Users/mconway/.rvm/gems/ruby-1.9.3-p125-falcon/bundler/gems/resque-fcaf987d0724/lib/resque/worker.rb:145:in `work'
from /Users/mconway/.rvm/gems/ruby-1.9.3-p125-falcon/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:332:in `block in spawn_worker!'
from /Users/mconway/.rvm/gems/ruby-1.9.3-p125-falcon/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:327:in `fork'
from /Users/mconway/.rvm/gems/ruby-1.9.3-p125-falcon/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:327:in `spawn_worker!'
from /Users/mconway/.rvm/gems/ruby-1.9.3-p125-falcon/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:287:in `block in spawn_missing_workers_for'
from /Users/mconway/.rvm/gems/ruby-1.9.3-p125-falcon/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:287:in `times'
from /Users/mconway/.rvm/gems/ruby-1.9.3-p125-falcon/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:287:in `spawn_missing_workers_for'
from /Users/mconway/.rvm/gems/ruby-1.9.3-p125-falcon/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:273:in `block in maintain_worker_count'
from /Users/mconway/.rvm/gems/ruby-1.9.3-p125-falcon/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:271:in `each'
from /Users/mconway/.rvm/gems/ruby-1.9.3-p125-falcon/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:271:in `maintain_worker_count'
from /Users/mconway/.rvm/gems/ruby-1.9.3-p125-falcon/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:191:in `start'
from /Users/mconway/.rvm/gems/ruby-1.9.3-p125-falcon/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:65:in `run'
from /Users/mconway/.rvm/gems/ruby-1.9.3-p125-falcon/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool/tasks.rb:17:in `block (2 levels) in <top (required)>'
from /Users/mconway/.rvm/gems/ruby-1.9.3-p125-falcon/gems/rake-0.9.2.2/lib/rake/task.rb:205:in `call'
from /Users/mconway/.rvm/gems/ruby-1.9.3-p125-falcon/gems/rake-0.9.2.2/lib/rake/task.rb:205:in `block in execute'
from /Users/mconway/.rvm/gems/ruby-1.9.3-p125-falcon/gems/rake-0.9.2.2/lib/rake/task.rb:200:in `each'
from /Users/mconway/.rvm/gems/ruby-1.9.3-p125-falcon/gems/rake-0.9.2.2/lib/rake/task.rb:200:in `execute'
from /Users/mconway/.rvm/gems/ruby-1.9.3-p125-falcon/gems/rake-0.9.2.2/lib/rake/task.rb:158:in `block in invoke_with_call_chain'
from /Users/mconway/.rvm/rubies/ruby-1.9.3-p125-falcon/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'
from /Users/mconway/.rvm/gems/ruby-1.9.3-p125-falcon/gems/rake-0.9.2.2/lib/rake/task.rb:151:in `invoke_with_call_chain'
from /Users/mconway/.rvm/gems/ruby-1.9.3-p125-falcon/gems/rake-0.9.2.2/lib/rake/task.rb:144:in `invoke'
from /Users/mconway/.rvm/gems/ruby-1.9.3-p125-falcon/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool/cli.rb:110:in `start_pool'
from /Users/mconway/.rvm/gems/ruby-1.9.3-p125-falcon/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool/cli.rb:16:in `run'
from /Users/mconway/.rvm/gems/ruby-1.9.3-p125-falcon/bundler/gems/resque-pool-774e1e384d5c/bin/resque-pool:5:in `<top (required)>'
from /Users/mconway/.rvm/gems/ruby-1.9.3-p125-falcon/bin/resque-pool:19:in `load'
from /Users/mconway/.rvm/gems/ruby-1.9.3-p125-falcon/bin/resque-pool:19:in `<main>'
Discussion: Use a barrier to ensure all clients move at the same time
I've been thinking about ways to coordinate failover and ensure that all clients move at the same time to the new master.
The Zookeeper programming tutorial talks about using barriers to ensure that a certain number of clients are ready before work is done.
What I've come up with for how to get this coordination is as follows.
Example tree
/
redis/
clients/
<...>
waiting_clients/
<...>
monitor_master
config
Definitions:
/redis/clients: A directory that contains an entry per active client process/redis/waiting_clients: A directory (usually empty) used to coordinate failover/redis/monitor_master: The lock file for the node monitor processes/redis/config: The JSON config data
Client Workflow
- When a client starts, register an ephemeral znode under a given root (say
/redis/clientsfor the sake of this discussion) - If
/redis/configexists, read the config and operate as usual - If
/redis/configdoesn't exist (or is removed), block redis commands and create an ephemeral node in/redis/waiting_clients - As soon as
/redis/configexists again, unblock redis commands and remove the entry from/redis/waiting_clients
Server Workflow
- When a Redis failure is detected or a failover is initiated, delete the entry for
/redis/config - Monitor
/redis/waiting_clientsand/redis/clientsuntil the number of entries in both directories is the same (which means everyone is blocked) - Create a new entry for
/redis/configwith the new data
Other thoughts
- It may be worth while to create some sort of timeout for how long to wait for all of the clients to participate, but I'm not sure if that is required โ because all of the clients are using ephemeral nodes, it shouldn't take long for zookeeper to kill the session, right? (we should test this)
- I've noticed that you changed the config znode to be ephemeral โ I personally would think I would rather have the znode be persistent โ if no node watchers are running, I would much rather have everyone still agreeing on a node instead of having no one to talk to.
Support Ruby 2.0
redis_failover does not shutdown properly in ruby 2.0+. This is due to a change in how signal handlers work. We should ensure that all tests pass and that redis_failover works properly in the latest stable Ruby release.
License missing from gemspec
RubyGems.org doesn't report a license for your gem. This is because it is not specified in the gemspec of your last release.
via e.g.
spec.license = 'MIT'
# or
spec.licenses = ['MIT', 'GPL-2']
Including a license in your gemspec is an easy way for rubygems.org and other tools to check how your gem is licensed. As you can imagine, scanning your repository for a LICENSE file or parsing the README, and then attempting to identify the license or licenses is much more difficult and more error prone. So, even for projects that already specify a license, including a license in your gemspec is a good practice. See, for example, how rubygems.org uses the gemspec to display the rails gem license.
There is even a License Finder gem to help companies/individuals ensure all gems they use meet their licensing needs. This tool depends on license information being available in the gemspec. This is an important enough issue that even Bundler now generates gems with a default 'MIT' license.
I hope you'll consider specifying a license in your gemspec. If not, please just close the issue with a nice message. In either case, I'll follow up. Thanks for your time!
Appendix:
If you need help choosing a license (sorry, I haven't checked your readme or looked for a license file), GitHub has created a license picker tool. Code without a license specified defaults to 'All rights reserved'-- denying others all rights to use of the code.
Here's a list of the license names I've found and their frequencies
p.s. In case you're wondering how I found you and why I made this issue, it's because I'm collecting stats on gems (I was originally looking for download data) and decided to collect license metadata,too, and make issues for gemspecs not specifying a license as a public service :). See the previous link or my blog post about this project for more information.
Undefined method 'job' for main object with Resque
Getting a really weird error after implementing Redis Failover. Every resque job is failing with:
undefined local variable or method `job' for main:Object
with no backtrace.
I've been trying to localize it down to either the Resque::Job class or Resque:::Worker class with little success. The only thing that has changed though is using redis failover.
Redis Failover is working perfectly otherwise. It would seem that all writes and reads are happening but none of my resque jobs seem to be able to work on jobs correctly.
I can't seem to replicate this in development which is also disconcerting. Only staging and production.
Hanging rake task due to lock contention when using multiple redis failover client connections in a threadpool
Here's another one for ya :)
Today we started noticing that Rake tasks were hanging after completion. We did a thread dump on one that was hung and saw this:
https://gist.github.com/b3852a8ea44cc1cf36e1
The code that ended up fixing it was changing this:
https://gist.github.com/95316d829d24627b6937
to this:
https://gist.github.com/9d70da788596c8192572
For Sidekiq server instances we are still using a connection pool of worker_thread_count + 2 and haven't seen this same issue. I did notice though on occasion that when closing a rails console with allot of open redis failover client connections open it'll hang. Perhaps it relates to connection cleanup?
Rework specs to work against a set of real Redis/ZooKeeper nodes as opposed to stubs
Right now the specs do a good job of testing core logic, but unfortunately they go to stubbed redis/zk clients. Ideally we should automatically launch a mini ZK/Redis cluster and perform the tests against that.
Continuously swapping master redis_node_manager
We're running 7 instances of redis node manager in production, one per app server. There was a connection timeout with zookeeper on a few clients ZK::Exceptions::OperationTimeOut: inputs: {:path=>"/redis_failover_nodes"} because we took down one of our app servers to move it to a different host. The loss of connection seemed to cause it to start switching between master node manager about once every 20 seconds with the error:
ZK::Exceptions::LockAssertionFailedError: we do not actually hold the lock
https://gist.github.com/3feb567ff0374be12757
Any ideas?
ruby1.9 ไธๅฏๅจๆฅ้
[root@DB2 ~]#redis_node_manager -n 10.46.73.15:6379,10.46.78.66:6379 -z 10.46.73.15:2181,10.46.78.66:2181
/usr/local/ruby/lib/ruby/1.9.1/yaml.rb:84:in <top (required)>': It seems your ruby installation is missing psych (for YAML output). To eliminate this warning, please install libyaml and reinstall your ruby. 2015-12-23 02:14:56 UTC RedisFailover 24708 INFO: Redis Node Manager v1.0.2 starting (ruby 1.9.3p551 (2014-11-13 revision 48407) [x86_64-linux]) 2015-12-23 02:14:56 UTC RedisFailover 24708 INFO: Created ZK node /redis_failover 2015-12-23 02:14:56 UTC RedisFailover 24708 INFO: Created ZK node /redis_failover/manager_node_state /usr/local/ruby/lib/ruby/gems/1.9.1/gems/redis_failover-1.0.2/lib/redis_failover/node_manager.rb:429:ingethostbyname': getaddrinfo: Name or service not known (SocketError)
from /usr/local/ruby/lib/ruby/gems/1.9.1/gems/redis_failover-1.0.2/lib/redis_failover/node_manager.rb:429:in manager_id' from /usr/local/ruby/lib/ruby/gems/1.9.1/gems/redis_failover-1.0.2/lib/redis_failover/node_manager.rb:454:incurrent_state_path'
from /usr/local/ruby/lib/ruby/gems/1.9.1/gems/redis_failover-1.0.2/lib/redis_failover/node_manager.rb:289:in spawn_watchers' from /usr/local/ruby/lib/ruby/gems/1.9.1/gems/redis_failover-1.0.2/lib/redis_failover/node_manager.rb:58:instart'
from /usr/local/ruby/lib/ruby/gems/1.9.1/gems/redis_failover-1.0.2/lib/redis_failover/runner.rb:13:in run' from /usr/local/ruby/lib/ruby/gems/1.9.1/gems/redis_failover-1.0.2/bin/redis_node_manager:6:in<top (required)>'
from /usr/local/ruby/bin/redis_node_manager:23:in load' from /usr/local/ruby/bin/redis_node_manager:23:in
Existing connections get dropped when I start a new process
If I've been running for a while ad the RedisFailover connection has gotten exercised, when I start a new process that uses RedisFailover, it causes the existing one to throw an exception:
==> log/resque-pool.log <==
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/redis-2.2.2/lib/redis.rb:93:in `[]': odd number of arguments for Hash (ArgumentError)
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/redis-2.2.2/lib/redis.rb:93:in `block in info'
from /usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/redis-2.2.2/lib/redis.rb:89:in `info'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/redis_failover-0.8.3/lib/redis_failover/client.rb:339:in `verify_role!'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/redis_failover-0.8.3/lib/redis_failover/client.rb:247:in `master'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/redis_failover-0.8.3/lib/redis_failover/client.rb:264:in `slave'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/redis_failover-0.8.3/lib/redis_failover/client.rb:433:in `client_for'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/redis_failover-0.8.3/lib/redis_failover/client.rb:220:in `dispatch'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/redis_failover-0.8.3/lib/redis_failover/client.rb:82:in `block (2 levels) in <class:Client>'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/redis-namespace-1.0.3/lib/redis/namespace.rb:213:in `method_missing'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-fcaf987d0724/lib/resque/worker.rb:27:in `all'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-fcaf987d0724/lib/resque.rb:311:in `workers'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:302:in `orphaned_count'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:316:in `worker_delta_for'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:272:in `block in maintain_worker_count'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:271:in `each'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:271:in `maintain_worker_count'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:212:in `block in join'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:207:in `loop'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:207:in `join'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:65:in `run'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool/tasks.rb:17:in `block (2 levels) in <top (required)>'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:205:in `call'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:205:in `block in execute'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:200:in `each'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:200:in `execute'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:158:in `block in invoke_with_call_chain'
from /usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:151:in `invoke_with_call_chain'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:144:in `invoke'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool/cli.rb:110:in `start_pool'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool/cli.rb:16:in `run'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/bin/resque-pool:5:in `<top (required)>'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bin/resque-pool:19:in `load'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bin/resque-pool:19:in `<main>'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/redis_failover-0.8.3/lib/redis_failover/client.rb:339:in `[]': can't convert String into Integer (TypeError)
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/redis_failover-0.8.3/lib/redis_failover/client.rb:339:in `verify_role!'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/redis_failover-0.8.3/lib/redis_failover/client.rb:247:in `master'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/redis_failover-0.8.3/lib/redis_failover/client.rb:264:in `slave'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/redis_failover-0.8.3/lib/redis_failover/client.rb:433:in `client_for'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/redis_failover-0.8.3/lib/redis_failover/client.rb:220:in `dispatch'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/redis_failover-0.8.3/lib/redis_failover/client.rb:82:in `block (2 levels) in <class:Client>'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/redis-namespace-1.0.3/lib/redis/namespace.rb:213:in `method_missing'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-fcaf987d0724/lib/resque/worker.rb:430:in `job'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-fcaf987d0724/lib/resque/worker.rb:362:in `unregister_worker'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-fcaf987d0724/lib/resque/worker.rb:145:in `ensure in work'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-fcaf987d0724/lib/resque/worker.rb:145:in `work'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:332:in `block in spawn_worker!'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:327:in `fork'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:327:in `spawn_worker!'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:287:in `block in spawn_missing_workers_for'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:287:in `times'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:287:in `spawn_missing_workers_for'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:273:in `block in maintain_worker_count'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:271:in `each'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:271:in `maintain_worker_count'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:191:in `start'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:65:in `run'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool/tasks.rb:17:in `block (2 levels) in <top (required)>'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:205:in `call'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:205:in `block in execute'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:200:in `each'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:200:in `execute'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:158:in `block in invoke_with_call_chain'
from /usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:151:in `invoke_with_call_chain'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:144:in `invoke'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool/cli.rb:110:in `start_pool'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool/cli.rb:16:in `run'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/bin/resque-pool:5:in `<top (required)>'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bin/resque-pool:19:in `load'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bin/resque-pool:19:in `<main>'
==> log/matt.log <==
ERROR 08:58:15 rails::RedisFailover [pid=31085]: Error while handling `del` - #<Errno::EAGAIN: Resource temporarily unavailable - Timeout reading from the socket>
ERROR 08:58:15 rails::RedisFailover [pid=31085]: /mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/pry-exception_explorer-0.1.9/lib/pry-exception_explorer/core_ext.rb:46:in `raise'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/redis-2.2.2/lib/redis/client.rb:170:in `rescue in read'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/redis-2.2.2/lib/redis/client.rb:161:in `read'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/redis-2.2.2/lib/redis/client.rb:46:in `block in call'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/redis-2.2.2/lib/redis/client.rb:142:in `block (2 levels) in process'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/redis-2.2.2/lib/redis/client.rb:250:in `ensure_connected'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/redis-2.2.2/lib/redis/client.rb:137:in `block in process'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/redis-2.2.2/lib/redis/client.rb:206:in `logging'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/redis-2.2.2/lib/redis/client.rb:136:in `process'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/redis-2.2.2/lib/redis/client.rb:46:in `call'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/redis-2.2.2/lib/redis.rb:681:in `block in del'
/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/redis-2.2.2/lib/redis.rb:680:in `del'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/redis_failover-0.8.3/lib/redis_failover/client.rb:220:in `dispatch'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/redis_failover-0.8.3/lib/redis_failover/client.rb:82:in `block (2 levels) in <class:Client>'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/redis-namespace-1.0.3/lib/redis/namespace.rb:213:in `method_missing'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-fcaf987d0724/lib/resque/stat.rb:50:in `clear'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-fcaf987d0724/lib/resque/worker.rb:374:in `unregister_worker'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-fcaf987d0724/lib/resque/worker.rb:145:in `ensure in work'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-fcaf987d0724/lib/resque/worker.rb:145:in `work'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:332:in `block in spawn_worker!'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:327:in `fork'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:327:in `spawn_worker!'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:287:in `block in spawn_missing_workers_for'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:287:in `times'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:287:in `spawn_missing_workers_for'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:273:in `block in maintain_worker_count'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:271:in `each'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:271:in `maintain_worker_count'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:191:in `start'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:65:in `run'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool/tasks.rb:17:in `block (2 levels) in <top (required)>'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:205:in `call'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:205:in `block in execute'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:200:in `each'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:200:in `execute'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:158:in `block in invoke_with_call_chain'
/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:151:in `invoke_with_call_chain'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:144:in `invoke'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool/cli.rb:110:in `start_pool'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool/cli.rb:16:in `run'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/bin/resque-pool:5:in `<top (required)>'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bin/resque-pool:19:in `load'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bin/resque-pool:19:in `<main>'
==> log/resque-pool.log <==
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/pry-exception_explorer-0.1.9/lib/pry-exception_explorer/core_ext.rb:46:in `raise': 756: unexpected token at 're (Resque::Helpers::DecodeException)
redis_git_sha1:00000000
redis_git_dirty:0
arch_bits:64
multiplexing_api:epoll
gcc_version:4.6.3
process_id:19181
uptime_in_seconds:1302
uptime_in_days:0
lru_clock:1579453
used_cpu_sys:0.61
used_cpu_user:0.39
used_cpu_sys_children:0.00
used_cpu_user_children:0.00
connected_clients:6
connected_slaves:0
client_longest_output_list:0
client_biggest_input_buf:0
blocked_clients:1
used_memory:771280
used_memory_human:753.20K
used_memory_rss:991232
used_memory_peak:799528
used_memory_peak_human:780.79K
mem_fragmentation_ratio:1.29
mem_allocator:jemalloc-2.2.5
loading:0
aof_enabled:0
changes_since_last_save:26
bgsave_in_progress:0
last_save_time:1336998988
bgrewriteaof_in_progress:0
total_connections_received:722
total_commands_processed:3760
expired_keys:0
evicted_keys:0
keyspace_hits:460
keyspace_misses:922
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:0
vm_enabled:0
role:master
db1:keys=4,expires=0
'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-fcaf987d0724/lib/resque/helpers.rb:32:in `rescue in decode'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-fcaf987d0724/lib/resque/helpers.rb:29:in `decode'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/resque-dynamic-queues-0.8.1/lib/resque/plugins/dynamic_queues/attributes.rb:16:in `get_dynamic_queue'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/resque-dynamic-queues-0.8.1/lib/resque/plugins/dynamic_queues/queues.rb:34:in `queues_with_dynamic'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/resque-queue-priority-0.6.1/lib/resque/plugins/queue_priority/priority.rb:17:in `queues_with_priority'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-fcaf987d0724/lib/resque/worker.rb:182:in `reserve'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/resque-concurrent-restriction-0.5.4/lib/resque/plugins/concurrent_restriction/resque_worker_extension.rb:19:in `reserve_with_restriction'
from /mnt/appleton-matt/releases/20120514124847/config/initializers/resque.rb:55:in `reserve'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-fcaf987d0724/lib/resque/worker.rb:119:in `block in work'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-fcaf987d0724/lib/resque/worker.rb:116:in `loop'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-fcaf987d0724/lib/resque/worker.rb:116:in `work'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:332:in `block in spawn_worker!'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:327:in `fork'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:327:in `spawn_worker!'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:287:in `block in spawn_missing_workers_for'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:287:in `times'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:287:in `spawn_missing_workers_for'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:273:in `block in maintain_worker_count'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:271:in `each'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:271:in `maintain_worker_count'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:191:in `start'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:65:in `run'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool/tasks.rb:17:in `block (2 levels) in <top (required)>'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:205:in `call'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:205:in `block in execute'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:200:in `each'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:200:in `execute'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:158:in `block in invoke_with_call_chain'
from /usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:151:in `invoke_with_call_chain'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:144:in `invoke'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool/cli.rb:110:in `start_pool'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool/cli.rb:16:in `run'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/bin/resque-pool:5:in `<top (required)>'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bin/resque-pool:19:in `load'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bin/resque-pool:19:in `<main>'
tail: log/resque-pool.log: file truncated
resque-pool-manager[599]: Resque Pool running in matt environment
:public is no longer used to avoid overloading Module#public, use :public_folder instead
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-fcaf987d0724/lib/resque/server.rb:12:in `<class:Server>'
resque-pool-manager[599]: started manager
resque-pool-manager[599]: Pool contains worker PIDs: [736]
resque-pool-worker[736]: Starting worker matt.appleton.backupify.com:736:@matt
==> log/matt.log <==
DEBUG 08:59:12 rails::RedisFactory [pid=736]: RedisFactory.reconnect start
DEBUG 08:59:12 rails::RedisFactory [pid=736]: RedisFactory.reconnect complete
==> log/resque-pool.log <==
resque-pool-worker[1772]: Starting worker matt.appleton.backupify.com:1772:@matt
==> log/matt.log <==
DEBUG 09:00:12 rails::RedisFactory [pid=1772]: RedisFactory.reconnect start
DEBUG 09:00:12 rails::RedisFactory [pid=1772]: RedisFactory.reconnect complete
DEBUG 09:03:13 rails::RedisFactory [pid=2436]: RedisFactory.reconnect start
DEBUG 09:03:13 rails::RedisFactory [pid=2436]: RedisFactory.reconnect complete
INFO 09:03:13 rails::Jobs::Dummy [pid=2436]: Started Dummy Job []
INFO 09:03:13 rails::Jobs::Dummy [pid=2436]: Dummy Job sleeping for 8 seconds
INFO 09:03:21 rails::Jobs::Dummy [pid=2436]: Ended Dummy Job []
DEBUG 09:04:01 rails::RedisFactory [pid=2674]: RedisFactory.reconnect start
DEBUG 09:04:01 rails::RedisFactory [pid=2674]: RedisFactory.reconnect complete
INFO 09:04:01 rails::Jobs::Dummy [pid=2674]: Started Dummy Job [{"error"=>"bad"}]
DEBUG 09:12:33 rails::Backupify::RedisCacheStore [pid=32207 user_id=1]: Cache read: https://matt.appleton.backupify.com/?
INFO 09:12:33 rails [pid=32207 user_id=1]:
Started GET "/" for 10.76.87.52 at 2012-05-14 09:12:33 -0400
INFO 09:12:33 rails::controllers [pid=32207 user_id=1]: Processing by WelcomeController#index as HTML
DEBUG 09:12:33 rails [pid=32207 user_id=1]: Existing local CAS session detected for "[email protected]". Previous ticket "ST-1337000242r236A6C6164B7DAA431" will be re-used.
DEBUG 09:12:33 rails::models [pid=32207]: User Load (2.4ms) SELECT "users".* FROM "users" WHERE "users"."cas_ident" = '[email protected]' LIMIT 1
DEBUG 09:12:33 rails::models [pid=32207 user_id=1]: Service Load (1.7ms) SELECT "services".* FROM "services" WHERE "services"."user_id" = 1
INFO 09:12:33 rails::controllers [pid=32207 user_id=1]: Redirected to https://matt.appleton.backupify.com/services/new
INFO 09:12:33 rails::controllers [pid=32207 user_id=1]: Completed 302 Found in 49ms (ActiveRecord: 4.1ms)
DEBUG 09:12:34 rails::Backupify::RedisCacheStore [pid=32207 user_id=1]: Cache read: https://matt.appleton.backupify.com/services/new?
INFO 09:12:34 rails [pid=32207 user_id=1]:
Started GET "/services/new" for 10.76.87.52 at 2012-05-14 09:12:34 -0400
INFO 09:12:34 rails::controllers [pid=32207 user_id=1]: Processing by ServicesController#new as HTML
DEBUG 09:12:34 rails [pid=32207 user_id=1]: Existing local CAS session detected for "[email protected]". Previous ticket "ST-1337000242r236A6C6164B7DAA431" will be re-used.
DEBUG 09:12:34 rails::models [pid=32207]: User Load (0.8ms) SELECT "users".* FROM "users" WHERE "users"."cas_ident" = '[email protected]' LIMIT 1
INFO 09:12:34 rails::controllers [pid=32207 user_id=1]: Rendered services/new.html.erb within layouts/application (0.8ms)
DEBUG 09:12:34 rails::models [pid=32207 user_id=1]: (1.7ms) SELECT COUNT(*) FROM "services" WHERE "services"."user_id" = 1
INFO 09:12:34 rails::controllers [pid=32207 user_id=1]: Rendered shared/_flashes.html.erb (0.1ms)
INFO 09:12:34 rails::controllers [pid=32207 user_id=1]: Completed 200 OK in 28ms (Views: 13.7ms | ActiveRecord: 2.5ms)
DEBUG 09:12:39 rails::Backupify::RedisCacheStore [pid=32207 user_id=1]: Cache read: https://matt.appleton.backupify.com/twitter_services/new?
INFO 09:12:39 rails [pid=32207 user_id=1]:
Started GET "/twitter_services/new" for 10.76.87.52 at 2012-05-14 09:12:39 -0400
INFO 09:12:39 rails::controllers [pid=32207 user_id=1]: Processing by TwitterServicesController#new as HTML
DEBUG 09:12:39 rails [pid=32207 user_id=1]: Existing local CAS session detected for "[email protected]". Previous ticket "ST-1337000242r236A6C6164B7DAA431" will be re-used.
DEBUG 09:12:39 rails::models [pid=32207]: User Load (0.8ms) SELECT "users".* FROM "users" WHERE "users"."cas_ident" = '[email protected]' LIMIT 1
INFO 09:12:40 rails::controllers [pid=32207 user_id=1]: Rendered twitter_services/new.html.erb within layouts/application (1.2ms)
DEBUG 09:12:40 rails::models [pid=32207 user_id=1]: (0.8ms) SELECT COUNT(*) FROM "services" WHERE "services"."user_id" = 1
INFO 09:12:40 rails::controllers [pid=32207 user_id=1]: Rendered shared/_flashes.html.erb (0.1ms)
INFO 09:12:40 rails::controllers [pid=32207 user_id=1]: Completed 200 OK in 325ms (Views: 22.1ms | ActiveRecord: 1.6ms)
DEBUG 09:12:41 rails::Backupify::RedisCacheStore [pid=32207 user_id=1]: Cache read: https://matt.appleton.backupify.com/?
INFO 09:12:41 rails [pid=32207 user_id=1]:
Started GET "/" for 10.76.87.52 at 2012-05-14 09:12:41 -0400
INFO 09:12:41 rails::controllers [pid=32207 user_id=1]: Processing by WelcomeController#index as HTML
DEBUG 09:12:41 rails [pid=32207 user_id=1]: Existing local CAS session detected for "[email protected]". Previous ticket "ST-1337000242r236A6C6164B7DAA431" will be re-used.
DEBUG 09:12:41 rails::models [pid=32207]: User Load (0.8ms) SELECT "users".* FROM "users" WHERE "users"."cas_ident" = '[email protected]' LIMIT 1
DEBUG 09:12:41 rails::models [pid=32207 user_id=1]: Service Load (0.5ms) SELECT "services".* FROM "services" WHERE "services"."user_id" = 1
INFO 09:12:41 rails::controllers [pid=32207 user_id=1]: Redirected to https://matt.appleton.backupify.com/services/new
INFO 09:12:41 rails::controllers [pid=32207 user_id=1]: Completed 302 Found in 9ms (ActiveRecord: 1.3ms)
DEBUG 09:12:41 rails::Backupify::RedisCacheStore [pid=32207 user_id=1]: Cache read: https://matt.appleton.backupify.com/services/new?
INFO 09:12:42 rails [pid=32207 user_id=1]:
Started GET "/services/new" for 10.76.87.52 at 2012-05-14 09:12:42 -0400
INFO 09:12:42 rails::controllers [pid=32207 user_id=1]: Processing by ServicesController#new as HTML
DEBUG 09:12:42 rails [pid=32207 user_id=1]: Existing local CAS session detected for "[email protected]". Previous ticket "ST-1337000242r236A6C6164B7DAA431" will be re-used.
DEBUG 09:12:42 rails::models [pid=32207]: User Load (0.7ms) SELECT "users".* FROM "users" WHERE "users"."cas_ident" = '[email protected]' LIMIT 1
INFO 09:12:42 rails::controllers [pid=32207 user_id=1]: Rendered services/new.html.erb within layouts/application (0.6ms)
DEBUG 09:12:42 rails::models [pid=32207 user_id=1]: (0.5ms) SELECT COUNT(*) FROM "services" WHERE "services"."user_id" = 1
INFO 09:12:42 rails::controllers [pid=32207 user_id=1]: Rendered shared/_flashes.html.erb (0.1ms)
INFO 09:12:42 rails::controllers [pid=32207 user_id=1]: Completed 200 OK in 10ms (Views: 6.2ms | ActiveRecord: 1.2ms)
DEBUG 09:12:42 rails::Backupify::RedisCacheStore [pid=32207 user_id=1]: Cache read: https://matt.appleton.backupify.com/services/new?
INFO 09:12:42 rails [pid=32207 user_id=1]:
Started GET "/services/new" for 10.76.87.52 at 2012-05-14 09:12:42 -0400
INFO 09:12:42 rails::controllers [pid=32207 user_id=1]: Processing by ServicesController#new as HTML
DEBUG 09:12:42 rails [pid=32207 user_id=1]: Existing local CAS session detected for "[email protected]". Previous ticket "ST-1337000242r236A6C6164B7DAA431" will be re-used.
DEBUG 09:12:42 rails::models [pid=32207]: User Load (0.8ms) SELECT "users".* FROM "users" WHERE "users"."cas_ident" = '[email protected]' LIMIT 1
INFO 09:12:42 rails::controllers [pid=32207 user_id=1]: Rendered services/new.html.erb within layouts/application (0.5ms)
DEBUG 09:12:42 rails::models [pid=32207 user_id=1]: (1.0ms) SELECT COUNT(*) FROM "services" WHERE "services"."user_id" = 1
INFO 09:12:43 rails::controllers [pid=32207 user_id=1]: Rendered shared/_flashes.html.erb (0.1ms)
INFO 09:12:43 rails::controllers [pid=32207 user_id=1]: Completed 200 OK in 13ms (Views: 9.3ms | ActiveRecord: 1.8ms)
DEBUG 09:12:44 rails::Backupify::RedisCacheStore [pid=32207 user_id=1]: Cache read: https://matt.appleton.backupify.com/?
INFO 09:12:44 rails [pid=32207 user_id=1]:
Started GET "/" for 10.76.87.52 at 2012-05-14 09:12:44 -0400
INFO 09:12:44 rails::controllers [pid=32207 user_id=1]: Processing by WelcomeController#index as HTML
DEBUG 09:12:44 rails [pid=32207 user_id=1]: Existing local CAS session detected for "[email protected]". Previous ticket "ST-1337000242r236A6C6164B7DAA431" will be re-used.
DEBUG 09:12:44 rails::models [pid=32207]: User Load (0.8ms) SELECT "users".* FROM "users" WHERE "users"."cas_ident" = '[email protected]' LIMIT 1
DEBUG 09:12:44 rails::models [pid=32207 user_id=1]: Service Load (0.5ms) SELECT "services".* FROM "services" WHERE "services"."user_id" = 1
INFO 09:12:44 rails::controllers [pid=32207 user_id=1]: Redirected to https://matt.appleton.backupify.com/services/new
INFO 09:12:44 rails::controllers [pid=32207 user_id=1]: Completed 302 Found in 66ms (ActiveRecord: 1.3ms)
DEBUG 09:12:44 rails::Backupify::RedisCacheStore [pid=32207 user_id=1]: Cache read: https://matt.appleton.backupify.com/services/new?
INFO 09:12:44 rails [pid=32207 user_id=1]:
Started GET "/services/new" for 10.76.87.52 at 2012-05-14 09:12:44 -0400
INFO 09:12:44 rails::controllers [pid=32207 user_id=1]: Processing by ServicesController#new as HTML
DEBUG 09:12:44 rails [pid=32207 user_id=1]: Existing local CAS session detected for "[email protected]". Previous ticket "ST-1337000242r236A6C6164B7DAA431" will be re-used.
DEBUG 09:12:44 rails::models [pid=32207]: User Load (0.7ms) SELECT "users".* FROM "users" WHERE "users"."cas_ident" = '[email protected]' LIMIT 1
INFO 09:12:44 rails::controllers [pid=32207 user_id=1]: Rendered services/new.html.erb within layouts/application (0.7ms)
DEBUG 09:12:44 rails::models [pid=32207 user_id=1]: (0.7ms) SELECT COUNT(*) FROM "services" WHERE "services"."user_id" = 1
INFO 09:12:44 rails::controllers [pid=32207 user_id=1]: Rendered shared/_flashes.html.erb (0.1ms)
INFO 09:12:44 rails::controllers [pid=32207 user_id=1]: Completed 200 OK in 13ms (Views: 6.9ms | ActiveRecord: 1.4ms)
INFO 09:14:23 rails [pid=6349]: Compiled jquery.js (2ms) (pid 6349)
INFO 09:14:23 rails [pid=6349]: Compiled jquery_ujs.js (0ms) (pid 6349)
INFO 09:14:23 rails [pid=6349]: Compiled jquery.fancybox-1.3.4.js (0ms) (pid 6349)
INFO 09:14:23 rails [pid=6349]: Compiled twitter/bootstrap/bootstrap-transition.js (0ms) (pid 6349)
INFO 09:14:23 rails [pid=6349]: Compiled twitter/bootstrap/bootstrap-alert.js (0ms) (pid 6349)
INFO 09:14:23 rails [pid=6349]: Compiled twitter/bootstrap/bootstrap-modal.js (0ms) (pid 6349)
INFO 09:14:23 rails [pid=6349]: Compiled twitter/bootstrap/bootstrap-dropdown.js (0ms) (pid 6349)
INFO 09:14:23 rails [pid=6349]: Compiled twitter/bootstrap/bootstrap-scrollspy.js (0ms) (pid 6349)
INFO 09:14:23 rails [pid=6349]: Compiled twitter/bootstrap/bootstrap-tab.js (0ms) (pid 6349)
INFO 09:14:23 rails [pid=6349]: Compiled twitter/bootstrap/bootstrap-tooltip.js (0ms) (pid 6349)
INFO 09:14:23 rails [pid=6349]: Compiled twitter/bootstrap/bootstrap-popover.js (0ms) (pid 6349)
INFO 09:14:24 rails [pid=6349]: Compiled twitter/bootstrap/bootstrap-button.js (0ms) (pid 6349)
INFO 09:14:24 rails [pid=6349]: Compiled twitter/bootstrap/bootstrap-collapse.js (0ms) (pid 6349)
INFO 09:14:24 rails [pid=6349]: Compiled twitter/bootstrap/bootstrap-carousel.js (0ms) (pid 6349)
INFO 09:14:24 rails [pid=6349]: Compiled twitter/bootstrap/bootstrap-typeahead.js (0ms) (pid 6349)
INFO 09:14:24 rails [pid=6349]: Compiled twitter/bootstrap.js (232ms) (pid 6349)
INFO 09:14:25 rails [pid=6349]: Compiled bootstrap.js (993ms) (pid 6349)
INFO 09:14:25 rails [pid=6349]: Compiled twitter_pdf_exports.js (19ms) (pid 6349)
INFO 09:14:25 rails [pid=6349]: Compiled application.js (1342ms) (pid 6349)
INFO 09:14:38 rails [pid=6349]: Compiled bootstrap_and_overrides.css (5130ms) (pid 6349)
INFO 09:14:41 rails [pid=6349]: Compiled jquery.fancybox-1.3.4.css (2797ms) (pid 6349)
INFO 09:14:41 rails [pid=6349]: Compiled twitter_pdf_exports.css (1ms) (pid 6349)
INFO 09:14:41 rails [pid=6349]: Compiled application.css (8163ms) (pid 6349)
INFO 09:15:12 rails [pid=6349]: Compiled jquery.js (2ms) (pid 6349)
INFO 09:15:12 rails [pid=6349]: Compiled jquery_ujs.js (0ms) (pid 6349)
INFO 09:15:12 rails [pid=6349]: Compiled jquery.fancybox-1.3.4.js (71ms) (pid 6349)
INFO 09:15:12 rails [pid=6349]: Compiled twitter/bootstrap/bootstrap-transition.js (0ms) (pid 6349)
INFO 09:15:12 rails [pid=6349]: Compiled twitter/bootstrap/bootstrap-alert.js (0ms) (pid 6349)
INFO 09:15:12 rails [pid=6349]: Compiled twitter/bootstrap/bootstrap-modal.js (0ms) (pid 6349)
INFO 09:15:12 rails [pid=6349]: Compiled twitter/bootstrap/bootstrap-dropdown.js (0ms) (pid 6349)
INFO 09:15:12 rails [pid=6349]: Compiled twitter/bootstrap/bootstrap-scrollspy.js (0ms) (pid 6349)
INFO 09:15:12 rails [pid=6349]: Compiled twitter/bootstrap/bootstrap-tab.js (0ms) (pid 6349)
INFO 09:15:12 rails [pid=6349]: Compiled twitter/bootstrap/bootstrap-tooltip.js (0ms) (pid 6349)
INFO 09:15:12 rails [pid=6349]: Compiled twitter/bootstrap/bootstrap-popover.js (0ms) (pid 6349)
INFO 09:15:12 rails [pid=6349]: Compiled twitter/bootstrap/bootstrap-button.js (0ms) (pid 6349)
INFO 09:15:12 rails [pid=6349]: Compiled twitter/bootstrap/bootstrap-collapse.js (0ms) (pid 6349)
INFO 09:15:12 rails [pid=6349]: Compiled twitter/bootstrap/bootstrap-carousel.js (0ms) (pid 6349)
INFO 09:15:12 rails [pid=6349]: Compiled twitter/bootstrap/bootstrap-typeahead.js (0ms) (pid 6349)
INFO 09:15:12 rails [pid=6349]: Compiled twitter/bootstrap.js (206ms) (pid 6349)
INFO 09:15:13 rails [pid=6349]: Compiled bootstrap.js (472ms) (pid 6349)
INFO 09:15:13 rails [pid=6349]: Compiled twitter_pdf_exports.js (18ms) (pid 6349)
INFO 09:15:13 rails [pid=6349]: Compiled application.js (1037ms) (pid 6349)
INFO 09:24:19 rails [pid=10500]: Compiled jquery.js (2ms) (pid 10500)
INFO 09:24:19 rails [pid=10500]: Compiled jquery_ujs.js (0ms) (pid 10500)
INFO 09:24:19 rails [pid=10500]: Compiled jquery.fancybox-1.3.4.js (0ms) (pid 10500)
INFO 09:24:19 rails [pid=10500]: Compiled twitter/bootstrap/bootstrap-transition.js (0ms) (pid 10500)
INFO 09:24:19 rails [pid=10500]: Compiled twitter/bootstrap/bootstrap-alert.js (0ms) (pid 10500)
INFO 09:24:19 rails [pid=10500]: Compiled twitter/bootstrap/bootstrap-modal.js (0ms) (pid 10500)
INFO 09:24:19 rails [pid=10500]: Compiled twitter/bootstrap/bootstrap-dropdown.js (0ms) (pid 10500)
INFO 09:24:19 rails [pid=10500]: Compiled twitter/bootstrap/bootstrap-scrollspy.js (0ms) (pid 10500)
INFO 09:24:19 rails [pid=10500]: Compiled twitter/bootstrap/bootstrap-tab.js (0ms) (pid 10500)
INFO 09:24:19 rails [pid=10500]: Compiled twitter/bootstrap/bootstrap-tooltip.js (0ms) (pid 10500)
INFO 09:24:19 rails [pid=10500]: Compiled twitter/bootstrap/bootstrap-popover.js (0ms) (pid 10500)
INFO 09:24:19 rails [pid=10500]: Compiled twitter/bootstrap/bootstrap-button.js (0ms) (pid 10500)
INFO 09:24:19 rails [pid=10500]: Compiled twitter/bootstrap/bootstrap-collapse.js (0ms) (pid 10500)
INFO 09:24:19 rails [pid=10500]: Compiled twitter/bootstrap/bootstrap-carousel.js (0ms) (pid 10500)
INFO 09:24:19 rails [pid=10500]: Compiled twitter/bootstrap/bootstrap-typeahead.js (0ms) (pid 10500)
INFO 09:24:19 rails [pid=10500]: Compiled twitter/bootstrap.js (160ms) (pid 10500)
INFO 09:24:19 rails [pid=10500]: Compiled bootstrap.js (134ms) (pid 10500)
INFO 09:24:19 rails [pid=10500]: Compiled twitter_pdf_exports.js (21ms) (pid 10500)
INFO 09:24:19 rails [pid=10500]: Compiled application.js (537ms) (pid 10500)
INFO 09:24:26 rails [pid=10500]: Compiled bootstrap_and_overrides.css (1270ms) (pid 10500)
INFO 09:24:26 rails [pid=10500]: Compiled jquery.fancybox-1.3.4.css (435ms) (pid 10500)
INFO 09:24:26 rails [pid=10500]: Compiled twitter_pdf_exports.css (2ms) (pid 10500)
INFO 09:24:26 rails [pid=10500]: Compiled application.css (1944ms) (pid 10500)
INFO 09:24:57 rails [pid=10500]: Compiled jquery.js (2ms) (pid 10500)
INFO 09:24:57 rails [pid=10500]: Compiled jquery_ujs.js (0ms) (pid 10500)
INFO 09:24:57 rails [pid=10500]: Compiled jquery.fancybox-1.3.4.js (0ms) (pid 10500)
INFO 09:24:57 rails [pid=10500]: Compiled twitter/bootstrap/bootstrap-transition.js (0ms) (pid 10500)
INFO 09:24:57 rails [pid=10500]: Compiled twitter/bootstrap/bootstrap-alert.js (0ms) (pid 10500)
INFO 09:24:57 rails [pid=10500]: Compiled twitter/bootstrap/bootstrap-modal.js (0ms) (pid 10500)
INFO 09:24:57 rails [pid=10500]: Compiled twitter/bootstrap/bootstrap-dropdown.js (0ms) (pid 10500)
INFO 09:24:57 rails [pid=10500]: Compiled twitter/bootstrap/bootstrap-scrollspy.js (0ms) (pid 10500)
INFO 09:24:57 rails [pid=10500]: Compiled twitter/bootstrap/bootstrap-tab.js (0ms) (pid 10500)
INFO 09:24:57 rails [pid=10500]: Compiled twitter/bootstrap/bootstrap-tooltip.js (0ms) (pid 10500)
INFO 09:24:57 rails [pid=10500]: Compiled twitter/bootstrap/bootstrap-popover.js (0ms) (pid 10500)
INFO 09:24:57 rails [pid=10500]: Compiled twitter/bootstrap/bootstrap-button.js (0ms) (pid 10500)
INFO 09:24:57 rails [pid=10500]: Compiled twitter/bootstrap/bootstrap-collapse.js (0ms) (pid 10500)
INFO 09:24:57 rails [pid=10500]: Compiled twitter/bootstrap/bootstrap-carousel.js (0ms) (pid 10500)
INFO 09:24:57 rails [pid=10500]: Compiled twitter/bootstrap/bootstrap-typeahead.js (0ms) (pid 10500)
INFO 09:24:57 rails [pid=10500]: Compiled twitter/bootstrap.js (178ms) (pid 10500)
INFO 09:24:57 rails [pid=10500]: Compiled bootstrap.js (123ms) (pid 10500)
INFO 09:24:57 rails [pid=10500]: Compiled twitter_pdf_exports.js (19ms) (pid 10500)
INFO 09:24:57 rails [pid=10500]: Compiled application.js (513ms) (pid 10500)
INFO 09:25:04 rails [pid=10500]: Compiled bootstrap_and_overrides.css (1712ms) (pid 10500)
INFO 09:25:04 rails [pid=10500]: Compiled jquery.fancybox-1.3.4.css (270ms) (pid 10500)
INFO 09:25:04 rails [pid=10500]: Compiled twitter_pdf_exports.css (1ms) (pid 10500)
INFO 09:25:04 rails [pid=10500]: Compiled application.css (2251ms) (pid 10500)
==> log/resque-pool.log <==
resque-pool-manager[599]: INT: immediate shutdown (graceful worker shutdown)
resque-pool-manager[599]: manager finished
==> log/resque-web.log <==
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/puma-1.3.0/lib/puma/server.rb:625:in `close': closed stream (IOError)
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/puma-1.3.0/lib/puma/server.rb:625:in `stop'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/puma-1.3.0/lib/puma/cli.rb:429:in `block in run'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/puma-1.3.0/lib/puma/cli.rb:435:in `call'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/puma-1.3.0/lib/puma/cli.rb:435:in `join'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/puma-1.3.0/lib/puma/cli.rb:435:in `run'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/puma-1.3.0/bin/puma:10:in `<top (required)>'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bin/puma:19:in `load'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bin/puma:19:in `<main>'
Puma 1.3.0 starting...
* Min threads: 0, max threads: 16
* Listening on tcp://0.0.0.0:5678
Use Ctrl-C to stop
24.61.131.238 - - [14/May/2012 09:04:13] "GET /dynamicqueues HTTP/1.1" 200 3244 0.3409
24.61.131.238 - - [14/May/2012 09:04:13] "GET /reset.css HTTP/1.1" 304 - 0.0161
24.61.131.238 - - [14/May/2012 09:04:13] "GET /style.css HTTP/1.1" 304 - 0.0008
24.61.131.238 - - [14/May/2012 09:04:13] "GET /jquery.relatize_date.js HTTP/1.1" 304 - 0.0006
24.61.131.238 - - [14/May/2012 09:04:13] "GET /ranger.js HTTP/1.1" 304 - 0.0006
24.61.131.238 - - [14/May/2012 09:04:13] "GET /jquery-1.3.2.min.js HTTP/1.1" 304 - 0.0007
24.61.131.238 - - [14/May/2012 09:04:13] "GET /favicon.ico HTTP/1.1" 200 1150 0.0011
24.61.131.238 - - [14/May/2012 09:04:15] "GET /retry HTTP/1.1" 200 2218 0.0377
24.61.131.238 - - [14/May/2012 09:04:17] "GET /queuepriority HTTP/1.1" 200 3640 0.0250
24.61.131.238 - - [14/May/2012 09:04:18] "GET /dynamicqueues HTTP/1.1" 200 3244 0.0075
24.61.131.238 - - [14/May/2012 09:04:19] "GET /delayed HTTP/1.1" 200 2137 0.0089
24.61.131.238 - - [14/May/2012 09:04:21] "GET /retry HTTP/1.1" 200 2218 0.0084
24.61.131.238 - - [14/May/2012 09:04:35] "GET /retry HTTP/1.1" 200 2218 0.0097
24.61.131.238 - - [14/May/2012 09:04:37] "GET /working HTTP/1.1" 200 1699 0.0103
24.61.131.238 - - [14/May/2012 09:04:37] "GET /overview HTTP/1.1" 200 2371 0.0129
24.61.131.238 - - [14/May/2012 09:04:37] "GET /poll.png HTTP/1.1" 304 - 0.0006
24.61.131.238 - - [14/May/2012 09:04:39] "GET /failed HTTP/1.1" 200 1419 0.0060
24.61.131.238 - - [14/May/2012 09:04:40] "GET /retry HTTP/1.1" 200 2218 0.0093
24.61.131.238 - - [14/May/2012 09:04:45] "GET /delayed HTTP/1.1" 200 2137 0.0083
- Gracefully stopping, waiting for requests to finish - Gracefully stopping, waiting for requests to finish
==> log/resque-pool.log <==
resque-pool-manager[11815]: Resque Pool running in matt environment
:public is no longer used to avoid overloading Module#public, use :public_folder instead
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-fcaf987d0724/lib/resque/server.rb:12:in `<class:Server>'
==> log/resque-web.log <==
:public is no longer used to avoid overloading Module#public, use :public_folder instead
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-fcaf987d0724/lib/resque/server.rb:12:in `<class:Server>'
==> log/resque-pool.log <==
resque-pool-manager[11815]: started manager
resque-pool-manager[11815]: Pool contains worker PIDs: [12384, 12387]
resque-pool-worker[12384]: Starting worker matt.appleton.backupify.com:12384:@matt
resque-pool-worker[12387]: Starting worker matt.appleton.backupify.com:12387:@matt
==> log/matt.log <==
DEBUG 09:25:50 rails::RedisFactory [pid=12384]: RedisFactory.reconnect start
DEBUG 09:25:50 rails::RedisFactory [pid=12387]: RedisFactory.reconnect start
DEBUG 09:25:50 rails::RedisFactory [pid=12387]: RedisFactory.reconnect complete
DEBUG 09:25:50 rails::RedisFactory [pid=12384]: RedisFactory.reconnect complete
DEBUG 09:30:43 rails::RedisFactory [pid=13740]: RedisFactory.reconnect start
DEBUG 09:30:43 rails::RedisFactory [pid=13740]: RedisFactory.reconnect complete
DEBUG 09:30:43 rails::RedisFactory [pid=13748]: RedisFactory.reconnect start
DEBUG 09:30:43 rails::RedisFactory [pid=13748]: RedisFactory.reconnect complete
DEBUG 09:30:43 rails::Backupify::RedisCacheStore [pid=13740]: Cache read: https://matt.appleton.backupify.com/?
DEBUG 09:30:43 rails::RedisFactory [pid=13754]: RedisFactory.reconnect start
DEBUG 09:30:43 rails::RedisFactory [pid=13754]: RedisFactory.reconnect complete
INFO 09:30:43 rails [pid=13740]:
Started GET "/" for 10.76.87.52 at 2012-05-14 09:30:43 -0400
DEBUG 09:30:43 rails::RedisFactory [pid=13760]: RedisFactory.reconnect start
DEBUG 09:30:43 rails::RedisFactory [pid=13760]: RedisFactory.reconnect complete
INFO 09:30:43 rails::controllers [pid=13740]: Processing by WelcomeController#index as HTML
DEBUG 09:30:43 rails [pid=13740]: Existing local CAS session detected for "[email protected]". Previous ticket "ST-1337000242r236A6C6164B7DAA431" will be re-used.
DEBUG 09:30:43 rails::RedisFactory [pid=13766]: RedisFactory.reconnect start
DEBUG 09:30:43 rails::RedisFactory [pid=13766]: RedisFactory.reconnect complete
DEBUG 09:30:43 rails::models [pid=13740]: User Load (37.4ms) SELECT "users".* FROM "users" WHERE "users"."cas_ident" = '[email protected]' LIMIT 1
DEBUG 09:30:43 rails::RedisFactory [pid=13772]: RedisFactory.reconnect start
DEBUG 09:30:43 rails::RedisFactory [pid=13772]: RedisFactory.reconnect complete
DEBUG 09:30:43 rails::RedisFactory [pid=13778]: RedisFactory.reconnect start
DEBUG 09:30:43 rails::RedisFactory [pid=13778]: RedisFactory.reconnect complete
DEBUG 09:30:43 rails::RedisFactory [pid=13786]: RedisFactory.reconnect start
DEBUG 09:30:43 rails::RedisFactory [pid=13786]: RedisFactory.reconnect complete
DEBUG 09:30:43 rails::RedisFactory [pid=13792]: RedisFactory.reconnect start
DEBUG 09:30:43 rails::RedisFactory [pid=13792]: RedisFactory.reconnect complete
DEBUG 09:30:43 rails::models [pid=13740 user_id=1]: Service Load (2.0ms) SELECT "services".* FROM "services" WHERE "services"."user_id" = 1
INFO 09:30:43 rails::controllers [pid=13740 user_id=1]: Redirected to https://matt.appleton.backupify.com/services/new
INFO 09:30:43 rails::controllers [pid=13740 user_id=1]: Completed 302 Found in 432ms (ActiveRecord: 71.7ms)
DEBUG 09:30:43 rails::RedisFactory [pid=13799]: RedisFactory.reconnect start
DEBUG 09:30:43 rails::RedisFactory [pid=13799]: RedisFactory.reconnect complete
DEBUG 09:30:43 rails::RedisFactory [pid=13805]: RedisFactory.reconnect start
DEBUG 09:30:43 rails::RedisFactory [pid=13805]: RedisFactory.reconnect complete
DEBUG 09:30:43 rails::RedisFactory [pid=13812]: RedisFactory.reconnect start
DEBUG 09:30:43 rails::RedisFactory [pid=13812]: RedisFactory.reconnect complete
DEBUG 09:30:44 rails::RedisFactory [pid=13818]: RedisFactory.reconnect start
DEBUG 09:30:44 rails::RedisFactory [pid=13818]: RedisFactory.reconnect complete
DEBUG 09:30:44 rails::Backupify::RedisCacheStore [pid=13805]: Cache read: https://matt.appleton.backupify.com/services/new?
INFO 09:30:44 rails [pid=13805]:
Started GET "/services/new" for 10.76.87.52 at 2012-05-14 09:30:44 -0400
DEBUG 09:30:44 rails::RedisFactory [pid=13825]: RedisFactory.reconnect start
DEBUG 09:30:44 rails::RedisFactory [pid=13825]: RedisFactory.reconnect complete
DEBUG 09:30:44 rails::RedisFactory [pid=13842]: RedisFactory.reconnect start
DEBUG 09:30:44 rails::RedisFactory [pid=13842]: RedisFactory.reconnect complete
INFO 09:30:44 rails::controllers [pid=13805]: Processing by ServicesController#new as HTML
DEBUG 09:30:44 rails [pid=13805]: Existing local CAS session detected for "[email protected]". Previous ticket "ST-1337000242r236A6C6164B7DAA431" will be re-used.
DEBUG 09:30:44 rails::RedisFactory [pid=13848]: RedisFactory.reconnect start
DEBUG 09:30:44 rails::models [pid=13805]: User Load (1.3ms) SELECT "users".* FROM "users" WHERE "users"."cas_ident" = '[email protected]' LIMIT 1
DEBUG 09:30:44 rails::RedisFactory [pid=13848]: RedisFactory.reconnect complete
DEBUG 09:30:44 rails::RedisFactory [pid=13854]: RedisFactory.reconnect start
DEBUG 09:30:44 rails::RedisFactory [pid=13854]: RedisFactory.reconnect complete
DEBUG 09:30:44 rails::RedisFactory [pid=13860]: RedisFactory.reconnect start
DEBUG 09:30:44 rails::RedisFactory [pid=13860]: RedisFactory.reconnect complete
DEBUG 09:30:44 rails::RedisFactory [pid=13866]: RedisFactory.reconnect start
DEBUG 09:30:44 rails::RedisFactory [pid=13866]: RedisFactory.reconnect complete
DEBUG 09:30:45 rails::RedisFactory [pid=13872]: RedisFactory.reconnect start
DEBUG 09:30:45 rails::RedisFactory [pid=13872]: RedisFactory.reconnect complete
INFO 09:30:45 rails::controllers [pid=13805 user_id=1]: Rendered services/new.html.erb within layouts/application (9.9ms)
DEBUG 09:30:47 rails::models [pid=13805 user_id=1]: (0.9ms) SELECT COUNT(*) FROM "services" WHERE "services"."user_id" = 1
INFO 09:30:47 rails::controllers [pid=13805 user_id=1]: Rendered shared/_flashes.html.erb (0.9ms)
INFO 09:30:47 rails::controllers [pid=13805 user_id=1]: Completed 200 OK in 3149ms (Views: 2910.0ms | ActiveRecord: 55.5ms)
DEBUG 09:31:48 rails::Backupify::RedisCacheStore [pid=13805 user_id=1]: Cache read: https://matt.appleton.backupify.com/services/new?
INFO 09:31:48 rails [pid=13805 user_id=1]:
Started GET "/services/new" for 10.76.87.52 at 2012-05-14 09:31:48 -0400
INFO 09:31:48 rails::controllers [pid=13805 user_id=1]: Processing by ServicesController#new as HTML
DEBUG 09:31:48 rails [pid=13805 user_id=1]: Existing local CAS session detected for "[email protected]". Previous ticket "ST-1337000242r236A6C6164B7DAA431" will be re-used.
DEBUG 09:31:48 rails::models [pid=13805]: User Load (18.3ms) SELECT "users".* FROM "users" WHERE "users"."cas_ident" = '[email protected]' LIMIT 1
INFO 09:31:48 rails::controllers [pid=13805 user_id=1]: Rendered services/new.html.erb within layouts/application (0.9ms)
DEBUG 09:31:48 rails::models [pid=13805 user_id=1]: (1.2ms) SELECT COUNT(*) FROM "services" WHERE "services"."user_id" = 1
INFO 09:31:48 rails::controllers [pid=13805 user_id=1]: Rendered shared/_flashes.html.erb (0.1ms)
INFO 09:31:48 rails::controllers [pid=13805 user_id=1]: Completed 200 OK in 31ms (Views: 7.5ms | ActiveRecord: 19.4ms)
DEBUG 09:31:51 rails::Backupify::RedisCacheStore [pid=13805 user_id=1]: Cache read: https://matt.appleton.backupify.com/twitter_services/new?
INFO 09:31:51 rails [pid=13805 user_id=1]:
Started GET "/twitter_services/new" for 10.76.87.52 at 2012-05-14 09:31:51 -0400
INFO 09:31:51 rails::controllers [pid=13805 user_id=1]: Processing by TwitterServicesController#new as HTML
DEBUG 09:31:51 rails [pid=13805 user_id=1]: Existing local CAS session detected for "[email protected]". Previous ticket "ST-1337000242r236A6C6164B7DAA431" will be re-used.
DEBUG 09:31:51 rails::models [pid=13805]: User Load (1.8ms) SELECT "users".* FROM "users" WHERE "users"."cas_ident" = '[email protected]' LIMIT 1
INFO 09:31:52 rails::controllers [pid=13805 user_id=1]: Rendered twitter_services/new.html.erb within layouts/application (1.6ms)
DEBUG 09:31:52 rails::models [pid=13805 user_id=1]: (1.1ms) SELECT COUNT(*) FROM "services" WHERE "services"."user_id" = 1
INFO 09:31:52 rails::controllers [pid=13805 user_id=1]: Rendered shared/_flashes.html.erb (0.1ms)
INFO 09:31:52 rails::controllers [pid=13805 user_id=1]: Completed 200 OK in 673ms (Views: 102.4ms | ActiveRecord: 2.8ms)
DEBUG 09:31:53 rails::Backupify::RedisCacheStore [pid=13805 user_id=1]: Cache read: https://matt.appleton.backupify.com/?
INFO 09:31:53 rails [pid=13805 user_id=1]:
Started GET "/" for 10.76.87.52 at 2012-05-14 09:31:53 -0400
INFO 09:31:53 rails::controllers [pid=13805 user_id=1]: Processing by WelcomeController#index as HTML
DEBUG 09:31:53 rails [pid=13805 user_id=1]: Existing local CAS session detected for "[email protected]". Previous ticket "ST-1337000242r236A6C6164B7DAA431" will be re-used.
DEBUG 09:31:53 rails::models [pid=13805]: User Load (0.8ms) SELECT "users".* FROM "users" WHERE "users"."cas_ident" = '[email protected]' LIMIT 1
DEBUG 09:31:53 rails::models [pid=13805 user_id=1]: Service Load (0.4ms) SELECT "services".* FROM "services" WHERE "services"."user_id" = 1
INFO 09:31:53 rails::controllers [pid=13805 user_id=1]: Redirected to https://matt.appleton.backupify.com/services/new
INFO 09:31:53 rails::controllers [pid=13805 user_id=1]: Completed 302 Found in 7ms (ActiveRecord: 1.3ms)
DEBUG 09:31:53 rails::Backupify::RedisCacheStore [pid=13805 user_id=1]: Cache read: https://matt.appleton.backupify.com/services/new?
INFO 09:31:53 rails [pid=13805 user_id=1]:
Started GET "/services/new" for 10.76.87.52 at 2012-05-14 09:31:53 -0400
INFO 09:31:53 rails::controllers [pid=13805 user_id=1]: Processing by ServicesController#new as HTML
DEBUG 09:31:53 rails [pid=13805 user_id=1]: Existing local CAS session detected for "[email protected]". Previous ticket "ST-1337000242r236A6C6164B7DAA431" will be re-used.
DEBUG 09:31:53 rails::models [pid=13805]: User Load (0.7ms) SELECT "users".* FROM "users" WHERE "users"."cas_ident" = '[email protected]' LIMIT 1
INFO 09:31:53 rails::controllers [pid=13805 user_id=1]: Rendered services/new.html.erb within layouts/application (0.7ms)
DEBUG 09:31:53 rails::models [pid=13805 user_id=1]: (1.6ms) SELECT COUNT(*) FROM "services" WHERE "services"."user_id" = 1
INFO 09:31:53 rails::controllers [pid=13805 user_id=1]: Rendered shared/_flashes.html.erb (0.1ms)
INFO 09:31:53 rails::controllers [pid=13805 user_id=1]: Completed 200 OK in 16ms (Views: 10.8ms | ActiveRecord: 2.4ms)
==> log/resque-pool.log <==
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/redis_failover-0.8.3/lib/redis_failover/client.rb:339:in `[]': can't convert String into Integer (TypeError)
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/redis_failover-0.8.3/lib/redis_failover/client.rb:339:in `verify_role!'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/redis_failover-0.8.3/lib/redis_failover/client.rb:247:in `master'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/redis_failover-0.8.3/lib/redis_failover/client.rb:264:in `slave'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/redis_failover-0.8.3/lib/redis_failover/client.rb:433:in `client_for'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/redis_failover-0.8.3/lib/redis_failover/client.rb:220:in `dispatch'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/redis_failover-0.8.3/lib/redis_failover/client.rb:82:in `block (2 levels) in <class:Client>'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/redis-namespace-1.0.3/lib/redis/namespace.rb:213:in `method_missing'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-fcaf987d0724/lib/resque/worker.rb:430:in `job'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-fcaf987d0724/lib/resque/worker.rb:362:in `unregister_worker'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-fcaf987d0724/lib/resque/worker.rb:145:in `ensure in work'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-fcaf987d0724/lib/resque/worker.rb:145:in `work'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:332:in `block in spawn_worker!'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:327:in `fork'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:327:in `spawn_worker!'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:287:in `block in spawn_missing_workers_for'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:287:in `times'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:287:in `spawn_missing_workers_for'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:273:in `block in maintain_worker_count'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:271:in `each'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:271:in `maintain_worker_count'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:191:in `start'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:65:in `run'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool/tasks.rb:17:in `block (2 levels) in <top (required)>'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:205:in `call'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:205:in `block in execute'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:200:in `each'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:200:in `execute'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:158:in `block in invoke_with_call_chain'
from /usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:151:in `invoke_with_call_chain'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:144:in `invoke'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool/cli.rb:110:in `start_pool'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool/cli.rb:16:in `run'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/bin/resque-pool:5:in `<top (required)>'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bin/resque-pool:19:in `load'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bin/resque-pool:19:in `<main>'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/pry-exception_explorer-0.1.9/lib/pry-exception_explorer/core_ext.rb:46:in `raise': 756: unexpected token at ' (Resque::Helpers::DecodeException)
redis_git_sha1:00000000
redis_git_dirty:0
arch_bits:64
multiplexing_api:epoll
gcc_version:4.6.3
process_id:19181
uptime_in_seconds:3552
uptime_in_days:0
lru_clock:1579678
used_cpu_sys:2.24
used_cpu_user:1.66
used_cpu_sys_children:0.00
used_cpu_user_children:0.00
connected_clients:6
connected_slaves:0
client_longest_output_list:0
client_biggest_input_buf:0
blocked_clients:1
used_memory:772392
used_memory_human:754.29K
used_memory_rss:929792
used_memory_peak:799528
used_memory_peak_human:780.79K
mem_fragmentation_ratio:1.20
mem_allocator:jemalloc-2.2.5
loading:0
aof_enabled:0
changes_since_last_save:78
bgsave_in_progress:0
last_save_time:1336998988
bgrewriteaof_in_progress:0
total_connections_received:2551
total_commands_processed:18176
expired_keys:0
evicted_keys:0
keyspace_hits:3117
keyspace_misses:3496
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:0
vm_enabled:0
role:master
db1:keys=11,expires=0
'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-fcaf987d0724/lib/resque/helpers.rb:32:in `rescue in decode'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-fcaf987d0724/lib/resque/helpers.rb:29:in `decode'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/resque-dynamic-queues-0.8.1/lib/resque/plugins/dynamic_queues/attributes.rb:12:in `get_dynamic_queue'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/resque-dynamic-queues-0.8.1/lib/resque/plugins/dynamic_queues/queues.rb:34:in `queues_with_dynamic'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/resque-queue-priority-0.6.1/lib/resque/plugins/queue_priority/priority.rb:17:in `queues_with_priority'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:243:in `block in reap_all_workers'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:237:in `loop'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:237:in `reap_all_workers'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:208:in `block in join'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:207:in `loop'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:207:in `join'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:65:in `run'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool/tasks.rb:17:in `block (2 levels) in <top (required)>'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:205:in `call'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:205:in `block in execute'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:200:in `each'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:200:in `execute'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:158:in `block in invoke_with_call_chain'
from /usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:151:in `invoke_with_call_chain'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:144:in `invoke'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool/cli.rb:110:in `start_pool'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool/cli.rb:16:in `run'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/bin/resque-pool:5:in `<top (required)>'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bin/resque-pool:19:in `load'
from /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bin/resque-pool:19:in `<main>'
==> log/matt.log <==
ERROR 09:35:45 rails::RedisFailover [pid=12384]: Error while handling `get` - #<Errno::EAGAIN: Resource temporarily unavailable - Timeout reading from the socket>
ERROR 09:35:45 rails::RedisFailover [pid=12384]: /mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/pry-exception_explorer-0.1.9/lib/pry-exception_explorer/core_ext.rb:46:in `raise'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/redis-2.2.2/lib/redis/client.rb:170:in `rescue in read'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/redis-2.2.2/lib/redis/client.rb:161:in `read'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/redis-2.2.2/lib/redis/client.rb:46:in `block in call'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/redis-2.2.2/lib/redis/client.rb:142:in `block (2 levels) in process'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/redis-2.2.2/lib/redis/client.rb:250:in `ensure_connected'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/redis-2.2.2/lib/redis/client.rb:137:in `block in process'
/mnt/appleton-matt/shared/
Feature Request: Force failover to a slave
It would be nice to have a way to force failover to a slave so it is coordinated better than just doing a kill -9 on the master.
Seeing as redis still doesn't have a read-only flag you can set on a server, I think it would require issuing a shutdown to the old master to ensure clients don't continue to write to the master that changes to a slave (and would also provide them with a way to get notification that they need to refresh the server list).
Stack trace with resque
I'm seeing the following when using redis_failover with resque (specifically from resque web ui) - I could be doing something wrong, haven't dug that deep yet.
ArgumentError - wrong number of arguments (2 for 1):
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/redis_failover-0.8.1/lib/redis_failover/client.rb:127:in respond_to_missing?' /mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/redis-namespace-1.0.3/lib/redis/namespace.rb:169:inrespond_to?'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/redis-namespace-1.0.3/lib/redis/namespace.rb:169:in respond_to?' /mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-fcaf987d0724/lib/resque.rb:58:inredis_id'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-fcaf987d0724/lib/resque/server/views/layout.erb:40:in `evaluate_source'
Starting redis_node_manager connection timeouts
Hi,
I was looking for a fail over package and found this package so I'm fairly new to this. I have started to experiment with it and I had entered on the command line
$ redis_node_manager -n localhost:6000,localhost:6001 -z localhost:7000,localhost:7001,localhost:7002 --with-chroot /tmp/
So after 30 seconds I would get this message continuously can you please help
ERROR: ZK error while attempting to manage nodes: #<Zookeeper::Exceptions::ContinuationTimeoutError: response for meth: :exists, args: [0, "/opt/", nil, nil], not received within 30 seconds>
No way to pass redis DB when setting up client
We use multiple redis DBs, (resque, cache, etc), and usually select them at client initialization time. There doesn't seem to be a way to pass in the db to the failover client - perhaps other init time options are missing as well?
If ZK has an old master, redis_failover will use it even though the redis instance does not exist
This issue may be particular to cloud environments where IP addresses are not very stable.
Somehow we managed to get into a state where ZK maintained a no-longer-in-service IP for the redis master. That is, we shutdown a redis node and brought up a whole new one, and ZK never noticed because that node also runs the redis_node_manager.
At startup, when the node manager gets this old IP from ZK it attempts to ask that Redis instance whether it is still the master. In our case, this times out. And when this happens redis_failover simply logs a warning and continues to use this old IP as master. Which basically causes all future client usage to fail. I believe this is the code in question
It would probably be better to:
- Check to see if IPs returned from ZK match the list of nodes provided at startup of the node manager, and expire from ZK the ones that don't. And/or....
- Return nil instead of master if the find_existing_master method rescues a NodeUnavailableError.
Full Failure of server
For the situation where zookeeper, redis and a node manager are on the same server, RedisFailover can't seem to reconcile the situation of losing an entire cluster of these three.
For example:
If I have two servers, each with an instance of redis, zookeeper and a redis_node manager with one in slave to the other and I completely kill the server with the master redis instance, the other server can't reconcile it because - I believe - it can't communicate with those zookeeper instances anymore. If zookeeper can't be communicated with anymore, isn't the best course of action to try and mark that redis instance as unavailable and switch the slave over to master?
Add subscribing part of Pub/Sub ops to REDIS_READ_OPS
Looks like SUBSCRIBE, PSUBSCRIBE and UNSUBSCRIBE can be added to REDIS_READ_OPS.
I can add a pull request if this is greenlighted.
Weird zookeeper error happening every second or so in the zookeeper logs
2012-10-03 16:13:45,653 - INFO [ProcessThread:-1:PrepRequestProcessor@419] - Got user-level KeeperException when processing sessionid:0x13a273564e2000f type:delete cxid:0x506c65f9 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/_zklocking/__redis_failover_nodes_lock Error:KeeperErrorCode = Directory not empty for /_zklocking/__redis_failover_nodes_lock
It doesn't seem to effect functionality, but it'll fill up the logs pretty fast. Any ideas?
Support Redis 2.8
Slaves are read-only in redis 2.8. This means that the existing approach of using blpop/del/lpush to monitor a node's liveness must be changed since these are write operations. This original approach was used to avoid having to busy-wait when checking nodes, but we may just need to go back to polling given how slaves behave in 2.8.
Feature Request: Abstract gathering and notification of servers in client
I'm a huge fan of zookeeper and think it would be really great to use it instead of the HTTP interface to notify clients of changes in topologies.
It looks like it would be pretty easy to structure the client in such a way that the gathering and notification of server changes was pluggable. One cool thing about using zookeeper is that clients could watch for changes to the redis server keys in zookeeper and be updated instantaniously when a change happens.
Proposal: Real-world test cases
I wanted to document some of the real-world test cases I've been envisioning for a test suite for this library.
The Setup
It seems like it would be pretty easy to setup a local environment to test some of this stuff:
- 3 zookeeper servers
- 2 redis servers
- 2 clients
- 2 node monitors
to give us a chance to kill or hang each component and make sure everything reacts appropriately.
Scenarios
Here is an incomplete list of tests that I think should be run against a real set of redis servers and clients.
- Kill a redis server with
SIGKILL(akill -9) โ ensure the failover happens immediately - Pause a redis server (causing a hang) with
SIGSTOPโ ensure the monitor process notices the hang and starts a failover - Kill the master monitor process with
SIGKILLโ ensure another monitor takes over - Pause the master monitor process with
SIGSTOPand then kill redis withSIGKILLโ How long does this take to failover?
Monitoring
While running these tests, it would be worthwhile for the redis clients to be constantly running SET commands against redis.
Tracking the average and max times for requests would be helpful in understanding how long failover really takes. Using my metriks library may be helpful in getting those statistics easily.
I envision the redis client processes having an at_exit defined that would output statistics like the number of keys set, the number of errors, and the average and max times per SET. We could easily compare the number of keys they thought they set with the number that the final master has, to see what sort of failures happened.
Error bringing back up failed master
I have master localhost:7777 and slave localhost:8888
I take down 7777 and everything switches over to 8888 without a problem. I bring 7777 back up and Redis properly switches itself to a slave of 8888 but I see this error in the log output of the leader redis_node_manager. Any idea what's wrong?
2013-12-30 06:55:01 UTC RedisFailover 12178 INFO: RedisFailover::NodeStrategy::Majority marking localhost:7777 as unavailable. Snapshot: Node localhost:7777 available by {}, unavailable by ["mikes-macbook-air.local-12374", "mikes-macbook-air.local-12178"] (0 up, 2 down)
2013-12-30 06:55:06 UTC RedisFailover 12178 INFO: Reconciling node localhost:7777
2013-12-30 06:55:06 UTC RedisFailover 12178 INFO: localhost:7777 is now a slave of localhost:8888
2013-12-30 06:55:06 UTC RedisFailover 12178 ERROR: Error handling state report for [<RedisFailover::Node localhost:7777>, :available]: #<RedisFailover::NodeUnavailableError: Node: RedisFailover::NodeUnavailableError: Node: Redis::CommandError: READONLY You can't write against a read only slave.>
2013-12-30 06:55:11 UTC RedisFailover 12178 INFO: Handling available node: localhost:7777
RedisFailover::Client.new should not throw an exception if the znode is not found
The client throws:
ZK::Exceptions::NoNode: inputs: {:path=>"/redis_failover_nodes"}
passing in a zk and having it close before redis-failover can trigger an infinite loop
If you close a ZK client and then redis-failover goes to close the same zk client (because it was passed in)... you can trigger an infite loop of
2013-01-17 19:44:00 UTC RedisFailover 28592 WARN: Caught Zookeeper::Exceptions::NotConnected 'Zookeeper::Exceptions::NotConnected' - retrying
redis_node_manager should start if no redis servers are available
Here's a stack trace:
lib/redis_failover/version.rb:2: warning: already initialized constant VERSION
Mon Aug 13 05:29:22 UTC 2012 RedisFailover 46108 INFO: Redis Node Manager v0.8.9 starting (ruby 1.8.7 (2011-12-28 MBARI 8/0x6770 on patchlevel 357) [i686-darwin11.3.0], MBARI 0x6770, Ruby Enterprise Edition 2011.12)
Mon Aug 13 05:29:22 UTC 2012 RedisFailover 46108 INFO: Waiting to become master Node Manager ...
Mon Aug 13 05:29:22 UTC 2012 RedisFailover 46108 INFO: Acquired master Node Manager lock
lib/redis_failover/node_manager.rb:222:in `discover_nodes': RedisFailover::NoMasterError (RedisFailover::NoMasterError)
from lib/redis_failover/runner.rb:14:in `join'
from lib/redis_failover/runner.rb:14:in `run'
from bin/redis_node_manager:6
from bin/redis_node_manager:23:in `load'
from bin/redis_node_manager:23
State of redis_failover
Hi,
is redis_failover still maintained? Is anybody using this still in production? I ask because the last coments on the issues #62, #64 are one year old now, but the fork from @fanhattan seems to see a lot of activity (and etcd-support which would be awesome). Maybe someone (@arohter, @ryanlecompte, @mperham) can comment on any further plans?
Thanks!
Allow Manual Failover in cluster bootstrap
When launching fresh clusters, when zk holds no 'nodes' data, and all redis nodes are in master state, client.manual_failover() will fail with an exception due to the build_client() call failing to find the 'nodes' data when initializing the client object.
At the moment we need to manually issue slaveof commands to redis nodes in order to bootstrap.
We should be able to just call a static class method (skipping build_clients) to force a bootstrap.
Support multiple Node Manager daemons
Right now only one Node Manager is supported. For added redundancy, we should have support for multiple Node Managers running across the network. At any given moment, only one will be in charge of monitoring the redis servers. When one fails, another Node Manager will be elected to monitor the redis servers. This should all be done via ZooKeeper.
Discussion: Adding daemonize and log rotate signal
Hi, Ryan.
I'm adding daemonize and logrotateing.
But, Its code seems like dirty, though.
- use class eval
https://github.com/kyohsuke/redis_failover/commit/a2888d41e77de89f827437801b95800bc8843905 - use class variables
https://github.com/kyohsuke/redis_failover/commit/8415c9911e44104e37088309fd197759367c51b7
Do you have any good idea?
Thanks
stuck in "Failed to discover master node"
We're running a setup of 3 zookeeper nodes and 2 redis nodes. On each of those 2 redis nodes, there's an instance of redis_node_manager running.
If the original redis master instance for any reason disappears (kill, crash, reboot), the other original slave redis node correctly becomes the new master node.
If that instance now, for any reason crashes as well and is not recovered within the configured retries but later, redis_node_manager gets into a dead loop.
2013-12-01 16:36:07 UTC RedisFailover 5425 ERROR: Failed to promote a new master after 3 attempts. 2013-12-01 16:36:13 UTC RedisFailover 5425 INFO: Monitoring redis nodes at (172.16.154.152:6379, 172.16.154.153:6379) 2013-12-01 16:36:13 UTC RedisFailover 5425 INFO: Waiting to become master Node Manager ... 2013-12-01 16:36:13 UTC RedisFailover 5425 INFO: Acquired master Node Manager lock. 2013-12-01 16:36:13 UTC RedisFailover 5425 INFO: Configured node strategy RedisFailover::NodeStrategy::Single 2013-12-01 16:36:13 UTC RedisFailover 5425 INFO: Configured failover strategy RedisFailover::FailoverStrategy::Latency 2013-12-01 16:36:13 UTC RedisFailover 5425 INFO: Required Node Managers to make a decision: 1 2013-12-01 16:36:13 UTC RedisFailover 5425 INFO: Master from existing znode config: none 2013-12-01 16:36:13 UTC RedisFailover 5425 WARN: Failed to discover master node: # In order to ensure a safe startup, redis_failover requires that all redis nodes be accessible, and only a single node indicating that it's the master. In order to fix this, you can perform a manual failover via redis_failover, or manually fix the individual redis servers. This discovery process will retry in 5s. 2013-12-01 16:36:18 UTC RedisFailover 5425 INFO: Master from existing znode config: none 2013-12-01 16:36:18 UTC RedisFailover 5425 WARN: Failed to discover master node: # In order to ensure a safe startup, redis_failover requires that all redis nodes be accessible, and only a single node indicating that it's the master. In order to fix this, you can perform a manual failover via redis_failover, or manually fix the individual redis servers. This discovery process will retry in 5s. 2013-12-01 16:36:23 UTC RedisFailover 5425 INFO: Master from existing znode config: none 2013-12-01 16:36:23 UTC RedisFailover 5425 WARN: Failed to discover master node: # In order to ensure a safe startup, redis_failover requires that all redis nodes be accessible, and only a single node indicating that it's the master. In order to fix this, you can perform a manual failover via redis_failover, or manually fix the individual redis servers. This discovery process will retry in 5s. ...
Is this intended behavior? If the master is not recoverable even announcing the original slave by slaveof no one to master did not resolve that issue.
Only replacing the defected master completely and reassigning the original IP and reconstructing the original master/slave setup did recover the node_manager.
I thought, it should just take one of the known nodes and promote it to master ?
Prevent reads on slaves
Is it possible to make it so that slaves are not used for reads at all?
I know I can set the slave-serve-stale-data setting on the slave to 'no' but I think this will involve one attempt to read from the slave, getting a reply of "SYNC with master in progress", then reading from the master. In the case, I think performance will be adversely affected.
Add informative procline
I'd like to be able to get very simple info out of redis_node_manager without having to do anything. I think the procline would be a great idea here:
Current procline:
/usr/local/rvm/rubies/ruby-1.9.3-p125/bin/ruby /usr/local/rvm/gems/ruby-1.9.3-p125/bin/redis_node_manager -c /opt/theclymb/shared/config/redis_node_manager_conf.yml -e production
Example:
redis_node_manager version=1.2.3 master=app-4.theclymb.com:6379
I'm not sold on the format, I was just thinking key=value pairs but anything is better than nothing. Sidekiq does this:
sidekiq 2.6.4 theclymb [0 of 10 busy]
Where 'theclymb' is the application tag associated with sidekiq so you can distinguish between sidekiqs for different rails apps on the same machine.
Feature request: Support Ruby 1.8
I'm still using Ruby 1.8, so I would love to see support for it.
I can't think of anything that would prevent it from working, so I'll definitely be trying to make it work.
Process locks up with failover client
This may not be the fault of redis_failover, but all else being the same, when I have it enabled, my resque workers lock up on start. Here are a couple thread dumps - in the first, the worker doesn't get past changing the Starting in the procline, in the second, it got to the "Waiting" procline, but never picks up jobs. I used gdb.rb to get the traces.
"Starting" procline:
[
[0] [
[ 0] "(eval):1:in `block in require_with_backports'",
[ 1] "(eval):1:in `collect'",
[ 2] "(eval):1:in `require_with_backports'",
[ 3] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/backports-2.5.3/lib/backports/tools.rb:314:in `require'",
[ 4] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/backports-2.5.3/lib/backports/tools.rb:314:in `require_with_backports'",
[ 5] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/activesupport-3.2.2/lib/active_support/dependencies.rb:251:in `block in require'",
[ 6] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/activesupport-3.2.2/lib/active_support/dependencies.rb:236:in `load_dependency'",
[ 7] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/activesupport-3.2.2/lib/active_support/dependencies.rb:251:in `require'",
[ 8] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/activesupport-3.2.2/lib/active_support/dependencies.rb:359:in `require_or_load'",
[ 9] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/activesupport-3.2.2/lib/active_support/dependencies.rb:502:in `load_missing_constant'",
[10] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/activesupport-3.2.2/lib/active_support/dependencies.rb:192:in `block in const_missing'",
[11] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/activesupport-3.2.2/lib/active_support/dependencies.rb:190:in `each'",
[12] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/activesupport-3.2.2/lib/active_support/dependencies.rb:190:in `const_missing'",
[13] "/mnt/appleton-matt/releases/20120510001131/config/initializers/resque.rb:48:in `reserve'",
[14] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-fcaf987d0724/lib/resque/worker.rb:119:in `block in work'",
[15] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-fcaf987d0724/lib/resque/worker.rb:116:in `loop'",
[16] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-fcaf987d0724/lib/resque/worker.rb:116:in `work'",
[17] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:332:in `block in spawn_worker!'",
[18] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:327:in `fork'",
[19] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:327:in `spawn_worker!'",
[20] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:287:in `block in spawn_missing_workers_for'",
[21] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:287:in `times'",
[22] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:287:in `spawn_missing_workers_for'",
[23] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:273:in `block in maintain_worker_count'",
[24] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:271:in `each'",
[25] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:271:in `maintain_worker_count'",
[26] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:191:in `start'",
[27] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:65:in `run'",
[28] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool/tasks.rb:17:in `block (2 levels) in <top (required)>'",
[29] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:205:in `call'",
[30] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:205:in `block in execute'",
[31] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:200:in `each'",
[32] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:200:in `execute'",
[33] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:158:in `block in invoke_with_call_chain'",
[34] "/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'",
[35] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:151:in `invoke_with_call_chain'",
[36] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:144:in `invoke'",
[37] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool/cli.rb:110:in `start_pool'",
[38] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool/cli.rb:16:in `run'",
[39] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/bin/resque-pool:5:in `<top (required)>'",
[40] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bin/resque-pool:19:in `load'",
[41] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bin/resque-pool:19:in `<main>'"
],
[1] [
[0] "/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/ruby/1.9.1/thread.rb:189:in `block in pop'",
[1] "<internal:prelude>:10:in `synchronize'",
[2] "/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/ruby/1.9.1/thread.rb:184:in `pop'",
[3] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zk-1.3.1/lib/zk/threadpool.rb:176:in `block (2 levels) in spawn_threadpool'"
],
[2] [
[0] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zookeeper-1.0.2/ext/c_zookeeper.rb:181:in `_iterate_event_delivery'",
[1] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zookeeper-1.0.2/ext/c_zookeeper.rb:172:in `_event_thread_body'"
],
[3] [
[0] "/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/ruby/1.9.1/thread.rb:189:in `block in pop'",
[1] "<internal:prelude>:10:in `synchronize'",
[2] "/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/ruby/1.9.1/thread.rb:184:in `pop'",
[3] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zookeeper-1.0.2/lib/zookeeper/common/queue_with_pipe.rb:32:in `pop'",
[4] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zookeeper-1.0.2/lib/zookeeper/common.rb:15:in `get_next_event'",
[5] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zookeeper-1.0.2/lib/zookeeper/common.rb:79:in `block (2 levels) in setup_dispatch_thread!'"
]
]
"Waiting" procilne:
[
[0] [
[ 0] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-fcaf987d0724/lib/resque/worker.rb:140:in `block in work'",
[ 1] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-fcaf987d0724/lib/resque/worker.rb:116:in `loop'",
[ 2] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-fcaf987d0724/lib/resque/worker.rb:116:in `work'",
[ 3] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:332:in `block in spawn_worker!'",
[ 4] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:327:in `fork'",
[ 5] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:327:in `spawn_worker!'",
[ 6] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:287:in `block in spawn_missing_workers_for'",
[ 7] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:287:in `times'",
[ 8] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:287:in `spawn_missing_workers_for'",
[ 9] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:273:in `block in maintain_worker_count'",
[10] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:271:in `each'",
[11] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:271:in `maintain_worker_count'",
[12] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:191:in `start'",
[13] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool.rb:65:in `run'",
[14] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool/tasks.rb:17:in `block (2 levels) in <top (required)>'",
[15] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:205:in `call'",
[16] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:205:in `block in execute'",
[17] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:200:in `each'",
[18] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:200:in `execute'",
[19] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:158:in `block in invoke_with_call_chain'",
[20] "/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'",
[21] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:151:in `invoke_with_call_chain'",
[22] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/rake-0.9.2.2/lib/rake/task.rb:144:in `invoke'",
[23] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool/cli.rb:110:in `start_pool'",
[24] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/lib/resque/pool/cli.rb:16:in `run'",
[25] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bundler/gems/resque-pool-774e1e384d5c/bin/resque-pool:5:in `<top (required)>'",
[26] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bin/resque-pool:19:in `load'",
[27] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/bin/resque-pool:19:in `<main>'"
],
[1] [
[ 0] "/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/ruby/1.9.1/monitor.rb:185:in `mon_enter'",
[ 1] "/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/ruby/1.9.1/monitor.rb:209:in `mon_synchronize'",
[ 2] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/log4r-1.1.10/lib/log4r/outputter/outputter.rb:130:in `synch'",
[ 3] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/log4r-gelf-0.8.2/lib/log4r-gelf/gelf_outputter.rb:85:in `canonical_log'",
[ 4] "(eval):3:in `error'",
[ 5] "(eval):8:in `block in error'",
[ 6] "(eval):8:in `each'",
[ 7] "(eval):8:in `error'",
[ 8] "(eval):9:in `error'",
[ 9] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/redis_failover-0.8.2/lib/redis_failover/client.rb:267:in `rescue in block in build_clients'",
[10] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/redis_failover-0.8.2/lib/redis_failover/client.rb:256:in `block in build_clients'",
[11] "/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'",
[12] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/redis_failover-0.8.2/lib/redis_failover/client.rb:253:in `build_clients'",
[13] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/redis_failover-0.8.2/lib/redis_failover/client.rb:169:in `handle_zk_event'",
[14] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/redis_failover-0.8.2/lib/redis_failover/client.rb:156:in `block in setup_zk'",
[15] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zk-1.3.1/lib/zk/subscription.rb:33:in `call'",
[16] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zk-1.3.1/lib/zk/subscription.rb:33:in `call'",
[17] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zk-1.3.1/lib/zk/event_handler.rb:304:in `block (2 levels) in safe_call'",
[18] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zk-1.3.1/lib/zk/threadpool.rb:178:in `call'",
[19] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zk-1.3.1/lib/zk/threadpool.rb:178:in `block (2 levels) in spawn_threadpool'"
],
[2] [
[0] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zookeeper-1.0.2/ext/c_zookeeper.rb:181:in `_iterate_event_delivery'",
[1] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zookeeper-1.0.2/ext/c_zookeeper.rb:172:in `_event_thread_body'"
],
[3] [
[ 0] "/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/ruby/1.9.1/logger.rb:568:in `block in write'",
[ 1] "/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'",
[ 2] "/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/ruby/1.9.1/logger.rb:559:in `write'",
[ 3] "/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/ruby/1.9.1/logger.rb:377:in `add'",
[ 4] "/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/ruby/1.9.1/logger.rb:452:in `error'",
[ 5] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zk-1.3.1/lib/zk/client/threaded.rb:244:in `block in raw_event_handler'",
[ 6] "/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'",
[ 7] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zk-1.3.1/lib/zk/client/threaded.rb:241:in `raw_event_handler'",
[ 8] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zk-1.3.1/lib/zk/event_handler.rb:145:in `process'",
[ 9] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zk-1.3.1/lib/zk/event_handler.rb:273:in `block in watcher_callback'",
[10] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zookeeper-1.0.2/lib/zookeeper/callbacks.rb:19:in `block in create'",
[11] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zookeeper-1.0.2/lib/zookeeper/callbacks.rb:26:in `block in initialize'",
[12] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zookeeper-1.0.2/lib/zookeeper/callbacks.rb:32:in `call'",
[13] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zookeeper-1.0.2/lib/zookeeper/callbacks.rb:32:in `call'",
[14] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zk-1.3.1/lib/zk/event_handler.rb:214:in `block (2 levels) in get_default_watcher_block'",
[15] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zk-1.3.1/lib/zk/event_handler.rb:213:in `tap'",
[16] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zk-1.3.1/lib/zk/event_handler.rb:213:in `block in get_default_watcher_block'",
[17] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zookeeper-1.0.2/lib/zookeeper/common.rb:155:in `call'",
[18] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zookeeper-1.0.2/lib/zookeeper/common.rb:155:in `dispatch_next_callback'",
[19] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zookeeper-1.0.2/lib/zookeeper/common.rb:79:in `block (2 levels) in setup_dispatch_thread!'"
],
[4] [
[ 0] "(eval):1:in `block in serialize_hash'",
[ 1] "(eval):1:in `collect'",
[ 2] "(eval):1:in `serialize_hash'",
[ 3] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/gelf-1.3.2/lib/gelf/notifier.rb:230:in `deflate'",
[ 4] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/gelf-1.3.2/lib/gelf/notifier.rb:230:in `serialize_hash'",
[ 5] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/gelf-1.3.2/lib/gelf/notifier.rb:206:in `datagrams_from_hash'",
[ 6] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/gelf-1.3.2/lib/gelf/notifier.rb:143:in `notify_with_level!'",
[ 7] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/gelf-1.3.2/lib/gelf/notifier.rb:118:in `notify!'",
[ 8] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/log4r-gelf-0.8.2/lib/log4r-gelf/gelf_outputter.rb:88:in `block in canonical_log'",
[ 9] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/log4r-1.1.10/lib/log4r/outputter/outputter.rb:130:in `block in synch'",
[10] "/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'",
[11] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/log4r-1.1.10/lib/log4r/outputter/outputter.rb:130:in `synch'",
[12] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/log4r-gelf-0.8.2/lib/log4r-gelf/gelf_outputter.rb:85:in `canonical_log'",
[13] "(eval):3:in `debug'",
[14] "(eval):8:in `block in debug'",
[15] "(eval):8:in `each'",
[16] "(eval):8:in `debug'",
[17] "(eval):9:in `debug'",
[18] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/redis_failover-0.8.2/lib/redis_failover/client.rb:287:in `fetch_nodes'",
[19] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/redis_failover-0.8.2/lib/redis_failover/client.rb:257:in `block in build_clients'",
[20] "/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'",
[21] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/redis_failover-0.8.2/lib/redis_failover/client.rb:253:in `build_clients'",
[22] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/redis_failover-0.8.2/lib/redis_failover/client.rb:169:in `handle_zk_event'",
[23] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/redis_failover-0.8.2/lib/redis_failover/client.rb:156:in `block in setup_zk'",
[24] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zk-1.3.1/lib/zk/subscription.rb:33:in `call'",
[25] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zk-1.3.1/lib/zk/subscription.rb:33:in `call'",
[26] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zk-1.3.1/lib/zk/event_handler.rb:304:in `block (2 levels) in safe_call'",
[27] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zk-1.3.1/lib/zk/threadpool.rb:178:in `call'",
[28] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zk-1.3.1/lib/zk/threadpool.rb:178:in `block (2 levels) in spawn_threadpool'"
],
[5] [
[0] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zookeeper-1.0.2/ext/c_zookeeper.rb:181:in `_iterate_event_delivery'",
[1] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zookeeper-1.0.2/ext/c_zookeeper.rb:172:in `_event_thread_body'"
],
[6] [
[ 0] "/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/ruby/1.9.1/monitor.rb:185:in `mon_enter'",
[ 1] "/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/ruby/1.9.1/monitor.rb:209:in `mon_synchronize'",
[ 2] "/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/ruby/1.9.1/logger.rb:559:in `write'",
[ 3] "/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/ruby/1.9.1/logger.rb:377:in `add'",
[ 4] "/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/ruby/1.9.1/logger.rb:452:in `error'",
[ 5] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zk-1.3.1/lib/zk/client/threaded.rb:244:in `block in raw_event_handler'",
[ 6] "/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'",
[ 7] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zk-1.3.1/lib/zk/client/threaded.rb:241:in `raw_event_handler'",
[ 8] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zk-1.3.1/lib/zk/event_handler.rb:145:in `process'",
[ 9] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zk-1.3.1/lib/zk/event_handler.rb:273:in `block in watcher_callback'",
[10] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zookeeper-1.0.2/lib/zookeeper/callbacks.rb:19:in `block in create'",
[11] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zookeeper-1.0.2/lib/zookeeper/callbacks.rb:26:in `block in initialize'",
[12] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zookeeper-1.0.2/lib/zookeeper/callbacks.rb:32:in `call'",
[13] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zookeeper-1.0.2/lib/zookeeper/callbacks.rb:32:in `call'",
[14] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zk-1.3.1/lib/zk/event_handler.rb:214:in `block (2 levels) in get_default_watcher_block'",
[15] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zk-1.3.1/lib/zk/event_handler.rb:213:in `tap'",
[16] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zk-1.3.1/lib/zk/event_handler.rb:213:in `block in get_default_watcher_block'",
[17] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zookeeper-1.0.2/lib/zookeeper/common.rb:155:in `call'",
[18] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zookeeper-1.0.2/lib/zookeeper/common.rb:155:in `dispatch_next_callback'",
[19] "/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/zookeeper-1.0.2/lib/zookeeper/common.rb:79:in `block (2 levels) in setup_dispatch_thread!'"
]
]
start-stop-daemon to start redis_failover how to save log
I try start it daemon by start-stop-daemon,because we want to see the log to know which redis is down and which tobe master!
It's ok!
But no output!
I try to add "> /var/log/redis_failover.log" into start-stop-daemon shell,but is bad!
So i think if you can add -l|--log option into redis_node_manager next version,it will so cool!
Thank you!
My start-stop-daemon shell(debian):
dev@template:/var/log$ cat /etc/init.d/redis_failover
!/bin/bash
ulimit -s 256
DAEMON=/usr/local/bin/redis_node_manager
DAEMON_OPTS="-C /etc/redis/redis_failover.yml"
PIDFILE=/var/run/goagent.pid
case "$1" in
start)
echo "Starting server"
/sbin/start-stop-daemon --start --pidfile $PIDFILE
--user root --group root
--background --make-pidfile --exec $DAEMON -- $DAEMON_OPTS
;;
stop)
echo "Stopping server"
/sbin/start-stop-daemon --stop --pidfile $PIDFILE --verbose
;;
restart)
echo "Restarting server"
$0 stop && $0 start
;;
*)
echo "Usage: /etc/init.d/redis_failover {start|stop|restart}"
exit 1
;;
esac
exit 0
Support SELECT and TTL commands
Hi, I was looking through the code for the failover client, and noticed that SELECT and TTL are not supported. These still have meaning in a failover environment, as many use SELECT to partition their Redis entries between different databases. Similarly, TTL is relevant to check how close to expiry a key is. Key expiration certainly seems like something that is needed.
What is your take on this? What was the rationale behind not supporting those operations in particular?
Thanks!
flu
Occasionally stale server selection
We've been testing failover and while, by and large, the server selection works as expected, occasionally we see failover not selecting the right master server even after a node manager has correctly broadcast the changes to our client.
Taking a look at the code, it seems like the conditional server selection here (where presence in the stack overrides master selection if :master_only is supplied):
https://github.com/ryanlecompte/redis_failover/blob/master/lib/redis_failover/client.rb#L458
coupled with .pop usage on the thread local client stack here:
https://github.com/ryanlecompte/redis_failover/blob/master/lib/redis_failover/client.rb#L475
might be causing it. We replaced the .pop usage with .clear and emptied the stack entirely when the method was called, and we stopped seeing incorrect client selection. Does this seem right to you? Is that actually a problem?
Stacktrace in resque web
Another resque web related stack trace:
NoMethodError - undefined method `[]' for nil:NilClass:
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/redis_failover-0.8.2/lib/redis_failover/client.rb:328:in `verify_role!'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/redis_failover-0.8.2/lib/redis_failover/client.rb:230:in `master'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/redis_failover-0.8.2/lib/redis_failover/client.rb:208:in `dispatch'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/redis_failover-0.8.2/lib/redis_failover/client.rb:82:in `block (2 levels) in <class:Client>'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/redis-namespace-1.0.3/lib/redis/namespace.rb:213:in `method_missing'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/resque-dynamic-queues-0.7.1/lib/resque/plugins/dynamic_queues/attributes.rb:36:in `block in set_dynamic_queues'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/redis-2.2.2/lib/redis.rb:1015:in `block (2 levels) in multi'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/redis-2.2.2/lib/redis.rb:978:in `block in pipelined'
/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/redis-2.2.2/lib/redis.rb:975:in `pipelined'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/redis-2.2.2/lib/redis.rb:1013:in `block in multi'
/usr/local/rvm/rubies/ruby-1.9.3-p125-falcon/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/redis-2.2.2/lib/redis.rb:1009:in `multi'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/redis_failover-0.8.2/lib/redis_failover/client.rb:208:in `dispatch'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/redis_failover-0.8.2/lib/redis_failover/client.rb:82:in `block (2 levels) in <class:Client>'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/redis-namespace-1.0.3/lib/redis/namespace.rb:184:in `method_missing'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/resque-dynamic-queues-0.7.1/lib/resque/plugins/dynamic_queues/attributes.rb:35:in `set_dynamic_queues'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/resque-dynamic-queues-0.7.1/lib/resque/plugins/dynamic_queues/server.rb:46:in `block in registered'
/mnt/appleton-matt/shared/bundle/ruby/1.9.1/gems/sinatra-1.3.2/lib/sinatra/base.rb:1212:in `call'
Discussion: Get rid of ZkClient class
I think there could be value in not adding a layer on top of the ZK class:
- It would be great to allow for ZK classes to be passed in to the redis client and have everything work properly
- I'm concerned that some of the retry logic may not play well with locking, etc.
It would be great to see if things can be made stable enough by just relying on the standard ZK event hooks to do the right thing...
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. ๐๐๐
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google โค๏ธ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.