getting Failed to connect to any given member about fluent-plugin-mongo

[repeatedly]

2014-12-04 13:26:58 +0100 [error]: unexpected error error_class=Mongo::ConnectionFailure error=#

no error message?
Could you show me entire error message with stacktrace?

[97-109-107]

Here's the full message:

2014-12-09 17:09:41 +0100 [info]: adding match pattern="log.**" type="copy"                                                                                                             [376/9753]
2014-12-09 17:09:41 +0100 [info]: adding match pattern="lively.**" type="copy"
2014-12-09 17:09:41 +0100 [error]: unexpected error error_class=Mongo::ConnectionFailure error=#<Mongo::ConnectionFailure: Failed to connect to any given member.>
  2014-12-09 17:09:41 +0100 [error]: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/mongo-1.8.6/lib/mongo/util/pool_manager.rb:182:in `connect_to_members'
  2014-12-09 17:09:41 +0100 [error]: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/mongo-1.8.6/lib/mongo/util/pool_manager.rb:52:in `block in connect'
  2014-12-09 17:09:41 +0100 [error]: <internal:prelude>:10:in `synchronize'
  2014-12-09 17:09:41 +0100 [error]: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/mongo-1.8.6/lib/mongo/util/pool_manager.rb:47:in `connect'
  2014-12-09 17:09:41 +0100 [error]: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/mongo-1.8.6/lib/mongo/mongo_replica_set_client.rb:181:in `block in connect'
  2014-12-09 17:09:41 +0100 [error]: <internal:prelude>:10:in `synchronize'
  2014-12-09 17:09:41 +0100 [error]: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/mongo-1.8.6/lib/mongo/mongo_replica_set_client.rb:170:in `connect'
  2014-12-09 17:09:41 +0100 [error]: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/mongo-1.8.6/lib/mongo/mongo_client.rb:645:in `setup'
  2014-12-09 17:09:41 +0100 [error]: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/mongo-1.8.6/lib/mongo/mongo_replica_set_client.rb:474:in `setup'
  2014-12-09 17:09:41 +0100 [error]: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/mongo-1.8.6/lib/mongo/mongo_replica_set_client.rb:147:in `initialize'
  2014-12-09 17:09:41 +0100 [error]: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-plugin-mongo-0.7.3/lib/fluent/plugin/out_mongo_replset.rb:51:in `new'
  2014-12-09 17:09:41 +0100 [error]: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-plugin-mongo-0.7.3/lib/fluent/plugin/out_mongo_replset.rb:51:in `get_connection'
  2014-12-09 17:09:41 +0100 [error]: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-plugin-mongo-0.7.3/lib/fluent/plugin/out_mongo.rb:190:in `get_or_create_collection'
  2014-12-09 17:09:41 +0100 [error]: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-plugin-mongo-0.7.3/lib/fluent/plugin/out_mongo.rb:85:in `start'
  2014-12-09 17:09:41 +0100 [error]: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.50/lib/fluent/plugin/out_copy.rb:50:in `block in start'
  2014-12-09 17:09:41 +0100 [error]: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.50/lib/fluent/plugin/out_copy.rb:49:in `each'
  2014-12-09 17:09:41 +0100 [error]: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.50/lib/fluent/plugin/out_copy.rb:49:in `start'
  2014-12-09 17:09:41 +0100 [error]: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.50/lib/fluent/match.rb:40:in `start'
  2014-12-09 17:09:41 +0100 [error]: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.50/lib/fluent/engine.rb:257:in `block in start'
  2014-12-09 17:09:41 +0100 [error]: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.50/lib/fluent/engine.rb:256:in `each'
  2014-12-09 17:09:41 +0100 [error]: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.50/lib/fluent/engine.rb:256:in `start'
  2014-12-09 17:09:41 +0100 [error]: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.50/lib/fluent/engine.rb:207:in `run'
  2014-12-09 17:09:41 +0100 [error]: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.50/lib/fluent/supervisor.rb:420:in `run_engine'
  2014-12-09 17:09:41 +0100 [error]: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.50/lib/fluent/supervisor.rb:133:in `block in start'
  2014-12-09 17:09:41 +0100 [error]: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.50/lib/fluent/supervisor.rb:246:in `call'
  2014-12-09 17:09:41 +0100 [error]: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.50/lib/fluent/supervisor.rb:246:in `main_process'
  2014-12-09 17:09:41 +0100 [error]: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.50/lib/fluent/supervisor.rb:221:in `block in supervise'
  2014-12-09 17:09:41 +0100 [error]: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.50/lib/fluent/supervisor.rb:220:in `fork'
  2014-12-09 17:09:41 +0100 [error]: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.50/lib/fluent/supervisor.rb:220:in `supervise'
  2014-12-09 17:09:41 +0100 [error]: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.50/lib/fluent/supervisor.rb:126:in `start'
  2014-12-09 17:09:41 +0100 [error]: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.50/lib/fluent/command/fluentd.rb:160:in `<top (required)>'
  2014-12-09 17:09:41 +0100 [error]: /usr/lib/fluent/ruby/lib/ruby/1.9.1/rubygems/custom_require.rb:55:in `require'
  2014-12-09 17:09:41 +0100 [error]: /usr/lib/fluent/ruby/lib/ruby/1.9.1/rubygems/custom_require.rb:55:in `require'
  2014-12-09 17:09:41 +0100 [error]: /usr/lib/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.50/bin/fluentd:6:in `<top (required)>'
  2014-12-09 17:09:41 +0100 [error]: /usr/lib/fluent/ruby/bin/fluentd:23:in `load'
  2014-12-09 17:09:41 +0100 [error]: /usr/lib/fluent/ruby/bin/fluentd:23:in `<top (required)>'
  2014-12-09 17:09:41 +0100 [error]: /usr/sbin/td-agent:7:in `load'
  2014-12-09 17:09:41 +0100 [error]: /usr/sbin/td-agent:7:in `<main>'
2014-12-09 17:09:41 +0100 [info]: shutting down fluentd
2014-12-09 17:09:41 +0100 [info]: process finished code=0
2014-12-09 17:09:41 +0100 [warn]: process died within 1 second. exit.

[repeatedly]

It seems mongo client judges mongo replica set nodes are dead.
Could you try Mongo::MongoReplicaSetClient.new(nodes) in simple ruby script?

[97-109-107]

I ran

#!/usr/bin/env ruby
require 'mongo' 

Mongo::MongoReplicaSetClient.new(['172.31.48.10:27999'])

with the following result:

/var/lib/gems/1.9.1/gems/mongo-1.11.1/lib/mongo/connection/pool_manager.rb:203:in 'connect_to_members': Failed to connect to any given member. (Mongo::ConnectionFailure)
from /var/lib/gems/1.9.1/gems/mongo-1.11.1/lib/mongo/connection/pool_manager.rb:70:in 'block in connect'
from <internal:prelude>:10:in 'synchronize'
from /var/lib/gems/1.9.1/gems/mongo-1.11.1/lib/mongo/connection/pool_manager.rb:65:in 'connect'
from /var/lib/gems/1.9.1/gems/mongo-1.11.1/lib/mongo/mongo_replica_set_client.rb:209:in 'block in connect'
from <internal:prelude>:10:in 'synchronize'
from /var/lib/gems/1.9.1/gems/mongo-1.11.1/lib/mongo/mongo_replica_set_client.rb:198:in 'connect'
from /var/lib/gems/1.9.1/gems/mongo-1.11.1/lib/mongo/mongo_client.rb:661:in 'setup'
from /var/lib/gems/1.9.1/gems/mongo-1.11.1/lib/mongo/mongo_replica_set_client.rb:513:in 'setup'
from /var/lib/gems/1.9.1/gems/mongo-1.11.1/lib/mongo/mongo_replica_set_client.rb:175:in 'initialize'
from ./check-mongo.rb:4:in 'new'
from ./check-mongo.rb:4:in '<main>'

Which I guess confirms your suspicion.

[repeatedly]

It seems client or your environment problem.
If updating mongo gem version doesn't fix the problem, I don't know why this problem occurs.

[97-109-107]

Thank you for you assistance, it was indeed caused by a characteristic of the mongo driver. It appears that you must use the exact name provided by rs.status() or connection will fail. This means if it isn't resolvable with DNS, you need to add it to your /etc/hosts on your connecting server. Once I did that the connection worked. - found the answer here: http://stackoverflow.com/questions/11714675/mongo-ruby-driver-cannot-connect-to-replica-set-primary