I've briefly looked through the code and I'm not seeing any support for logging what em-mongo is doing. Has that been left out for a reason, or is it simply not implemented yet? I'm new to the project and am having a little trouble getting a feel for the history, design decisions and roadmap. Can any one comment on this question?

You chaps might be interested in a patch I made for em-mongo to make it's API synchronous using fibers:

https://gist.github.com/20e49e0282783c4f5603

I've been using it with MongoMapper.

please tag all releases so that it is easy to download tarballs of releases from github.

@jsaak feel free to yank the notice at the top if you like ...

Any chance we'll be seeing remote host support in the near future so we can use services like MongoHQ?

http://github.com/dj2/em-mongo/commit/3b649a0a985635c35c2a5e4af527cc0cf00fc0fe

Had to add support for this in em-synchrony, but to work properly requires that the above is merged as well.

Hey there,

After explicitly calling #close on an EM::Mongo::Database, it looks like em-mongo is reconnecting from #unbind. Had a nasty problem today with Mongo running out of connections from that. :(

--Mike

$ cat /tmp/foo.rb

!/usr/bin/env ruby

require 'rubygems'
require 'em-mongo'

EM.run do
db = EM::Mongo::Connection.new.db('peach_test')
collection = db.collection('hotels')
EM.next_tick do
found = collection.find do |x|
p x
EM.stop
end
end
end

$ ruby /tmp/foo.rb
/Library/Ruby/Gems/1.8/gems/em-mongo-0.2.9/lib/em-mongo/connection.rb:197:in times': no block given (LocalJumpError) from /Library/Ruby/Gems/1.8/gems/em-mongo-0.2.9/lib/em-mongo/connection.rb:197:innext_response'
from /Library/Ruby/Gems/1.8/gems/em-mongo-0.2.9/lib/em-mongo/connection.rb:163:in receive_data' from /Library/Ruby/Gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:256:inrun_machine'
from /Library/Ruby/Gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:256:in `run'
from /tmp/foo.rb:6

$ ruby --version
ruby 1.8.6 (2009-06-08 patchlevel 369) [universal-darwin9.0]

$ gem list | grep mongo
em-mongo (0.2.9)
mongo (1.0, 0.19.1)
mongo_ext (0.19.1)
mongo_mapper (0.7.1)

https://github.com/bcg/em-mongo/blob/master/lib/em-mongo/connection.rb#L228

If reconnect_in is false, then @retries never gets incremented and hence the on_unbind never gets invoked - the connection just hangs indefinitely. Seems like, in fact, the @retries logic is completely useless: if you specify reconnect then it always reconnects, and if you don't reconnect then you hang forever.

a) drop the retries counter all together
b) use retries counter within the reconnect block to keep track of failed retries.. after which, the on_unbind should be invoked.

Find on the collection doesn't seem to return the full set of documents, using no limit or limit set to 0 it always returns 101 documents. If I specify a larger limit it does return it exactly but unbounded it does not seem to return the full collection. Can be easily demonstrated with:

cnt = 0
col.find({}) do |res|
  res.each do |doc|
    cnt += 1
    puts cnt
  end
end

Using several modifier operations in an update causes a crash:

http://gist.github.com/395485

It would be great to support connecting to replica sets.

http://www.mongodb.org/display/DOCS/Connecting+Drivers+to+Replica+Sets

There seems to be an issue with paths in the latest em-mongo gem. I get:

>> require 'eventmachine' #=> true
>> require 'em-synchrony' #=> true
>> require 'em-mongo'
LoadError: no such file to load -- eventmachine
    from <internal:lib/rubygems/custom_require>:33:in `require'
    from <internal:lib/rubygems/custom_require>:33:in `rescue in require'
    from <internal:lib/rubygems/custom_require>:29:in `require'
    from /Users/adriaant/.rvm/gems/ruby-1.9.2-p0@birdgrinder/gems/em-mongo-0.3.2/lib/em-mongo.rb:7:in `ensure in <top (required)>'
    from /Users/adriaant/.rvm/gems/ruby-1.9.2-p0@birdgrinder/gems/em-mongo-0.3.2/lib/em-mongo.rb:8:in `<top (required)>'
    from <internal:lib/rubygems/custom_require>:33:in `require'
    from <internal:lib/rubygems/custom_require>:33:in `rescue in require'
    from <internal:lib/rubygems/custom_require>:29:in `require'
    from (irb):3
    from /Users/adriaant/.rvm/rubies/ruby-1.9.2-p0/bin/irb:17:in `<main>'

If I replace the first 13 lines with:
require "eventmachine"
require "bson"

then it works ok.

are everwhere

Thanks for this great gem - could you please make the latest verison available via rubygems.org as well?
Thanks, Dominik

Is there a way to select only the fields you need, rather than fetching whole documents from the database? This could dramatically improve performance in some cases.

The driver currently lacks support for server-side sorting, documented here:
http://api.mongodb.org/wiki/current/Advanced%20Queries.html#AdvancedQueries-{{sort%28%29}}
and here:
http://www.mongodb.org/display/DOCS/Sorting+and+Natural+Order
Please add!

Whenever EM::stop is run, the unbind handler seems to produce this stacktrace:

/Users/wolfram/.rvm/gems/ruby-1.9.3-p448@handl-server/gems/em-mongo-0.4.3/lib/em-mongo/collection.rb:739:in `block in safe_send': undefined method `docs' for :disconnected:Symbol (NoMethodError)
from /Users/wolfram/.rvm/gems/ruby-1.9.3-p448@handl-server/gems/em-mongo-0.4.3/lib/em-mongo/connection.rb:186:in `call'
from /Users/wolfram/.rvm/gems/ruby-1.9.3-p448@handl-server/gems/em-mongo-0.4.3/lib/em-mongo/connection.rb:186:in `block in unbind'
from /Users/wolfram/.rvm/gems/ruby-1.9.3-p448@handl-server/gems/em-mongo-0.4.3/lib/em-mongo/connection.rb:186:in `each'
from /Users/wolfram/.rvm/gems/ruby-1.9.3-p448@handl-server/gems/em-mongo-0.4.3/lib/em-mongo/connection.rb:186:in `unbind'
from /Users/wolfram/.rvm/gems/ruby-1.9.3-p448@handl-server/gems/eventmachine-1.0.3/lib/eventmachine.rb:1440:in `event_callback'
from /Users/wolfram/.rvm/gems/ruby-1.9.3-p448@handl-server/gems/eventmachine-1.0.3/lib/eventmachine.rb:194:in `release_machine'
from /Users/wolfram/.rvm/gems/ruby-1.9.3-p448@handl-server/gems/eventmachine-1.0.3/lib/eventmachine.rb:194:in `run'
from /Users/wolfram/.rvm/gems/ruby-1.9.3-p448@handl-server/gems/em-synchrony-1.0.3/lib/em-synchrony.rb:38:in `synchrony'

Looking at the code of EMConnection#unbind#, it seems that it's the @responses array holds something other than callable objects:

    def unbind
      if @is_connected
        @responses.values.each { |resp| resp.call(:disconnected) }
        ...

What goes wrong here?

Update: I noticed that this only happens if an insert statement was run immediately prior to calling close on the connection/prior to EM::stop. If there are no pending writes, it unbinds cleanly.

I ran the example and it does not seem to work on Ruby 2.5.1 and MongoDB 3.6.2.

Is there any documentation on how to make it work?

Here's where it's at in the Mongo ruby driver:

http://github.com/mongodb/mongo-ruby-driver/blob/master/lib/mongo/db.rb#L90

Sorry I can't get to a patch at the moment...

I am trying to bulk update a number of documents in a collection:

def initialize
  @db            = EM::Mongo::Connection.new('localhost').db('ats_development')
  @listings      = @db.collection('listings')
end

def update
  @listings.update({ state: 'changed' }, { state: 'processing' }, {:multi => true})
end

I'm facing two problem here. First, if I set :multi to true, no document is udpdated at all. Second, if I omit :multi the first document is updated with it's new state, but all other attributes of the record are set to nil.

I followed the docs in the source. Am I doing something wrong here, or is this actually a bug?

When I run rake spec:integration:exhaustive all tests pass, even if I change several should into should_not.

As a confirmation an example which proves database_spec.rb is wrong:

require 'em-mongo'
require 'eventmachine'

EM.run do
  db = EM::Mongo::Connection.new('localhost').db('my_database')
  collection = db.collection('my_collection')

  EM.next_tick do
    db.collections.callback do |c|
      puts c.first.inspect
      raise unless c.first.kind_of? EM::Mongo::Collection
    end

    EM.add_timer(1) { EM.stop }
  end
end

My understanding is that when EM::Mongo::Connection is instantiated with the timeout parameter that an error should be raised if a connection cannot be established in that time. It looks like that's not happening.

Looking at the initializer for Connection confirms that the errback should raise the "failure with mongodb server..." error. The errback is never called, though. The reason is that the timeout timer is cancelled when the connection's unbind method calls set_deferred_status(nil). So the timeout timer never fires.

You can confirm the problem by stopping your mongod process and running the following code. I think it should raise an error, but it does not.

require 'em-mongo'
require 'eventmachine'

EM.run do
  EM::Mongo::Connection.new('localhost', 27017, 1)
end

It's not obvious to me how, exactly, to fix this. Any suggestions?

Hello,

I'm seeing an issue where query callbacks are never invoked when multiple queries are pending. I've added a spec (which hangs) + some debug printing to my fork

http://github.com/jarib/em-mongo

but I'm not really sure how to debug further. Any help is greatly appreciated.

I was looking through the specs for usage info and I thought I saw some code that didn't look right. I ran the tests and got this:

https://gist.github.com/785832

(the code I thought didn't look right was actually OK, however)

Hi

I'm using em-mongo with and without Goliath. There is a weird thing that I'm not able to close a connection:

I've tried conn.close almost everywhere (in and out of callback, out of method, in EM proc etc.), it always keep connection alive which consumes resources on server box

here is a snippet of problem I've reproduced:

def load_items
  org = Fiber.current
  conn = EM::Mongo::Connection.new('localhost')
  database = conn.db('mydb')
  coll = database.collection('itemscoll')
  curs = coll.find({"active" => true}).defer_as_a.callback do |docs|
    docs.each do |doc|
      $items << doc["data"].to_s unless $existing.include?(doc["data"].to_s)
    end
    conn.close
    org.resume
  end
  return Fiber.yield
end

EM.run do
  Fiber.new{
    load_items  
    EM.stop
  }.resume
end

bson version ['>= 1.1.3'] does not suffice as 2.X.X does not work.

I'll try to dig into this in more detail later, but I thought I'd capture it. EM-Mongo doesn't seem to work under JRuby, BSON issues:

NativeException in 'EventMachine::Mongo::Collection should handle multiple pending queries'
org.bson.BSONException: should be impossible
org/bson/BSONDecoder.java:35:in decode' org/bson/BSONDecoder.java:35:indecode'
/home/nathan/.rvm/gems/jruby-1.6.2/gems/bson-1.3.1-java/lib/../lib/bson/bson_java.rb:16:in deserialize' org/jruby/RubyRange.java:407:ineach'
org/jruby/RubyEnumerable.java:706:in collect' /home/nathan/.rvm/gems/jruby-1.6.2/gems/eventmachine-0.12.10-java/lib/eventmachine.rb:1447:inevent_callback'
/home/nathan/.rvm/gems/jruby-1.6.2/gems/eventmachine-0.12.10-java/lib/jeventmachine.rb:82:in eventCallback' /home/nathan/.rvm/gems/jruby-1.6.2/gems/eventmachine-0.12.10-java/lib/jeventmachine.rb:101:inrun_machine'
/home/nathan/.rvm/gems/jruby-1.6.2/gems/eventmachine-0.12.10-java/lib/eventmachine.rb:256:in run' /home/nathan/.rvm/gems/jruby-1.6.2/bundler/gems/em-spec-25509a6efa0c/lib/em-spec/rspec.rb:24:inem'
/home/nathan/.rvm/gems/jruby-1.6.2/bundler/gems/em-spec-25509a6efa0c/lib/em-spec/rspec.rb:62:in instance_eval' org/jruby/ext/Timeout.java:79:intimeout'
org/jruby/RubyArray.java:1602:in each' org/jruby/RubyArray.java:1602:ineach'
org/jruby/RubyKernel.java:1063:in `load'

and newer files in spec directory are missing from gem. Please update the gem with all files in spec directory.

The example in the readme will not work when first run by someone experimenting with ruby, eventmachine and em-mongo.

The code should be as follows:

require 'rubygems' # Missing in original 
require 'em-mongo'

EM.run do
  db = EM::Mongo::Connection.new.db('db')
  collection = db.collection('test')
  EM.next_tick do
    doc = {"hello" => "world"}
    id = collection.insert(doc)
    collection.find({'_id' => id}) do |res| # Original codes was collection.find('_id' => id]) do |res|
      puts res.inspect
      EM.stop
    end
    collection.remove(doc)
  end
end

I would do a fork, modify pull request cycle and bask in the reflected glory of the git history but I am not at the computer with my key at the moment.

When trying to switch from mongo to em-mongo, I think I discovered a performance issue when handling documents that contain field bigger than 1 kilobyte. The script below takes a size in bytes as argument, inserts 1000 documents with a data field that has the given size, and logs the time loading these documents takes with mongo and em-mongo.

Here are the results I get on my MacBook Air (Core 2 Duo 1.6 Ghz):

$ ruby em_mongo_vs_mongo.rb 1000
0.000: Requesting with mongo
0.094: mongo: returned 1000000 bytes
0.000: Requesting with em_mongo
0.191: em_mongo: returned 1000000 bytes
$ ruby em_mongo_vs_mongo.rb 10000
0.000: Requesting with mongo
0.154: mongo: returned 10000000 bytes
0.000: Requesting with em_mongo
2.568: em_mongo: returned 10000000 bytes
$ ruby em_mongo_vs_mongo.rb 100000
0.000: Requesting with mongo
0.887: mongo: returned 100000000 bytes
0.000: Requesting with em_mongo
32.195: em_mongo: returned 100000000 bytes

As you can see, em-mongo needs quite a lot of time when the documents get bigger. Is this a bug, or have I done something wrong in the code?

require 'mongo'
require 'em-mongo'

def log(message)
  now = Time.now.to_f
  @last ||= now
  puts "%2.3f: #{message}" % (now - @last)
  @last = now
end

EM.run do
  mongo = Mongo::Connection.new.db("em_mongo_vs_mongo")
  em_mongo = EM::Mongo::Connection.new.db("em_mongo_vs_mongo")

  coll_mongo = mongo.collection('test_data')
  coll_em_mongo = em_mongo.collection('test_data')

  coll_mongo.remove()

  (1..1000).each do |index|
    coll_mongo.insert({id:index, data:(" " * ARGV[0].to_i)})
  end

  log("Requesting with mongo")
  cursor = coll_mongo.find({}, :fields => ['id', 'data'])
  count = 0
  cursor.each do |doc|
    if doc # unnecessary with mongo, but just to make the comparison fair
      count += doc['data'].to_s.size
    else
      log("mongo: screwed up")
      EM.next_tick do
        EM.stop
      end
    end
  end
  log("mongo: returned #{count} bytes")

  log("Requesting with em_mongo")
  cursor = coll_em_mongo.find({}, :fields => ['id', 'data'])
  count = 0
  cursor.each do |doc|
    if doc
      count += doc['data'].to_s.size
    else
      log("em_mongo: returned #{count} bytes")
      EM.next_tick do
        EM.stop
      end
    end
  end
end

I am using both em-mongo and mongoid in my application. The mongoid gem loads the mongo gem, which has the same Mongo:Conversions class. em-mongo should either require the mongo gem and use the class from there or use a unique namespace so there is no collision. The result is only a warning, but it's annoying.

/Users/canavese/.rvm/gems/ruby-1.9.2-p0@banjo_server/gems/em-mongo-0.3.6/lib/em-mongo/conversions.rb:24: warning: already initialized constant ASCENDING_CONVERSION
/Users/canavese/.rvm/gems/ruby-1.9.2-p0@banjo_server/gems/em-mongo-0.3.6/lib/em-mongo/conversions.rb:25: warning: already initialized constant DESCENDING_CONVERSION

In collection.rb, I'd like to suggest to use
obj['_id'] ||= BSON::ObjectId.new unless obj[:_id]
instead of
obj['_id'] ||= BSON::ObjectId.new
since someone might use symbols as keys in the passed hash.

I also recommend to use a newer version of em-spec:
gem "em-spec", :git => "git://github.com/mloughran/em-spec.git", :branch => "rspec2"

A fork with the above changes and which passes the rspec tests is at https://github.com/adriaant/em-mongo

Currently you cannot pass options to the create_collection command.

This is mentioned in the documentation but that might be something left over from the regular Ruby Mongo driver

Is this by choice or should I try adding this to submit a pull request?

Using:

• MRI 1.9.3p0
• em-mongo 0.4.1

Seeing this error:

 NoMethodError:
   undefined method `defer_as_a' for #<EventMachine::Mongo::Cursor:0x000001024f9338>

Pulled latest from github and the method is there, but fresh gem install sees the above error.

Would be great to have the distinct command implemented

http://www.mongodb.org/display/DOCS/Aggregation#Aggregation-Distinct

collection.find only returns max 101 results, and a higher :limit causes a Buffer Overflow exception to be raised:

http://gist.github.com/395122