igrigorik / em-synchrony

Fiber aware EventMachine clients and convenience classes

em-synchrony's Issues

em-mongo goes crazy with mongo 1.4

I am just trying to report as much as I know about this issue. We have a Goliath app using em-mongo. After upgrading the mongo driver to 1.4 thousands of Goliath processes were created which took down the application. If you don't have time to debug this issue as we don't, I would suggest just releasing a new version of em-mongo that depends on mongo < 1.4

not useable with mysqlplus -- always raises error

require 'em-activerecord'

RuntimeError: Missing EM-Synchrony dependency: gem install em-mysqlplus
from /home/micha/.rvm/gems/ruby-1.9.2-p0/gems/em-synchrony-0.1.5/lib/em-synchrony/em-mysqlplus.rb:4:in rescue in <top (required)>' from /home/micha/.rvm/gems/ruby-1.9.2-p0/gems/em-synchrony-0.1.5/lib/em-synchrony/em-mysqlplus.rb:1:in<top (required)>'
from internal:lib/rubygems/custom_require:29:in require' from <internal:lib/rubygems/custom_require>:29:inrequire'
from /home/micha/.rvm/gems/ruby-1.9.2-

The original error should e given, to not swallow any other error.
The problem afaik is the require 'mysqlplus' which does not exist in mysqlplus @ http://github.com/igrigorik/em-mysqlplus/tree/master/lib/

Former connection lost when we call 'acquire' twice in the same fiber

From #72

*   if a "sync" call is made, then fiber is paused until operation is done and connection is automatically returned to the pool

This is not always true i think.
if the 'sync' call invoke another sync call, the first conn wont be returned back to the pool till the second finished.
In this case, the connection should be reused.

As the current implementation, we will lost the former connection if we call the 'acquire' method twice:

def acquire(fiber)
  if conn = @available.pop
    @reserved[fiber.object_id] = conn
    conn
  else
    Fiber.yield @pending.push fiber
    acquire(fiber)
  end
end

and still 'release' twice:

def execute(async)
  f = Fiber.current

  begin
    conn = acquire(f)
    yield conn
  ensure
    release(f) if not async
  end
end

def release(fiber)
  @available.push(@reserved.delete(fiber.object_id))

  if pending = @pending.shift
    pending.resume
  end
end

then push a 'nil' to the list 'available'.

This is a case to reproduce the problem:

require 'em-synchrony'

CACHE_HASH = {}

class CacheClient < Hash

  # the following three methods are all sync
  # get value from CACHE_HASH
  def get(key)
    puts "#{self} get: #{key}"
    CACHE_HASH[key]
  end

  # get value from CACHE_HASH
  # if value does not exist
  # then yield to accept a value
  # then put it into the CACHE_HASH
  # and return the value
  def fetch(key)
    puts "#{self} fetch: #{key}"
    if value = get(key)
      value
    else
      value = yield
      set key, value
      value
    end
  end

  # put the value into CACHE_HASH
  def set(key, value)
    puts "#{self} set: #{key}, value: #{value}"
    CACHE_HASH[key] = value
  end

  def inspect
    "#{self.to_s}<#{super}>"
  end

  def to_s
    "#{self.class.name}@#{self.object_id}"
  end
end

CACHE = EM::Synchrony::ConnectionPool.new :size => 2 do
  CacheClient.new
end

class CacheUtil
  class << self

    def get(key)
      CACHE.get key
    end

    def fetch(key)
      CACHE.fetch key
    end

    def set(key, value)
      CACHE.set key, value
    end
  end
end

def log(key)
  puts "+" * 20
  puts key
  puts "available: #{CACHE.instance_variable_get(:@available).inspect}"
  puts "available size: #{CACHE.instance_variable_get(:@available).size}"
  puts "reserved: #{CACHE.instance_variable_get(:@reserved).inspect}"
  puts "reserved size: #{CACHE.instance_variable_get(:@reserved).size}"
  puts "CACHE_HASH: #{CACHE_HASH.inspect}"
  puts "-" * 20
end

log "0"
# the methods are all sync
CACHE.set "key2", "value2"  # put "key2" into cache
CACHE.fetch "key1" do       # get "key1", not exist;        acquire
  log "1"
  # what we put into key1 are based on the value of key2
  value2 = CACHE.get "key2" # get "key2";                   acquire
                            #                               release
  log "2"

  "just_for_test:" + value2 # generate value for "key1" based on value of "key2"
                            # put the value into "key1"
end                                                       # release
log "3"

result:

++++++++++++++++++++
0
available: [CacheClient@5756940<{}>, CacheClient@5756920<{}>]
available size: 2
reserved: {}
reserved size: 0
CACHE_HASH: {}
--------------------
CacheClient@5756920 set: key2, value: value2
CacheClient@5756920 fetch: key1
CacheClient@5756920 get: key1
++++++++++++++++++++
1
available: [CacheClient@5756940<{}>]
available size: 1
reserved: {5755000=>CacheClient@5756920<{}>}
reserved size: 1
CACHE_HASH: {"key2"=>"value2"}
--------------------
CacheClient@5756940 get: key2
++++++++++++++++++++
2
available: [CacheClient@5756940<{}>]
available size: 1
reserved: {}
reserved size: 0
CACHE_HASH: {"key2"=>"value2"}
--------------------
CacheClient@5756920 set: key1, value: just_for_test:value2
++++++++++++++++++++
3
available: [CacheClient@5756940<{}>, nil]
available size: 2
reserved: {}
reserved size: 0
CACHE_HASH: {"key2"=>"value2", "key1"=>"just_for_test:value2"}
--------------------

Error while running Synchrony Iterator example

I get the same result when running the MultiIterator as well.

Error

/Users/bmckim/.rbenv/versions/1.9.2-p290/lib/ruby/gems/1.9.1/gems/em-synchrony-1.0.0/lib/em-synchrony.rb:25: [BUG] Bus Error
ruby 1.9.2p290 (2011-07-09 revision 32553) [x86_64-darwin11.1.0]

-- control frame ----------
c:0010 p:---- s:0032 b:0032 l:000031 d:000031 CFUNC  :resume
c:0009 p:0021 s:0029 b:0029 l:000ef8 d:000538 BLOCK  /Users/bmckim/.rbenv/versions/1.9.2-p290/lib/ruby/gems/1.9.1/gems/em-synchrony-1.0.0/lib/em-synchrony.rb:25
c:0008 p:---- s:0027 b:0027 l:000026 d:000026 FINISH
c:0007 p:---- s:0025 b:0025 l:000024 d:000024 CFUNC  :call
c:0006 p:---- s:0023 b:0023 l:000022 d:000022 CFUNC  :run_machine
c:0005 p:0248 s:0020 b:0020 l:000019 d:000019 METHOD /Users/bmckim/.rbenv/versions/1.9.2-p290/lib/ruby/gems/1.9.1/gems/eventmachine-1.0.0.beta.4/lib/eventmachine.rb:179
c:0004 p:0056 s:0013 b:0013 l:000ef8 d:000ef8 METHOD /Users/bmckim/.rbenv/versions/1.9.2-p290/lib/ruby/gems/1.9.1/gems/em-synchrony-1.0.0/lib/em-synchrony.rb:27
c:0003 p:0039 s:0006 b:0006 l:000358 d:001d38 EVAL   em-sync.rb:4
c:0002 p:---- s:0004 b:0004 l:000003 d:000003 FINISH
c:0001 p:0000 s:0002 b:0002 l:000358 d:000358 TOP   
---------------------------
-- Ruby level backtrace information ----------------------------------------
em-sync.rb:4:in `<main>'
/Users/bmckim/.rbenv/versions/1.9.2-p290/lib/ruby/gems/1.9.1/gems/em-synchrony-1.0.0/lib/em-synchrony.rb:27:in `synchrony'
/Users/bmckim/.rbenv/versions/1.9.2-p290/lib/ruby/gems/1.9.1/gems/eventmachine-1.0.0.beta.4/lib/eventmachine.rb:179:in `run'
/Users/bmckim/.rbenv/versions/1.9.2-p290/lib/ruby/gems/1.9.1/gems/eventmachine-1.0.0.beta.4/lib/eventmachine.rb:179:in `run_machine'
/Users/bmckim/.rbenv/versions/1.9.2-p290/lib/ruby/gems/1.9.1/gems/eventmachine-1.0.0.beta.4/lib/eventmachine.rb:179:in `call'
/Users/bmckim/.rbenv/versions/1.9.2-p290/lib/ruby/gems/1.9.1/gems/em-synchrony-1.0.0/lib/em-synchrony.rb:25:in `block in synchrony'
/Users/bmckim/.rbenv/versions/1.9.2-p290/lib/ruby/gems/1.9.1/gems/em-synchrony-1.0.0/lib/em-synchrony.rb:25:in `resume'

-- C level backtrace information -------------------------------------------
0   ruby                                0x000000010682f53e rb_vm_bugreport + 110
1   ruby                                0x0000000106724193 report_bug + 259
2   ruby                                0x0000000106724301 rb_bug + 161
3   ruby                                0x00000001067c6452 sigbus + 18
4   libsystem_c.dylib                   0x00007fff8cff7cfa _sigtramp + 26
5   ???                                 0x0000000106a05680 0x0 + 4406138496

[NOTE]
You may have encountered a bug in the Ruby interpreter or extension libraries.
Bug reports are welcome.
For details: http://www.ruby-lang.org/bugreport.html

Abort trap: 6

Code

require "em-synchrony"
require "em-synchrony/em-http"

EM.synchrony do
    concurrency = 2
    urls = ['http://google.com', 'http://yahoo.com']

    # iterator will execute async blocks until completion, .each, .inject also work!
    results = EM::Synchrony::Iterator.new(urls, concurrency).map do |url, iter|

        # fire async requests, on completion advance the iterator
        http = EventMachine::HttpRequest.new(url).aget
        http.callback { iter.return(http) }
        http.errback { iter.return(http) }
    end

    p results # all completed requests
    EventMachine.stop
end

Gemfile

gem 'eventmachine', '>=1.0.0.beta'
gem 'em-http-request', '1.0.0'
gem 'em-synchrony', '1.0.0'

Gems

*** LOCAL GEMS ***

addressable (2.2.6)
bundler (1.0.18)
em-http-request (1.0.0)
em-socksify (0.1.0)
em-synchrony (1.0.0)
eventmachine (1.0.0.beta.4)
http_parser.rb (0.5.2)

EM::Synchrony::TCPSocket - provide #readline

Hi! I try to get ruby-net-nntp running with em-synchrony through the TCPSocket replacement. All that would be needed is a #readline method to get single lines from the input stream.

Are you able to provide that?

-alex

Gem dependencies

Hi,

the em-synchrony gem does not depend of em-http-request, em-mysqlplus or remcached, but it's not possible to use it without them (LoadError when I require it). It could be nice to add these dependencies to the gem.

bug in em-synchrony/em-mongo

I am using em-synchrony/em-mongo

on goliath trying to issue find or first return with an error :

[9838:ERROR] 2011-07-13 14:22:20 :: wrong number of arguments (5 for 6)
[9838:ERROR] 2011-07-13 14:22:20 :: /usr/local/lib/ruby/gems/1.9.1/gems/em-mongo-0.3.6/lib/em-mongo/connection.rb:102:in find' /usr/local/lib/ruby/gems/1.9.1/gems/em-synchrony-0.3.0.beta.1/lib/em-synchrony/em-mongo.rb:34:infind'
/usr/local/lib/ruby/gems/1.9.1/gems/em-synchrony-0.3.0.beta.1/lib/em-synchrony/em-mongo.rb:41:in `first'

code:

mongo_db.db('thumbnail-db').collection('thumbnails').first({"url" => url,"width" => width.to_i}, {})

Does the connection_pool determine when a method is async properly?

https://github.com/igrigorik/em-synchrony/blob/master/lib/em-synchrony/connection_pool.rb#L66

If my client has a method 'add', it will be treated as a 'async' method. This might be not as we expected.

   it "should treat add method as sync" do
     class Client
       def add
       end
     end

     pool = EM::Synchrony::ConnectionPool.new :size => 1 do
       Client.new
     end

     EM.run do
       EM.synchrony do
         pool.add

         EM.stop
       end
     end
   end

Have we a better way to do this? Can we use the result returned by the method which we called?

unexpected "can't yield from root fiber" error

# Gemfile
source :rubygems

gem "em-synchrony", :git => 'git://github.com/igrigorik/em-synchrony'
gem "eventmachine", :git => 'git://github.com/eventmachine/eventmachine'
gem "em-http-request", :git => 'git://github.com/igrigorik/em-http-request'

# example-aget.rb
require 'rubygems'
require 'bundler/setup'
require 'em-synchrony'
require 'em-synchrony/em-http'

EM.synchrony do
    pages = EM::Synchrony::Iterator.new(1..2, 2).map do |num, iter|
        page = EM::HttpRequest.new('http://google.com').aget :redirects => 1
        page.callback { iter.return(page) }
    end
    p "Callbacks. Fetched pages: #{pages}"
    EventMachine.stop
end

# example-get.rb
require 'rubygems'
require 'bundler/setup'
require 'em-synchrony'
require 'em-synchrony/em-http'

EM.synchrony do
    pages = EM::Synchrony::Iterator.new(1..2, 2).map do |num, iter|
        page = EM::HttpRequest.new('http://google.com').get :redirects => 1
        iter.return(page)
    end
    p "No callbacks. Fetched pages: #{pages}"
    EventMachine.stop
end

this code works...

EM.synchrony do
    page = EM::HttpRequest.new('http://google.com').get :redirects => 1
    p "No callbacks. Fetched page: #{page}"
    EventMachine.stop
end

example-aget.rb works fine too.

However, when I try to run example-get.rb from above, I get the following error.

$ ruby example-get.rb
(eval):10:in `yield': can't yield from root fiber (FiberError)
    from (eval):10:in `get'
    from test.rb:8:in `block (2 levels) in <main>'
    .....

Is this expected behavior, incorrect use on my part, or a bug in .get used in conjunction with EM::Synchrony::Iterator?

Rename Multi#perform to Multi#join or similar?

I noticed that since Multi#perform is implemented as just Fiber.yield, that it's really a join-type operation instead of a perform like other multi interfaces follow. Consider the following code:

require "em-synchrony/em-http"
EventMachine.synchrony do
    multi = EventMachine::Synchrony::Multi.new
    multi.add :a, EventMachine::HttpRequest.new("http://www.postrank.com").aget
    multi.add :b, EventMachine::HttpRequest.new("http://www.postrank.com").apost

    # ADDED LINE!
    # All three requests will complete here if there's time.
    response = EventMachine::HttpRequest.new("http://www.google.com/").get

    res = multi.perform

    p "Look ma, no callbacks, and parallel HTTP requests!"
    p res

    EventMachine.stop
end

ActiveRecord 3.1.0 and no connection_pool.column_defaults

Hello!

I think I've spotted a bug in em-synchrony, although it may be related to another library. My service is using ActiveRecord outside of rails and it runs fine using database 'mysql2'. However, using 'em_mysql2' adapter and issue a ActiveRecord::Create I'm met with the exception:

NoMethodError - undefined method `[]' for nil:NilClass:
 /var/lib/gems/1.9.1/gems/activerecord-3.1.0/lib/active_record/base.rb:717:in `column_defaults'
 /var/lib/gems/1.9.1/gems/activerecord-3.1.0/lib/active_record/persistence.rb:327:in `attributes_from_column_definition'
 /var/lib/gems/1.9.1/gems/activerecord-3.1.0/lib/active_record/locking/optimistic.rb:69:in `attributes_from_column_definition'
 /var/lib/gems/1.9.1/gems/activerecord-3.1.0/lib/active_record/base.rb:1544:in `initialize'
 /var/lib/gems/1.9.1/gems/activerecord-3.1.0/lib/active_record/base.rb:508:in `new'
 /var/lib/gems/1.9.1/gems/activerecord-3.1.0/lib/active_record/base.rb:508:in `create'

Browsing through the source of ActiveRecord I see the code causing the exception:
# Returns a hash where the keys are column names and the values are
# default values when instantiating the AR object for this table.
def column_defaults
connection_pool.column_defaults[table_name]
end

... the connection_pool.column_defaults is Nil.

mysql is 0.3.7
em-synchrony is 1.0.0
activerecord is 3.1.0

Thanks for any ideas!

Wrap in Fiber EM callbacks

em-synchrony works great when within a fiber you want to make use of an async library in a non-callback way. Alas, the EM callbacks are not wrapped within a fiber (except for the timers callbacks, where were recently added), which means you can't use em-synchrony within them without some work.

And example of where this is useful is something like the AMQP gem, where one subscribers to messages from a queue for further processing. The processing itself occurs as a result of a callback from EM, and this callback is not wrapped in a fiber, which means you can use em-synchrony to do further work (e.g. send the message over HTTP), without a bit of leg work.

You'd probably have to patch EM. event_callback and EM. run_deferred_callbacks.

Insane error message when no acceptor is found

If an invalid host is passed to EM for listening, like this:

EM::run do
  EM::synchrony do
    EM::start_server '*', 1234, Echo
  end
end

It shows a completely unrelated error message like this:

/home/whitequark/.rvm/gems/ruby-1.9.2-p290/gems/em-synchrony-1.0.0/lib/em-synchrony.rb:52:in `resume': dead fiber called (FiberError)
    from /home/whitequark/.rvm/gems/ruby-1.9.2-p290/gems/em-synchrony-1.0.0/lib/em-synchrony.rb:52:in `block in sync'
    from /home/whitequark/.rvm/gems/ruby-1.9.2-p290/gems/eventmachine-1.0.0.beta.4/lib/em/deferrable.rb:158:in `call'
    from /home/whitequark/.rvm/gems/ruby-1.9.2-p290/gems/eventmachine-1.0.0.beta.4/lib/em/deferrable.rb:158:in `set_deferred_status'
    from /home/whitequark/.rvm/gems/ruby-1.9.2-p290/gems/eventmachine-1.0.0.beta.4/lib/em/deferrable.rb:198:in `fail'
    from /home/whitequark/.rvm/gems/ruby-1.9.2-p290/gems/redis-2.2.2/lib/redis/connection/synchrony.rb:34:in `rescue in receive_data'
    from /home/whitequark/.rvm/gems/ruby-1.9.2-p290/gems/redis-2.2.2/lib/redis/connection/synchrony.rb:29:in `receive_data'
    from /home/whitequark/.rvm/gems/ruby-1.9.2-p290/gems/eventmachine-1.0.0.beta.4/lib/eventmachine.rb:179:in `run_machine'
    from /home/whitequark/.rvm/gems/ruby-1.9.2-p290/gems/eventmachine-1.0.0.beta.4/lib/eventmachine.rb:179:in `run'
    from ./server.rb:34:in `<main>'

(In my case, a EM'd Redis connection was also present.)

SSL doesn't work after setting TCPSocket = EventMachine::Synchrony::TCPSocket

minimum to reproduce:

EM.synchrony do
  TCPSocket = EventMachine::Synchrony::TCPSocket
  obj = Net::HTTP.new("google.com", 443)
  obj.use_ssl = true
  obj.verify_mode = OpenSSL::SSL::VERIFY_NONE
  obj.get("/")
  EM.stop
end

triggers:

ruby-1.9.2-p290/lib/ruby/1.9.1/net/http.rb:658:in `initialize': wrong argument type EventMachine::Synchrony::TCPSocket (expected File) (TypeError)
    from /Users/topper/.rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/net/http.rb:658:in `new'
    from /Users/topper/.rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/net/http.rb:658:in `connect'
    from /Users/topper/.rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/net/http.rb:637:in `do_start'
    from /Users/topper/.rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/net/http.rb:626:in `start'
    from /Users/topper/.rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/net/http.rb:1168:in `request'
    from /Users/topper/.rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/net/http.rb:888:in `get'

Using synchrony clients within a EM timer results in an error

Using synchrony clients (such as the Mysql2::EM::Fiber::Client) within a EM timer results (e.g. by using EM.add_periodic_timer) results in "`yield': can't yield from root fiber (FiberError)". Presumably this is because EM executed the block in the timer outside of the Fiber created by em-synchrony.

Wrapping the code in the timer block within a Fiber solves the error. E.g.

f = Fiber.current
Fiber.new {
  block.call
  f.transfer
}.transfer

em-synchrony should monkey-patch the EM timer methods to wrap timer blocks in Fibers or provide new timer methods that do so.

Add Bitly support API v3

http://github.com/mccoder/em-synchrony/commit/0dce0e61217b482f7da7c9705355c2314f5ad94a

Example added in Readme.

Running the specs

I'd like to run the specs! :)

$ bundle
Using addressable (2.2.6) 
Using bson (1.3.1) 
Using bson_ext (1.3.1) 
Using diff-lcs (1.1.2) 
Using hiredis (0.3.2) 
Using em-hiredis (0.1.0) 
Using eventmachine (1.0.0.beta.3) from git://github.com/eventmachine/eventmachine.git (at master) with native extensions 
Using em-socksify (0.1.0) 
Using http_parser.rb (0.5.1) 
Using em-http-request (1.0.0.beta.4) from git://github.com/igrigorik/em-http-request (at master) 
Using em-mongo (0.3.6) 
Using em-mysqlplus (0.1.5) 
Using em-redis (0.3.0) 
Using mongo (1.3.1) 
Using mysqlplus (0.1.2) 
Using remcached (0.4.1) 
Using rspec-core (2.6.4) 
Using rspec-expectations (2.6.0) 
Using rspec-mocks (2.6.0) 
Using rspec (2.6.0) 
Using bundler (1.0.13) 
Your bundle is complete! Use `bundle show [gemname]` to see where a bundled gem is installed.
$ rake
rake aborted!
undefined method `desc' for #<Bundler::GemHelper:0x8a5a160>

(See full trace by running task with --trace)

So I added Rake to the Gemfile to be able to do this:

$ bundle exec rake
/home/lars/.rvm/rubies/ruby-1.9.2-p180/bin/ruby -S bundle exec rspec ./spec/iterator_spec.rb ./spec/mysqlplus_spec.rb ./spec/hiredis_spec.rb ./spec/mongo_spec.rb ./spec/memcache_spec.rb ./spec/redis_spec.rb ./spec/remcached_spec.rb ./spec/timer_spec.rb ./spec/connection_pool_spec.rb ./spec/em-mongo_spec.rb ./spec/beanstalk_spec.rb ./spec/inlinesync_spec.rb ./spec/tcpsocket_spec.rb ./spec/http_spec.rb
/home/lars/workspace/em-synchrony/spec/iterator_spec.rb:1:in `require': no such file to load -- spec/helper/all (LoadError)
    from /home/lars/workspace/em-synchrony/spec/iterator_spec.rb:1:in `<top (required)>'
    from /home/lars/.rvm/gems/ruby-1.9.2-p180@em-synchrony/gems/rspec-core-2.6.4/lib/rspec/core/configuration.rb:419:in `load'
    from /home/lars/.rvm/gems/ruby-1.9.2-p180@em-synchrony/gems/rspec-core-2.6.4/lib/rspec/core/configuration.rb:419:in `block in load_spec_files'
    from /home/lars/.rvm/gems/ruby-1.9.2-p180@em-synchrony/gems/rspec-core-2.6.4/lib/rspec/core/configuration.rb:419:in `map'
    from /home/lars/.rvm/gems/ruby-1.9.2-p180@em-synchrony/gems/rspec-core-2.6.4/lib/rspec/core/configuration.rb:419:in `load_spec_files'
    from /home/lars/.rvm/gems/ruby-1.9.2-p180@em-synchrony/gems/rspec-core-2.6.4/lib/rspec/core/command_line.rb:18:in `run'
    from /home/lars/.rvm/gems/ruby-1.9.2-p180@em-synchrony/gems/rspec-core-2.6.4/lib/rspec/core/runner.rb:80:in `run_in_process'
    from /home/lars/.rvm/gems/ruby-1.9.2-p180@em-synchrony/gems/rspec-core-2.6.4/lib/rspec/core/runner.rb:69:in `run'
    from /home/lars/.rvm/gems/ruby-1.9.2-p180@em-synchrony/gems/rspec-core-2.6.4/lib/rspec/core/runner.rb:11:in `block in autorun'
rake aborted!
ruby -S bundle exec rspec ./spec/iterator_spec.rb ./spec/mysqlplus_spec.rb ./spec/hiredis_spec.rb ./spec/mongo_spec.rb ./spec/memcache_spec.rb ./spec/redis_spec.rb ./spec/remcached_spec.rb ./spec/timer_spec.rb ./spec/connection_pool_spec.rb ./spec/em-mongo_spec.rb ./spec/beanstalk_spec.rb ./spec/inlinesync_spec.rb ./spec/tcpsocket_spec.rb ./spec/http_spec.rb failed

Tasks: TOP => default => spec
(See full trace by running task with --trace)

I'd like to write a patch for EM::Synchrony.sync, are the specs known to be broken?

README example error: undefined method `get' for module `EventMachine::HTTPMethods' (NameError)

I put the first example in the README (section "Fiber-aware Iterator: mixing sync / async code") into a file, installed em-synchrony 0.3.0.beta.1, and ran it:

$ ruby em-synchrony-example.rb http://google.com/ http://drnicwilliams.com/
/Users/drnic/.rvm/gems/ruby-1.9.2-p180/gems/em-synchrony-0.3.0.beta.1/lib/em-synchrony/em-http.rb:21:in `class_eval': undefined method `get' for module `EventMachine::HTTPMethods' (NameError)
    from /Users/drnic/.rvm/gems/ruby-1.9.2-p180/gems/em-synchrony-0.3.0.beta.1/lib/em-synchrony/em-http.rb:21:in `class_eval'
    from /Users/drnic/.rvm/gems/ruby-1.9.2-p180/gems/em-synchrony-0.3.0.beta.1/lib/em-synchrony/em-http.rb:21:in `block in '
    from /Users/drnic/.rvm/gems/ruby-1.9.2-p180/gems/em-synchrony-0.3.0.beta.1/lib/em-synchrony/em-http.rb:9:in `each'
    from /Users/drnic/.rvm/gems/ruby-1.9.2-p180/gems/em-synchrony-0.3.0.beta.1/lib/em-synchrony/em-http.rb:9:in `'
    from /Users/drnic/.rvm/gems/ruby-1.9.2-p180/gems/em-synchrony-0.3.0.beta.1/lib/em-synchrony/em-http.rb:8:in `'
    from /Users/drnic/.rvm/gems/ruby-1.9.2-p180/gems/em-synchrony-0.3.0.beta.1/lib/em-synchrony/em-http.rb:7:in `'
    from /Users/drnic/.rvm/rubies/ruby-1.9.2-p180/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:58:in `require'
    from /Users/drnic/.rvm/rubies/ruby-1.9.2-p180/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:58:in `rescue in require'
    from /Users/drnic/.rvm/rubies/ruby-1.9.2-p180/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:35:in `require'
    from em-synchrony-example.rb:2:in `'

My script is:

require "rubygems"
require "em-synchrony/em-http"

EM.synchrony do
  urls = ARGV
  concurrency = urls.size

  # iterator will execute async blocks until completion, .each, .inject also work!
  results = EM::Synchrony::Iterator.new(urls, concurrency).map do |url, iter|

    # fire async requests, on completion advance the iterator
    http = EventMachine::HttpRequest.new(url).aget
    http.callback { iter.return(http) }
    http.errback { iter.return(http) }
  end

  p results # all completed requests
  EventMachine.stop
end

How did I screw this up?`

sample code doesn't work

Update

It looks like that I have to add:

require "em-synchrony"

so the following is a bit of a noob question :-$

EOF_UPDATE

I've tried the script mentioned in the readme:

require "em-synchrony/em-mysqlplus"

EventMachine.synchrony do
    db = EventMachine::Synchrony::ConnectionPool.new(size: 2) do
        EventMachine::MySQL.new(host: "localhost")
    end

    multi = EventMachine::Synchrony::Multi.new
    multi.add :a, db.aquery("select sleep(1)")
    multi.add :b, db.aquery("select sleep(1)")
    res = multi.perform

    p "Look ma, no callbacks, and parallel MySQL requests!"
    p res

    EventMachine.stop
end

and the result of executing it is:

$ ./basics.rb 
./basics.rb:14:in `<main>': undefined method `synchrony' for EventMachine:Module (NoMethodError)

I've tried the most basic EM script too but it works fine:

require 'eventmachine'
require 'em-mysqlplus'

EventMachine.run do
  conn = EventMachine::MySQL.new(host: 'localhost')
  query = conn.query("select sleep(1)")
  query.callback {|res| p res.all_hashes }
  query.errback {|res| p res.all_hashes }
  puts "executing.."
end

the output is:

$ ./basics.rb 
executing..
[{"sleep(1)"=>"0"}]

My environment is the following:

$ uname -a
Darwin devuby 10.6.0 Darwin Kernel Version 10.6.0: Wed Nov 10 18:13:17 PST 2010; root:xnu-1504.9.26~3/RELEASE_I386 i386 i386
$ rvm --version

rvm 1.2.5 by Wayne E. Seguin ([email protected]) [http://rvm.beginrescueend.com/]

$ ruby --version
ruby 1.9.2p136 (2010-12-25 revision 30365) [x86_64-darwin10.5.0]
$ gem --version
1.5.0

$ gem list
*** LOCAL GEMS ***
em-mysqlplus (0.1.5)
em-synchrony (0.2.0)
eventmachine (0.12.10)
mysqlplus (0.1.2)

Maurizio

em-mongo authentication no longer working

Authenticating to a mongo database does not seem to work when running it from within synchrony, I think I have pinpointed the problem though!

#authenticate does a find for a nonce from the $cmd database, and tries to execute some stuff from within the callback:

auth_resp = self.collection(SYSTEM_COMMAND_COLLECTION).first({'getnonce' => 1})
auth_resp.callback do |res|
...
end

Because .first gets overriden by synchrony, it does not have a callback method and the object is a BSON::OrderedHash. Using .callback on that throws this error:

em-mongo-fc3267511ddc/lib/em-mongo/database.rb:343:in `authenticate': undefined method `callback' for {"nonce"=>"684c385308963754", "ok"=>1.0}:BSON::OrderedHash (NoMethodError)

The authenticate method is quite extensive so I'm not sure what is the best way to go here.

I'll ping the author of em-mongo to see if he has any thoughts on this! :-)

assumes active support or a file that is not always loaded

em-mongo references silence_warnings

When I require it I get a failure.

require 'em-synchrony'
require 'em-synchrony/mongo'
require 'em-synchrony/mongoid'

This is easy enough to workaround for the moment by adding these methods myself.

potential problem with FiberedMonitor.synchronize

HI,
While experimenting with with em_mysql2 adapater I stumbled upon something annoying, while I agree it is an edge case I wonder if someone has a fix, here is my test case: https://gist.github.com/1330609

The problem is that by replacing the synchronize call with simply calling the block if you create multiple fibers asking for the connection they will all get one (I did not looked into this but I suppose the fiber is suspended during the actual connection to mysql). the net result is that you can can have more connections than your maximum in the connection pool, while this is not a major problem it still annoys me.

In my test case I defined a pool of 1 connection with activerecord and by setting the DELAY constant to 0 (or removing the add_timer call) you can get as many connections as you want in the pool although the limit is still 1, just change the "5.times" part.

ActiveRecord::Base.reset_column_information throws nil reference error

Steps to reproduce:

1. Rails 3.1, mysql2 latest, em-synchrony 1.0.0.
2. In migration do Model.reset_column_information and run rake db:migrate

Exception occurs in active_record/connection_adapters/abstract/connection_poolin #clear_table_cache! in #152: right here

Exception is 'undefined method "delete" for nil:NilClass'.

redis-rb, EventMachine::Synchrony::ConnectionPool and pipelined command

When I run the following code :

link to gist: https://gist.github.com/1174519

I keep getting this error :
`block in pipelined': no block given (yield) (LocalJumpError)

Am I missing something?

AR patch / synchrony

it looks like the AR monkey patch is missing something. If I perform a Widget.create(:name => 'top_posts') in the current em-synchrony/spec/activerecord_spec.rb I get this error :

1. Fiberized ActiveRecord driver for mysql2 should establish AR connection
   Failure/Error: Widget.create(:name => 'top_posts')
   NoMethodError:
   undefined method `[]' for nil:NilClass

   /home/toma/.rvm/gems/ruby-1.9.2-p180/gems/activerecord-3.1.0.rc8/lib/active_record/base.rb:717:in`column_defaults'

   /home/toma/.rvm/gems/ruby-1.9.2-p180/gems/activerecord-3.1.0.rc8/lib/active_record/persistence.rb:327:in `attributes_from_column_definition'

   /home/toma/.rvm/gems/ruby-1.9.2-p180/gems/activerecord-3.1.0.rc8/lib/active_record/locking/optimistic.rb:69:in`attributes_from_column_definition'

   /home/toma/.rvm/gems/ruby-1.9.2-p180/gems/activerecord-3.1.0.rc8/lib/active_record/base.rb:1544:in `initialize'

   /home/toma/.rvm/gems/ruby-1.9.2-p180/gems/activerecord-3.1.0.rc8/lib/active_record/base.rb:508:in`new'

   /home/toma/.rvm/gems/ruby-1.9.2-p180/gems/activerecord-3.1.0.rc8/lib/active_record/base.rb:508:in `create'

   ./spec/activerecord_spec.rb:22:in`block (3 levels) in <top (required)>'

   ./lib/em-synchrony.rb:25:in `call'

   ./lib/em-synchrony.rb:25:in`block (2 levels) in synchrony

It seems the same issue as brianmario/mysql2#202

Regards.

A few concerns about the method replacement pattern

I've been working on integrating EM::Synchrony/Fiber-style interfaces into several EM-based libraries I'm developing/maintaining, and I wanted to bring up for discussion some concerns I have about promoting the "replace methods with synchrony versions" pattern.

The main concern I have is that this is clearly Action at a distance. I've seen this in my own code, and it introduces bugs that can be difficult to track down. The main problem with the method-replacement style is that it works if none of the methods being replaced depend on another method being replaced.

Let's say I have the following (contrived) class:

class Foo < Blah
  def bark
    EM::DefaultDeferrable.new.tap do |dfr|
      EM.next_tick do
        dfr.succeed("woof!")
      end
    end
  end

  def bark_loudly
    EM::DefaultDeferrable.new.tap do |dfr|
      bark.callback do |str|
        dfr.callback(str.upcase)
      end
    end
  end
end

This is a fairly common pattern, the problem comes when someone replaces the bark and bark_loudly methods naievly.

class Foo < Blah
  %w[bark bark_loudly].each do |name|
    class_eval(<<-EOS, __FILE__, __LINE__+1) 
      alias :"a#{name}" :#{name}
      def #{name}
        EM::Synchrony.sync(a#{name})
      end
    EOS
  end
end

This will work in the case of the bark method, but bark_loudly will break because it's calling bark expecting to get a Deferrable back, but instead it gets a string.

(As a side note, the alias line above will cause problems if the class is reloaded, as it will overwrite the original abark method with the now-sync bark method. A guard should be added to ensure that abark isn't already defined.)

I've had this happen to me when using the WebMock library, as it (somewhat frustratingly) rewrites HTTPMethods if Synchrony is defined. My code is depending on the asynchronous nature of EM::HttpRequest, and their assumption breaks my code. Not only that, it breaks my code in a way that's very difficult for me to track down, because no exception is thrown, it's just my code hangs forever. Now, I fully understand that this isn't part of em-synchrony, however I'm concerned because I think that em-synchrony is an important project, and I feel promoting this kind of pattern may cause more problems than it solves.

Unless all of the libraries being used understand how to communicate with the EM::Synchrony-rewritten methods, then there will be problems. What I've taken to doing in my own code is writing a synchrony-aware wrapper-class around the async class. Both the sync and async classes define #to_synchrony and #to_async methods which allow easy switching between styles. This preserves the original methods, and will not create "spooky action at a distance" in code that doesn't know about Synchrony, and yet will make it convenient for people who want to have easy access to the sync version.

The wrapper style isn't as slick as method replacement because it's somewhat more explicit, however, implicitly changing the definition of methods is changing global state in a way that can have unforseen consequences.

problem with recursive nested iterator

with this code

require 'rubygems'
require 'eventmachine'
require "em-synchrony"
require "em-synchrony/em-http"
require "em-synchrony/iterator"
require 'nokogiri'

EM.synchrony do
concurrency = 2
urls = ["http://www.x1.com", "http://www.x2.com"]

iterator will execute async blocks until completion, .each, .inject also work!

results = EM::Synchrony::Iterator.new(urls, concurrency).map do |url, iter|

# fire async requests, on completion advance the iterator
http = EventMachine::HttpRequest.new(url).aget
http.callback { iter.return(http) }

end

result2 = EM::Synchrony::Iterator.new(results, concurrency).map do |client, iter|
doc=Nokogiri::XML(client.response)

results_xml=nil
Fiber.new do

  results_xml = EM::Synchrony::Iterator.new(doc.search("*"), concurrency).map do |node, iter2|
    iter2.return(node.name)
  end


end.resume

iter.return(results_xml)

end

p result2

EventMachine.stop
end

result2 is an array of nil. if like if second iterator don't return to first iterator the result. where i have write wrong code? is right or synchroni can only use of one level of profondity in stack?

Reusing EM::HttpRequest.new inside EM::Synchrony

Hi,

The script below only executes the first request. The second one is not triggered. I'm not sure if this is really supposed to work.

EM.synchrony {
  bot = EventMachine::HttpRequest.new("www.example.com")
  res1 = EM::Synchrony.sync(bot.ahead :path => "/url/blop.asp")
  res2 = EM::Synchrony.sync(bot.apost :path => "/url/foo.asp")
  EM.stop
}

I tried using :keepalive => true in both requests but didnt work as well. For some reason keepalive option added ~10s between both requests. I can provide a tcpdump if needed.

Thanks

em-synchrony/activerecord should not require mysql2

I am using the EM-compatable PostgreSQL driver, not MySQL.

EventMachine::Synchrony::TCPServer#read returns partial reads

Calls to read with a length parameters should only return a string shorter than requests if the stream's EOF has been reached. See http://www.ruby-doc.org/core-1.9.2/IO.html#method-i-read.

The current read implementation returns as there is any data available, without waiting for enough data as requested. The current behavior is sufficient to emulate recv, but not read. It breaks applications that expect read to wait for enough data to arrive.

Dead Fiber Called problems with mysql2 & em-synchrony

Hi there,

I randomly get this exception in a mysql2/fiber_pool/em-syncrhony/rails3.1 setup:

/home/rails/sites/adserver/shared/bundle/ruby/1.9.1/gems/mysql2-0.3.7/lib/active_record/connection_adapters/em_mysql2_adapter.rb:53:in `resume': dead fiber called (FiberError)
    from /home/rails/sites/adserver/shared/bundle/ruby/1.9.1/gems/mysql2-0.3.7/lib/active_record/connection_adapters/em_mysql2_adapter.rb:53:in `block in query'
    from /home/rails/sites/adserver/shared/bundle/ruby/1.9.1/gems/eventmachine-1.0.0.beta.3/lib/em/deferrable.rb:155:in `call'
    from /home/rails/sites/adserver/shared/bundle/ruby/1.9.1/gems/eventmachine-1.0.0.beta.3/lib/em/deferrable.rb:155:in `set_deferred_status'
    from /home/rails/sites/adserver/shared/bundle/ruby/1.9.1/gems/eventmachine-1.0.0.beta.3/lib/em/deferrable.rb:194:in `fail'
    from /home/rails/sites/adserver/shared/bundle/ruby/1.9.1/gems/mysql2-0.3.7/lib/active_record/connection_adapters/em_mysql2_adapter.rb:38:in `rescue in notify_readable'
    from /home/rails/sites/adserver/shared/bundle/ruby/1.9.1/gems/mysql2-0.3.7/lib/active_record/connection_adapters/em_mysql2_adapter.rb:33:in `notify_readable'
    from /home/rails/sites/adserver/shared/bundle/ruby/1.9.1/gems/eventmachine-1.0.0.beta.3/lib/eventmachine.rb:199:in `run_machine'
    from /home/rails/sites/adserver/shared/bundle/ruby/1.9.1/gems/eventmachine-1.0.0.beta.3/lib/eventmachine.rb:199:in `run'
    from /home/rails/sites/adserver/shared/bundle/ruby/1.9.1/gems/thin-1.2.11/lib/thin/backends/base.rb:61:in `start'
    from /home/rails/sites/adserver/shared/bundle/ruby/1.9.1/gems/thin-1.2.11/lib/thin/server.rb:159:in `start'
    from /home/rails/sites/adserver/shared/bundle/ruby/1.9.1/gems/thin-1.2.11/lib/thin/controllers/controller.rb:86:in `start'
    from /home/rails/sites/adserver/shared/bundle/ruby/1.9.1/gems/thin-1.2.11/lib/thin/runner.rb:185:in `run_command'
    from /home/rails/sites/adserver/shared/bundle/ruby/1.9.1/gems/thin-1.2.11/lib/thin/runner.rb:151:in `run!'
    from /home/rails/sites/adserver/shared/bundle/ruby/1.9.1/gems/thin-1.2.11/bin/thin:6:in `<top (required)>'
    from /home/rails/sites/adserver/shared/bundle/ruby/1.9.1/bin/thin:19:in `load'
    from /home/rails/sites/adserver/shared/bundle/ruby/1.9.1/bin/thin:19:in `<main>'

is this a known problem or is there any solution for this?
I already had it without em-synchrony (just fiber-pool & mysql2).
I'm trying to find this beast as it happens in production only - so far :)

-Tobias

Fiber errors when using next_tick?

When I run the below code, it dies with a Fiber error, In my EM code I have lots of callbacks that run with timers and call themselves, how would I do this using em-synchrony?

/usr/local/lib/ruby/gems/1.9.1/gems/em-synchrony-1.0.0/lib/em-synchrony/em-redis.rb:42:in `yield': can't yield from root fiber (FiberError)

require "em-synchrony"
require "em-synchrony/em-redis"

EM.synchrony do
r = EM::Protocols::Redis.connect
i = 0

test = lambda {
puts i
r.rpush "afoo", "s1"
r.rpush "afoo", "s2"
i += 1
EM.next_tick &test
}

test.call
#EventMachine.stop

end

Request Timing out with Empty Response

I'm using em-synchrony in Rails3 to perform head requests on about a hundred or so remote urls using EM::HttpRequest. The request runs for about 20 seconds and then returns an empty response to the browser, but continues to run in the background emitting log-lines.

Everything works perfectly if I constrain the number of URLs to around 10 or so, and running thin with the -D flag still demonstrates the same behaviour. The synchronous version of this action runs for over a minute but completes correctly. I also tried bumping up the size of the rack fiber pool, but the same thing happens at around the same time.

Any suggestions would be greatly appreciated. The code in question is available here:
https://github.com/simon-engledew/itube

ruby version: ruby 1.9.2p136 via. RVM on a mac
server: thin 1.2.7
em-synchrony: 0.2.0

Potential improvement to EM::Synchrony.sync's handling of EM::Deferrable called with multiple arguments

Hi!

I'm working on providing an em-synchrony friendly version of ZK::EventMachine, but have run into trouble with the way that EM::Synchrony.sync handles callbacks that have their succeed method invoked with multiple arguments. I make extensive use of this pattern throughout my code, so for me this is a pretty important thing to have working :)

Below I have an example fix for sync. I created a fork and branch of em-synchrony and made the change, but I couldn't figure out how to set up all of the required dependencies to run the specs.

I've included below a small demonstration of the problem (for me) and the proposed fix. If it's easier for you, you can also check out a gist of this code and run it that way.

The change is pretty minor, if you can instruct me on how to configure mongo and mysql, etc. so I can run the test suite, i'd be happy to. If you don't mind making the change in your environment and seeing if there are any ill effects, that'd work too.

#!/usr/bin/env ruby

require 'rubygems'
require 'eventmachine'
require 'em-synchrony'

module EventMachine
  module Synchrony
    def self.new_sync(df)
      f = Fiber.current
      xback = proc {|*args|
        if f == Fiber.current
          return *args
        else
          f.resume(*args)
        end
      }
      df.callback &xback
      df.errback &xback

      Fiber.yield
    end
  end
end

def gimme_deferreds
  a, b = EM::DefaultDeferrable.new, EM::DefaultDeferrable.new

  EM.next_tick do
    a.succeed(1, 2, [:foo, :bar])
    b.succeed(:single_arg)
  end

  return a, b
end


def current_implementation
  EM.synchrony do
    a, b = gimme_deferreds

    rval = EM::Synchrony.sync(a)
    puts "a rval: #{rval.inspect}"

    rval = EM::Synchrony.sync(b)
    puts "b rval: #{rval.inspect}"

    EM.next_tick { EM.stop_event_loop }
  end
end

def proposed_fix
  EM.synchrony do
    a, b = gimme_deferreds

    rval = EM::Synchrony.new_sync(a)
    puts "a rval: #{rval.inspect}"

    rval = EM::Synchrony.new_sync(b)
    puts "b rval: #{rval.inspect}"

    EM.next_tick { EM.stop_event_loop }
  end
end


if __FILE__ == $0
  puts <<-EOS
call signature: 
a.succeed(1, 2, [:foo, :bar])
b.succeed(:single_arg)

  EOS

  puts "this is the return value of the current implementation:"
  current_implementation

  puts "\nthis is the proposed change:"
  proposed_fix
end

__END__

call signature:
a.succeed(1, 2, [:foo, :bar])
b.succeed(:single_arg)

this is the return value of the current implementation:
a rval: 1
b rval: :single_arg

this is the proposed change:
a rval: [1, 2, [:foo, :bar]]
b rval: :single_arg

Thanks!
Jonathan

Add Bitly support

http://github.com/mccoder/em-synchrony/commit/4f24fed2d779fd535dd538ec07a5c288449a2475 - Bitly patch
http://github.com/mccoder/em-synchrony/commit/84e8e319640dba657052f94b88a06358df93c999 - Example

ConnectionPool (tries to) add callback to Fixnum (em-redis)

ConnectionPool thinks it got a defferable from em-redis, while it actually got a Fixnum.

require "em-synchrony"
require "em-synchrony/em-redis"

EM.synchrony do
  redis = EM::Synchrony::ConnectionPool.new(size: 2) do
    EM::Protocols::Redis.connect
  end

  redis.ainfo { |result| puts results }
end

Run this and you get this error:

/home/david/.bundler/ruby/1.9.1/em-synchrony-4224adb118eb/lib/em-synchrony/connection_pool.rb:73:in block in method_missing': undefined methodcallback' for 14:Fixnum (NoMethodError)
from /home/david/.bundler/ruby/1.9.1/em-synchrony-4224adb118eb/lib/em-synchrony/connection_pool.rb:23:in execute' from /home/david/.bundler/ruby/1.9.1/em-synchrony-4224adb118eb/lib/em-synchrony/connection_pool.rb:68:inmethod_missing'
from main.rb:15:in block in <main>' from /home/david/.bundler/ruby/1.9.1/em-synchrony-4224adb118eb/lib/em-synchrony.rb:24:incall'
from /home/david/.bundler/ruby/1.9.1/em-synchrony-4224adb118eb/lib/em-synchrony.rb:24:in `block (2 levels) in synchrony'

Error when with latest merging

Hi Ilya,

I kept commiting on my fork after my initial pull request.

My last commit jnak@3d28544 was not supposed to be merged in the master tree. Plus, I've found a workaround to use em-sync with blather without patching em-sync itself.

I've already left a comment there #37, but you closed the discussion.

Sorry about that,
Julien

EM.sync with sinatra_async

I'm trying to get em-synchrony to run under Thin/Sinatra in conjunction with sinatra_async. Unfortunately, EM.synchrony doesn't seem to play well when it's executed within a running reactor.

Is there a simple way for me to implement EM.sync within a currently active EventMachine loop?

'auth' command getting borked in em-redis.rb

The em-redis wrapper does some work to make redis commands starting with 'a' fiber aware. But it makes sure 'add' is not messed with. However, 'add' is not the only command starting with 'a', 'auth' does as well and is getting munged into the invalid 'uth' command when trying to use a password.

/home/ec2-user/.rvm/gems/ruby-1.9.2-p180/gems/em-redis-0.3.0/lib/em-redis/redis_protocol.rb:335:in `block in initialize': Redis server returned error code: ERR unknown command 'uth' (EventMachine::Protocols::Redis::RedisError)

Better connection limit logic for ActiveRecord

Here's a thread for bit of context:
https://groups.google.com/d/msg/goliath-io/smLYeLYWHOA/WX9Fq6gj_O4J

Basically i am trying to use activerecord with em-synchrony/activerecord in a Goliath app. Problem is unlike rails, I can't just do establish_connection for each request and not expect some exceptions in case of too many connections at once or long running queries at once. Now that the the activerecord driver has moved into this repo from mysql2, need to add some sort of connection pool logic within context of fiber. Anyway, I don't understand nuts and bolts of it underneath, but here's some sample code to reproduce the problem:
https://gist.github.com/1185433 (app with endpoint and db config)
https://gist.github.com/1185438 (gems used and script to hit the endpoint in parallel to invoke this).

Running the script I get intermitted exceptions like:

Mysql2::Error: This connection is still waiting for a result, try again once you have the result: SELECT sleep(1.0)

I cant reproduce now but I also got 'Too many connections' exception earlier.
*This looks very similar to issue mentioned here, #51
Feel free to delete if dup.

EM::Synchrony::Iterator hanging

I am using EM::Synchrony::Iterator to expand shortened urls and I found that it would periodically hang. I can't figure out what exactly causes it. It does not happen if I use Typhoeus::Hydra. I made a little test script that shows the hang. It occurs at random points, but it seems to happen always when the last url in the batch was processed. i.e. the iterator block fails to return control.

See https://gist.github.com/770795 for test code and https://gist.github.com/770794 for a list of sample urls. There's also an url in the sample urls list that has a non-resolvable domain (http://bits4babies.com/blog/?p=1741) which will also hang the iterator.

ActiveRecord with Mysql2 outside Rack apps

Hi everyone,

I have an issue that I believe is not 100% specific to em-synchrony, but I didn't know where to post it.

I'm trying to reuse my ActiveRecords models with em-synchrony and em_mysql2 outside of rails. This means that I don't use @mperham rack-fiber_pool. Because of this, it seems that AR will use as many mysql connections as fibers calling the DB, unlike the em-synchrony connexion pool. I tried using the :pool parameter (which defaults to 5 anyway), but it does not change anything.

Am I doing something wrong ? Does anyone use async AR outside of a rack app ?
Some help here would be much appreciated :)

Cheers,
Julien

Stack level too deep with em-redis

I can't get em-synchrony to work with em-redis.

require 'em-synchrony'
require 'em-synchrony/em-redis'
EM.synchrony do
  redis = EM::Protocols::Redis.connect
  redis.set('a', 'foobar')
  EM.stop
end

results in :

/Users/ivan/.rvm/gems/ruby-1.9.2-p136/gems/em-synchrony-0.2.0/lib/em-synchrony/em-redis.rb:20: stack level too deep (SystemStackError)

What am I missing ?
Thanks in advance for your help.

silence_warnings usage in for regular mongo gem

I'm having some trouble running em-synchrony with the regular mongo gem.

The silence_warnings call in mongo.rb is causing a undefined method error for Object.
silence_warnings is defined in the all spec helper.

So it seems to me the code shouldn't be wrapped in that method, or I might be missing something obvious here :)

em-dbi ?

Thanks for providing em-synchrony.

How difficult would it be to produce an em-sequel-dbi gem in order to enjoy the benefits of non-blocking db i/o with a wider range of db engines? Do you know of anyone working on such a gem?

Thanks in advance.

An EM.schedule-esque for Fibers

One thing I've found handy while integrating/migrating my code to use EM::Synchrony is the following:

module FiberHelper
  mattr_accessor :root_fiber, :instance_writer => false, :instance_reader => false

  def self.root_fiber?
    root_fiber == Fiber.current
  end

  def ensure_fiber(&blk)
    if FiberHelper.root_fiber?
      Fiber.new { blk.call }.resume
    else
      blk.call
    end
  end
end

FiberHelper.root_fiber ||= Fiber.current

If we're already running in some fibrous context, one doesn't need to do the wrapping. Much in the same way that EM.schedule is handy when you're trying to ensure that some thread doesn't trample your async code: if you're already in the reactor, great! otherwise, make things safe.

This way, as I'm going through and converting methods one at a time, I can just wrap the body in ensure_fiber and I don't have to worry about the calling context.

Thought it might make a decent convenience feature (with a slightly better name, perhaps).

Next gem release?

The latest gem release of em-synchrony is from April. Any chance of releasing a new version? I'd love to start making use of the em-redis bits.

em-synchrony/em-mongo should work with both current em-mongo (callback'ed find) and upcoming em-mongo 0.4.0 (cursor'ed find)

the upcoming em-mongo version uses a different interface to #find and #first. A patch to follow later today that lets it work with both.

em_mysql2 undefined method `callback`

When I try to run my test suite using Rails 3.1.0 I with:

gem "em-synchrony", git: "git://github.com/igrigorik/em-synchrony.git", require: [
"em-synchrony",
"em-synchrony/em-http",
"em-synchrony/activerecord"
]

in my Gemfile, and using the em_mysql2 driver it errors before when it tries to run the first test and aborts. although if I revert back to mysql2 driver it works perfectly, this is the full trace on the tests

bundle exec rake test --trace
** Invoke test (first_time)
** Execute test
** Invoke test:units (first_time)
** Invoke test:prepare (first_time)
** Invoke db:test:prepare (first_time)
** Invoke db:abort_if_pending_migrations (first_time)
** Invoke environment (first_time)
** Execute environment
** Execute db:abort_if_pending_migrations
** Invoke test:functionals (first_time)
** Invoke test:prepare
** Execute test:functionals
Loaded suite /Users/ChristopherHein/.rvm/gems/ruby-1.9.2-p290@global/gems/rake-0.9.2/lib/rake/rake_test_loader
Started

PagesControllerTest:
ERROR should get index (0.03s)
NoMethodError: undefined method callback' for nil:NilClass /Users/ChristopherHein/.rvm/gems/ruby-1.9.2-p290@chrishein/gems/activesupport-3.1.0/lib/active_support/whiny_nil.rb:48:inmethod_missing'

Finished in 0.036990 seconds.

1 tests, 0 assertions, 0 failures, 2 errors, 0 skips
** Invoke test:integration (first_time)
** Invoke test:prepare
** Execute test:integration
Errors running test:units, test:functionals!

I might be missing something, but it seems to work fine when I'm using rails 3.1.0.rc4 like your Async-Rails but it seems to be something with the full release of rails...

Any input/suggestions would be greatly appreciated.
Thanks for all the amazing things you hack on btw!

Breaks existing code, no?

Wouldn't it be better to define blocking methods in em-http, e.g. sget, instead of overriding the existing get and keeping the old one as aget. Right now I can't use em-synchrony because other libraries using em-http breaks.