Feature: parallel specs about autotest

[grosser]

It would also be amazing if you fork it and add it :)

[grosser]

I think the hack has to go here, so rspec should be patched ?

[snelson]

I wanna make this work ... just to understand, it looks like me need to have rspec look for the parallel option and change the command it makes to use parallel_spec if its present?

[grosser]

jep

[snelson]

I got this working on top of Rspec master, it will need to be backported to 2.5 and such ...

thisbythem/rspec-core@de074af

Wanted to get your thoughts ...

Since this is a fork enhancement of the "real" autotest, I'd think the rspec peeps will be resistant to support a feature of a fork. Perhaps there's another way we can get this to work. Almost seems like rspec autotest support should be pulled out of rspec core, but that's another issue. Monkey patch in your fork? Separate gem with monkey patch? Thoughts?

[snelson]

I suppose it wouldn't hurt to ask first. Posted a message to the rspec group that's awaiting moderation.

[grosser]

If you can make it work via a monkey-patch in this autotest fork that would be the easiest way for users to get it (right after rspec accepting the patch)

[snelson]

Putting this in the project's .autotest almost works:

require 'autotest/rspec2'

class Autotest::Rspec2 < Autotest
  def make_test_cmd(files_to_test)
    if files_to_test.empty?
      ''
    elsif options[:parallel] and files_to_test.size > 1
      "#{prefix}parallel_spec #{normalize(files_to_test).keys.flatten.join(' ')}"
    else
      "#{prefix}#{ruby}#{suffix} -S #{SPEC_PROGRAM} --tty #{normalize(files_to_test).keys.flatten.map { |f| "'#{f}'"}.join(' ')}"
    end
  end
end

The problem is, the parallel_spec command never finishes. I've been trying to debug this all day and haven't been able to get it working, but here's what I've found ...

When parallel_spec is called out to from a ruby script, this is when it does not finish. A simple test file in the root of my rails/rspec2 project looked like this:

# test.rb

cmd = "parallel_spec spec/models/*_spec.rb"
open("| #{cmd}", "r") do |f|
  until f.eof? do
    c = f.getc or break
    putc (c.is_a?(Fixnum) ? c.chr : c)
  end
end

As with autotest, this runs the specs, but never terminates. Drilling down into parallel_tests, it seems to be hanging indefinitely on the "fetch_output" method. Maybe something to do with output buffering in multiple nested threads?

Each parallel thread never gets past this line: https://github.com/grosser/parallel_tests/blob/master/lib/parallel_tests.rb#L45

@grosser, any insight would be much appreciated.

Thanks!
Scott

[grosser]

hmm fetch_outpout should return nil / EOF when the child finishes, ill try to see if I find something, thanks for the great digging/pointers :)

[snelson]

In my debugging, I had dug into the fetch method, and inside of it, this is the line where it wouldn't get past:

https://github.com/grosser/parallel_tests/blob/master/lib/parallel_tests.rb#L102

If there's anything you can think of to try, I'd be happy to do the work.

[snelson]

Ok, forget everything I said about fetch_output ...

It seems that the parallel threads are getting hung up. That is, wait_for_threads in the parallel gem isn't finishing. Even if I remove the call to klass.run_tests in the bin/parallel_test, and put in an empty hash {}, it still hangs. If all I do change :in_processes to :in_threads, at L75 of bin/parallel_test, the problem goes away.

https://github.com/grosser/parallel_tests/blob/master/bin/parallel_test#L75

So, it doesn't work when Parallel uses in_processes, but does work when in_threads. This gets us a little closer to a solution, but why would the threads hang when in_processes. Also, I've noticed my cpus get slammed regardless of which method I use, so it seems like in_threads is still using all my cores (which is good).

[grosser]

your test.rb script works for me :)

[grosser]

does normal y = Parallel.map([1,2,3,4,5]){|x| x+1 } work inside your project <-> maybe something weird is going on.
Maybe give it a try on another project / an empty project.

[snelson]

Weird! We've tried it on two different projects and 4 different computers with varied cpus. The issue was inconsistent though, like every once in a long while it would work, but most of the time it would hang. I'll try what you suggested above.

[snelson]

Ok, I've setup the simplest possible example I could here:

https://github.com/snelson/parallel_issue_example

It is Rails 3.1 with a basic Test::Unit test that tries your above example in 4 different scenarios. The last one, running in a subprocess with in_processes and a puts before the Parallel.map call is the one that seems to fail. But, again its inconsistent. I made this on my desktop, which has 8 cores, and it was hanging everytime on the last example. The first time I ran it on my laptop (2 cores), it passed without hanging, but then on every subsequent run, it hangs.

I should also mention that these are both OSX lion machines (10.7.1) and have tested on both Ruby 1.9.2 p180 and p290.

[snelson]

I also just tried inside of an Ubuntu 11.04 VM, with both 4 and 8 cpus allotted, and got the same thing. It runs the first time, but then hangs on subsequent runs.

[snelson]

FYI, I just ran it on a High CPU instance at Engine Yard (8 cps, 1.9.2p180), and it worked every time ... no hanging. So maybe its an OSX and possibly Lion only issue? Humph. I'll see if I can find a 10.6 machine to try it on somewhere.

[grosser]

tests pass for me, just asking some mac guys to try to

git clone https://github.com/snelson/parallel_issue_example.git
cd parallel_issue_example
rvm 1.9.2
bundle
rake test

maybe they get the same problem...

[snelson]

I've tried it on Snow Leopard and get the same hanging issue as Lion with 1.9.2. But ... on REE 1.8.7-2011.03, it works on both Snow Leopard and Lion, with no hanging.

[snelson]

I've confirmed that this commit is where the problem started:

grosser/parallel@5babcac

If I use the commit before (555eb), then the tests finish and pass on all machines I've tested.

I've pushed up a passing branch here for anyone to try:

https://github.com/snelson/parallel_issue_example/tree/passing

[grosser]

great work :D

There is nothing I can spot in this changeset, just reverting it would be kind of ugly :<
Maybe using Thread.new do instead of the in_threads could do the trick...

Can you strace a hanging process and find out where exactly it hangs,
or e.g. put logging all over the place to see where things stop/hang.

[snelson]

Its hanging at the call to write_to_pipe here: https://github.com/grosser/parallel/blob/master/lib/parallel.rb#L143

I puts a line before and after it and it doesn't put the after line. Not sure why or what needs to be done.

[grosser]

seems to me like the pipe is already closed or the thread is dead, therefore no-one is listening -> it blocks
some ideas off the top of my head:

• try to see what its trying to write / what it wrote+received earlier
• try to remove the pipe closing (in the ensure block)
• try to put some logging at the end of the thread code

[snelson]

I've been trying to figure this out with a spec on the parallel gem, and the weirdest thing is that if I do this, in a spec case:

require File.expand_path('spec/spec_helper')

num_processes = Parallel.processor_count
puts "BEFORE"
#num_processes = Parallel.processor_count
Parallel.map([1,2,3,4,5,6,7,8], :in_processes => num_processes) { |x| x+1 } # HANGS
puts "AFTER"

it will hang, but if I do this:

require File.expand_path('spec/spec_helper')

#num_processes = Parallel.processor_count
puts "BEFORE"
num_processes = Parallel.processor_count
Parallel.map([1,2,3,4,5,6,7,8], :in_processes => num_processes) { |x| x+1 } # WORKS
puts "AFTER"

it won't. Additionally, if I do this:

require File.expand_path('spec/spec_helper')

puts "BEFORE"
Parallel.map([1,2,3,4,5,6,7,8], :in_processes => 8) { |x| x+1 } # HANGS
puts "AFTER"

It will hang, but if if I do this:

require File.expand_path('spec/spec_helper')

puts "BEFORE"
Parallel.processor_count
Parallel.map([1,2,3,4,5,6,7,8], :in_processes => 8) { |x| x+1 } # WORKS
puts "AFTER"

it won't hang! Its as if as long as there is a system call right before parallel.map runs and after any output, it will not hang. Just for fun I tried this:

require File.expand_path('spec/spec_helper')

puts "BEFORE"
`which ruby`
Parallel.map([1,2,3,4,5,6,7,8], :in_processes => 8) { |x| x+1 } # WORKS
puts "AFTER"

and, it does not hang!

I've tried some of the things you've suggested above but am having a hard time getting anywhere with it. I'll keep poking.

[grosser]

wow, thats an amazingly nasty bug :D
try to run the commands that are run inside processor_count, maybe they hang or block...

[joshpencheon]

I've been having problems too. I'm on OS X Lion, using RVM. Running the autotest-standalone suite:

$ rvm current
ruby-1.9.2-p290

$ rake
[...] 
ERROR test: integration green run should run in parallel.  (90.06s) 

That test hung, I interrupted after 90s, hence the error.

$ rvm use system
Now using system ruby.

$ ruby -v
ruby 1.8.7 (2010-01-10 patchlevel 249) [universal-darwin11.0] 

$ rake
[...]
  1) Failure:
test: integration green run should run in parallel. (TestAutotestIntegration)
[./test/test_autotest_integration.rb:90:in `__bind_1315858415_414298'
 /Library/Ruby/Gems/1.8/gems/shoulda-2.11.3/lib/shoulda/context.rb:382:in `call'
 /Library/Ruby/Gems/1.8/gems/shoulda-2.11.3/lib/shoulda/context.rb:382:in `test: integration green run should run in parallel. ']:
<""> expected to be =~
</YES/>.

So with system ruby, the test fails outright.

When I was digging into this over the weekend, I found it would tend to hang here - I will investigate further!

[joshpencheon]

@snelson, I've run your test app. I'm on a dual-core (no hyper-threading) machine running 10.7.1 and 1.9.2p290.

snelson-parallel_issue_example$ rake

Loaded suite rake-0.9.2/lib/rake/rake_test_loader
Started

ParallelTest:
     PASS a: in process (0.84s) 
     PASS b: as a subprocess (0.75s) 
running in 2 threads
     PASS c: as a subprocess with puts in threads (0.70s) 
running in 2 processes
^CParallel execution interrupted, exiting ...Parallel execution interrupted, exiting ...
Parallel execution interrupted, exiting ...

Parallel execution interrupted, exiting ...
parallel-0.5.8/lib/parallel.rb:249:in `kill': No such process (Errno::ESRCH)
    from parallel-0.5.8/lib/parallel.rb:249:in `block (2 levels) in kill_on_ctrl_c'
    from parallel-0.5.8/lib/parallel.rb:249:in `each'
    from parallel-0.5.8/lib/parallel.rb:249:in `block in kill_on_ctrl_c'
    from parallel-0.5.8/lib/parallel.rb:211:in `call'
    from parallel-0.5.8/lib/parallel.rb:211:in `join'
    from parallel-0.5.8/lib/parallel.rb:211:in `block in wait_for_threads'
    from parallel-0.5.8/lib/parallel.rb:209:in `each'
    from parallel-0.5.8/lib/parallel.rb:209:in `wait_for_threads'
    from parallel-0.5.8/lib/parallel.rb:20:in `in_threads'
    from parallel-0.5.8/lib/parallel.rb:132:in `work_in_processes'
    from parallel-0.5.8/lib/parallel.rb:57:in `map'
    from snelson-parallel_issue_example-1eccea0/test/fixtures/parallel_subprocess_with_puts_in_processes.rb:7:in `<main>'
    ERROR d: as a subprocess with puts in processes (74.53s) 
          Errno::ESRCH: No such process
          parallel-0.5.8/lib/parallel.rb:249:in `kill'


Finished in 76.831958 seconds.

4 tests, 3 assertions, 0 failures, 1 errors, 0 skips
Errors running test:units!

So I'm getting the same issue that you describe.
But, then if I use 1.8.7:

snelson-parallel_issue_example-1eccea0$ rvm use system
Now using system ruby.

snelson-parallel_issue_example-1eccea0$ ruby -v

ruby 1.8.7 (2010-01-10 patchlevel 249) [universal-darwin11.0]

snelson-parallel_issue_example-1eccea0$ rake
Loaded suite rake-0.9.2/lib/rake/rake_test_loader
Started
....
Finished in 1.992668 seconds.

4 tests, 4 assertions, 0 failures, 0 errors

No problems. It repeatedly passed all four on 1.8.7.

So my experiences so far:
1.9.2 - autotest -p hangs, as does Parallel :in_processes option.
1.8.7 - autotest -p test fails (not hangs), but Parallel seems to work fine.

[morganchristiansson]

I'm having the same issue with parallel when used from parallel_tests which I posted about here: grosser/parallel_tests#1

[swrobel]

Did anyone ever figure out how to get rspec working in parallel?

[grosser]

Nope, I was hoping it's fixed because nobody complained :)

If you can reproduce it, please try to fix it :D

[swrobel]

Not even sure how to test it. It seems to me that autotest -p doesn't run parallel specs though...