Coder Social home page Coder Social logo

rspec-retry's Introduction

RSpec::Retry Build Status

RSpec::Retry adds a :retry option for intermittently failing rspec examples. If an example has the :retry option, rspec will retry the example the specified number of times until the example succeeds.

Compatibility

Rspec Version Rspec-Retry Version
> 3.8 0.6.2 but untested
> 3.3, <= 3.8 0.6.2            
3.2       0.4.6            
2.14.8       0.4.4            

Maintenance Expectations

NoRedInk used to be a Ruby shop, and we open-sourced this in the hope that it will help other people. However, we've been moving away from Ruby for some time now, and only plan to do maintenance on this repo when we have an internal need for it. We don't plan to add any new functionality, and expect that it will fall behind the latest versions of Ruby and Rspec.

That said, if you're reading this and you need this gem to do something new, feel free to fork it and publish your own gem! If you open an issue here to let us know about your fork, we can add a link to it from this repo to help folks find something that's more actively maintained.

Installation

Add this line to your application's Gemfile:

gem 'rspec-retry', group: :test # Unlike rspec, this doesn't need to be included in development group

And then execute:

$ bundle

Or install it yourself as:

$ gem install rspec-retry

require in spec_helper.rb

# spec/spec_helper.rb
require 'rspec/retry'

RSpec.configure do |config|
  # show retry status in spec process
  config.verbose_retry = true
  # show exception that triggers a retry if verbose_retry is set to true
  config.display_try_failure_messages = true

  # run retry only on features
  config.around :each, :js do |ex|
    ex.run_with_retry retry: 3
  end

  # callback to be run between retries
  config.retry_callback = proc do |ex|
    # run some additional clean up task - can be filtered by example metadata
    if ex.metadata[:js]
      Capybara.reset!
    end
  end
end

Usage

it 'should randomly succeed', :retry => 3 do
  expect(rand(2)).to eq(1)
end

it 'should succeed after a while', :retry => 3, :retry_wait => 10 do
  expect(command('service myservice status')).to eq('started')
end
# run spec (following log is shown if verbose_retry options is true)
# RSpec::Retry: 2nd try ./spec/lib/random_spec.rb:49
# RSpec::Retry: 3rd try ./spec/lib/random_spec.rb:49

Calling run_with_retry programmatically

You can call ex.run_with_retry(opts) on an individual example.

Configuration

  • :verbose_retry(default: false) Print retry status
  • :display_try_failure_messages (default: false) If verbose retry is enabled, print what reason forced the retry
  • :default_retry_count(default: 1) If retry count is not set in an example, this value is used by default. Note that currently this is a 'try' count. If increased from the default of 1, all examples will be retried. We plan to fix this as a breaking change in version 1.0.
  • :default_sleep_interval(default: 0) Seconds to wait between retries
  • :clear_lets_on_failure(default: true) Clear memoized values for lets before retrying
  • :exceptions_to_hard_fail(default: []) List of exceptions that will trigger an immediate test failure without retry. Takes precedence over :exceptions_to_retry
  • :exceptions_to_retry(default: []) List of exceptions that will trigger a retry (when empty, all exceptions will)
  • :retry_callback(default: nil) Callback function to be called between retries

Environment Variables

  • RSPEC_RETRY_RETRY_COUNT can override the retry counts even if a retry count is set in an example or default_retry_count is set in a configuration.

Contributing

  1. Fork it
  2. Create your feature branch (git checkout -b my-new-feature)
  3. Commit your changes (git commit -am 'Added some feature')
  4. Push to the branch (git push origin my-new-feature)
  5. Create a pull request

rspec-retry's People

Contributors

anthonywoo avatar arkham avatar brianhicks avatar dthorsen avatar dwbutler avatar eitoball avatar georgedrummond avatar hkariti avatar ianks avatar james-dominy avatar joneshf avatar jonrowe avatar justin808 avatar maximeg avatar michaelglass avatar mmorast avatar moredip avatar olleolleolle avatar omnibs avatar p avatar reiz avatar ryanoglesby08 avatar sunflat avatar swrobel avatar tdeo avatar trevorcreech avatar varyform avatar y-yagi avatar y310 avatar yasyf avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

rspec-retry's Issues

RSpec HTML report formatting

Noticed the following when using RSpec's HTMLFormatter:

image

Is the HTMLFormatter supported? Going to start going through the code tomorrow to see how easy it might be to get more meaningful failure output but haven't messed with RSpec's formatters before. So if anyone has an idea of where to make the change and wants to share I'd definitely appreciate it.

Confusing default behavior, possibly due to `default_retry_count`

I just started using this gem, and I set up the following configuration:

RSpec.configure do |config|
  config.verbose_retry = true # show retry status in spec process
  config.default_retry_count = 4
  config.exceptions_to_retry = [Net::ReadTimeout] # only retry if this exception was thrown
end

I didn't tag any examples with :retry, because I had been looking at this answer on Stackoverflow, and got the impression that if I had this configuration, all examples that failed due to a Net::ReadTimeout error would be retried up to 4 times.

When I ran my test suite, my ouput included this:

Athlete logs a climb
  from the dashboard
[status message] Turning on truncation for the following example.
    for a nearby gym (PENDING: need to work out geolocation stubbing...)
[status message] Turning on truncation for the following example.
    for a recent gym (FAILED - 1)

User edits profile:
[status message] Turning on truncation for the following example.
  name can be changed (FAILED - 2)
[status message] Turning on truncation for the following example.
  user can select roles (FAILED - 3)

User signs out
[status message] Turning on truncation for the following example.
2nd Try error in ./spec/features/users/logout_spec.rb:4:
 Net::ReadTimeout 

RSpec::Retry: 2nd try ./spec/features/users/logout_spec.rb:4
  with valid session

# ... (omitted output) ...

Failures:

  1) Athlete logs a climb from the dashboard for a recent gym
     Got 0 failures and 2 other errors:

     1.1) Failure/Error: visit root_path

          Net::ReadTimeout:
            Net::ReadTimeout
          # ./spec/support/helpers/feature_helper.rb:14:in `capybara_login'
          # ./spec/features/athlete_logs_climb_spec.rb:62:in `block (3 levels) in <top (required)>'
          # ------------------
          # --- Caused by: ---
          # IO::EAGAINWaitReadable:
          #   Resource temporarily unavailable - read would block
          #   ./spec/support/helpers/feature_helper.rb:14:in `capybara_login'

     1.2) Failure/Error: raise Net::ReadTimeout

          Net::ReadTimeout:
            Net::ReadTimeout
          # /home/runner/.rbenv/versions/2.2.2/lib/ruby/2.2.0/net/protocol.rb:158:in `rescue in rbuf_fill'
          # /home/runner/.rbenv/versions/2.2.2/lib/ruby/2.2.0/net/protocol.rb:152:in `rbuf_fill'
          # /home/runner/.rbenv/versions/2.2.2/lib/ruby/2.2.0/net/protocol.rb:134:in `readuntil'

# ... (omitted output) ...

So I'm not sure whether those first examples that failed were retried 3 times, but the verbose message just wasn't printed, or whether they weren't retried at all. Based on how long those examples took to run compared to how long they normally take, I would suspect that they were each retried 3 times, but the message just wasn't output... is this intended behavior?

Also, the readme makes it sound like you have to either add the :retry tag to the example, or do something like this:

  # run retry only on features
  config.around :each, :js do |ex|
    ex.run_with_retry retry: 3
  end

in order for any retrying to occur, but it looks like if you specify a default_retry_count, this applies to all examples, not just the tagged ones.

Get startup error from spork...

Without requiring rspec in retry.rb, I get the below stack error when sport is started from guard.

What am I doing wrong? Can you add the require 'rspec' to retry.rb?
Take a look at the very bottom at the Bundler code that causes this issue.
Note, I figured out that I can modify application.rb to include 'rspec' before the call to bundler, but it seems that it would be preferable to simply require 'rspec' in retry.rb.

retry.rb

require 'rspec/retry/version'
require 'rspec_ext/rspec_ext'

ADD NEXT LINE
require 'rspec'
<<<<<< END ADD
module RSpec
class Retry
def self.apply
RSpec.configure do |config|
config.add_setting :verbose_retry, :default => false

23:26:28 - INFO - Guard uses GNTP to send notifications.
23:26:28 - INFO - Guard uses Emacs to send notifications.
23:26:28 - INFO - Guard uses Tmux to send notifications.
23:26:28 - INFO - Guard uses TerminalTitle to send notifications.
23:26:28 - INFO - Starting Spork for RSpec
Using RSpec, Rails
Preloading Rails environment
undefined method configure' for RSpec:Module (NoMethodError) /Users/justin/.rvm/gems/ruby-1.9.3-p327@rails320/gems/rspec-retry-0.2.0/lib/rspec/retry.rb:7:inapply'
/Users/justin/.rvm/gems/ruby-1.9.3-p327@rails320/gems/rspec-retry-0.2.0/lib/rspec/retry.rb:53:in <top (required)>' /Users/justin/.rvm/gems/ruby-1.9.3-p327@rails320/gems/bundler-1.3.4/lib/bundler/runtime.rb:81:inrequire'
/Users/justin/.rvm/gems/ruby-1.9.3-p327@rails320/gems/bundler-1.3.4/lib/bundler/runtime.rb:81:in rescue in block in require' /Users/justin/.rvm/gems/ruby-1.9.3-p327@rails320/gems/bundler-1.3.4/lib/bundler/runtime.rb:66:inblock in require'
/Users/justin/.rvm/gems/ruby-1.9.3-p327@rails320/gems/bundler-1.3.4/lib/bundler/runtime.rb:59:in each' /Users/justin/.rvm/gems/ruby-1.9.3-p327@rails320/gems/bundler-1.3.4/lib/bundler/runtime.rb:59:inrequire'
/Users/justin/.rvm/gems/ruby-1.9.3-p327@rails320/gems/bundler-1.3.4/lib/bundler.rb:132:in require' /Users/justin/j/blink/bpos/config/application.rb:9:in<top (required)>'
/Users/justin/.rvm/gems/ruby-1.9.3-p327@rails320/gems/spork-rails-3.2.1/lib/spork/app_framework/rails.rb:49:in require' /Users/justin/.rvm/gems/ruby-1.9.3-p327@rails320/gems/spork-rails-3.2.1/lib/spork/app_framework/rails.rb:49:inpreload_rails'
/Users/justin/.rvm/gems/ruby-1.9.3-p327@rails320/gems/spork-rails-3.2.1/lib/spork/app_framework/rails.rb:7:in preload' /Users/justin/.rvm/gems/ruby-1.9.3-p327@rails320/gems/spork-1.0.0rc3/lib/spork/test_framework.rb:134:inblock in preload'
/Users/justin/.rvm/gems/ruby-1.9.3-p327@rails320/gems/spork-1.0.0rc3/lib/spork.rb:62:in exec_prefork' /Users/justin/.rvm/gems/ruby-1.9.3-p327@rails320/gems/spork-1.0.0rc3/lib/spork/test_framework.rb:120:inpreload'
/Users/justin/.rvm/gems/ruby-1.9.3-p327@rails320/gems/spork-1.0.0rc3/lib/spork/run_strategy/forking.rb:25:in preload' /Users/justin/.rvm/gems/ruby-1.9.3-p327@rails320/gems/spork-1.0.0rc3/lib/spork/runner.rb:74:inrun'
/Users/justin/.rvm/gems/ruby-1.9.3-p327@rails320/gems/spork-1.0.0rc3/lib/spork/runner.rb:10:in run' /Users/justin/.rvm/gems/ruby-1.9.3-p327@rails320/gems/spork-1.0.0rc3/bin/spork:10:in<top (required)>'
/Users/justin/.rvm/gems/ruby-1.9.3-p327@rails320/bin/spork:19:in load' /Users/justin/.rvm/gems/ruby-1.9.3-p327@rails320/bin/spork:19:in

'
/Users/justin/.rvm/gems/ruby-1.9.3-p327@rails320/bin/ruby_noexec_wrapper:14:in eval' /Users/justin/.rvm/gems/ruby-1.9.3-p327@rails320/bin/ruby_noexec_wrapper:14:in'
23:26:58 - ERROR - Could not start Spork server for RSpec after 30 seconds. I will continue waiting for a further 60 seconds.

23:27:58 - ERROR - Could not start Spork server for RSpec. Make sure you can use it manually first.

23:27:58 - INFO - Guard::RSpec is running
23:27:58 - INFO - Running all specs
No DRb server is running. Running in local process instead ...

Here's what I have in application.rb
if defined?(Bundler)

If you precompile assets before deploying to production, use this line

Bundler.require(*Rails.groups(:assets => %w(development test)))

If you want your assets lazily compiled in production, use this line

Bundler.require(:default, :assets, Rails.env)

end

Retry does not work for some cases in latest rspec

The example given in the readme does not work in rspec 3.5 :

it 'should randomly succeed', :retry => 3 do
  expect(rand(2)).to eq(1)
end

When the expectation does not match, seems rspec does not treat it as an exception anymore. so example.exception will still be nil and this line will pass: https://github.com/NoRedInk/rspec-retry/blob/master/lib/rspec/retry.rb#L116

However, if it's a real exception like below, the retry works at expected:

it 'should randomly succeed', :retry => 3 do
  raise Network:Timeout
end

Attaching 'retry: 3' to a specify block does not override the exception list

I have a specify block where I specified a retry count:

specify 'my test name', retry: 3 do
...
end

However, in my config, I have specified

config.exceptions_to_retry = [Errno::EPIPE, EOFError]

Because I have specified an exception list, the explicit 'retry' on my test doesn't work. The reason I specified 'exceptions_to_retry' is that I globally wanted to retry any/all of my tests if they hit that just that exception; however, in the case that I had a particular test that was flaky, I would like to add a 'retry' param to that test. Am I missing something? Is this already baked in, or is it a feature request?

rspec-retry-0.5.1 not working with latest rspec-core

When trying to run one of my tests, the following error is displayed:

C:/Ruby23/lib/ruby/2.3.0/rubygems/specification.rb:2284:in `raise_if_conflicts': Unable to activate
rspec-retry-0.5.1, because rspec-core-3.5.3 conflicts with rspec-core (<= 3.5, > 3.3) (Gem::Confli
ctError)

Can you make your version check just be greater than 3.3... I'm guessing that's the problem here,

additional maintainers

Hey there! I think this gem is used by enough people to be helped by additional maintainers. Thoughts? I volunteer myself but think the role would be better for folks who have actually worked on the gem / created PRs. @y310 do you think you would be amenable to that?

CircleCI still detects RC 1 even though rspec-retry ultimately succeeded

I'm using CircleCI, and my spec/support/rspec_retry.rb is:

# Deal with intermittent Selenium timeouts
# From: https://github.com/jnicklas/capybara/issues/1305#issuecomment-112535264

RSpec.configure do |config|
  # show retry status in spec process
  config.verbose_retry = true

  # Try twice (retry once)
  config.default_retry_count = 3

  # Only retry when Selenium raises Net::ReadTimeout
  config.exceptions_to_retry = [Net::ReadTimeout]
end

Sometimes I still get the Net::ReadTimeout, rspec-retry does it's job and the test passes, but RSpec still gives return code 1, which make the build fail.

rspec-retry

Any thoughts on how to configure rspec-retry, RSpec, and/or CircleCI together so that when rspec-retry succeeds I still get RC 0?

Thanks for the work you do on this gem.

Successful Retries Never Output Exceptions

I recently moved from rspec-rerun to rspec-retry. I'm using Capybara & Selenium Rspec Tests. These can occasionally be flakey, which is why I am glad for this gem's existence.

However, I try to improve my tests to retry at last resort. Perhaps I'm not waiting long enough for a page to load, etc.

Unfortunately rspec-retry doesn't give me insight as to why the test failed in the first place. If it passes on attempt 3, I'll never know why 1 & 2 failed.

Would the community be opposed to including the exception to the verbose_retry output?

My fear is that this can make the test output rather unreadable at times in the log - especially if it's a big exception. Perhaps a better solution would be to include a snippet when the test finishes with stats about the exceptions rspec-retry caught.

I can fork, experiment and submit a PR, but curious on your thoughts before I do this.

Bad permissions on rspec/retry.rb

Hi there - if I install from the gem, all the files in the gem directory have permissions -rw-r--r--, with the exception of lib/rspec/retry.rb, which is -rw-------. This means that if the gem is installed into a root-owned directory, for example, regular users can't read that file.

I'm not sure where this problem comes from - if I download the source and run rake build, files in the resulting gem have the correct permissions.

Invoking database cleaner between retries

We have a flaky test suite of >2500 tests that invokes DatabaseCleaner between tests. Despite having a retry count of 3, we're seeing issues like unique constraint violated and record already exists...

Is there a way to explicitly invoke DatabaseCleaner between each retry attempt?

irb_context warning in rails console

irb: warn: can't alias context from irb_context.
Shows up in rails console if this gem is installed.

rspec gem versions installed (from Gemfile.lock)
rspec-core (3.5.4) rspec-expectations (3.5.0) rspec-mocks (3.5.0) rspec-rails (3.5.2) rspec-retry (0.5.3) rspec-support (3.5.0)

Similar issue in rspec-rails -- not sure if its related though
rspec/rspec-rails#1645

No Red INK

NO RED INK

No Red Ink has issues that need to be addressed. For example, if it's not exactly one of the ways they
have it, then that means your answer is wrong. However, I will have answers that are grammatically correct and they will still be marked wrong....... it is really irritating. I will look at other problems just like it and those are correct, but this one isn't even though it follows the same exact guidelines. To be honest, I believe it's their way of making sure you don't get 100% since nothing can ever be "perfect". It needs to be fixed. I'm tired of getting 90%'s or 95%'s on tests that I deserve an 100% for just because there's something wrong with their system.

rspec-retry-0.5.5 not working with latest rspec-core (3.7.0)

Failure/Error: raise Gem::LoadError, "Unable to activate #{self.full_name}, because #{y}"
Gem::LoadError:
Unable to activate rspec-retry-0.5.5, because rspec-core-3.7.0 conflicts with rspec-core (< 3.7, > 3.3)

/usr/share/rubygems/rubygems/specification.rb:2030:in raise_if_conflicts' /usr/share/rubygems/rubygems/specification.rb:1176:in activate'
/usr/share/rubygems/rubygems.rb:186:in rescue in try_activate' /usr/share/rubygems/rubygems.rb:183:in try_activate'

NameError: uninitialized constant Net Did you mean? Set

rspec-retry 0.5.4
ruby 2.3.0
rspec 3.6.0

This is my configuration:

RSpec.configure do |config|
  config.verbose_retry = true
  config.display_try_failure_messages = true
  config.clear_lets_on_failure = true
  config.default_retry_count = 2
  config.exceptions_to_retry = [Net::ReadTimeout]

I am getting the following error:

Failure/Error: config.exceptions_to_retry = [Net::ReadTimeout]

NameError:
  uninitialized constant Net
  Did you mean?  Set

Any idea why this error is thrown? If I require 'net/http' it does not throw that error buty it will not retry when the exception I am looking for is thrown.

Failing to catch Net::ReadTimeout

It seems this is unable to retry on certain exceptions like Net::ReadTimout. Here's my stack trace:

     1.1) Failure/Error: find('[href="/videos"]').click

          Net::ReadTimeout:
            Net::ReadTimeout



          # /Users/mhluska/.rvm/gems/ruby-2.5.1/gems/webmock-3.3.0/lib/webmock/http_lib_adapters/net_http.rb:97:in `block in request'
          # /Users/mhluska/.rvm/gems/ruby-2.5.1/gems/webmock-3.3.0/lib/webmock/http_lib_adapters/net_http.rb:110:in `block in request'
          # /Users/mhluska/.rvm/gems/ruby-2.5.1/gems/webmock-3.3.0/lib/webmock/http_lib_adapters/net_http.rb:109:in `request'
          # /Users/mhluska/.rvm/gems/ruby-2.5.1/gems/selenium-webdriver-3.14.0/lib/selenium/webdriver/remote/http/default.rb:121:in `response_for'
          # /Users/mhluska/.rvm/gems/ruby-2.5.1/gems/selenium-webdriver-3.14.0/lib/selenium/webdriver/remote/http/default.rb:76:in `request'
          # /Users/mhluska/.rvm/gems/ruby-2.5.1/gems/selenium-webdriver-3.14.0/lib/selenium/webdriver/remote/http/common.rb:62:in `call'
          # /Users/mhluska/.rvm/gems/ruby-2.5.1/gems/selenium-webdriver-3.14.0/lib/selenium/webdriver/remote/bridge.rb:164:in `execute'
          # /Users/mhluska/.rvm/gems/ruby-2.5.1/gems/selenium-webdriver-3.14.0/lib/selenium/webdriver/remote/oss/bridge.rb:584:in `execute'
          # /Users/mhluska/.rvm/gems/ruby-2.5.1/gems/selenium-webdriver-3.14.0/lib/selenium/webdriver/remote/oss/bridge.rb:562:in `find_elements_by'
          # /Users/mhluska/.rvm/gems/ruby-2.5.1/gems/selenium-webdriver-3.14.0/lib/selenium/webdriver/common/search_context.rb:78:in `find_elements'
          # /Users/mhluska/.rvm/gems/ruby-2.5.1/gems/capybara-3.8.2/lib/capybara/selenium/driver.rb:77:in `find_css'
          # /Users/mhluska/.rvm/gems/ruby-2.5.1/gems/capybara-3.8.2/lib/capybara/node/base.rb:97:in `find_css'
          # /Users/mhluska/.rvm/gems/ruby-2.5.1/gems/capybara-3.8.2/lib/capybara/queries/selector_query.rb:137:in `find_nodes_by_selector_format'
          # /Users/mhluska/.rvm/gems/ruby-2.5.1/gems/capybara-3.8.2/lib/capybara/queries/selector_query.rb:102:in `block in resolve_for'
          # /Users/mhluska/.rvm/gems/ruby-2.5.1/gems/capybara-3.8.2/lib/capybara/node/base.rb:77:in `synchronize'
          # /Users/mhluska/.rvm/gems/ruby-2.5.1/gems/capybara-3.8.2/lib/capybara/queries/selector_query.rb:101:in `resolve_for'
          # /Users/mhluska/.rvm/gems/ruby-2.5.1/gems/capybara-3.8.2/lib/capybara/node/finders.rb:288:in `block in synced_resolve'
          # /Users/mhluska/.rvm/gems/ruby-2.5.1/gems/capybara-3.8.2/lib/capybara/node/base.rb:82:in `synchronize'
          # /Users/mhluska/.rvm/gems/ruby-2.5.1/gems/capybara-3.8.2/lib/capybara/node/finders.rb:286:in `synced_resolve'
          # /Users/mhluska/.rvm/gems/ruby-2.5.1/gems/capybara-3.8.2/lib/capybara/node/finders.rb:33:in `find'
          # /Users/mhluska/.rvm/gems/ruby-2.5.1/gems/capybara-3.8.2/lib/capybara/session.rb:728:in `block (2 levels) in <class:Session>'
          # /Users/mhluska/.rvm/gems/ruby-2.5.1/gems/capybara-3.8.2/lib/capybara/dsl.rb:51:in `block (2 levels) in <module:DSL>'
          # ./spec/system/affiliate_spec.rb:34:in `block (2 levels) in <main>'
          # /Users/mhluska/.rvm/gems/ruby-2.5.1/gems/rspec-retry-0.6.1/lib/rspec/retry.rb:123:in `block in run'
          # /Users/mhluska/.rvm/gems/ruby-2.5.1/gems/rspec-retry-0.6.1/lib/rspec/retry.rb:110:in `loop'
          # /Users/mhluska/.rvm/gems/ruby-2.5.1/gems/rspec-retry-0.6.1/lib/rspec/retry.rb:110:in `run'
          # /Users/mhluska/.rvm/gems/ruby-2.5.1/gems/rspec-retry-0.6.1/lib/rspec_ext/rspec_ext.rb:12:in `run_with_retry'
          # ./spec/spec_helper.rb:123:in `block in setup_rspec_retry'
          # /Users/mhluska/.rvm/gems/ruby-2.5.1/gems/rspec-retry-0.6.1/lib/rspec/retry.rb:123:in `block in run'
          # /Users/mhluska/.rvm/gems/ruby-2.5.1/gems/rspec-retry-0.6.1/lib/rspec/retry.rb:110:in `loop'
          # /Users/mhluska/.rvm/gems/ruby-2.5.1/gems/rspec-retry-0.6.1/lib/rspec/retry.rb:110:in `run'
          # /Users/mhluska/.rvm/gems/ruby-2.5.1/gems/rspec-retry-0.6.1/lib/rspec_ext/rspec_ext.rb:12:in `run_with_retry'
          # /Users/mhluska/.rvm/gems/ruby-2.5.1/gems/rspec-retry-0.6.1/lib/rspec/retry.rb:37:in `block (2 levels) in setup'

It looks like this library calls ex.run which throws the Net::ReadTimeout error and RSpec fails to put it into example.exception? Would this benefit from a begin/rescue around the ex.run call for these catastrophic cases?

"Retry count" is really a try count

Retry count appears to be misnamed and is actually an attempt/try count.

retry count = 0: example is not run at all.

retry count = 1: example is run once (no retry).

retry count = 2: example is run once, and if fails, is retried once.

Features for a "randomly passing" mindset?

I like this project, but I don't want to simply retry and forget flaky tests.
I like the philosophy from No more flaky tests on the Go team - I think of them as "randomly passing", not "randomly failing".

Continuous Integration at Google Scale (PDF) also has a clear plan:

Solutions (Google does all of these):

  • Fix them!!!
    • Difficult - requires developer time
  • Hide them
    • Retry causing delays
    • Identify infrastructure flakes
    • Use metrics to ignore
  • Track them
    • Provide metrics to prioritize fix / hide

Currently rspec retry only does "retry - causing delays". It'd be good add some other features that would support this.

Some possible examples:

A summary report: which tests were retried? How many times?
Exporting/Importing metrics from previous runs: did a test become flaky that was stable in the past?
Automatic quarantine: run stable tests first and make sure they succeed. Run tests known to be flaky only if the stable tests pass.

Procsy is not a class (TypeError)

ruby -v
ruby 2.1.10p492 (2016-04-01 revision 54464) [x86_64-darwin15.0]
rails -v
Rails 4.0.0

Just running rspec throws

/Users/lacoste/.rbenv/versions/2.1.10/lib/ruby/gems/2.1.0/gems/rspec-retry-0.4.6/lib/rspec_ext/rspec_ext.rb:10:in `<class:Example>': Procsy is not a class (TypeError)
	from /Users/lacoste/.rbenv/versions/2.1.10/lib/ruby/gems/2.1.0/gems/rspec-retry-0.4.6/lib/rspec_ext/rspec_ext.rb:3:in `<module:Core>'
	from /Users/lacoste/.rbenv/versions/2.1.10/lib/ruby/gems/2.1.0/gems/rspec-retry-0.4.6/lib/rspec_ext/rspec_ext.rb:2:in `<module:RSpec>'
	from /Users/lacoste/.rbenv/versions/2.1.10/lib/ruby/gems/2.1.0/gems/rspec-retry-0.4.6/lib/rspec_ext/rspec_ext.rb:1:in `<top (required)>'
	from /Users/lacoste/.rbenv/versions/2.1.10/lib/ruby/gems/2.1.0/gems/activesupport-4.0.0/lib/active_support/dependencies.rb:228:in `require'
	from /Users/lacoste/.rbenv/versions/2.1.10/lib/ruby/gems/2.1.0/gems/activesupport-4.0.0/lib/active_support/dependencies.rb:228:in `block in require'
	from /Users/lacoste/.rbenv/versions/2.1.10/lib/ruby/gems/2.1.0/gems/activesupport-4.0.0/lib/active_support/dependencies.rb:213:in `load_dependency'
	from /Users/lacoste/.rbenv/versions/2.1.10/lib/ruby/gems/2.1.0/gems/activesupport-4.0.0/lib/active_support/dependencies.rb:228:in `require'
	from /Users/lacoste/.rbenv/versions/2.1.10/lib/ruby/gems/2.1.0/gems/rspec-retry-0.4.6/lib/rspec/retry.rb:3:in `<top (required)>'
	from /Users/lacoste/.rbenv/versions/2.1.10/lib/ruby/gems/2.1.0/gems/bundler-1.14.6/lib/bundler/runtime.rb:105:in `require'
	from /Users/lacoste/.rbenv/versions/2.1.10/lib/ruby/gems/2.1.0/gems/bundler-1.14.6/lib/bundler/runtime.rb:105:in `rescue in block in require'
	from /Users/lacoste/.rbenv/versions/2.1.10/lib/ruby/gems/2.1.0/gems/bundler-1.14.6/lib/bundler/runtime.rb:82:in `block in require'
	from /Users/lacoste/.rbenv/versions/2.1.10/lib/ruby/gems/2.1.0/gems/bundler-1.14.6/lib/bundler/runtime.rb:75:in `each'
	from /Users/lacoste/.rbenv/versions/2.1.10/lib/ruby/gems/2.1.0/gems/bundler-1.14.6/lib/bundler/runtime.rb:75:in `require'
	from /Users/lacoste/.rbenv/versions/2.1.10/lib/ruby/gems/2.1.0/gems/bundler-1.14.6/lib/bundler.rb:107:in `require'
	from /Users/lacoste/dev/clients/thoroughcare/config/application.rb:8:in `<top (required)>'
	from /Users/lacoste/dev/clients/thoroughcare/config/environment.rb:2:in `require'
	from /Users/lacoste/dev/clients/thoroughcare/config/environment.rb:2:in `<top (required)>'
	from /Users/lacoste/dev/clients/thoroughcare/spec/spec_helper.rb:3:in `require'
	from /Users/lacoste/dev/clients/thoroughcare/spec/spec_helper.rb:3:in `<top (required)>'
	from /Users/lacoste/dev/clients/thoroughcare/spec/controllers/admin/acp_consultation_controller_spec.rb:1:in `require'
	from /Users/lacoste/dev/clients/thoroughcare/spec/controllers/admin/acp_consultation_controller_spec.rb:1:in `<top (required)>'
	from /Users/lacoste/.rbenv/versions/2.1.10/lib/ruby/gems/2.1.0/gems/rspec-core-2.14.8/lib/rspec/core/configuration.rb:896:in `load'
	from /Users/lacoste/.rbenv/versions/2.1.10/lib/ruby/gems/2.1.0/gems/rspec-core-2.14.8/lib/rspec/core/configuration.rb:896:in `block in load_spec_files'
	from /Users/lacoste/.rbenv/versions/2.1.10/lib/ruby/gems/2.1.0/gems/rspec-core-2.14.8/lib/rspec/core/configuration.rb:896:in `each'
	from /Users/lacoste/.rbenv/versions/2.1.10/lib/ruby/gems/2.1.0/gems/rspec-core-2.14.8/lib/rspec/core/configuration.rb:896:in `load_spec_files'
	from /Users/lacoste/.rbenv/versions/2.1.10/lib/ruby/gems/2.1.0/gems/rspec-core-2.14.8/lib/rspec/core/command_line.rb:22:in `run'
	from /Users/lacoste/.rbenv/versions/2.1.10/lib/ruby/gems/2.1.0/gems/rspec-core-2.14.8/lib/rspec/core/runner.rb:80:in `run'

it vs its

sorry for the rspec noob question. How would rspec-retry be run when using its? The examples you give are only with it.

`exceptions_to_retry` does not retry child exceptions

Hi, I'm running into an issue with retrying exceptions. I've added the following configuration to Rspec:

config.exceptions_to_retry = [Selenium::WebDriver::Error::WebDriverError]

The intention was to catch a whole bunch of Selenium errors that all descend from WebDriverError. (A biggy is Selenium::WebDriver::Error::StaleElementReferenceError). My expectation was that child exceptions would be caught and retried, similar to how Ruby's rescue works.

Unfortunately, it appears that only specific error classes are retried, as indicated by this line:

https://github.com/NoRedInk/rspec-retry/blob/master/lib/rspec/retry.rb#L57

I propose replacing it with something like:

break unless exceptions_to_retry.any? do |exception_klass|
  example.exception.class < exception_klass
end

base on retriable gem

The retriable gem has a lot going for it:

  1. it's widely used (with about 2MM downloads)
  2. it implements exponential backoff with randomized variation
  3. it's got a clean interface

Would you be interested in adopting the retriable gem? If so maybe I cold take a whack at a PR.

Exponential Backoff

Tests with intermediate network failures have a tendency to pass if an exponential random backoff is used.
Would you mind if we added it?

Proposal: more flexible API by allowing configuration on indivudual errors

The current configuration API is nice, but it has one major shortcoming: specific errors can't specify different retry behavior. This is often desirable since certain errors have different requirements. Here is some pseudo-code to show what I mean:

example.run_with_retry retrriables:[
  { exception: Capybara::ElementNotFound, retry_wait: 0, retry: 3 },
  { exception: RateLimitError, retry_wait: 30, retry: 1 }
]

This type of configuration would make dealing with errors much more robust IMO. Anywho, love the gem and just figured I would give some feedback 😄

Cheers,
Ian

Issues with Rspec tests for Travis CI randomly failing from time to time

Referenced here as well: https://github.com/rspec/rspec/issues/29 / Might be an rspec issue - Not that well versed on ruby and rspec to diagnose.


See example Travis CI build: https://travis-ci.org/duracelltomi/gtm4wp/jobs/380080993 - Randomly it all goes through but sometimes fails for no reason, and would work again on some builds. Example repo where the test code is kept is: https://github.com/koconder/wordpress-test-template/tree/master/spec

In the travis ci build you can see

[Thu May 17 08:02:50 2018] 127.0.0.1:41748 Invalid request (Unexpected EOF)

Relating to:

RSpec::Retry: 2nd try ./baseline.rb:21
RSpec::Retry: 3rd try ./baseline.rb:21
F
RSpec::Retry: 2nd try ./baseline.rb:25
RSpec::Retry: 3rd try ./baseline.rb:25
F
RSpec::Retry: 2nd try ./baseline.rb:30
RSpec::Retry: 3rd try ./baseline.rb:30
F
Cleaning up...
Failures:
  1) WordPress: http://localhost:12000/ -  frontpage Healthy status code 200, 301, 302, 503
     Failure/Error: expect(page).to have_status_of [200,301,302,503]
       expected #<Capybara::Session> to have status of 200, 301, 302, and 503
     # ./baseline.rb:22:in `block (3 levels) in <top (required)>'
     # /home/travis/.rvm/gems/ruby-2.4.1/gems/rspec-retry-0.4.0/lib/rspec/retry.rb:43:in `block (3 levels) in apply'
     # /home/travis/.rvm/gems/ruby-2.4.1/gems/rspec-retry-0.4.0/lib/rspec/retry.rb:34:in `times'
     # /home/travis/.rvm/gems/ruby-2.4.1/gems/rspec-retry-0.4.0/lib/rspec/retry.rb:34:in `block (2 levels) in apply'
  2) WordPress: http://localhost:12000/ -  frontpage Page includes stylesheets
     Failure/Error: expect(page).to have_css
       expected #<Capybara::Session> to have css
     # ./baseline.rb:26:in `block (3 levels) in <top (required)>'
     # /home/travis/.rvm/gems/ruby-2.4.1/gems/rspec-retry-0.4.0/lib/rspec/retry.rb:43:in `block (3 levels) in apply'
     # /home/travis/.rvm/gems/ruby-2.4.1/gems/rspec-retry-0.4.0/lib/rspec/retry.rb:34:in `times'
     # /home/travis/.rvm/gems/ruby-2.4.1/gems/rspec-retry-0.4.0/lib/rspec/retry.rb:34:in `block (2 levels) in apply'
  3) WordPress: http://localhost:12000/ -  frontpage Page includes javascript
     Failure/Error: expect(page).to have_js
       expected #<Capybara::Session> to have js
     # ./baseline.rb:31:in `block (3 levels) in <top (required)>'
     # /home/travis/.rvm/gems/ruby-2.4.1/gems/rspec-retry-0.4.0/lib/rspec/retry.rb:43:in `block (3 levels) in apply'
     # /home/travis/.rvm/gems/ruby-2.4.1/gems/rspec-retry-0.4.0/lib/rspec/retry.rb:34:in `times'
     # /home/travis/.rvm/gems/ruby-2.4.1/gems/rspec-retry-0.4.0/lib/rspec/retry.rb:34:in `block (2 levels) in apply'
Finished in 6 minutes 41 seconds (files took 0.70182 seconds to load)
3 examples, 3 failures
Failed examples:
rspec ./baseline.rb:21 # WordPress: http://localhost:12000/ -  frontpage Healthy status code 200, 301, 302, 503
rspec ./baseline.rb:25 # WordPress: http://localhost:12000/ -  frontpage Page includes stylesheets
rspec ./baseline.rb:30 # WordPress: http://localhost:12000/ -  frontpage Page includes javascript
The command "cd wp-tests/spec && bundle exec rspec *.rb" exited with 1.

Also not sure if there is a ruby/rvm issue with the:

sh: 1: wp: not found
sh: 1: wp: not found
sh: 1: wp: not found

Feature Request: Integrate (or allow integration with) ActiveSupport::Notifications

I'm working on an app where there's a massive amount of random test failures due to environmental issues (large SOA app, shared final integration test environment, and lots and lots of teams deploying their services frequently)

We're trying to gauge the problem by instrumenting retries with ActiveSupport::Notifications, we have our own retrying logic localized to individual cases (e.g async message handlers, eventually refreshing a UI) and also retry handlers within our SitePrism page objects, these are already instrumented. But lacking is the general "retried the whole spec" scope.

Thoughts?

retry_callback not working

Here's my spec helper:

require "codeclimate-test-reporter"
require 'rspec/retry'

CodeClimate::TestReporter.start
FakeBraintree.activate!

RSpec.configure do |config|
  config.full_backtrace=false

  config.before do
    FakeBraintree.clear!
  end

  config.include ShowMeTheCookies, :type => :feature

  config.expect_with :rspec do |expectations|
    expectations.include_chain_clauses_in_custom_matcher_descriptions = true
  end

  config.expect_with :rspec do |c|
    c.syntax = :expect
  end

  # rspec-mocks config goes here. You can use an alternate test double
  # library (such as bogus or mocha) by changing the `mock_with` option here.
  config.mock_with :rspec do |mocks|
    mocks.verify_partial_doubles = true
    mocks.verify_doubled_constant_names = true
    # Prevents you from mocking or stubbing a method that does not exist on
    # a real object. This is generally recommended, and will default to
    # `true` in RSpec 4.
    config.around(:each, type: :helper) do |ex|
      config.mock_with :rspec do |mocks|
        mocks.verify_partial_doubles = false
        ex.run
        mocks.verify_partial_doubles = true
      end
    end
  end

  # show retry status in spec process
  config.verbose_retry = true
  # show exception that triggers a retry if verbose_retry is set to true
  config.display_try_failure_messages = true

  # run retry only on features
  config.around :each, :js do |ex|
    ex.run_with_retry retry: 3
  end

  # callback to be run between retries
  config.retry_callback = proc do |ex|
    # restart phantomjs between retries
    if ex.metadata[:js]
      restart_phantomjs
    end
  end

  # Run specs in random order to surface order dependencies. If you find an
  # order dependency and want to debug it, you can fix the order by providing
  # the seed, which is printed after each run.
  #     --seed 1234
  config.order = :random

  config.raise_errors_for_deprecations!
end

def restart_phantomjs
  puts "-> Restarting phantomjs: iterating through capybara sessions..."
  session_pool = Capybara.send('session_pool')
  session_pool.each do |mode,session|
    msg = "  => #{mode} -- "
    driver = session.driver
    if driver.is_a?(Capybara::Poltergeist::Driver)
      msg += "restarting"
      driver.restart
    else
      msg += "not poltergeist: #{driver.class}"
    end
    puts msg
  end
end

But it produces an error because retry_callback does not exist:

╰─$ rspec spec/features/referral_landing_page_spec.rb:43                                                                     1 ↵
bundler: failed to load command: rspec (/Users/csalvato/Dropbox/csalvato-home-dropbox/Development/Territory/vendor/bundle/ruby/2.3.0/bin/rspec)
NoMethodError: undefined method `retry_callback=' for #<RSpec::Core::Configuration:0x007fa296601180>
  /Users/csalvato/Dropbox/csalvato-home-dropbox/Development/Territory/spec/spec_helper.rb:52:in `block in <top (required)>'
  /Users/csalvato/Dropbox/csalvato-home-dropbox/Development/Territory/vendor/bundle/ruby/2.3.0/gems/rspec-core-3.4.4/lib/rspec/core.rb:97:in `configure'
  /Users/csalvato/Dropbox/csalvato-home-dropbox/Development/Territory/spec/spec_helper.rb:7:in `<top (required)>'
  /Users/csalvato/Dropbox/csalvato-home-dropbox/Development/Territory/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.6/lib/active_support/dependencies.rb:274:in `require'
  /Users/csalvato/Dropbox/csalvato-home-dropbox/Development/Territory/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.6/lib/active_support/dependencies.rb:274:in `block in require'
  /Users/csalvato/Dropbox/csalvato-home-dropbox/Development/Territory/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.6/lib/active_support/dependencies.rb:240:in `load_dependency'
  /Users/csalvato/Dropbox/csalvato-home-dropbox/Development/Territory/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.6/lib/active_support/dependencies.rb:274:in `require'
  /Users/csalvato/Dropbox/csalvato-home-dropbox/Development/Territory/spec/rails_helper.rb:13:in `<top (required)>'
  /Users/csalvato/Dropbox/csalvato-home-dropbox/Development/Territory/spec/features/referral_landing_page_spec.rb:1:in `require'
  /Users/csalvato/Dropbox/csalvato-home-dropbox/Development/Territory/spec/features/referral_landing_page_spec.rb:1:in `<top (required)>'
  /Users/csalvato/Dropbox/csalvato-home-dropbox/Development/Territory/vendor/bundle/ruby/2.3.0/gems/rspec-core-3.4.4/lib/rspec/core/configuration.rb:1361:in `load'
  /Users/csalvato/Dropbox/csalvato-home-dropbox/Development/Territory/vendor/bundle/ruby/2.3.0/gems/rspec-core-3.4.4/lib/rspec/core/configuration.rb:1361:in `block in load_spec_files'
  /Users/csalvato/Dropbox/csalvato-home-dropbox/Development/Territory/vendor/bundle/ruby/2.3.0/gems/rspec-core-3.4.4/lib/rspec/core/configuration.rb:1359:in `each'
  /Users/csalvato/Dropbox/csalvato-home-dropbox/Development/Territory/vendor/bundle/ruby/2.3.0/gems/rspec-core-3.4.4/lib/rspec/core/configuration.rb:1359:in `load_spec_files'
  /Users/csalvato/Dropbox/csalvato-home-dropbox/Development/Territory/vendor/bundle/ruby/2.3.0/gems/rspec-core-3.4.4/lib/rspec/core/runner.rb:106:in `setup'
  /Users/csalvato/Dropbox/csalvato-home-dropbox/Development/Territory/vendor/bundle/ruby/2.3.0/gems/rspec-core-3.4.4/lib/rspec/core/runner.rb:92:in `run'
  /Users/csalvato/Dropbox/csalvato-home-dropbox/Development/Territory/vendor/bundle/ruby/2.3.0/gems/rspec-core-3.4.4/lib/rspec/core/runner.rb:78:in `run'
  /Users/csalvato/Dropbox/csalvato-home-dropbox/Development/Territory/vendor/bundle/ruby/2.3.0/gems/rspec-core-3.4.4/lib/rspec/core/runner.rb:45:in `invoke'
  /Users/csalvato/Dropbox/csalvato-home-dropbox/Development/Territory/vendor/bundle/ruby/2.3.0/gems/rspec-core-3.4.4/exe/rspec:4:in `<top (required)>'
  /Users/csalvato/Dropbox/csalvato-home-dropbox/Development/Territory/vendor/bundle/ruby/2.3.0/bin/rspec:23:in `load'
  /Users/csalvato/Dropbox/csalvato-home-dropbox/Development/Territory/vendor/bundle/ruby/2.3.0/bin/rspec:23:in `<top (required)>'

Looked at the gem code and can't figure out why. Any advice?

Idea: refactor so that it can be used programatically

It'd be nice to do this:

# Call `retry` around each example in this context
around(:each) { |example| retry example, 3 }

This will be useful to attach custom retry behavior to some tags:

# Attempt to retry any failing Selenium tests due to some net errors
RSpec.configure do |config|
  config.around :each, :selenium do |example|
    retry example, 3, exceptions: [Net::ConnectError]
  end
end
# elsewhere:
describe "your feature", :selenium do
  it "should work" do
    # run this 3x
  end
end

At the moment, to do what the above is supposed to do, you'll need to do describe "your scenario", exceptions_to_retry: [Net::ConnectError], retry: 3, selenium: true do, which will be very cumbersome to type on every spec that needs it.

How to use with vagrant?

I use vagrant-serverspec to run tests on Vagrant boxes.

How can I install this gem to make it useable by the serverspec examples?

Thanks.

Retry until fail

i sometimes have to debug finicky tests that can pass or fail depending on timeouts or race conditions. while reasoning about these underlying issues is certainly what will ultimately lead to better, consistent specs, i often find myself wishing i could programmatically run them as many times as i wish, until either the specified number of runs has been reached or the test fails. rspec-retry runs until a failing spec passes. i think it makes sense for rspec-retry to also be able to run a spec until it fails.

for a proof of concept, please see my branch. all a user has to do is tag the spec with :retry_until_fail => true and some number of retries, e.g. :retry => 10. i haven't yet written any specs, but i would be happy to do so if they would be of interest.

could it use for turnip

If could. which syntax can I put it to my code

  Scenario: testing, :retry => 3 ???
    Then click "a" with text "MY mainframe" in frame 'mainFrame'

thanks :)

Consolidate messages when using both `verbose_retry` and `display_try_failure_messages`

When I use both of those configuration options, this is what I see:

User signs out
2nd Try error in ./spec/features/users/logout_spec.rb:4:
 Net::ReadTimeout 

RSpec::Retry: 2nd try ./spec/features/users/logout_spec.rb:4
  with valid session

Lines 2-3 make it sound like the 2nd try failed due to a Net::ReadTimeout, when really the 1st try failed for that reason. Also, lines 2 and 5 are essentially the same and it would be nice to not have to see both of them. I feel like the following is clearer:

User signs out

RSpec::Retry: 2nd try running due to Net::ReadTimeout (from line ./spec/features/users/logout_spec.rb:4)
  with valid session

And if they have only configured verbose_retry, and not display_try_failure_messages, then maybe they don't need to see the line of the error, and just this is enough:

User signs out

RSpec::Retry: 2nd try
  with valid session

Retry skips include_context calls

I have an include_context call before my it statement that I want to retry, this include_context contains a before_all. When the test fails, the it-statement is reran, but the before_all inside the include_context is not. Also if there is a failure in the include_context, then the test is not reran at all. Here is an example

describe "Cool Operation", :retry => 10, :verbose_retry => true,  do

  include_context('cool setup - before all')

  it 'does cool stuff' do
    expect(@var_setup_in_before_all).to eq(CoolObject)
  end
end

I have tried setting the :retry option on the 'it' statement and the 'describe' statement, but neither will retry the include_context. Does this gem support this feature?

Rspec Retry causing error: Errno::EPIPE: Broken pipe

We currently use rspec-retry with the following configuration in our spec_helper.rb file:

require "codeclimate-test-reporter"
require 'rspec/retry'

CodeClimate::TestReporter.start
FakeBraintree.activate!

RSpec.configure do |config|
  config.before do
    FakeBraintree.clear!
  end

  config.include ShowMeTheCookies, :type => :feature

  config.expect_with :rspec do |expectations|
    expectations.include_chain_clauses_in_custom_matcher_descriptions = true
  end

  config.expect_with :rspec do |c|
    c.syntax = :expect
  end

  # rspec-mocks config goes here. You can use an alternate test double
  # library (such as bogus or mocha) by changing the `mock_with` option here.
  config.mock_with :rspec do |mocks|
    mocks.verify_partial_doubles = true
    mocks.verify_doubled_constant_names = true
    # Prevents you from mocking or stubbing a method that does not exist on
    # a real object. This is generally recommended, and will default to
    # `true` in RSpec 4.
    config.around(:each, type: :helper) do |ex|
      config.mock_with :rspec do |mocks|
        mocks.verify_partial_doubles = false
        ex.run
        mocks.verify_partial_doubles = true
      end
    end
  end

  # show retry status in spec process
  config.verbose_retry = true
  # show exception that triggers a retry if verbose_retry is set to true
  config.display_try_failure_messages = true

  # run retry only on features
  config.around :each, :js do |ex|
    ex.run_with_retry retry: 3
  end

  # Run specs in random order to surface order dependencies. If you find an
  # order dependency and want to debug it, you can fix the order by providing
  # the seed, which is printed after each run.
  #     --seed 1234
  config.order = :random

  config.raise_errors_for_deprecations!
end

On our CI we sporadically see these errors on various different specs:

Got 0 failures and 3 other errors:

     1.1) Failure/Error: visit root_path(r: @campaign_code, utm_campaign: "foo", utm_source: "bar", utm_medium: "baz")
          
          Errno::EPIPE:
            Broken pipe
          # ./spec/features/funnel/checkout_spec.rb:22:in `block (2 levels) in <top (required)>'
          # ./spec/spec_helper.rb:46:in `block (2 levels) in <top (required)>'

     1.2) Failure/Error: ex.run_with_retry retry: 3
          
          Errno::EPIPE:
            Broken pipe
          # ./spec/spec_helper.rb:46:in `block (2 levels) in <top (required)>'

     1.3) Failure/Error: ex.run_with_retry retry: 3
          
          Errno::EPIPE:
            Broken pipe
          # ./spec/spec_helper.rb:46:in `block (2 levels) in <top (required)>'

While the specs are always different, the pattern is the same:

  1. Visit some path - get Broken Pipe
  2. ex.run_with_retry retry: 3 - get Broken Pipe
  3. ex.run_with_retry retry: 3 - get Broken Pipe

Since we are doing 3 tries, it looks like there's a broken pipe on the first try on line 46 of spec_helper.rb:

  # run retry only on features
  config.around :each, :js do |ex|
    ex.run_with_retry retry: 3
  end

I'm having a hell of a time debugging this issue, but it seems like something is sporadically breaking our retry process. I haven't found anyone else with this issue through Google, and am not quite sure how to debug it from here.

Any thoughts?

Feature proposal: Notification

I read this blog post "Flaky Tests: Are You Sure You Want to Rerun Them?" by @semaphoreci and it was inspiring.

The cost of fixing of flickering tests is sometimes very high. Hence, I am using rspec-retry for chrome headless feature tests as sometimes, this is the only way to get a green build. Pragmatic way :)

With this issue, I'd like to discuss a new feature that I can implement:

Feature: Notification
  When a test is silently failing and passes after being re-run by `rspec-retry` - it's flacky / flickering / randomly failing - I want to be notified, So I can address it and make my build faster.

  Background:
  * Number of retries: 2

  Scenario: "Real failure" => Do not notify
    When a real failure occurs 🔴🔴, (2 failed attempts)
    Then it should not notify the developer as there could be hundreds

  Scenario: 1 "Flickering"" => Notify
    When a test is flickering 🔴💚, (1 failed attempt followed by a "passed")
    Then it should notify the developer with the BUILD url and details about the test which failed

I have already implemented a Slack notification so it's so much work.

NameError: uninitialized constant Net Did you mean? Set

rspec-retry 0.5.4
ruby 2.3.0
rspec 3.6.0

I am getting the following error:

Failure/Error: config.exceptions_to_retry = [Net::ReadTimeout]

NameError:
  uninitialized constant Net
  Did you mean?  Set

Any idea why this error is thrown?

Restrictive rspec-core < 3.7 dependency

Now that rspec-core 3.7 is out, can we test/change the dependencies in this lib? Is there any reason going forward we shouldn't use pessimistic versioning like ~> 3.3?

merge with rspec-repeat?

@dwbutler @rstacruz

thoughts on merging with rspec-repeat? I think rspec-retry has better SEO / name recognition but rspec-repeat has a cleaner codebase. From that perspective: Would be happy to either

  • bring on @rstacruz as a maintainer and release rspec-repeat as the next big-version-number version of rspec-retry.
    or
  • deprecate rspec-retry and point people towards rspec-repeat in the readme.

My requirements would be

  • ensure feature parity (which might be done)
  • ensure adequate test coverage
  • ensure open issues are migrated to the appropriate repository

thoughts, you two?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.