grosser / maxitest

Minitest + all the features you always wanted.

Minitest + all the features you always wanted.

Features

• Ctrl+c stops tests and prints failures
• pastable rerun snippet for failures (disabled/integrated on rails 5)
• multiple before & after blocks
• before :all blocks
• around blocks around { |t| Dir.chdir(...) { t.call } }
• red-green output (disabled/integrated on rails 5)
• mtest executable to run by line number and by folder (disabled/integrated on rails 5)
• full backtrace for errors and assertions with verbose (-v)
• let!
• let_all execute once for all tests in a class and it's subclasses
• order_dependent! to make your tests run in given order
• Maxitest.static_class_order = true no longer sort tests class/sub-classes in random order
• context for more expression
• pending { assert false } is skip when it fails, but fails when it passes
• implicit subject via require 'maxitest/implicit_subject'
• xit to skip test (also does not call setup or teardown)
• with_env to change environment variables during test run
• capture_stdout and capture_stderr to capture stdout or stderr but not both (like capture_io does)
• require 'maxitest/timeout' to make hanging tests fail after Maxitest.timeout seconds
• require 'maxitest/threads' fail tests that leave extra threads running
• require 'maxitest/global_must' (before autorun) disable deprecation on global must_* or global_expectations gem

Install

gem install maxitest

Usage

require "maxitest/autorun"

# ... normal minitest tests ...
describe MyClass do
  describe "#my_method" do
    it "passes" do
      _(MyClass.new.my_method).must_equal 1
    end
  end
end

pending

• pending "need to fix" do to show why something is pending
• pending "need to fix", if: ENV["CI"] do to only skip on CI (if something is supposed to work locally)

with_env

Use during test: with_env FOO: "bar do ... Use as around block: with_env FOO: "bar"

context

Use as alias for describe

describe "#my_method" do
  context "with bad state" do
    before { errors += 1 }
    it "fails" # ...
  end
end

capture_stdout / capture_stderr

output = capture_stdout { puts 1 }
_(output).must_equal "1\n"

Development

• everything vendored into 1 gem to avoid dependency madness
• tested via rspec to avoid messing up our own tests by accident
• fixes should go back to the original libraries
• restrictive minitest dependency so nothing breaks by accident
• ruby >=3.0
• rake bundle to update all vendored gems

Author

• running by line number from minitest-line
• around from minitest-around
• mtest from testrbl
• red-green from minitest-rg

Michael Grosser
[email protected]
License: MIT

maxitest's Issues

`mtest --help` doesn't

I was trying to figure out how to invoke mtest.

% bundle exec mtest --help   
ruby -rbundler/setup --help -e 
Usage: ruby [switches] [--] [programfile] [arguments]

Check for leaked processes and file descriptors

The existing check for leaked threads is very useful.
It is also a common mistake to forget to close opened file descriptors or join forked processes. Wdyt about adding such checks?
Processes are not as much a popular problem, as file descriptors.

I made a little investigation on how to get the list of those prior to opening this issues. Unfortunately, it is not as easy as to get the list of threads 😅 And we need to care to not introduce a performance hit, since these will be run before and after each test case. I propose some of the "solutions", but maybe you know how to do it better.

To get the list of opened file descriptors, we can:

1. Run something like ObjectSpace.each_object(IO) { ... }
2. Monkey patch File#open/File#close to track fds.
3. Run some unix command like lsof
4. Iterate over some max number (100, for example) and check if it is a valid fd

To get the list of child processes:

1. Monkey patching fork/join/etc
2. Run some unix command like ps/pgrep, query /proc file system

Gem dependency

Hey, I just saw the line everything vendored into 1 gem to avoid dependency madness in README.

In case of minitest-around. On which version of minitest should it depend in order to be a "valid" dependency in maxitest?
Maybe I should use the same restrictive dependency on minitest as you did with maxitest?

before :all / after :all / around :all support

This is mentioned in the TODO in the README. You may want to look into integrating minitest-hooks (https://github.com/jeremyevans/minitest-hooks) instead of minitest-around, as minitest-hooks has support for before :all / after :all / around :all hooks, as well as around hooks, and doesn't implement around via Fiber hacks.

Drop support for legacy minitest and rubies

I propose that we refactor the codebase by dropping support for EOL rubies and minitest. This move will simplify the maintenance efforts and make test more robust.

Users on other hand can upgrade their gems or rubies to benefit from all the new things we add.

We will avoid doing conditions like : https://github.com/grosser/maxitest/blob/master/lib/maxitest/threads.rb#L3

wdyt ?

Test failure with minitest 5.16

rspec spec/
......................F..............

Failures:

  1) Maxitest extra threads fails on extra and passes on regular
     Failure/Error:
       result.gsub(/\d\.\d+/, "0.0").should include <<-OUT.gsub(/^\s+/, "")
         threads#test_0001_is fine without extra threads = 0.0 s = .
         threads#test_0002_fails on extra threads = 0.0 s = F
         threads#test_0003_can kill extra threads = 0.0 s = .
         threads#test_0004_can wait for extra threads = 0.0 s = .
       OUT

       expected "Run options: -v --seed 31005\n\n# Running:\n\nthreads#test_0001_is fine without extra threads = 0.0 ...5\nmtest spec/cases/threads.rb:19\nmtest spec/cases/threads.rb:25\nmtest spec/cases/threads.rb:31\n" to include "threads#test_0001_is fine without extra threads = 0.0 s = .\nthreads#test_0002_fails on extra thread...0003_can kill extra threads = 0.0 s = .\nthreads#test_0004_can wait for extra threads = 0.0 s = .\n"
       Diff:
       @@ -1,158 +1,315 @@
       -threads#test_0001_is fine without extra threads = 0.0 s = .\nthreads#test_0002_fails on extra threads = 0.0 s = F\nthreads#test_0003_can kill extra threads = 0.0 s = .\nthreads#test_0004_can wait for extra threads = 0.0 s = .\n
       +Run options: -v --seed 31005
       +
       +# Running:
       +
       +threads#test_0001_is fine without extra threads = 0.0 s = F
       +threads#test_0002_fails on extra threads = 0.0 s = F
       +threads#test_0003_can kill extra threads = 0.0 s = F
       +threads#test_0004_can wait for extra threads = 0.0 s = F
       +
       +Finished in 0.0s, 500.0 runs/s, 500.0 assertions/s.
       +
       +  1) Failure:
       +threads#test_0001_is fine without extra threads [spec/cases/threads.rb:16]:
       +Minitest::Assertion: Expected: 3
       +  Actual: 33
       +    /usr/lib/ruby/gems/0.0.0/gems/minitest-0.0.3/lib/minitest/assertions.rb:183:in `assert'
       +    /usr/lib/ruby/gems/0.0.0/gems/minitest-0.0.3/lib/minitest/assertions.rb:218:in `assert_equal'
       +    /usr/lib/ruby/gems/0.0.0/gems/minitest-0.0.3/lib/minitest/spec.rb:28:in `must_equal'
       +    /build/ruby-maxitest/src/maxitest-0.0.0/lib/maxitest/global_must.rb:14:in `must_equal'
       +    spec/cases/threads.rb:8:in `assert_correct_threads'
       +    spec/cases/threads.rb:16:in `block (2 levels) in <main>'
       +    /usr/lib/ruby/gems/0.0.0/gems/minitest-0.0.3/lib/minitest/test.rb:98:in `block (3 levels) in run'
       +    /usr/lib/ruby/gems/0.0.0/gems/minitest-0.0.3/lib/minitest/test.rb:195:in `capture_exceptions'
       +    /build/ruby-maxitest/src/maxitest-0.0.0/lib/maxitest/trap.rb:4:in `capture_exceptions'
       +    /usr/lib/ruby/gems/0.0.0/gems/minitest-0.0.3/lib/minitest/test.rb:95:in `block (2 levels) in run'
       +    /usr/lib/ruby/gems/0.0.0/gems/minitest-0.0.3/lib/minitest.rb:296:in `time_it'
       +    /usr/lib/ruby/gems/0.0.0/gems/minitest-0.0.3/lib/minitest/test.rb:94:in `block in run'
       +    /usr/lib/ruby/gems/0.0.0/gems/minitest-0.0.3/lib/minitest.rb:391:in `on_signal'
       +    /usr/lib/ruby/gems/0.0.0/gems/minitest-0.0.3/lib/minitest/test.rb:243:in `with_info_handler'
       +    /usr/lib/ruby/gems/0.0.0/gems/minitest-0.0.3/lib/minitest/test.rb:93:in `run'
       +    /build/ruby-maxitest/src/maxitest-0.0.0/lib/maxitest/vendor/around.rb:47:in `run'
       +    /build/ruby-maxitest/src/maxitest-0.0.0/lib/maxitest/trap.rb:23:in `run'
       +    /usr/lib/ruby/gems/0.0.0/gems/minitest-0.0.3/lib/minitest.rb:1059:in `run_one_method'
       +    /usr/lib/ruby/gems/0.0.0/gems/minitest-0.0.3/lib/minitest.rb:365:in `run_one_method'
       +    /usr/lib/ruby/gems/0.0.0/gems/minitest-0.0.3/lib/minitest.rb:352:in `block (2 levels) in run'
       +    /usr/lib/ruby/gems/0.0.0/gems/minitest-0.0.3/lib/minitest.rb:351:in `each'
       +    /usr/lib/ruby/gems/0.0.0/gems/minitest-0.0.3/lib/minitest.rb:351:in `block in run'
       +    /usr/lib/ruby/gems/0.0.0/gems/minitest-0.0.3/lib/minitest.rb:391:in `on_signal'
       +    /usr/lib/ruby/gems/0.0.0/gems/minitest-0.0.3/lib/minitest.rb:378:in `with_info_handler'
       +    /usr/lib/ruby/gems/0.0.0/gems/minitest-0.0.3/lib/minitest.rb:350:in `run'
       +    /usr/lib/ruby/gems/0.0.0/gems/minitest-0.0.3/lib/minitest.rb:182:in `block in __run'
       +    /usr/lib/ruby/gems/0.0.0/gems/minitest-0.0.3/lib/minitest.rb:182:in `map'
       +    /usr/lib/ruby/gems/0.0.0/gems/minitest-0.0.3/lib/minitest.rb:182:in `__run'
       +    /usr/lib/ruby/gems/0.0.0/gems/minitest-0.0.3/lib/minitest.rb:159:in `run'
       +    /usr/lib/ruby/gems/0.0.0/gems/minitest-0.0.3/lib/minitest.rb:83:in `block in autorun'
       +
       +  2) Failure:
       +threads#test_0002_fails on extra threads [spec/cases/threads.rb:20]:
       +Minitest::Assertion: Expected: 3
       +  Actual: 33
       +    /usr/lib/ruby/gems/0.0.0/gems/minitest-0.0.3/lib/minitest/assertions.rb:183:in `assert'
       +    /usr/lib/ruby/gems/0.0.0/gems/minitest-0.0.3/lib/minitest/assertions.rb:218:in `assert_equal'
       +    /usr/lib/ruby/gems/0.0.0/gems/minitest-0.0.3/lib/minitest/spec.rb:28:in `must_equal'
       +    /build/ruby-maxitest/src/maxitest-0.0.0/lib/maxitest/global_must.rb:14:in `must_equal'
       +    spec/cases/threads.rb:8:in `assert_correct_threads'
       +    spec/cases/threads.rb:20:in `block (2 levels) in <main>'
       +    /usr/lib/ruby/gems/0.0.0/gems/minitest-0.0.3/lib/minitest/test.rb:98:in `block (3 levels) in run'
       +    /usr/lib/ruby/gems/0.0.0/gems/minitest-0.0.3/lib/minitest/test.rb:195:in `capture_exceptions'
       +    /build/ruby-maxitest/src/maxitest-0.0.0/lib/maxitest/trap.rb:4:in `capture_exceptions'
       +    /usr/lib/ruby/gems/0.0.0/gems/minitest-0.0.3/lib/minitest/test.rb:95:in `block (2 levels) in run'
       +    /usr/lib/ruby/gems/0.0.0/gems/minitest-0.0.3/lib/minitest.rb:296:in `time_it'
       +    /usr/lib/ruby/gems/0.0.0/gems/minitest-0.0.3/lib/minitest/test.rb:94:in `block in run'
       +    /usr/lib/ruby/gems/0.0.0/gems/minitest-0.0.3/lib/minitest.rb:391:in `on_signal'
       +    /usr/lib/ruby/gems/0.0.0/gems/minitest-0.0.3/lib/minitest/test.rb:243:in `with_info_handler'
       +    /usr/lib/ruby/gems/0.0.0/gems/minitest-0.0.3/lib/minitest/test.rb:93:in `run'
       +    /build/ruby-maxitest/src/maxitest-0.0.0/lib/maxitest/vendor/around.rb:47:in `run'
       +    /build/ruby-maxitest/src/maxitest-0.0.0/lib/maxitest/trap.rb:23:in `run'
       +    /usr/lib/ruby/gems/0.0.0/gems/minitest-0.0.3/lib/minitest.rb:1059:in `run_one_method'
       +    /usr/lib/ruby/gems/0.0.0/gems/minitest-0.0.3/lib/minitest.rb:365:in `run_one_method'
       +    /usr/lib/ruby/gems/0.0.0/gems/minitest-0.0.3/lib/minitest.rb:352:in `block (2 levels) in run'
       +    /usr/lib/ruby/gems/0.0.0/gems/minitest-0.0.3/lib/minitest.rb:351:in `each'
       +    /usr/lib/ruby/gems/0.0.0/gems/minitest-0.0.3/lib/minitest.rb:351:in `block in run'
       +    /usr/lib/ruby/gems/0.0.0/gems/minitest-0.0.3/lib/minitest.rb:391:in `on_signal'
       +    /usr/lib/ruby/gems/0.0.0/gems/minitest-0.0.3/lib/minitest.rb:378:in `with_info_handler'
       +    /usr/lib/ruby/gems/0.0.0/gems/minitest-0.0.3/lib/minitest.rb:350:in `run'
       +    /usr/lib/ruby/gems/0.0.0/gems/minitest-0.0.3/lib/minitest.rb:182:in `block in __run'
       +    /usr/lib/ruby/gems/0.0.0/gems/minitest-0.0.3/lib/minitest.rb:182:in `map'
       +    /usr/lib/ruby/gems/0.0.0/gems/minitest-0.0.3/lib/minitest.rb:182:in `__run'
       +    /usr/lib/ruby/gems/0.0.0/gems/minitest-0.0.3/lib/minitest.rb:159:in `run'
       +    /usr/lib/ruby/gems/0.0.0/gems/minitest-0.0.3/lib/minitest.rb:83:in `block in autorun'
       +
       +  3) Failure:
       +threads#test_0003_can kill extra threads [spec/cases/threads.rb:26]:
       +Minitest::Assertion: Expected: 3
       +  Actual: 33
       +    /usr/lib/ruby/gems/0.0.0/gems/minitest-0.0.3/lib/minitest/assertions.rb:183:in `assert'
       +    /usr/lib/ruby/gems/0.0.0/gems/minitest-0.0.3/lib/minitest/assertions.rb:218:in `assert_equal'
       +    /usr/lib/ruby/gems/0.0.0/gems/minitest-0.0.3/lib/minitest/spec.rb:28:in `must_equal'
       +    /build/ruby-maxitest/src/maxitest-0.0.0/lib/maxitest/global_must.rb:14:in `must_equal'
       +    spec/cases/threads.rb:8:in `assert_correct_threads'
       +    spec/cases/threads.rb:26:in `block (2 levels) in <main>'
       +    /usr/lib/ruby/gems/0.0.0/gems/minitest-0.0.3/lib/minitest/test.rb:98:in `block (3 levels) in run'
       +    /usr/lib/ruby/gems/0.0.0/gems/minitest-0.0.3/lib/minitest/test.rb:195:in `capture_exceptions'
       +    /build/ruby-maxitest/src/maxitest-0.0.0/lib/maxitest/trap.rb:4:in `capture_exceptions'
       +    /usr/lib/ruby/gems/0.0.0/gems/minitest-0.0.3/lib/minitest/test.rb:95:in `block (2 levels) in run'
       +    /usr/lib/ruby/gems/0.0.0/gems/minitest-0.0.3/lib/minitest.rb:296:in `time_it'
       +    /usr/lib/ruby/gems/0.0.0/gems/minitest-0.0.3/lib/minitest/test.rb:94:in `block in run'
       +    /usr/lib/ruby/gems/0.0.0/gems/minitest-0.0.3/lib/minitest.rb:391:in `on_signal'
       +    /usr/lib/ruby/gems/0.0.0/gems/minitest-0.0.3/lib/minitest/test.rb:243:in `with_info_handler'
       +    /usr/lib/ruby/gems/0.0.0/gems/minitest-0.0.3/lib/minitest/test.rb:93:in `run'
       +    /build/ruby-maxitest/src/maxitest-0.0.0/lib/maxitest/vendor/around.rb:47:in `run'
       +    /build/ruby-maxitest/src/maxitest-0.0.0/lib/maxitest/trap.rb:23:in `run'
       +    /usr/lib/ruby/gems/0.0.0/gems/minitest-0.0.3/lib/minitest.rb:1059:in `run_one_method'
       +    /usr/lib/ruby/gems/0.0.0/gems/minitest-0.0.3/lib/minitest.rb:365:in `run_one_method'
       +    /usr/lib/ruby/gems/0.0.0/gems/minitest-0.0.3/lib/minitest.rb:352:in `block (2 levels) in run'
       +    /usr/lib/ruby/gems/0.0.0/gems/minitest-0.0.3/lib/minitest.rb:351:in `each'
       +    /usr/lib/ruby/gems/0.0.0/gems/minitest-0.0.3/lib/minitest.rb:351:in `block in run'
       +    /usr/lib/ruby/gems/0.0.0/gems/minitest-0.0.3/lib/minitest.rb:391:in `on_signal'
       +    /usr/lib/ruby/gems/0.0.0/gems/minitest-0.0.3/lib/minitest.rb:378:in `with_info_handler'
       +    /usr/lib/ruby/gems/0.0.0/gems/minitest-0.0.3/lib/minitest.rb:350:in `run'
       +    /usr/lib/ruby/gems/0.0.0/gems/minitest-0.0.3/lib/minitest.rb:182:in `block in __run'
       +    /usr/lib/ruby/gems/0.0.0/gems/minitest-0.0.3/lib/minitest.rb:182:in `map'
       +    /usr/lib/ruby/gems/0.0.0/gems/minitest-0.0.3/lib/minitest.rb:182:in `__run'
       +    /usr/lib/ruby/gems/0.0.0/gems/minitest-0.0.3/lib/minitest.rb:159:in `run'
       +    /usr/lib/ruby/gems/0.0.0/gems/minitest-0.0.3/lib/minitest.rb:83:in `block in autorun'
       +
       +  4) Failure:
       +threads#test_0004_can wait for extra threads [spec/cases/threads.rb:32]:
       +Minitest::Assertion: Expected: 3
       +  Actual: 33
       +    /usr/lib/ruby/gems/0.0.0/gems/minitest-0.0.3/lib/minitest/assertions.rb:183:in `assert'
       +    /usr/lib/ruby/gems/0.0.0/gems/minitest-0.0.3/lib/minitest/assertions.rb:218:in `assert_equal'
       +    /usr/lib/ruby/gems/0.0.0/gems/minitest-0.0.3/lib/minitest/spec.rb:28:in `must_equal'
       +    /build/ruby-maxitest/src/maxitest-0.0.0/lib/maxitest/global_must.rb:14:in `must_equal'
       +    spec/cases/threads.rb:8:in `assert_correct_threads'
       +    spec/cases/threads.rb:32:in `block (2 levels) in <main>'
       +    /usr/lib/ruby/gems/0.0.0/gems/minitest-0.0.3/lib/minitest/test.rb:98:in `block (3 levels) in run'
       +    /usr/lib/ruby/gems/0.0.0/gems/minitest-0.0.3/lib/minitest/test.rb:195:in `capture_exceptions'
       +    /build/ruby-maxitest/src/maxitest-0.0.0/lib/maxitest/trap.rb:4:in `capture_exceptions'
       +    /usr/lib/ruby/gems/0.0.0/gems/minitest-0.0.3/lib/minitest/test.rb:95:in `block (2 levels) in run'
       +    /usr/lib/ruby/gems/0.0.0/gems/minitest-0.0.3/lib/minitest.rb:296:in `time_it'
       +    /usr/lib/ruby/gems/0.0.0/gems/minitest-0.0.3/lib/minitest/test.rb:94:in `block in run'
       +    /usr/lib/ruby/gems/0.0.0/gems/minitest-0.0.3/lib/minitest.rb:391:in `on_signal'
       +    /usr/lib/ruby/gems/0.0.0/gems/minitest-0.0.3/lib/minitest/test.rb:243:in `with_info_handler'
       +    /usr/lib/ruby/gems/0.0.0/gems/minitest-0.0.3/lib/minitest/test.rb:93:in `run'
       +    /build/ruby-maxitest/src/maxitest-0.0.0/lib/maxitest/vendor/around.rb:47:in `run'
       +    /build/ruby-maxitest/src/maxitest-0.0.0/lib/maxitest/trap.rb:23:in `run'
       +    /usr/lib/ruby/gems/0.0.0/gems/minitest-0.0.3/lib/minitest.rb:1059:in `run_one_method'
       +    /usr/lib/ruby/gems/0.0.0/gems/minitest-0.0.3/lib/minitest.rb:365:in `run_one_method'
       +    /usr/lib/ruby/gems/0.0.0/gems/minitest-0.0.3/lib/minitest.rb:352:in `block (2 levels) in run'
       +    /usr/lib/ruby/gems/0.0.0/gems/minitest-0.0.3/lib/minitest.rb:351:in `each'
       +    /usr/lib/ruby/gems/0.0.0/gems/minitest-0.0.3/lib/minitest.rb:351:in `block in run'
       +    /usr/lib/ruby/gems/0.0.0/gems/minitest-0.0.3/lib/minitest.rb:391:in `on_signal'
       +    /usr/lib/ruby/gems/0.0.0/gems/minitest-0.0.3/lib/minitest.rb:378:in `with_info_handler'
       +    /usr/lib/ruby/gems/0.0.0/gems/minitest-0.0.3/lib/minitest.rb:350:in `run'
       +    /usr/lib/ruby/gems/0.0.0/gems/minitest-0.0.3/lib/minitest.rb:182:in `block in __run'
       +    /usr/lib/ruby/gems/0.0.0/gems/minitest-0.0.3/lib/minitest.rb:182:in `map'
       +    /usr/lib/ruby/gems/0.0.0/gems/minitest-0.0.3/lib/minitest.rb:182:in `__run'
       +    /usr/lib/ruby/gems/0.0.0/gems/minitest-0.0.3/lib/minitest.rb:159:in `run'
       +    /usr/lib/ruby/gems/0.0.0/gems/minitest-0.0.3/lib/minitest.rb:83:in `block in autorun'
       +
       +4 runs, 4 assertions, 4 failures, 0 errors, 0 skips
       +
       +Focus on failing tests:
       +mtest spec/cases/threads.rb:15
       +mtest spec/cases/threads.rb:19
       +mtest spec/cases/threads.rb:25
       +mtest spec/cases/threads.rb:31
     # ./spec/maxitest_spec.rb:159:in `block (3 levels) in <top (required)>'

Finished in 8.43 seconds (files took 0.05067 seconds to load)
37 examples, 1 failure

Failed examples:

rspec ./spec/maxitest_spec.rb:155 # Maxitest extra threads fails on extra and passes on regular

allow to make classes order dependent too

order_dependent! does not work when using multiple describe blocks ... fix that

r = Minitest::Runnable.runnables
def r.shuffle
  self
end

mtest -v crashes

% mtest -v
/Users/mperham/.gem/ruby/3.2.0/gems/maxitest-4.4.0/lib/maxitest/vendor/testrbl.rb:19:in `run_from_cli': uninitialized constant Maxitest::VERSION (NameError)

          puts Maxitest::VERSION
                       ^^^^^^^^^
	from /Users/mperham/.gem/ruby/3.2.0/gems/maxitest-4.4.0/bin/mtest:12:in `<top (required)>'
	from /Users/mperham/.gem/ruby/3.2.0/bin/mtest:25:in `load'
	from /Users/mperham/.gem/ruby/3.2.0/bin/mtest:25:in `<main>'

Some sort of changelog?

I picked up 5.0 this morning but can't find any documentation about what changed. Can you provide a changes.md, a milestone in GitHub Issues, or some other summary of what users might need to know about the upgrade? For instance, Sidekiq 7 supports Ruby 2.7 but maxitest 5.0 appears to be >= 3 only so I can't use 5.x.

Documentation

Please, update documentation with examples of all constructions, such as let!, pending, etc. I cannot use 'pending' properly - just get an syntax error...

`before :all` / `after :all` / `around :all` support

after :all + around :all

for after :all:
we need to track some kind of class hierarchy since a sub-class should not trigger the patent-class to run it's after :all hook

describe a do
  after :all { foo }
  it foo
  # not here
  describe b do
    it bar
    # but here
  end
end

especially with random test (can be disabled) + random class ordering (cannot be disabled without hackery) that will be tricky
also the usefulness is pretty low since there could be lots of tests that don't belong to the class with after :all that are run in between

for around :all:
once after :all works that should be the same fiber hackery like we use for the current around

Documentation missing

It would be great if the usage of maxitest could be explained with minimal working example codes.

Thanks!

Thanks for your efforts on this! It brings everything I wanted from minitest, but were too afraid to ask 😄

❤️ 💛 💚 💜

Maxitest may interfere with other reporters?

I'm having an issue using maxitest in conjunction with minitest-reporters. It appears that using both causes minitest-reporters to not work correctly. When using their RubyMine reporter, exceptions occur.

I created a minimal project to demonstrate this at https://github.com/icantrap/maxitest-sample. Have you guys seen something similar with other custom minitest reporters?

consider including metadata/tagging feature?

The one feature I always wanted that is not in here is test 'tagging', as in:

it "something", some_tag: "something" do

(Or I suppose you could do that in test-unit-style do as def something_test(some_tag => 'something'), although I never have).

It is useful for things like:

• tagging certain tests to be automatically wrapped by vcr setup/teardown (typical vcr rspec usage pattern)
• tagging certain tests to use capybara javascript driver setup/teardown (typical capybara rspec usage pattern)
• whatever other custom behavior you want, once you have tests/examples tagged, it's easy to write your own before/after/around's that do certain things with certain tags.

Here's one existing implementations of adding metadata/tagging to minitest, I'm not sure how robustly or cleanly it's written.

https://github.com/wojtekmach/minitest-metadata

As metadata-tags are a feature you can then use to build other things, having a particular implementation 'blessed' by the maxitest extension project would be useful to establish common ground to build these other things, with some degree of support expectation etc.