adamchainz / nose-randomly

:nose: Nose plugin to randomly order tests and control `random.seed`

nose-randomly

---

Unmaintained: I'm no longer maintaining this package because I haven't used nose for some time, and nose itself has not seen a release since 2015, nor a commit since 2016 (nearly 3 years at time of writing). If you want to continue maintenance please contact me.

---

Nose plugin to randomly order tests and control random.seed. (Also available for pytest).

Features

All of these features are on by default but can be disabled with flags.

• Randomly shuffles the submodules, TestCase classes + test functions when loading a module of tests.
• Randomly shuffles the test functions inside a TestCase when loading it.
• Resets random.seed() at the start of every test case and test to a fixed number - this defaults to time.time() from the start of your test run, but you can pass in --randomly-seed to repeat a randomness-induced failure.
• If factory boy is installed, its random state is reset at the start of every test. This allows for repeatable use of its random 'fuzzy' features.
• If faker is installed, its random state is reset at the start of every test. This is also for repeatable fuzzy data in tests - factory boy uses faker for lots of data.

About

Randomness in testing can be quite powerful to discover hidden flaws in the tests themselves, as well as giving a little more coverage to your system.

By randomly ordering the tests, the risk of surprising inter-test dependencies is reduced - a technique used in many places, for example Google's C++ test runner googletest.

By resetting the random seed to a repeatable number for each test, tests can create data based on random numbers and yet remain repeatable, for example factory boy's fuzzy values. This is good for ensuring that tests specify the data they need and that the tested system is not affected by any data that is filled in randomly due to not being specified.

Requirements

Tested with:

• Python 2.7, 3.6
• The latest version of Nose

Usage

Install from pip with:

pip install nose-randomly

Nose will automatically find the plugin.

To activate it on your test run, use the --with-randomly flag, for example:

nosetests -v --with-randomly

The output will start with an extra line that tells you the random seed that is being used:

Using --randomly-seed=1234
test_D (abcd_tests.Tests) ... ok
...

If the tests then fail due to ordering or randomly created data, you can then restart them with that seed:

nosetests -v --with-randomly --randomly-seed=1234

You can disable behaviours you don't like with the following flags:

• --randomly-dont-shuffle-modules - turn off the shuffling of the contents of modules
• --randomly-dont-shuffle-cases - turn off the shuffling of test functions inside TestCase classes
• --randomly-dont-reset-seed - turn off the reset of random.seed() at the start of every test

Background

nose has an unmerged pull request from 2009 to add random ordering functionality. This is available in plugin format in the nose-randomize package. It works quite well but I found that since it replaces all of the test loading machinery inside nose, it can interact badly with other plugins. This plugin was developed as a thinner layer to achieve the same thing, plus the random seed resetting which was not available before.

License

• BSD licensed, see LICENSE file
• Logo by Christian Mohr from the Noun Project (link).

nose-randomly's Issues

Logo

Every cool OSS project has a logo

When using randomly with other plugins, there is no random execution order (anymore)

I suspect that this is due to the missing priority in the main plugin class, so nose executes the plugin at the wrong time, when test execution order already has been determined.

This is just a guess on what might be going wrong here. I'm definitely sure about the observation though.

with randomly:
https://travis-ci.org/marscher/PyEMMA/jobs/171220051#L1330
without:
https://travis-ci.org/markovmodel/PyEMMA/jobs/170559868#L1337

random_seed isn't printed when passed --randomly-dont-reset-seed

When running randomly with --randomly-dont-reset-seed, the seed used (for randomly ordering tests) isn't printed. As a result, I can't replicate test ordering.

Here's the relevant code:

nose-randomly/nose_randomly/plugin.py

Lines 80 to 84 in 5411d41

	if self.options.reset_seed:
	print(
	"Using --randomly-seed={seed}".format(seed=self.options.seed),
	file=self.output_stream
	)

I think this is a bug? If you agree, I'm happy to open a PR with a fix.

Tests dropped with some seeds

Hi,

We are using nose-randomly with a django app. I hit a strange behaviour : with some seeds, tests runned drops to 3 tests instead of e.g 900 tests. I can reproduce this with presetting de seed with --randomly-seed.

I don't know how to debug this. Can you give me a hint ? I'll gladly propose a PR !

Regards,

Nose-randomly and setuptools don't get along

Trying to run ./setup.py nosetests with the nose-randomly plugin installed causes setuptools to immediately.

Here's the traceback:

% ./setup.py nosetests    
Traceback (most recent call last):
  File "./setup.py", line 304, in <module>
    test_suite='nose.collector',
  File "/usr/lib64/python2.7/distutils/core.py", line 137, in setup
    ok = dist.parse_command_line()
  File "/usr/lib/python2.7/site-packages/setuptools/dist.py", line 299, in parse_command_line
    result = _Distribution.parse_command_line(self)
  File "/usr/lib64/python2.7/distutils/dist.py", line 467, in parse_command_line
    args = self._parse_command_opts(parser, args)
  File "/usr/lib/python2.7/site-packages/setuptools/dist.py", line 609, in _parse_command_opts
    nargs = _Distribution._parse_command_opts(self, parser, args)
  File "/usr/lib64/python2.7/distutils/dist.py", line 564, in _parse_command_opts
    (args, opts) = parser.getopt(args[1:])
  File "/usr/lib64/python2.7/distutils/fancy_getopt.py", line 253, in getopt
    self._grok_option_table()
  File "/usr/lib64/python2.7/distutils/fancy_getopt.py", line 171, in _grok_option_table
    "must be a string of length >= 2") % long
distutils.errors.DistutilsGetoptError: invalid long option 'with-randomly': must be a string of length >= 
2

My steps were:

1. pip install nose-randomly
2. set test_suite='nose.collector' in setup()
3. Run setup.py

Test on Python 3.5, fix 'development status' in setup.py

We should be Python 3.5 compatible