criteo-forks / pytest-runner

Setup scripts can use pytest-runner to add setup.py test support for pytest runner.

Deprecation Notice

pytest-runner depends on deprecated features of setuptools and relies on features that break security mechanisms in pip. For example 'setup_requires' and 'tests_require' bypass pip --require-hashes. See also pypa/setuptools#1684.

It is recommended that you:

• Remove 'pytest-runner' from your setup_requires, preferably removing the setup_requires option.
• Remove 'pytest' and any other testing requirements from tests_require, preferably removing the tests_requires option.
• Select a tool to bootstrap and then run tests such as tox

Usage

• Add 'pytest-runner' to your 'setup_requires'. Pin to '>=2.0,<3dev' (or similar) to avoid pulling in incompatible versions.
• Include 'pytest' and any other testing requirements to 'tests_require'.
• Invoke tests with setup.py pytest.
• Pass --index-url to have test requirements downloaded from an alternate index URL (unnecessary if specified for easy_install in setup.cfg).
• Pass additional py.test command-line options using --addopts.
• Set permanent options for the python setup.py pytest command (like index-url) in the [pytest] section of setup.cfg.
• Set permanent options for the py.test run (like addopts or pep8ignore) in the [pytest] section of pytest.ini or tox.ini or put them in the [tool:pytest] section of setup.cfg. See pytest issue 567.
• Optionally, set test=pytest in the [aliases] section of setup.cfg to cause python setup.py test to invoke pytest.

Example

The most simple usage looks like this in setup.py:

setup(
    setup_requires=[
        'pytest-runner',
    ],
    tests_require=[
        'pytest',
    ],
)

Additional dependencies require to run the tests (e.g. mock or pytest plugins) may be added to tests_require and will be downloaded and required by the session before invoking pytest.

Follow this search on github for examples of real-world usage.

Standalone Example

This technique is deprecated - if you have standalone scripts you wish to invoke with dependencies, use pip-run.

Although pytest-runner is typically used to add pytest test runner support to maintained packages, pytest-runner may also be used to create standalone tests. Consider this example failure, reported in jsonpickle #117 or this MongoDB test demonstrating a technique that works even when dependencies are required in the test.

Either example file may be cloned or downloaded and simply run on any system with Python and Setuptools. It will download the specified dependencies and run the tests. Afterward, the the cloned directory can be removed and with it all trace of invoking the test. No other dependencies are needed and no system configuration is altered.

Then, anyone trying to replicate the failure can do so easily and with all the power of pytest (rewritten assertions, rich comparisons, interactive debugging, extensibility through plugins, etc).

As a result, the communication barrier for describing and replicating failures is made almost trivially low.

Considerations

Conditional Requirement

Because it uses Setuptools setup_requires, pytest-runner will install itself on every invocation of setup.py. In some cases, this causes delays for invocations of setup.py that will never invoke pytest-runner. To help avoid this contingency, consider requiring pytest-runner only when pytest is invoked:

needs_pytest = {'pytest', 'test', 'ptr'}.intersection(sys.argv)
pytest_runner = ['pytest-runner'] if needs_pytest else []

# ...

setup(
    #...
    setup_requires=[
        #... (other setup requirements)
    ] + pytest_runner,
)

Security Contact

To report a security vulnerability, please use the Tidelift security contact. Tidelift will coordinate the fix and disclosure.