spyder-ide / spyder-unittest

A plugin for Spyder to run tests and view the results

Spyder-Unittest

Project information

Build status

Copyright © 2014 Spyder Project Contributors

Description

Spyder-unittest is a plugin that integrates popular unit test frameworks with Spyder, allowing you to run test suites and view the results in the IDE.

The plugin supports the unittest module in the Python standard library as well as the pytest and nose2 testing frameworks. Support for pytest is most complete at the moment.

Installation

The unittest plugin is available in the spyder-ide channel in Anaconda and in PyPI, so it can be installed with the following commands:

• Using Anaconda: conda install -c spyder-ide spyder-unittest
• Using pip: pip install spyder-unittest

All dependencies will be automatically installed. You have to restart Spyder before you can use the plugin.

Usage

The plugin adds an item Run unit tests to the Run menu in Spyder. Click on this to run the unit tests. After you specify the testing framework and the directory under which the tests are stored, the tests are run. The Unit testing window pane (displayed at the top of this file) will pop up with the results. If you are using pytest, you can double-click on a test to view it in the editor.

If you want to run tests in a different directory or switch testing frameworks, click Configure in the Options menu (cogwheel icon), which is located in the upper right corner of the Unit testing pane.

Feedback

Bug reports, feature requests and other ideas are more than welcome on the issue tracker. Use the Spyder Google Group or our Gitter Chatroom for general discussion.

Development

Development of the plugin is done at https://github.com/spyder-ide/spyder-unittest . You can install the development version of the plugin by cloning the git repository and running pip install ., possibly with the --editable flag.

The plugin has the following dependencies:

• spyder (obviously), at least version 4.0
• lxml
• the testing framework that you will be using: pytest and/or nose2

In order to run the tests distributed with this plugin, you need nose2, pytest and pytest-qt. If you use Python 2, you also need mock.

You are very welcome to submit code contributions in the form of pull requests to the issue tracker. GitHub is configured to run pull requests automatically against the test suite and against several automatic style checkers using ciocheck. The style checkers can be rather finicky so you may want to install ciocheck locally and run them before submitting the code.

Contributing

Everyone is welcome to contribute! The document Contributing to Spyder also applies to the unittest plugin.

We are grateful to the entire Spyder community for their support, without which this plugin and the whole of Spyder would be a lot less awesome.

More information

Main Website

Download Spyder (with Anaconda)

Spyder Github

Troubleshooting Guide and FAQ

Development Wiki

Gitter Chatroom

Google Group

@Spyder_IDE on Twitter

@SpyderIDE on Facebook

Support Spyder on OpenCollective

spyder-unittest's Issues

KeyError: 'system-err'

When I run pytest unit tests using this plugin on an internal package of mine, I get in the internal console

Traceback (most recent call last):
  File "c:\users\bilderbuchi\desktop\coding\spyder-unittest\spyder_unittest\backend\testrunner.py", line 162, in finished
    testresults = self.load_data()
  File "c:\users\bilderbuchi\desktop\coding\spyder-unittest\spyder_unittest\backend\testrunner.py", line 206, in load_data
    category = STATUS_TO_CATEGORY[status]
KeyError: 'system-err'

The code/tests is not public, so I can't offer a good repro, but from the key I guess something in the tests goes wrong, and that status is not coded in yet? Vanilla pytest on the package works correctly, as do vanilla pytest tests and Spyder-run tests on spyder-unittest.

Make code translation friendly?

Use _() whenever gui text is exposed

Find translators for the project? (We could ask the same translators of spyder)

Changes for compatibility with new undocking behavior of Spyder

As discussed in Spyder PR #3824

Add issues/PRs templates

Point to Spyder wiki if necessary

pytest statuses "expected-fail" and "unexpectedly passing" not yet reflected in Category

Hi!

Just to note this (can't currently work on this), the pytest statuses "expected to fail" (XFAIL, shown as an "x" in test results) and "“unexpectedly passing” (XPASS) are not yet available in https://github.com/spyder-ide/spyder-unittest/blob/52af625e6a526c218d3a7930fdd348ab5716c724/spyder_unittest/backend/testrunner.py#L28, and probably should be at some point.
See also http://doc.pytest.org/en/latest/skipping.html?highlight=expected%20fail#skip-and-xfail-dealing-with-tests-that-cannot-succeed

Rename repo to spyder-unittest

Also rename main module spyder_unittesting to spyder_unitest?

Remove QuantifiedCode

Will be shut down 22 July.

Add the test suite to the release tarball

Please consider adding the tests to the release tarball uploaded on PyPI. PyPI is the primary source for packaging on Linux distributions and having the test suite would be highly beneficial for QA / CI purposes.

Thanks,
Ghis

Add support for running DocTest

From @spyder-issues-migrator on February 17, 2015 23:33

From [email protected] on 2015-01-27T19:21:34Z

Doctest is a useful built-in library in Python.

I would find it very useful if Spyder could run doctest on the active editor file.

Original issue: http://code.google.com/p/spyderlib/issues/detail?id=2144

Copied from original issue: spyder-ide/spyder#2144

Add unittest support

untittest/unittest2 are still used by people, we could support them.

It should wait for #3.

Add changelog

@jitseniesen, please use our own loghub project to create a changelog for this project.

Thanks!

Add a screenshot to the README

Its nice to see how the plugin looks/will look even if not finished (but specially when finished)

so we can add a img_src folder to put some screenshots so the README can link to those files

Use sensible precision when displaying test times

TimeoutError when running tests

I am getting the following error log when running the tests as part of the package build process on Debian:

cd /<<PKGBUILDDIR>>/.pybuild/pythonX.Y_3.5/build; python3.5 -m pytest /<<PKGBUILDDIR>>/spyder_unittest
============================= test session starts ==============================
platform linux -- Python 3.5.3, pytest-3.0.6, py-1.4.32, pluggy-0.4.0
PyQt5 5.7 -- Qt runtime 5.7.1 -- Qt compiled 5.7.1
rootdir: /<<PKGBUILDDIR>>, inifile: 
plugins: xvfb-1.0.0, qt-2.1.0
collected 13 items

../../../spyder_unittest/backend/tests/test_testrunner.py ....
../../../spyder_unittest/widgets/tests/test_configdialog.py .......
../../../spyder_unittest/widgets/tests/test_unittestgui.py FF

=================================== FAILURES ===================================
_________________ test_run_tests_and_display_results[py.test] __________________

qtbot = <pytestqt.qtbot.QtBot object at 0x7fa7ba718b00>
tmpdir = local('/tmp/pytest-of-gvaillan/pytest-0/test_run_tests_and_display_res0')
monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7fa7ba7186a0>
framework = 'py.test'

    @pytest.mark.parametrize('framework', ['py.test', 'nose'])
    def test_run_tests_and_display_results(qtbot, tmpdir, monkeypatch, framework):
        """Basic check."""
        os.chdir(tmpdir.strpath)
        testfilename = tmpdir.join('test_foo.py').strpath
    
        with open(testfilename, 'w') as f:
            f.write("def test_ok(): assert 1+1 == 2\n"
                    "def test_fail(): assert 1+1 == 3\n")
    
        MockQMessageBox = Mock()
        monkeypatch.setattr('spyder_unittest.widgets.unittestgui.QMessageBox',
                            MockQMessageBox)
    
        widget = UnitTestWidget(None)
        qtbot.addWidget(widget)
        config = Config(wdir=tmpdir.strpath, framework=framework)
        with qtbot.waitSignal(widget.sig_finished, timeout=10000, raising=True):
>           widget.run_tests(config)
E           pytestqt.exceptions.TimeoutError: Signal sig_finished() not emitted after 10000 ms

/<<PKGBUILDDIR>>/spyder_unittest/widgets/tests/test_unittestgui.py:44: TimeoutError
___________________ test_run_tests_and_display_results[nose] ___________________

qtbot = <pytestqt.qtbot.QtBot object at 0x7fa7ba744c88>
tmpdir = local('/tmp/pytest-of-gvaillan/pytest-0/test_run_tests_and_display_res1')
monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7fa7ba744e10>
framework = 'nose'

    @pytest.mark.parametrize('framework', ['py.test', 'nose'])
    def test_run_tests_and_display_results(qtbot, tmpdir, monkeypatch, framework):
        """Basic check."""
        os.chdir(tmpdir.strpath)
        testfilename = tmpdir.join('test_foo.py').strpath
    
        with open(testfilename, 'w') as f:
            f.write("def test_ok(): assert 1+1 == 2\n"
                    "def test_fail(): assert 1+1 == 3\n")
    
        MockQMessageBox = Mock()
        monkeypatch.setattr('spyder_unittest.widgets.unittestgui.QMessageBox',
                            MockQMessageBox)
    
        widget = UnitTestWidget(None)
        qtbot.addWidget(widget)
        config = Config(wdir=tmpdir.strpath, framework=framework)
        with qtbot.waitSignal(widget.sig_finished, timeout=10000, raising=True):
>           widget.run_tests(config)
E           pytestqt.exceptions.TimeoutError: Signal sig_finished() not emitted after 10000 ms

/<<PKGBUILDDIR>>/spyder_unittest/widgets/tests/test_unittestgui.py:44: TimeoutError
============================ pytest-warning summary ============================
WC1 /<<PKGBUILDDIR>>/spyder_unittest/backend/tests/test_testrunner.py cannot collect test class 'TestRunner' because it has a __init__ constructor
=========== 2 failed, 11 passed, 1 pytest-warnings in 20.48 seconds ============

This happens regardless of whether pytest is called via xvfb-run or via the xvfb plugin for pytest.

Plugin shouldn't inherit from widget (?)

Consider whether it's better to do composition instead

Be smarter about abbreviating test names

As suggested in #75, do not just chop off the module name when displaying the test name, but abbreviate it. Maybe use just one letter, maybe use the shortest prefix to make it unique.

Unclutter GUI

Not much space is left for the test results. Try to use only one line above the results instead of the two we have now (line 1: filename, folder icon, start, stop; line 2: collapse, expand, date, output).

Correctly parse output of unittest tests containing print statements

After PR #72, the output of tests as printed to stdout is displayed with the last tests, instead of with the test that actually produces the output. Fix this.

I think this may require us to implement our own test runner instead of relying on python -m unittest.

Display test results as they come in

Use project dir as default test suite

Double clicking on test name should take you somewhere useful

Maybe to location of test?

Clear up file / dir confusion

It looks like the combobox only accepts directories, but only works with files ?!? Related error message:

.../spyder_unittesting/widgets/unittestinggui.py", line 220, in start
    self.process.setWorkingDirectory(wdir)
TypeError: setWorkingDirectory(self, str): argument 1 has unexpected type 'bool'

Extend plugin to compute and display coverage

This is still some way off.

Add a changelog

Use https://github.com/spyder-ide/loghub

We could also use this plugin as an opportunity to test an automatic release script I am working on.

Define an api module

The idea for the new api decoupling is that every plugin on spyder (even third party) or at least the one we make, should include an api module and define there any public api we want to expose.

Since this is a language related plugin, I need to brief you a bit on how I think we can handle that.

But so far we need to have a language agnostic functionality assuming that a unit test suite gives a list of tests (failed, passed, xfailed, xpassed) plus some optional coverage.

Depend on nose and pytest at installation

Right now, the plugin offers to run pytest or nose on a folder or your choice. There is however nothing forcing users to have any, one or both test frameworks available in their environment. Regardless, the selection box offer all test frameworks, which may lead to problems later.

Could this be made a bit more adaptive somehow? For instance, by greying out selection of test frameworks for which a test import failed?

Otherwise, perhaps there should be an install dependency on nose and pytest?

Do first release

Once plugin is in a decent state, put it on PyPI (and Conda?) and publicize

• publish 0.1.beta on PyPI
• publish 0.1.beta on conda (the beta versions are mainly to test packaging)
• publish 0.1 on PyPI
• publish 0.1 on conda
• send announcement

Avoid qtbot.wait in test

See spyder_unittest/widgets/tests/test_unittestgui.py.

---

This refers to the qtbot.wait in that file

Add function for registering test frameworks

The idea is that every test framework has an associated test runner, which follows the interface of the current TestRunner class. This class will be split in two classes: PyTestTestRunner and NoseTestRunner. The plugin has a function register_test_framework which takes the name of the framework and the test runner class.

This makes it easier to extend to other frameworks like unittest (cf. #4)

Create a conda package

Nitpicks for setup.py

Remove 3.3 and 3.4 support from classifiers and add 3.6

Don't display test time when using unittest

Unittest does not report test times. At the moment, a spurious time of zero is displayed.

Also test against spyder's master branch in CI

At the moment, the CI test script installed spyder via conda. It would be nice if we could also test against the current master branch on github to ensure that the plugin remains compatible with the development version.

Normalized copyright information

setup.py lists "The Spyder Project Contributors"

LICENSE.txt lists "Joseph Martinot-Lagarde"

The copyright headers in the source files list "Spyder Project Developers"

Please normalize this. Right now, the copyright / authorship information is confusing.

Also, please consider adding dates to the copyright headers.

Allow a user to run a specific unit test

Is it possible for a user to run a specific unit test, rather than all tests in the directory?
If not, I think it would be a useful addition since I don't want to rerun all unit tests to see if I fixed the specific one I am debugging (I currently use nose).

Update Appveyor Badge for new username spyder-ide

Update README

The plugin can be used in alpha state, so the doc could get some love (see the other plugins for examples)

Add an option to choose the test suite

pytest and nostesets are harcoded and need a code edit to be switched. There should be a config page to chose the one wanted.

Point contributors to ciocheck

Explain automatic tests that need to be passed before PR is accepted; either in readme or contributing file

Save before running tests (?)

Same as Spyder saves before running a script. Or maybe it's better to ask the user first?

Implement automatic testing

Separate test runner from widget

Better design, I think

Dependency of tests on pytestqt

I know it's a small thing, but the dependency on pytestqt for running the tests on this should probably be noted somewhere.

Check whether test framework is installed

so that we can give a good error message if it is not

Use python interpreter from preferences to run tests

Use <python> -m pytest (for py.test) to run the tests, where <python> is the Python executable set in the preferences. This is useful for systems with more than one Python interpreter and it also resolves the Debian issue from #64.

Use unittest's API for running tests

Same as #50 but for unittest

Extend CI to do coverage testing

Save selected test framework

The test framework (nose, py.test, unittest) should be saved between invocations of Spyder so that the user does not have to specify the test framework again after quitting Spyder. This should probably be done per project.

I guess this requires changes to Spyder ...

Abbreviate test names

They are currently too long for the widget

Use py.test's API to run tests

At the moment, we parse the JUnitXml output, which has the advantage that we can use the same code for py.test and nose. However, the JUnitXml output does not have all the information we need. In particular, this issue is a prerequisite for #12, #13 and #47.