Support PyTests about pybuilder

[mriehl]

That's currently not possible. However, writing a plugin should be very easy.

I had a quick look but I couldn't find any documentation on using py.test from within python (e.G. to access the test results or the builtin test collector), though module inspection clearly shows that this is possible.
The documentation seems to be very focused on the command-line calls.

If you can point me in the right direction, I'll come up with a plugin within the next few days.

[fatuhoku]

Totally! Every flipping test framework assumes that a coder will run through their special tests using their special command line program. I'm afraid I'm no expert on the matter either; but when I get time too I'll try to link you to the right places.

I think the Python testing ecosystem is very fragmented and many tools are ill-designed. They don't just ... compose, you know? The focus on just running tests on the command line with no foresight of integrating with other tools feels... amateur.

[mriehl]

I get that feeling too, although it seems to apply especially well to test frameworks!

I have opened a ticket and will keep you updated.

Regards,
max

[fatuhoku]

A simple way out of this: #16.
Let the user run py.test with whatever flags they normally do; forward the test output to console. Observe only the process result. If there were test failures the process return code is set to a non-zero value. That should be enough to fail the build.

[mriehl]

I got it to work in a plugin but the API is not clean at all and you cannot prevent pytest from writing to stdout (unless you use mock... in production code.....).
I'll start working on the exec plugin right away.

[fatuhoku]

Haha, that sounds awesome. Thank you @mriehl.

[mriehl]

Can now be done with

use_plugin("exec")

and

project.set_property("run_unit_tests_command", "py.test %s" % project.expand_path("$dir_source_unittest_python"))

[obestwalter]

Oh nice ... so I guess this won't be necessary anymore: https://github.com/obestwalter/pytestplugindev/blob/master/build.py

I'll have a look at the sphinx plugin instead.

[obestwalter]

o.k. so when setting these properties

project.set_property("dir_source_unittest_python", "tests")
project.set_property("unittest_module_glob", "test_*")
project.set_property("unittest_test_method_prefix", "test")
project.set_property(
    "run_unit_tests_command",
    "py.test %s" % project.expand_path("$dir_source_unittest_python"))

It works but I get no output from the failed test (even with -v):

$ pyb -v                                                   
PyBuilder version 0.10.36
Build started at 2014-11-17 00:26:54
------------------------------------------------------------
[INFO]  Building ehw version 1.0-SNAPSHOT
[INFO]  Executing build in /home/obestwalter/work/presentation_stack/linpy3toolchain
[INFO]  Going to execute task publish
------------------------------------------------------------
BUILD FAILED - exec plugin command py.test     
/bla/linpy3toolchain/tests for run_unit_tests exited with nonzero code 1
------------------------------------------------------------
Build finished at 2014-11-17 00:26:54
Build took 0 seconds (571 ms)

Is there a way to get at the output? If not I guess I better get that pytest plugin polished and working and we can use that instead.

... or can that use_plugin machinery be expanded to collect stdout and stderr, like I am doing it in the pytest plugin?

[mriehl]

You can set this per task (here it's run_unit_tests) :

project.set_property("run_unit_tests_propagate_stdout", True)
project.set_property("run_unit_tests_propagate_stderr", True)

To get the errors, stderr might be enough.

[obestwalter]

o.k. thanks, works like a charm - test results in pycharm are reported on stdout though.

Nice addition, this exec - makes a lot of plugins unnecessary I guess.

$ pyb                                             
PyBuilder version 0.10.36
[...]
[INFO]  ============================= test session starts ==============================
[INFO]  platform linux -- Python 3.4.2 -- py-1.4.26 -- pytest-2.6.4
[INFO]  collected 1 items
[INFO]  
[INFO]  tests/test_ehw.py F
[INFO]  
[INFO]  =================================== FAILURES ===================================
[INFO]  ___________________________________ test_ehw ___________________________________
[INFO]  
[INFO]      def test_ehw():
[INFO]  >       pytest.fail('BOOM!')
[INFO]  E       Failed: BOOM!
[INFO]  
[INFO]  tests/test_ehw.py:5: Failed
[INFO]  =========================== 1 failed in 0.01 seconds ===========================
[INFO]  
[INFO]  ----- end of verbatim  output -----
------------------------------------------------------------
BUILD FAILED - exec plugin command py.test /bla/linpy3toolchain/tests for run_unit_tests exited with nonzero code 1
------------------------------------------------------------

[mriehl]

You're welcome! One of the arguments in favor of a specialized plugin is that we are able to unify the output. For py.test I guess this is not that important because you always want to see the test results no matter what.

[vincentclaes]

I am also using pytest with pybuild. when I build pybuilder fails to import my source code when going through my tests.
i get for each module something like:
←[1m[INFO] ←[0;0m ERROR collecting src/tests/unittest/python/data_merger/controller/comparator_autom_params_test.py
←[1m[INFO] ←[0;0m src\tests\unittest\python\data_merger\controller\comparator_autom_params_test.py:6: in
←[1m[INFO] ←[0;0m from resources.diff_table import DiffTable
←[1m[INFO] ←[0;0m E ImportError: No module named resources.diff_table

this is part of my build.py file:

@init
def set_properties(project):
project.set_property("dir_source_main_python", r"src\main\python\data_merger")
project.set_property("dir_source_integrationtest_python", r"src\tests\integrationtest")
project.set_property("dir_source_unittest_python", r"src\tests\unittest")
project.set_property("unittest_module_glob", "*test.py")
project.set_property("unittest_test_method_prefix", "test")
project.set_property("run_unit_tests_command",
"py.test %s" % project.expand_path("$dir_source_unittest_python"))
project.set_property("run_unit_tests_propagate_stdout", True)
project.set_property("run_unit_tests_propagate_stderr", True)
project.set_property("teamcity_output", True)

I am using pycharm on windows 7 and in the path "src\main\python\data_merger" the folder data_merger is marked as source.

thanks in advance

[mriehl]

Hi,

@Vinnie1986 This looks like the PYTHONPATH environment variable is not set in the subprocess environment (my guess would be that on windows this is not automatic).

Can you inspect what the value of sys.path is when your tests run? (you will need to import sys.
Most probably we will need to pass the environment explicitly which is an easy fix. Just want to be sure this is indeed the problem.

[vincentclaes]

@mriehl Thanks for you quick reply. This was idd the issue, I have added the full path of "src\main\python\data_merger" to my pythonpath and the tests run fine now.

[AlexeySanko]

Hi.
Unfortunately, adding each time source folder to PYTHONPATH manually is not very usefull.
I modified a little bit solution of @obestwalter. Do You have any ideas how to improve it?

import sys
from pybuilder.core import use_plugin, init, task, after, Author, description as task_desc
from pybuilder.utils import assert_can_execute

use_plugin('python.core')
use_plugin('python.unittest')
use_plugin('python.install_dependencies')
use_plugin('python.flake8')
use_plugin('python.coverage')
use_plugin('python.distutils')
use_plugin('exec')

...

@after('prepare')
def assert_pytest_is_executable(logger):
    logger.debug('Checking if pytest is executable.')
    assert_can_execute(command_and_arguments=['py.test', '--version'],
                       prerequisite='pytest',
                       caller='plugin python.pytest')


@task
@task_desc('Runs unit tests based on Python pytest module.')
def run_unit_tests(project, logger):
    import pytest

    logger.info('Run tests (pytest)')
    from pybuilder.plugins.python.unittest_plugin \
        import _register_test_and_source_path_and_return_test_dir \
        as push_test_path_to_syspath
    test_dir = push_test_path_to_syspath(project, sys.path, 'unittest')
    try:
        args = [test_dir]
        if project.get_property('verbose'):
            args.append('-s')
            args.append('-v')
        ret = pytest.main(args)
        if ret:
            raise BuildFailedException('pytest unittests failed')
    except:
        raise

    logger.info('Run tests (pytest)')
    from pybuilder.plugins.python.unittest_plugin \
        import _register_test_and_source_path_and_return_test_dir \
        as push_test_path_to_syspath
    test_dir = push_test_path_to_syspath(project, sys.path, 'unittest')
    try:
        args = [test_dir]
        if verbose:
            args.append('-s')
            args.append('-v')
        ret = pytest.main(args)
        if ret:
            raise BuildFailedException('pytest unittests failed')
    except:
        raise

[AlexeySanko]

External plugin created based this workaround: https://github.com/AlexeySanko/pybuilder_pytest

[arcivanov]

@AlexeySanko I would be interested in bringing this into core as I may need this myself.

[AlexeySanko]

Hi, @arcivanov .
Puf... I originaly planned to add it to core but decided that it will be too. :))
I'm planning to add couple features. So I can add it as basic plugin to core in parallel.

I created issues into pytest repo: AlexeySanko/pybuilder_pytest#5