Create pytest-testslide about testslide

[fornellas]

Draft on this branch: https://github.com/facebookincubator/TestSlide/tree/pytest.

[deathowl]

@fornellas i've checked out the pytest branch. Draft seems to be working well, except:

-               import pytest_testslide
+               from pytest_testslide import testslide

Also i dont get the

FIXME aggregated failures

in the actual module, i've created a case where we expect failure, failure seems fine to me:

"    def test_patch_attribute_raises_on_private(testslide)
>       testslide.patch_attribute(sample_module.SomeClass, "_private_attr", "newval")

test_pass.py:42:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/home/deathowl/work/TestSlide/pytest-testslide/pytest_testslide.py:43: in patch_attribute
    return testslide_module.patch_attribute.patch_attribute(*args, **kwargs)
/home/deathowl/work/TestSlide/testslide/patch_attribute.py:82: in patch_attribute
    _bail_if_private(attribute, allow_private)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

candidate = '_private_attr', allow_private = False

    def _bail_if_private(candidate: str, allow_private: False):
        if (
            candidate.startswith("_")
            and not allow_private
            and not (candidate.startswith("__") and candidate.endswith("__"))
        ):
            raise ValueError(
>               f"It's disencouraged to patch/mock private interfaces.\n"
                "This would result in way too coupled tests and implementation. "
                "Please consider using patterns like dependency injection instead. "
                "If you really need to do this use the allow_private=True argument."
            )
E           ValueError: It's disencouraged to patch/mock private interfaces.
E           This would result in way too coupled tests and implementation. Please consider using patterns like dependency injection instead. If you really need 

/home/deathowl/work/TestSlide/testslide/lib.py:18: ValueError
=========================== short test summary info ============================
FAILED test_pass.py::test_patch_attribute_raises_on_private - ValueError: It'...
========================= 1 failed, 11 passed in 0.21s =========================
====================================================================================================================================================== warnings 
tests/test_pytest_testslide.py::test_pass
  /home/deathowl/work/TestSlide/.venv/lib/python3.7/site-packages/_pytest/compat.py:333: PytestDeprecationWarning: The TerminalReporter.writer attribute is depr
  See https://docs.pytest.org/en/latest/deprecations.html\#terminalreporter-writer for more information.
    return getattr(object, name, default)

-- Docs: https://docs.pytest.org/en/latest/warnings.html
=================================================================================================================================================== short test s
FAILED pytest-testslide/tests/test_pytest_testslide.py::test_pass - assert '12 passed' in "============================= test session starts ===================

[fornellas]

FIXME aggregated failures

IIRC, this has to do with the fact that TestSlide can report multiple failures from a single test execution:

https://testslide.readthedocs.io/en/master/test_runner/index.html#multiple-failures-report

TestSlide uses this class for that:

https://github.com/facebookincubator/TestSlide/blob/b5ae6dae00f3981c3bae1fe8fd940b988804dd42/testslide/__init__.py#L165

and has some code at the runner, to better format this exception, something that pytest will miss.

In retrospect, it would have probably been a better design to have AggregatedExceptions.__str__ deal with generating a nice combine exception message, and not have testslide runner specific code to deal with it. This would allow pytest to "just work" I bet. This needs checking though, because we have this issue #101, where unittest hides multi-line exceptions...

[deathowl]

@fornellas I added a test for aggregated exceptions in https://github.com/deathowl/TestSlide/tree/pytest
Except for the strace including too much testslide/pytest-testslide stuff, it seems fine to me.
What do you think?

➜  TestSlide git:(pytest) ✗ PYTHONPATH=/Users/death0wl/TestSlide/pytest-testslide:/Users/death0wl/TestSlide python3.7 -m pytest pytest-testslide/tests
============================================================================================================================ test session starts =============================================================================================================================
platform darwin -- Python 3.7.5, pytest-6.0.1, py-1.9.0, pluggy-0.13.1
rootdir: /Users/death0wl/TestSlide/pytest-testslide
plugins: typeguard-2.9.1
collected 1 item

pytest-testslide/tests/test_pytest_testslide.py F                                                                                                                                                                                                                      [100%]

================================================================================================================================== FAILURES ==================================================================================================================================
_________________________________________________________________________________________________________________________________ test_pass __________________________________________________________________________________________________________________________________

testdir = <Testdir local('/private/var/folders/yg/lkl4dm_113qcry5sf0pjzpgm0000gn/T/pytest-of-death0wl/pytest-45/test_pass0')>

    def test_pass(testdir):
        testdir.makepyfile(
            """
    		from pytest_testslide import testslide
    		from tests import sample_module
    		from testslide import StrictMock

    		def test_has_mock_callable(testslide):
    			testslide.mock_callable

    		def test_mock_callable_assertion_works(testslide):
    			testslide.mock_callable

    		def test_mock_callable_unpaches(testslide):
    			testslide.mock_callable

    		def test_has_mock_async_callable(testslide):
    			testslide.mock_async_callable

    		def test_mock_async_callable_assertion_works(testslide):
    			testslide.mock_async_callable

    		def test_mock_async_callable_unpaches(testslide):
    			testslide.mock_async_callable

    		def test_has_mock_constructor(testslide):
    			testslide.mock_constructor

    		def test_mock_constructor_assertion_works(testslide):
    			testslide.mock_constructor

    		def test_mock_constructor_unpaches(testslide):
    			testslide.mock_constructor

    		def test_has_patch_attribute(testslide):
    			testslide.patch_attribute

    		def test_patch_attribute_unpaches(testslide):
    			testslide.patch_attribute

    		def test_aggregated_exceptions(testslide):
    			mocked_cls = StrictMock(sample_module.CallOrderTarget)
    			testslide.mock_callable(mocked_cls, 'f1')\
    				.for_call("a").to_return_value("mocked")\
    				.and_assert_called_once()
    			testslide.mock_callable(mocked_cls, 'f1')\
    				.for_call("b").to_return_value("mocked2")\
    				.and_assert_called_once()
    			assert sample_module.CallOrderTarget("a").f1("a") == "mocked"
    		"""
        )
        result = testdir.runpytest("-v")
        assert "11 passed" in result.stdout.str()
>       assert result.ret == 0
E       assert <ExitCode.TESTS_FAILED: 1> == 0
E        +  where <ExitCode.TESTS_FAILED: 1> = <RunResult ret=ExitCode.TESTS_FAILED len(stdout.lines)=77 len(stderr.lines)=0 duration=0.15s>.ret

/Users/death0wl/TestSlide/pytest-testslide/tests/test_pytest_testslide.py:60: AssertionError
---------------------------------------------------------------------------------------------------------------------------- Captured stdout call ----------------------------------------------------------------------------------------------------------------------------
============================= test session starts ==============================
platform darwin -- Python 3.7.5, pytest-6.0.1, py-1.9.0, pluggy-0.13.1 -- /opt/homebrew/opt/python37/bin/python3.7
cachedir: .pytest_cache
rootdir: /private/var/folders/yg/lkl4dm_113qcry5sf0pjzpgm0000gn/T/pytest-of-death0wl/pytest-45/test_pass0
plugins: typeguard-2.9.1
collecting ... collected 12 items

test_pass.py::test_has_mock_callable PASSED                              [  8%]
test_pass.py::test_mock_callable_assertion_works PASSED                  [ 16%]
test_pass.py::test_mock_callable_unpaches PASSED                         [ 25%]
test_pass.py::test_has_mock_async_callable PASSED                        [ 33%]
test_pass.py::test_mock_async_callable_assertion_works PASSED            [ 41%]
test_pass.py::test_mock_async_callable_unpaches PASSED                   [ 50%]
test_pass.py::test_has_mock_constructor PASSED                           [ 58%]
test_pass.py::test_mock_constructor_assertion_works PASSED               [ 66%]
test_pass.py::test_mock_constructor_unpaches PASSED                      [ 75%]
test_pass.py::test_has_patch_attribute PASSED                            [ 83%]
test_pass.py::test_patch_attribute_unpaches PASSED                       [ 91%]
test_pass.py::test_aggregated_exceptions FAILED                          [100%]
test_pass.py::test_aggregated_exceptions ERROR                           [100%]

==================================== ERRORS ====================================
_______________ ERROR at teardown of test_aggregated_exceptions ________________

    @pytest.fixture
    def testslide():
        with _TestSlideFixture() as testslide_fixture:
>           yield testslide_fixture

/Users/death0wl/TestSlide/pytest-testslide/pytest_testslide.py:50:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/Users/death0wl/TestSlide/pytest-testslide/pytest_testslide.py:23: in __exit__
    assertion()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

    def assertion():
        if times != self.call_count:
            raise AssertionError(
                (
                    "calls did not match assertion.\n"
                    "{}, {}:\n"
                    "  expected: called exactly {} time(s) with {}"
                    "  received: {} call(s)"
                ).format(
                    _format_target(self.target),
                    repr(self.method),
                    times,
                    self._args_message(),
>                   self.call_count,
                )
            )
E           AssertionError: calls did not match assertion.
E           <StrictMock 0x102F199D0 template=tests.sample_module.CallOrderTarget /private/var/folders/yg/lkl4dm_113qcry5sf0pjzpgm0000gn/T/pytest-of-death0wl/pytest-45/test_pass0/test_pass.py:39>, 'f1':
E             expected: called exactly 1 time(s) with arguments:
E               ('a',)
E             received: 0 call(s)

/Users/death0wl/TestSlide/testslide/mock_callable.py:261: AssertionError
=================================== FAILURES ===================================
__________________________ test_aggregated_exceptions __________________________

testslide = <pytest_testslide._TestSlideFixture object at 0x102f19810>

    def test_aggregated_exceptions(testslide):
    	mocked_cls = StrictMock(sample_module.CallOrderTarget)
    	testslide.mock_callable(mocked_cls, 'f1')				.for_call("a").to_return_value("mocked")				.and_assert_called_once()
    	testslide.mock_callable(mocked_cls, 'f1')				.for_call("b").to_return_value("mocked2")				.and_assert_called_once()
>   	assert sample_module.CallOrderTarget("a").f1("a") == "mocked"
E    assert "f1: 'a'" == 'mocked'
E      - mocked
E      + f1: 'a'

test_pass.py:42: AssertionError
=========================== short test summary info ============================
FAILED test_pass.py::test_aggregated_exceptions - assert "f1: 'a'" == 'mocked'
ERROR test_pass.py::test_aggregated_exceptions - AssertionError: calls did no...
==================== 1 failed, 11 passed, 1 error in 0.09s =====================
========================================================================================================================== short test summary info ===========================================================================================================================
FAILED pytest-testslide/tests/test_pytest_testslide.py::test_pass - assert <ExitCode.TESTS_FAILED: 1> == 0
============================================================================================================================= 1 failed in 0.16s ==============================================================================================================================

[fornellas]

test_aggregated_exceptions should cover it. But I think it is still only reporting the first exception that surfaces, and not reporting all of them. I think you can put test_aggregated_exceptions code at a regular TestCase, and run testslide on it to get the reference error. As long as pytest matches that, we're gold.