facebook / testslide

A Python test framework

A test framework for Python that enable unit testing / TDD / BDD to be productive and enjoyable.

Its well behaved mocks with thorough API validations catches bugs both when code is first written or long in the future when it is changed.

The flexibility of using them with existing unittest.TestCase or TestSlide's own test runner let users get its benefits without requiring refactoring existing code.

Quickstart

Install:

pip install TestSlide

Scaffold the code you want to test backup.py:

class Backup:
  def delete(self, path):
    pass

Write a test case backup_test.py describing the expected behavior:

import testslide, backup, storage

class TestBackupDelete(testslide.TestCase):
  def setUp(self):
    super().setUp()
    self.storage_mock = testslide.StrictMock(storage.Client)
    # Makes storage.Client(timeout=60) return self.storage_mock
    self.mock_constructor(storage, 'Client')\
      .for_call(timeout=60)\
      .to_return_value(self.storage_mock)

  def test_delete_from_storage(self):
    # Set behavior and assertion for the call at the mock
    self.mock_callable(self.storage_mock, 'delete')\
      .for_call('/file/to/delete')\
      .to_return_value(True)\
      .and_assert_called_once()
    backup.Backup().delete('/file/to/delete')

TestSlide's StrictMock, mock_constructor() and mock_callable() are seamlessly integrated with Python's TestCase.

Run the test and see the failure:

TestSlide's mocks failure messages guide you towards the solution, that you can now implement:

import storage

class Backup:
  def __init__(self):
    self.storage = storage.Client(timeout=60)

  def delete(self, path):
    self.storage.delete(path)

And watch the test go green:

It is all about letting the failure messages guide you towards the solution. There's a plethora of validation inside TestSlide's mocks, so you can trust they will help you iterate quickly when writing code and also cover you when breaking changes are introduced.

Full documentation

There's a lot more that TestSlide can offer, please check the full documentation at https://testslide.readthedocs.io/ to learn more.

Requirements

• Linux
• Python 3

Join the TestSlide community

TestSlide is open source software, contributions are very welcome!

See the CONTRIBUTING file for how to help out.

License

TestSlide is MIT licensed, as found in the LICENSE file.

Terms of Use

https://opensource.facebook.com/legal/terms

Privacy Policy

https://opensource.facebook.com/legal/privacy

Copyright

Copyright © 2021 Meta Platforms, Inc

testslide's Issues

Multi line exception message hidden by Python unittest runner

TestSlide exceptions often have a multi-line message. While TestSlide properly shows them in full, Python unittest test runner seem to swallow all but the first exception line. This makes it difficult for users to understand why things fail.

This can potentially be fixed by detecting when not running with TestSlide test runner (or when running with Python unttest test runner) and change all exceptions to be single line.

CC @pallotron

Async support for StrictMock

StrictMock has tons of mechanics under the hood, and some of them might break with async.

Let's add test cases for async stuff at tests/strict_mock_testslide.py, such as:

• async instance methods.
• async static methods.
• async class methods.

These tests should cover the same ground we already have there, such as signature validation.

`with_wrapper` does not support `mock_constructor`

from testslide.dsl import context


class MyClass(object):
    __slots__ = ["value"]

    def __init__(self, value):
        self.value = value


@context  # noqa
def mock_constructor_with_wrapper_test(context):

    @context.sub_context
    def intercept_context(context):
        @context.before
        def intercept_call_to_constructer(self):
            def wrapper(original_callable, *args, **kwargs):
                obj = original_callable(*args, **kwargs)
                obj.value += 1
                return obj

            self.mock_constructor(
                __name__, "MyClass"
            ).with_wrapper(wrapper).and_assert_called_once()

        @context.example
        def call_constructor(self):
            obj = MyClass(value=1)
            self.assertEqual(obj.value, 2)

    @context.example
    def call_constructor(self):
        obj = MyClass(value=1)
        self.assertEqual(obj.value, 1)

Error

Failures:

  0) mock constructor with wrapper test, intercept context: call constructor
    1) TypeError: original_callable() takes at least 1 argument (0 given)
      File "experimental/shihaoxu/testslide/mock_constructor_with_wrapper_test.py", line 31, in call_constructor
        obj = MyClass(value=1)
      File "experimental/shihaoxu/testslide/mock_constructor_with_wrapper_test.py", line 21, in wrapper
        obj = original_callable(*args, **kwargs)

`before_once`: add docs and failure test

before_once is not documented and has no test case covering failures. Both needs addressing.

When a before_once hook fails, the expectation is that all tests that depends on it should fail with the same signal. I believe currently, the first test will get the failure, but others will not run the hook, and execute as if the hook worked.

Check DSL function argument names

Put check in place for all DSL functions to ensure they're declared with proper arguments.

This should be OK:

@context.sub_context
def when_something(context):
  ...

but this should yield a failure:

@context.sub_context
def when_something(self):
  ...

This avoids common user mistakes.

The function signatures can be checked with https://docs.python.org/3/library/inspect.html#inspect.signature.

How to do conditional Skip?

How to skip a TestSlide example, if I known in certain environment the test is expceted to fail or does not make sense.
Like the Python unittest module API.
https://fburl.com/ljc095sq

DSL debug

Sometimes it is hard to understand what hooks (before / after / around) are being executed. It would be nice to have a --debug option that would make the test runner show what hook is being executed, something like this perhaps:

Some Context
  before: do_something() file.py:33
    Some Example
      example: some_example() file.py:38
      PASS
  after: cleanup() file.py:45

Support for classes that define `__new__` for `mock_constructor`

It currently has no support for classes with __new__, and will reject patching those.

IIRC, it was tough getting around Python idiosyncrasies to make mock_constructor work, and I willingly decided not to implement __new__ at first to simplify things, but this is potentially doable.

Fail `mock_callable()` / `mock_async_callable()` with private methods

Generically speaking, mocking private interfaces is bad it breaks the Liskov substitution principle: if the private interface of the class changes, tests break. Using dependency injection or the registry pattern are ways around this.

Let's help users write better tests, and fail mock_callable / mock_async_callable if the method being mocked is private (eg: _some_method), by raising ValueError. The error message should educate users on why this is a bad pattern, how to go around it, and if it is a very special snow flake, they can still do it by adding the parameter allow_private=True.

Improve mock_constructor() old pointer error message

The current error message:

BaseException: Attribute getting after the class has been used with mock_constructor() is not supported! After using mock_constructor() you must get a pointer to the new mocked class...

Is hard to understand. Let's:

• Make this message clearer.
• Add links to the docs. This requires having a place where the version is defined so the doc link can be constructed correctly https://testslide.readthedocs.io/en/$VERSION/mock_constructor/index.html.

context.nest_context("") won't get the context set up

Example code

from testslide.dsl import context

@context  # noqa
def nest_context_test(context):

    @context.shared_context
    def shared_context(context):
        @context.before
        def setup_value(self):
            self.value = 1

    @context.sub_context
    def sub_context(context):
        context.nest_context("shared context")

        @context.example
        def assert_value_accessible(self):
            self.value

Error

nest context test
  sub context
    assert value accessible: AttributeError: Context 'sub context' has no attribute 'value'

Failures:

  1) nest context test, sub context: assert value accessible
    1) AttributeError: Context 'sub context' has no attribute 'value'
      File "experimental/shihaoxu/testslide/nest_context_test.py", line 18, in assert_value_accessible
        self.value

Type check callable returns

We currently only type check callable arguments from mock_callable(), mock_async_callable(), mock_constructor() and StrictMock. Let's also check the return value of the call if type checking is available.

Type checker fails with mocks

This PR added type checking for function calls. It works as expected, but it is failing when a call is made using a mock, but it should not.

Suggestion on how to fix this:

• Have a constant with a list of known mock classes: Python's unittest (eg: the ones listed here) and StrictMock.
• Have a map of mock class to a function that extracts its template:
  • For StrictMock, get the given template (IIRC mock_callable already does this, there's a trick there).
  • For Python's unittest, get the spec.

Using that:

• If the given argument is a mock:
  • Without a template / spec, then do no type checking.
  • With a template, do type checking using the template.

Fix argument type validation for *args

This:

testslide.lib._validate_function_signature(
  object.__new__, (1, 2, 3), {"four": 5, "six": 6}
)

should pass, but is failing with:

IndexError: list index out of range
File "testslide/lib.py", line 78, in _validate_function_signature
  argname = list(signature.parameters.keys())[idx]

This is blocking #141 .

Check misuse of mock callable/constructor

Add check if mock_callable / mock_constructor are being called without proper setup. For example, if testslide.TestCase is inherited, and setUp() is overloaded, but it does not call super().setUp(), mock_callable & mock_constructor fail with weird errors.

Drop Python 2 support

Python 2 is dying soon, let's drop support for in and open the road for async stuff at #19.

Fix StrictMock attribute's attribute access.

This should work:

from testslide import StrictMock
from unittest.mock import Mock
mock = StrictMock(template=Template)
mock.whatever = Mock()
mock.whatever(1)
mock.whatever.assert_called_once_with(1)

Fail mock_callable().to_return_value() with coroutines

mock_async_callable() is the way to go for callables that return coroutines. Let's add checks to mock_callable() for the behaviors that return something (return_value, return_values, with_implementation etc) so that it fails if the return is a coroutine.

Simplify sharing contexts at top level

If we have a shared context defined in one file:

def some_shared_context(context):
  @context.before
  def before(self):
    pass

and we want to use it at different files, we need to do this:

from wherever import some_shared_context

@context
def first(context):
  context.shared_context(some_shared_context)
  context.merge_context("some shared context")

and on the other file:

from wherever import some_shared_context

@context
def second(context):
  context.shared_context(some_shared_context)
  context.merge_context("some shared context")

We can somewhat easily modify the merge_context function to accept the function directly, simplifying things:

from wherever import some_shared_context

@context
def first(context):
  context.merge_context(some_shared_context)

and on the other file

from wherever import some_shared_context

@context
def second(context):
  context.merge_context(some_shared_context)

Add type checking for non callable attributes at StrictMock

We currently only check type for callable attributes for StrictMock instances.

Let's add type checking when setting non-callable attributes.

Suggest mock_(async)_callable from StrictMock

StrictMock raises when an attribute is accessed, but has not been previously defined

https://github.com/facebookincubator/TestSlide/blob/8d516487247b6591c5a310c3e069ddb32d822d5f/testslide/strict_mock.py#L70

The message simply mentions "setting" the attribute. Let's improve this message to, if the mock has a template, and the attribute is callable, suggest using mock_callable / mock_async_callable to set it.

Validate if mock_callable() target has left over references

mock_constructor has validation logic to find out if the class being mocked has other references that are not from the module where it lives. This is important, as there's no guarantees it is gonna work otherwise.

mock_callable has the same potential problem when mocking functions at modules for example: if there are left over references such as from module import func, it will not work.

Let's add such validation to mock_callable so users don't shoot themselves in the foot.

This logic can be added here.

Focus & Skip broken for unittest.TestCase decorated by @patch

For unittest.TestCase classes that are decorated with https://docs.python.org/3/library/unittest.mock.html#patch, when we do ftest or xtest, the decorator does to patch these methods, and we get a TypeError when running the test.

This happens because @patch is evaluated on the class loading, and it patches all methods there that begin with "test". Since ftest and xtest don't match the search, they're not decorated, and become broken.

context.memoize: check if argument is callable and has correct signature

Both cases below must raise when declared, not when executed:

context.memoize(
    not_callable=True,
    callable_but_wrong_signature=lambda: False,
  )

Support subclassing StrictMock

This should work:

from testslide import StrictMock

class Template:
  def __eq__(self, other):
    return False

class TemplateStrictMock(StrictMock):
  def __init__(self):
    super().__init__(template=Template)

  def __eq__(self, other):
    return False

mock = TemplateStrictMock()
assert (mock == mock) is False

But is currently failing with:

testslide.strict_mock.UndefinedAttribute: '__eq__' is not set.
<StrictMock 0x7F6A9E5F4D90 template=__main__.Template test.py:9> must have a value set for this attribute if it is going to be accessed.

Fix StrictMock with bool()

Given:

In [1]: from testslide import StrictMock                                      
In [2]: class A: 
   ...:     def __len__(self): 
   ...:         return 0                                                      
In [3]: m1 = StrictMock(A)                                                    
In [4]: m2 = StrictMock(A) 

bool(), from its docs, looks for __bool__, which is not defined, then it looks for __len__, which is defined in the template class, thus it should be invoked. Given the default behavior of StrictMock, it should raise UndefinedBehavior on __len__ access, bool(m1) should raise, but it does not:

In [5]: bool(m1)                                                              
Out[5]: True
In [6]: bool(m2)                                                              
Out[6]: True

Instead, it returns true, as the default value for bool (as if both __bool__ and __len__ are not defined). If we define a behavior for __len__, things works as expected:

In [7]: m1.__len__ = lambda: 0                                                                                                                                
In [8]: bool(m1)                                                                                                                                              
Out[8]: False

However, it breaks other instances:

In [9]: bool(m2)                                                                                                                                              
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-9-a2b1c204a39c> in <module>
----> 1 bool(m2)
~/.pyenv/versions/3.6.6/lib/python3.6/site-packages/testslide/strict_mock.py in __get__(self, instance, _owner)
    119         else:
    120             raise AttributeError(
--> 121                 "{}:\n  Object has no attribute '{}'".format(repr(instance), self.name)
    122             )
    123 
AttributeError: <StrictMock 0x7F27960CD4E0 template=__main__.A - <ipython-input-4-9a7b27e5dca0>:1>:
  Object has no attribute '__len__'

This seems to happen because the way bool is imlpemented. When we do m1.__len__ = lambda: 0, under the hood, StrictMock puts _DescriptorProxy at StrictMock dictionary. _DescriptorProxy correctly implements __get__ to give the correct value for each instance, but bool() does not follow the descriptor protocol, and simply checks if the class dictionary has __len__. If it happens to implement __get__ (as _DescriptorProxy does), and the instance in particular does not have the attribute, it simply raises the AttributeError from _DescriptorProxy.

From __bool__() docs, I'd expect for it do do something like this:

def bool(obj):
  if hasattr(obj, "__bool__"):
    return getattr(obj, "__bool__")()
  if hasattr(obj, "__len__"):
    return not (return getattr(obj, "__len__")() == 0)
  return True

Thus, if a particular instance does not have __len__ (eg: __get__ raises AttributeError), it sholud work. However, it seems it is doing this:

def bool(obj):
  if "__bool__" in type(obj).__dict__:
    return getattr(obj, "__bool__")()
  if "__bool__" in type(obj).__dict__:
    return not (return getattr(obj, "__len__")() == 0)
  return True

The workaround at StrictMock's end (to not depend on any upstream patch), can be:

• Override StrictMock.__new__ to dynamically create a subclass of StrictMock for each instance.
• Populate this subclass magic methods (__len__, __bool__ etc) accordingly to the template given.

This should make magic methods per StrictMock instance, thus covering the bool() case, and any other similar scenario. This is a similar solution to what mock_constructor() does to go workaround a similar __new__ bug in Python 3.x.

Execution time should be from process start

Currently the test execution time discards the time spent in between the process started and the tests starting to run. This discarded time is often associated with slow to import modules, which are hidden from users.

Let's change this logic to consider the start time as the moment the process initiates, rather than the moment the tests start to run.

Fix TestCase loading with nested classes

TestSlide loads TestCase classes using this logic. Upstream Python, does it like this. This means, that If the module has this:

class Outer:
  class Inner(TestCase):
    def test_something(self):
      pass

Python won't consider Inner.test_something as part of the test, but TestSlide will.

This can be easily fixed by refactoring the logic at TestSlide's side to match Python's, however, it won't cover other cases such as load_tests().

The proper fix for this, is to use Python's TestLoader in the same fashion as Python unittent does form its CLI.

Fix setUpModule and tearDownModule

Currently the test runner does not respect https://docs.python.org/3/library/unittest.html#setupmodule-and-teardownmodule, this needs fixing.

This is somewhat related to #51 , as we should probably reuse Python's test loader / runner, and hook TestSlide into them, so we can avoid similar potential issues in the future.

This will likely require implementing @context.before_once and @context.after_once at TestSlide's DSL.

Can't configure TestCase

TestSlide has a hacky support for TestCase.assert* methods. While this works, it prevents users from tweaking things such as maxDiff.

The proposed solution to fix this is:

• Drop the hacky self._test_case.
• Make usage of the TestCase integration, perhaps by adding doing self.add_test_case() at Context.__init__, and assigning it to self.test_case.
• Keep the __getattr__ logic, but forward it to self.test_case.

With that, users should be able to do self.test_case.maxDiff = new_value.

Omitting to_return_value changes expectation checks

See the following examples. In both cases the mocked function gets called by the tested code.

self.mock_callable(
    mock_service, "func"
).for_call(expected_request).to_return_value(None).and_assert_called_once()
# passes, all good

self.mock_callable(
    mock_service, "func"
).for_call(expected_request).and_assert_called_once()
# error: UnexpectedCallReceived: <StrictMock ...>, 'function_name': Excepted not to be called!

I'm not sure if omitting to_return_value should be allowed or not, but it looks like the error message is a bit detached from the expectations anyway. The error message says Excepted not to be called! when clearly I have an .and_assert_called_once().

(Btw I just noticed there is a typo in the error message Excepted -> Expected.)

mock_callable: Allow usage of not called without behavior defined.

Allow usage of:

and_assert_called_exactly(0)
and_assert_not_called()

without defining a explicit behavior. Currently mock_callable will force a behavior to be defined for those cases, even though it will never be executed.

Implement StrictMock method return type validation

This feature is currently missing. This branch:
https://github.com/facebookincubator/TestSlide/pull/new/strict_mock_return_type_validation
has tests covering those cases.

mock_callable(): Fail on the spot when an extra call is made

When we use:

and_assert_called_exactly()
and_assert_called_once()
and_assert_called_twice()
and_assert_called_at_most()
and_assert_not_called()

We want that when an extra call is made, that it fails right on, as opposed to currently, that it only fails the after assertion.

Add type checking for the build

Let's add type checking for the build process. We can start adding typing to all functions on top of that.

Automate release process

This is mostly manual as described at RELEASE.md. We need to:

• Using semantic versioning, be able to do major, minor and patch releases, calculating the new version based on the previous.
• Unify version at setup.py and docs/conf.py.
• Ensure Travis CI build is OK for new release.
• Ensure readthedocs.org build is OK for each release.

disabling_signature_validation=True breaks callable arguments

This:

import asyncio
import testslide

class AsyncContextManager:
  async def __aenter__(self):
    return self

  async def __aexit__(self, *args):
    pass

mock = testslide.StrictMock(
  template=AsyncContextManager,
  default_context_manager=True,
  signature_validation=False,
)

async def test():
  async with mock as cm:
    pass

asyncio.run(test(), debug=True)

Fails:

TypeError: aenter() takes 0 positional arguments but 1 was given

We need to:

• Refactor the tests, so we get all test cases to run with disabling_signature_validation=True as well (let's see what else maybe broken).
• Fix what needs fixing.

Add code coverage test for TestSlide's own code

Add code coverage for executed tests and badge for https://coveralls.io/.

Tests are executed in two steps:

• First, some "core" components (including the DSL) are tested with Python's unittest: make unittest_tests.
• Second, most tests run using TestSlide DSL itself: make testslide_tests.

Coverage report must merge both coverage reports into a single report.

Add argument type checking

mock_callable / mock_async_callable / mock_constructor / StrictMock validate call arguments, and raise TypeError if the mock receives a call with arguments that does not match the template's method signature (implemented here https://fburl.com/6siit7j3).

Python has typing information available for callables, we can extend the signature validation to also do type validation. This would allow a test to fail on wrong types being passed, which aids development.

Fail mock_callable / mock_constructor configuration if it already received a call

By design, mock_callable() should be used like this:

self.mock_callable(target, 'attr')\
  .for_call(something)\
  .to_return_value(None)\
  .and_assert_called_once()
do_target_attr_call()

But sometimes people misuse it like:

mock = self.mock_callable(target, 'attr')\
  .for_call(something)\
  .to_return_value(None)
do_target_attr_call()
mock.and_assert_called_once()

Although this works right now, it is highly undesirable, as this undocumented usage pattern blocks implementation of #3, which can allow unexpected calls to fail on the spot, not after the test finishes.

Let's make mock_callable / mock_constructor fail if it is attempted to receive extra configuration, after it received the first call, which should prevent and educate users on this.

Add options to disable type checking

Currently StrictMock reuses the signature validation flag signature_validation to enable/disable type checking and mock_callable(), mock_async_callable() and mock_constructor() have no option to disable type checknig.

Let's add a type_validation=True option to allow users to disable type checking:

• StrictMock: at __new__ & __initt__ plus a test to cover this.
• mock_callable(): add option plus a test to cover this.
• mock_async_callable() and mock_constructor(): as they re-use mock_callable engine, make sure they're passing the option along.

Add type validation for `mock_constructor`

Currently there's no type checking for mock_constructor, let's add it, and include 2 tests at least:

• Fails with wrong type.
• Passes with wrong type and type_validation=False.

Add argument matchers for mock_callable / mock_constructor

Add things like:

class regex_matching(object):
    def __init__(self, pattern):
        self.pattern = pattern

    def __eq__(self, other):
        if not isinstance(other, str):
            return False
        return re.match(self.pattern, other)

    def __str__(self):
        return self.pattern

    def __repr__(self):
        return f"regex_matching({repr(self.pattern)})"

self.mock_callable(some_object, "some_method")\
    .for_call(regex_matching("^some.*thing$"))\
    .to_return_value(None)

This allows for more relaxed mock_callable validation, meaning less need to change test code when things are refactored.

Other argument matchers to consider implementing:
https://relishapp.com/rspec/rspec-mocks/v/3-8/docs/setting-constraints/matching-arguments
https://relishapp.com/rspec/rspec-expectations/v/3-8/docs/built-in-matchers

Update documentation

• StrictMock
  • type_validation attribute (at __init__ and docs/)
• mock_callable, mock_async_callable, mock_constructor:
  • • type_validation attribute.

Validate that contexts are not async defs

This should yield a failure:

@context
def top(context):
  @context.sub_context
  async def sub(context):
     ...

But is currently being silently ignored. Let's turn this into a failure.

Check:

• @context
• @context.sub_context
• @context.shared_context

Support for async functions with mock_callable()

Support for async functions with mock_callable().

Here's a half working patch for that, as a starting point.

To prove that any solution works, all relevant tests at tests/mock_callable_testslide.py must be updated to to cover async cases (eg: async function at module, async instance method, async class method etc).

PS: previous PR #8

Create pytest-testslide

Add TestSlide support for pytest:

• Create the PyPI package.
• Add integration tests to Makefile for building, installing and using it.
• Add documentation at doc/ regarding pytest integration.

StrictMock can be used simply by importing it, just document it.

Patching tools require special pytest integration, which can be done with @pytest.fixture. Naming the fixture testslide makes sense:

def test_something(testslide):
  testslide.mock_callable(...)

but this contrasts with its usage at TestSlide's DSL and Python's Unittest, which access them via self.mock_callable(...), so this is TBD.

The fixture implementation can be copy / pasta from testslide.TestCase.

[easy] Tag for 1.3.3 missing

Should be commit ab3b8b6 I think.

Async support for the DSL

Add async support for DSL:

@context.example
async def async_example(self):
  pass

@context.before
async def async_before(self):
  pass

@context.after
async def async_after(self):
  pass

@context.around
async def async_around(self, example):
  example()

@context.memoize
async def async_memoize(self):
  pass

@contex.function
async def asyc_function(self):
  pass

Fix broken link to Github from readthedocs

From https://testslide.readthedocs.io/en/latest/, the link to "Edit on Github" is currently broken.