facebook / testslide Goto Github PK
View Code? Open in Web Editor NEWA Python test framework
Home Page: https://github.com/facebook/TestSlide
License: MIT License
A Python test framework
Home Page: https://github.com/facebook/TestSlide
License: MIT License
Currently the test runner does not respect https://docs.python.org/3/library/unittest.html#setupmodule-and-teardownmodule, this needs fixing.
This is somewhat related to #51 , as we should probably reuse Python's test loader / runner, and hook TestSlide into them, so we can avoid similar potential issues in the future.
This will likely require implementing @context.before_once
and @context.after_once
at TestSlide's DSL.
before_once
is not documented and has no test case covering failures. Both needs addressing.
When a before_once
hook fails, the expectation is that all tests that depends on it should fail with the same signal. I believe currently, the first test will get the failure, but others will not run the hook, and execute as if the hook worked.
Support for async functions with mock_callable()
.
Here's a half working patch for that, as a starting point.
To prove that any solution works, all relevant tests at tests/mock_callable_testslide.py
must be updated to to cover async cases (eg: async function at module, async instance method, async class method etc).
PS: previous PR #8
Currently there's no type checking for mock_constructor
, let's add it, and include 2 tests at least:
type_validation=False
.See the following examples. In both cases the mocked function gets called by the tested code.
self.mock_callable(
mock_service, "func"
).for_call(expected_request).to_return_value(None).and_assert_called_once()
# passes, all good
self.mock_callable(
mock_service, "func"
).for_call(expected_request).and_assert_called_once()
# error: UnexpectedCallReceived: <StrictMock ...>, 'function_name': Excepted not to be called!
I'm not sure if omitting to_return_value
should be allowed or not, but it looks like the error message is a bit detached from the expectations anyway. The error message says Excepted not to be called!
when clearly I have an .and_assert_called_once()
.
(Btw I just noticed there is a typo in the error message Excepted
-> Expected
.)
Add check if mock_callable / mock_constructor are being called without proper setup. For example, if testslide.TestCase is inherited, and setUp() is overloaded, but it does not call super().setUp(), mock_callable & mock_constructor fail with weird errors.
from testslide.dsl import context
class MyClass(object):
__slots__ = ["value"]
def __init__(self, value):
self.value = value
@context # noqa
def mock_constructor_with_wrapper_test(context):
@context.sub_context
def intercept_context(context):
@context.before
def intercept_call_to_constructer(self):
def wrapper(original_callable, *args, **kwargs):
obj = original_callable(*args, **kwargs)
obj.value += 1
return obj
self.mock_constructor(
__name__, "MyClass"
).with_wrapper(wrapper).and_assert_called_once()
@context.example
def call_constructor(self):
obj = MyClass(value=1)
self.assertEqual(obj.value, 2)
@context.example
def call_constructor(self):
obj = MyClass(value=1)
self.assertEqual(obj.value, 1)
Error
Failures:
0) mock constructor with wrapper test, intercept context: call constructor
1) TypeError: original_callable() takes at least 1 argument (0 given)
File "experimental/shihaoxu/testslide/mock_constructor_with_wrapper_test.py", line 31, in call_constructor
obj = MyClass(value=1)
File "experimental/shihaoxu/testslide/mock_constructor_with_wrapper_test.py", line 21, in wrapper
obj = original_callable(*args, **kwargs)
Add code coverage for executed tests and badge for https://coveralls.io/.
Tests are executed in two steps:
make unittest_tests
.make testslide_tests
.Coverage report must merge both coverage reports into a single report.
TestSlide exceptions often have a multi-line message. While TestSlide properly shows them in full, Python unittest test runner seem to swallow all but the first exception line. This makes it difficult for users to understand why things fail.
This can potentially be fixed by detecting when not running with TestSlide test runner (or when running with Python unttest test runner) and change all exceptions to be single line.
CC @pallotron
TestSlide loads TestCase classes using this logic. Upstream Python, does it like this. This means, that If the module has this:
class Outer:
class Inner(TestCase):
def test_something(self):
pass
Python won't consider Inner.test_something
as part of the test, but TestSlide will.
This can be easily fixed by refactoring the logic at TestSlide's side to match Python's, however, it won't cover other cases such as load_tests()
.
The proper fix for this, is to use Python's TestLoader
in the same fashion as Python unittent does form its CLI.
Given:
In [1]: from testslide import StrictMock
In [2]: class A:
...: def __len__(self):
...: return 0
In [3]: m1 = StrictMock(A)
In [4]: m2 = StrictMock(A)
bool(), from its docs, looks for __bool__
, which is not defined, then it looks for __len__
, which is defined in the template class, thus it should be invoked. Given the default behavior of StrictMock, it should raise UndefinedBehavior
on __len__
access, bool(m1)
should raise, but it does not:
In [5]: bool(m1)
Out[5]: True
In [6]: bool(m2)
Out[6]: True
Instead, it returns true, as the default value for bool
(as if both __bool__
and __len__
are not defined). If we define a behavior for __len__
, things works as expected:
In [7]: m1.__len__ = lambda: 0
In [8]: bool(m1)
Out[8]: False
However, it breaks other instances:
In [9]: bool(m2)
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
<ipython-input-9-a2b1c204a39c> in <module>
----> 1 bool(m2)
~/.pyenv/versions/3.6.6/lib/python3.6/site-packages/testslide/strict_mock.py in __get__(self, instance, _owner)
119 else:
120 raise AttributeError(
--> 121 "{}:\n Object has no attribute '{}'".format(repr(instance), self.name)
122 )
123
AttributeError: <StrictMock 0x7F27960CD4E0 template=__main__.A - <ipython-input-4-9a7b27e5dca0>:1>:
Object has no attribute '__len__'
This seems to happen because the way bool
is imlpemented. When we do m1.__len__ = lambda: 0
, under the hood, StrictMock puts _DescriptorProxy at StrictMock
dictionary. _DescriptorProxy
correctly implements __get__
to give the correct value for each instance, but bool()
does not follow the descriptor protocol, and simply checks if the class dictionary has __len__
. If it happens to implement __get__
(as _DescriptorProxy
does), and the instance in particular does not have the attribute, it simply raises the AttributeError
from _DescriptorProxy
.
From __bool__()
docs, I'd expect for it do do something like this:
def bool(obj):
if hasattr(obj, "__bool__"):
return getattr(obj, "__bool__")()
if hasattr(obj, "__len__"):
return not (return getattr(obj, "__len__")() == 0)
return True
Thus, if a particular instance does not have __len__
(eg: __get__
raises AttributeError
), it sholud work. However, it seems it is doing this:
def bool(obj):
if "__bool__" in type(obj).__dict__:
return getattr(obj, "__bool__")()
if "__bool__" in type(obj).__dict__:
return not (return getattr(obj, "__len__")() == 0)
return True
The workaround at StrictMock's end (to not depend on any upstream patch), can be:
StrictMock.__new__
to dynamically create a subclass of StrictMock for each instance.__len__
, __bool__
etc) accordingly to the template given.This should make magic methods per StrictMock
instance, thus covering the bool()
case, and any other similar scenario. This is a similar solution to what mock_constructor() does to go workaround a similar __new__
bug in Python 3.x.
Add async support for DSL:
@context.example
async def async_example(self):
pass
@context.before
async def async_before(self):
pass
@context.after
async def async_after(self):
pass
@context.around
async def async_around(self, example):
example()
@context.memoize
async def async_memoize(self):
pass
@contex.function
async def asyc_function(self):
pass
If we have a shared context defined in one file:
def some_shared_context(context):
@context.before
def before(self):
pass
and we want to use it at different files, we need to do this:
from wherever import some_shared_context
@context
def first(context):
context.shared_context(some_shared_context)
context.merge_context("some shared context")
and on the other file:
from wherever import some_shared_context
@context
def second(context):
context.shared_context(some_shared_context)
context.merge_context("some shared context")
We can somewhat easily modify the merge_context
function to accept the function directly, simplifying things:
from wherever import some_shared_context
@context
def first(context):
context.merge_context(some_shared_context)
and on the other file
from wherever import some_shared_context
@context
def second(context):
context.merge_context(some_shared_context)
When we use:
and_assert_called_exactly()
and_assert_called_once()
and_assert_called_twice()
and_assert_called_at_most()
and_assert_not_called()
We want that when an extra call is made, that it fails right on, as opposed to currently, that it only fails the after assertion.
mock_callable / mock_async_callable / mock_constructor / StrictMock validate call arguments, and raise TypeError
if the mock receives a call with arguments that does not match the template's method signature (implemented here https://fburl.com/6siit7j3).
Python has typing information available for callables, we can extend the signature validation to also do type validation. This would allow a test to fail on wrong types being passed, which aids development.
We currently only type check callable arguments from mock_callable()
, mock_async_callable()
, mock_constructor()
and StrictMock
. Let's also check the return value of the call if type checking is available.
Currently the test execution time discards the time spent in between the process started and the tests starting to run. This discarded time is often associated with slow to import modules, which are hidden from users.
Let's change this logic to consider the start time as the moment the process initiates, rather than the moment the tests start to run.
How to skip a TestSlide example, if I known in certain environment the test is expceted to fail or does not make sense.
Like the Python unittest module API.
https://fburl.com/ljc095sq
We currently only check type for callable attributes for StrictMock
instances.
Let's add type checking when setting non-callable attributes.
This feature is currently missing. This branch:
https://github.com/facebookincubator/TestSlide/pull/new/strict_mock_return_type_validation
has tests covering those cases.
StrictMock
type_validation
attribute (at __init__
and docs/
)mock_callable
, mock_async_callable
, mock_constructor
:
type_validation
attribute.mock_constructor has validation logic to find out if the class being mocked has other references that are not from the module where it lives. This is important, as there's no guarantees it is gonna work otherwise.
mock_callable has the same potential problem when mocking functions at modules for example: if there are left over references such as from module import func
, it will not work.
Let's add such validation to mock_callable so users don't shoot themselves in the foot.
This logic can be added here.
StrictMock has tons of mechanics under the hood, and some of them might break with async.
Let's add test cases for async stuff at tests/strict_mock_testslide.py
, such as:
These tests should cover the same ground we already have there, such as signature validation.
This:
import asyncio
import testslide
class AsyncContextManager:
async def __aenter__(self):
return self
async def __aexit__(self, *args):
pass
mock = testslide.StrictMock(
template=AsyncContextManager,
default_context_manager=True,
signature_validation=False,
)
async def test():
async with mock as cm:
pass
asyncio.run(test(), debug=True)
Fails:
TypeError: aenter() takes 0 positional arguments but 1 was given
We need to:
disabling_signature_validation=True
as well (let's see what else maybe broken).Python 2 is dying soon, let's drop support for in and open the road for async stuff at #19.
mock_async_callable()
is the way to go for callables that return coroutines. Let's add checks to mock_callable()
for the behaviors that return something (return_value
, return_values
, with_implementation
etc) so that it fails if the return is a coroutine.
from testslide.dsl import context
@context # noqa
def nest_context_test(context):
@context.shared_context
def shared_context(context):
@context.before
def setup_value(self):
self.value = 1
@context.sub_context
def sub_context(context):
context.nest_context("shared context")
@context.example
def assert_value_accessible(self):
self.value
nest context test
sub context
assert value accessible: AttributeError: Context 'sub context' has no attribute 'value'
Failures:
1) nest context test, sub context: assert value accessible
1) AttributeError: Context 'sub context' has no attribute 'value'
File "experimental/shihaoxu/testslide/nest_context_test.py", line 18, in assert_value_accessible
self.value
Put check in place for all DSL functions to ensure they're declared with proper arguments.
This should be OK:
@context.sub_context
def when_something(context):
...
but this should yield a failure:
@context.sub_context
def when_something(self):
...
This avoids common user mistakes.
The function signatures can be checked with https://docs.python.org/3/library/inspect.html#inspect.signature.
Allow usage of:
and_assert_called_exactly(0)
and_assert_not_called()
without defining a explicit behavior. Currently mock_callable will force a behavior to be defined for those cases, even though it will never be executed.
The current error message:
BaseException: Attribute getting after the class has been used with mock_constructor() is not supported! After using mock_constructor() you must get a pointer to the new mocked class...
Is hard to understand. Let's:
https://testslide.readthedocs.io/en/$VERSION/mock_constructor/index.html
.Generically speaking, mocking private interfaces is bad it breaks the Liskov substitution principle: if the private interface of the class changes, tests break. Using dependency injection or the registry pattern are ways around this.
Let's help users write better tests, and fail mock_callable
/ mock_async_callable
if the method being mocked is private (eg: _some_method
), by raising ValueError
. The error message should educate users on why this is a bad pattern, how to go around it, and if it is a very special snow flake, they can still do it by adding the parameter allow_private=True
.
Currently StrictMock
reuses the signature validation flag signature_validation
to enable/disable type checking and mock_callable()
, mock_async_callable()
and mock_constructor()
have no option to disable type checknig.
Let's add a type_validation=True
option to allow users to disable type checking:
StrictMock
: at __new__
& __initt__
plus a test to cover this.mock_callable()
: add option plus a test to cover this.mock_async_callable()
and mock_constructor()
: as they re-use mock_callable engine, make sure they're passing the option along.This should work:
from testslide import StrictMock
class Template:
def __eq__(self, other):
return False
class TemplateStrictMock(StrictMock):
def __init__(self):
super().__init__(template=Template)
def __eq__(self, other):
return False
mock = TemplateStrictMock()
assert (mock == mock) is False
But is currently failing with:
testslide.strict_mock.UndefinedAttribute: '__eq__' is not set.
<StrictMock 0x7F6A9E5F4D90 template=__main__.Template test.py:9> must have a value set for this attribute if it is going to be accessed.
StrictMock raises when an attribute is accessed, but has not been previously defined
The message simply mentions "setting" the attribute. Let's improve this message to, if the mock has a template, and the attribute is callable, suggest using mock_callable / mock_async_callable to set it.
Sometimes it is hard to understand what hooks (before / after / around) are being executed. It would be nice to have a --debug option that would make the test runner show what hook is being executed, something like this perhaps:
Some Context
before: do_something() file.py:33
Some Example
example: some_example() file.py:38
PASS
after: cleanup() file.py:45
Add TestSlide support for pytest:
Makefile
for building, installing and using it.doc/
regarding pytest integration.StrictMock can be used simply by importing it, just document it.
Patching tools require special pytest integration, which can be done with @pytest.fixture. Naming the fixture testslide
makes sense:
def test_something(testslide):
testslide.mock_callable(...)
but this contrasts with its usage at TestSlide's DSL and Python's Unittest, which access them via self.mock_callable(...)
, so this is TBD.
The fixture implementation can be copy / pasta from testslide.TestCase.
By design, mock_callable() should be used like this:
self.mock_callable(target, 'attr')\
.for_call(something)\
.to_return_value(None)\
.and_assert_called_once()
do_target_attr_call()
But sometimes people misuse it like:
mock = self.mock_callable(target, 'attr')\
.for_call(something)\
.to_return_value(None)
do_target_attr_call()
mock.and_assert_called_once()
Although this works right now, it is highly undesirable, as this undocumented usage pattern blocks implementation of #3, which can allow unexpected calls to fail on the spot, not after the test finishes.
Let's make mock_callable / mock_constructor fail if it is attempted to receive extra configuration, after it received the first call, which should prevent and educate users on this.
Should be commit ab3b8b6 I think.
Add things like:
class regex_matching(object):
def __init__(self, pattern):
self.pattern = pattern
def __eq__(self, other):
if not isinstance(other, str):
return False
return re.match(self.pattern, other)
def __str__(self):
return self.pattern
def __repr__(self):
return f"regex_matching({repr(self.pattern)})"
self.mock_callable(some_object, "some_method")\
.for_call(regex_matching("^some.*thing$"))\
.to_return_value(None)
This allows for more relaxed mock_callable validation, meaning less need to change test code when things are refactored.
Other argument matchers to consider implementing:
https://relishapp.com/rspec/rspec-mocks/v/3-8/docs/setting-constraints/matching-arguments
https://relishapp.com/rspec/rspec-expectations/v/3-8/docs/built-in-matchers
From https://testslide.readthedocs.io/en/latest/, the link to "Edit on Github" is currently broken.
It currently has no support for classes with __new__
, and will reject patching those.
IIRC, it was tough getting around Python idiosyncrasies to make mock_constructor
work, and I willingly decided not to implement __new__
at first to simplify things, but this is potentially doable.
Let's add type checking for the build process. We can start adding typing to all functions on top of that.
This is mostly manual as described at RELEASE.md. We need to:
TestSlide has a hacky support for TestCase.assert*
methods. While this works, it prevents users from tweaking things such as maxDiff
.
The proposed solution to fix this is:
self._test_case
.TestCase integration
, perhaps by adding doing self.add_test_case()
at Context.__init__
, and assigning it to self.test_case
.__getattr__
logic, but forward it to self.test_case
.With that, users should be able to do self.test_case.maxDiff = new_value
.
This PR added type checking for function calls. It works as expected, but it is failing when a call is made using a mock, but it should not.
Suggestion on how to fix this:
StrictMock
, get the given template
(IIRC mock_callable already does this, there's a trick there).spec
.Using that:
For unittest.TestCase classes that are decorated with https://docs.python.org/3/library/unittest.mock.html#patch, when we do ftest or xtest, the decorator does to patch these methods, and we get a TypeError when running the test.
This happens because @patch is evaluated on the class loading, and it patches all methods there that begin with "test". Since ftest and xtest don't match the search, they're not decorated, and become broken.
Both cases below must raise when declared, not when executed:
context.memoize(
not_callable=True,
callable_but_wrong_signature=lambda: False,
)
This should work:
from testslide import StrictMock
from unittest.mock import Mock
mock = StrictMock(template=Template)
mock.whatever = Mock()
mock.whatever(1)
mock.whatever.assert_called_once_with(1)
This:
testslide.lib._validate_function_signature(
object.__new__, (1, 2, 3), {"four": 5, "six": 6}
)
should pass, but is failing with:
IndexError: list index out of range
File "testslide/lib.py", line 78, in _validate_function_signature
argname = list(signature.parameters.keys())[idx]
This is blocking #141 .
This should yield a failure:
@context
def top(context):
@context.sub_context
async def sub(context):
...
But is currently being silently ignored. Let's turn this into a failure.
Check:
@context
@context.sub_context
@context.shared_context
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.