simplistix / sybil Goto Github PK
View Code? Open in Web Editor NEWAutomated testing for the examples in your documentation.
Home Page: https://sybil.readthedocs.io/en/latest/
License: Other
Automated testing for the examples in your documentation.
Home Page: https://sybil.readthedocs.io/en/latest/
License: Other
I am looking sybil as a replacement for the built in doctest collector in pytest, so I can more easily extend the functionality with setup code blocks and namespaces, all the good things sybil supports.
The one thing I am not clear about is if sybil can parse blocks from docstrings inside the python source? Which is something the doctest collector can do.
I'd love to have support for the testcode
and testoutput
directives from sphinx.ext.doctest
since we already have a good amount of long-form documentation that uses them and like to be able to add some explanatory text before the output, which is harder to do with doctest
blocks. You said in the docs adding more doctest features wouldn't be hard, so thought I'd ask! Thanks for your work!
I see that there is an existing branch for this, https://github.com/simplistix/sybil/tree/typing. Let me know if help is welcome on that. It would be nice to not have to ignore_missing_imports = true
for sybil
for strict mypy
to work on my projects.
I used Sybil to test a simple docstring, but got an unexpected error.
conftest.py
from sybil import Sybil
from sybil.parsers.rest import DocTestParser
pytest_collect_file = Sybil(
parsers=[DocTestParser()],
patterns=['*.py'],
).pytest()
module.py
def f():
"""Do something.
Example:
>>> x = 3
>>> x
3
"""
pass
Traceback
❯ pytest
================================================================================================ test session starts ================================================================================================
platform darwin -- Python 3.10.9, pytest-7.2.2, pluggy-1.0.0
rootdir: /Users/balaji/Documents/sybil-example, configfile: pytest.ini
collected 0 items / 1 error
====================================================================================================== ERRORS =======================================================================================================
____________________________________________________________________________________________ ERROR collecting module.py _____________________________________________________________________________________________
.venv/lib/python3.10/site-packages/sybil/integration/pytest.py:104: in collect
self.document = self.sybil.parse(Path(self.fspath.strpath))
.venv/lib/python3.10/site-packages/sybil/sybil.py:143: in parse
return type_.parse(str(path), *self.parsers, encoding=self.encoding)
.venv/lib/python3.10/site-packages/sybil/document.py:54: in parse
for region in parser(document):
.venv/lib/python3.10/site-packages/sybil/parsers/abstract/doctest.py:41: in __call__
source, options, want, exc_msg = self._parse_example(m, name, lineno)
/opt/homebrew/Cellar/[email protected]/3.10.9/Frameworks/Python.framework/Versions/3.10/lib/python3.10/doctest.py:721: in _parse_example
self._check_prefix(want_lines, ' '*indent, name,
/opt/homebrew/Cellar/[email protected]/3.10.9/Frameworks/Python.framework/Versions/3.10/lib/python3.10/doctest.py:806: in _check_prefix
raise ValueError('line %r of the docstring for %s has '
E ValueError: line 8 of the docstring for /Users/balaji/Documents/sybil-example/module.py has inconsistent leading whitespace: ' """'
============================================================================================== short test summary info ==============================================================================================
ERROR module.py - ValueError: line 8 of the docstring for /Users/balaji/Documents/sybil-example/module.py has inconsistent leading whitespace: ' """'
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
================================================================================================= 1 error in 0.05s ==================================================================================================
Dependencies
❯ pip freeze
attrs==22.2.0
exceptiongroup==1.1.0
iniconfig==2.0.0
packaging==23.0
pluggy==1.0.0
pytest==7.2.2
sybil==4.0.1
tomli==2.0.1
I spoke with Chris about this elsewhere, discovered another need.
Sometimes you have chunks of samples in docs that you don't want executed. They're un-runnable pseudocode. It would be nice to have a way to say "don't test these". Chris suggested a couple of variations of Sphinx directives.
Other times you want to skip when tox tests under certain interpreters. For example I have a snippet with a dataclass, in a package with optional dataclass support. I want the equivalent of:
@pytest.mark.skipif(sys.version_info < (3, 6), reason="requires python3.6 or higher")
Although it would be unique to pytest, it would be cool to have a Sphinx directive for skipif with the directive arguments matching the signature of skipif.
Hi!
it would be cool to have a quick way to go to https://sybil.readthedocs.io/en/latest/ from here. The project link is still unfilled and might make a great place for it.
Best, Sebastian
The first example from the documentation does not include the CodeBlockParser()
, so doesn't pick up code-block
directives.
https://sybil.readthedocs.io/en/latest/use.html
Here's what I'm using:
from doctest import ELLIPSIS
from sybil import Sybil
from sybil.parsers.codeblock import CodeBlockParser
from sybil.parsers.doctest import DocTestParser
from sybil.parsers.skip import skip
from sybil.parsers.capture import parse_captures
pytest_collect_file = Sybil(
parsers=[
CodeBlockParser(),
DocTestParser(optionflags=ELLIPSIS),
parse_captures,
skip,
],
pattern="test_*.rst",
).pytest()
P.S. Nice library!
I have .rst
files I'd like tested, but they in child directories under the root. I'd like a pattern of something like '**/*.rst' using the improved glob support in Python post-3.5
If that's not feasible, can path optional be a sequence of places to look?
I am not sure if it is due to me using it wrong or simply a missing check, but i am getting this error:
.direnv/python-3.11.2/lib64/python3.11/site-packages/sybil/integration/pytest.py:104: in collect
self.document = self.sybil.parse(Path(self.fspath.strpath))
.direnv/python-3.11.2/lib64/python3.11/site-packages/sybil/sybil.py:144: in parse
return type_.parse(str(path), *self.parsers, encoding=self.encoding)
.direnv/python-3.11.2/lib64/python3.11/site-packages/sybil/document.py:196: in parse
for start, end, text in cls.extract_docstrings(document.text):
.direnv/python-3.11.2/lib64/python3.11/site-packages/sybil/document.py:182: in extract_docstrings
punc_size = punc.end() - punc.start()
E AttributeError: 'NoneType' object has no attribute 'end'
While investigating, it seems to be that the result of the regex search is never checked before accessing it:
https://github.com/simplistix/sybil/blob/master/sybil/document.py#L183
Thus it should fail for any file that doesn't have the expected documentation type.
Several tests fail when building sybil-2.0.1 against python-3.10.0_beta2, attaching the output. Haven't checked if this is only a problem with tests or an actual issue with Sybil.
sybil-2.0.1-test.log
I have a section of docs which use variable annotations: https://github.com/pauleveritt/wired/tree/dataclasses_tutorial/docs/tutorials/dcdi
The index.rst
has a skip at the top. However, simply parsing the file under Python 3.5 causes a SyntaxError
. I need to teach the Sybil collector to skip it.
I tried putting a conftest.py
in that directory which sniffs the Python interpreter version and does an exclude. However, It seems to augment the Sybil instance, not replace it.
Hi!
Sybil is awesome! I would like to use Sybil for testing code examples in the documentation of plone.api https://github.com/plone/plone.api.
Plone needs a TestCase layer with an app instance to work on. I understand that Sybil can be initialized with a setup(namespace)
. But it's quite tricky to pass the app untouched from example to example. So after several approaches I ended up with the following suggestion: Allow the sybil.integration.unitttest.TestCase to be initialized not only with a single example, but optional also with a list of examples.
What's your opinion?
Reproducer:
# conftest.py
from sybil import Sybil
from sybil.parsers.rest import DocTestParser
sybil = Sybil(
parsers=[DocTestParser()],
patterns=["*.py"],
)
pytest_collect_file = sybil.pytest()
In pkg/foo.py
(note: intentionally no pkg/__init__.py
file present):
def func():
"""
>>> 1 + 1
2
"""
When letting doctest import the packages it works (although the test is executed twice, once by pytest doctest plugin and again by sybil)
$ pytest --doctest-modules -v
====================================== test session starts =======================================
platform darwin -- Python 3.11.4, pytest-7.4.0, pluggy-1.2.0 -- /private/tmp/s/.venv/bin/python
cachedir: .pytest_cache
rootdir: /private/tmp/s
collected 2 items
pkg/foo.py::line:3,column:1 PASSED [ 50%]
pkg/foo.py::foo.func PASSED [100%]
======================================= 2 passed in 0.01s ========================================
However when disabling the doctest plugin, as the sybil docs advise to do, the imports fail:
$ pytest -p no:doctest
=========================================================== test session starts ===========================================================
platform darwin -- Python 3.11.4, pytest-7.4.0, pluggy-1.2.0
rootdir: /private/tmp/s
collected 1 item
pkg/foo.py F [100%]
================================================================ FAILURES =================================================================
_________________________________________________________ foo.py line=3 column=1 __________________________________________________________
self = <sybil.document.PythonDocStringDocument object at 0x10a329d10>
example = <Example path=/private/tmp/s/pkg/foo.py line=3 column=1 using <sybil.evaluators.doctest.DocTestEvaluator object at 0x10a346550>>
def evaluator(self, example: Example) -> Optional[str]:
"""
Imports the document's source file as a Python module when the first
:class:`~sybil.example.Example` from it is evaluated.
"""
> module = import_path(Path(self.path))
.venv/lib/python3.11/site-packages/sybil/document.py:153:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
path = PosixPath('/private/tmp/s/pkg/foo.py')
def import_path(path: Path):
container = path
while True:
container = container.parent
if not (container / INIT_FILE).exists():
break
relative = path.relative_to(container)
if relative.name == INIT_FILE:
parts = tuple(relative.parts)[:-1]
else:
parts = tuple(relative.parts)[:-1]+(relative.stem,)
module = '.'.join(parts)
try:
return importlib.import_module(module)
except ImportError as e:
> raise ImportError(
f'{module!r} not importable from {path} as:\n{type(e).__name__}: {e}'
) from None
E ImportError: 'foo' not importable from /private/tmp/s/pkg/foo.py as:
E ModuleNotFoundError: No module named 'foo'
.venv/lib/python3.11/site-packages/sybil/python.py:40: ImportError
========================================================= short test summary info =========================================================
FAILED pkg/foo.py::line:3,column:1 - ImportError: 'foo' not importable from /private/tmp/s/pkg/foo.py as:
============================================================ 1 failed in 0.02s ============================================================
I try to use Sybil with a Markdown file that contains normal Python code block and doctests.
Testing doctests does not work in several ways though and I am not sure whether I am doing something wrong, or whether there is a (documentation) issue in Sybil.
I am using:
docs/index.md
# Test
## 1
```
>>> print(2)
1
```
## 2
```python
>>> print(2)
2
```
## 3
```{code-block} python
>>> print(3)
3
```
## 4
```{doctest}
>>> print(4)
4
```
docs/conf.py
project = "Test"
author = "Monty"
release = "1.0.0"
version = "1.0"
extensions = [
"myst_parser",
]
templates_path = ["_templates"]
exclude_patterns = ["_build", "Thumbs.db", ".DS_Store"]
conftest.py
from doctest import ELLIPSIS
from sybil import Sybil
from sybil.parsers.myst import DocTestDirectiveParser, PythonCodeBlockParser
from sybil.parsers.rest import DocTestParser
pytest_collect_file = Sybil(
parsers=[
DocTestParser(optionflags=ELLIPSIS),
DocTestDirectiveParser(optionflags=ELLIPSIS),
PythonCodeBlockParser(),
],
patterns=["*.md"],
).pytest()
{doctest}
directive and will not show it, but the docs state otherwisePythonCodeBlockParser
, only examples 2 and 3 are collected, but sucessfully testedDocTestDirectiveParser
, only example 4 is collected, but successfully tested. But it leads to Sphinx errors (see 1.).DocTestParser
, all four examples are collected, but doctest considers the closing three backticks as part of the example and fails with sth. like 1\n'''
!= 1`.DocTestParser
and PythonCodeBlockParser
(which I'd like to do in my real project, Sybil fails because of overlapping regions. I can "fix" this by removing the python
/{code-block} python
from examples 2 and 3, but then I get the same error as in point 4. again.I am getting some test failures, and tried to understand since Travis is passing, but still didn't figure out. Any hints?
=================================== FAILURES ===================================
________________________________ test_unittest _________________________________
capsys = <_pytest.capture.CaptureFixture object at 0x7f70d00cfd68>
def test_unittest(capsys):
runner = TextTestRunner(verbosity=2, stream=sys.stdout)
path = join(functional_test_dir, 'functional_unittest')
main = unittest_main(
exit=False, module=None, testRunner=runner,
argv=['x', 'discover', '-v', '-t', path, '-s', path]
)
out, err = capsys.readouterr()
assert err == ''
out = Finder(out)
> common_checks(out)
tests/test_functional.py:144:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/test_functional.py:120: in common_checks
out.then_find('sybil teardown 4\nsybil setup')
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <tests.test_functional.Finder object at 0x7f70d00fbfd0>
substring = 'sybil teardown 4\nsybil setup'
def then_find(self, substring):
> assert substring in self.text[self.index:]
E AssertionError: assert 'sybil teardown 4\nsybil setup' in 'fail.rst,line:12,column:1 ... 3\nok\nsybil teardown 4\n\n============================================================...------------------------------------------------------------\nRan 8 tests in 0.001s\n\nFAILED (failures=1, errors=1)\n'
tests/test_functional.py:19: AssertionError
__________________________________ test_nose ___________________________________
capsys = <_pytest.capture.CaptureFixture object at 0x7f70d00e4c18>
def test_nose(capsys):
class ResultStoringMain(NoseMain):
def runTests(self):
self.testRunner = NoseRunner(stream=sys.stdout,
verbosity=self.config.verbosity,
config=self.config)
self.result = self.testRunner.run(self.test)
main = ResultStoringMain(
module=None,
argv=['x', '-vs', join(functional_test_dir, 'nose')]
)
assert main.result.testsRun == 9
assert len(main.result.failures) == 1
assert len(main.result.errors) == 1
out, err = capsys.readouterr()
assert err == ''
out = Finder(out)
> common_checks(out)
tests/test_functional.py:170:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/test_functional.py:120: in common_checks
out.then_find('sybil teardown 4\nsybil setup')
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <tests.test_functional.Finder object at 0x7f70d0085198>
substring = 'sybil teardown 4\nsybil setup'
def then_find(self, substring):
> assert substring in self.text[self.index:]
E AssertionError: assert 'sybil teardown 4\nsybil setup' in 'fail.rst,line:12,column:1 ... 3\nok\nsybil teardown 4\nnose.test_other.test_it ... ok\n\n============================...------------------------------------------------------------\nRan 9 tests in 0.001s\n\nFAILED (errors=1, failures=1)\n'
tests/test_functional.py:19: AssertionError
_________________ TestSybil.test_all_paths_with_base_directory _________________
self = <tests.test_sybil.TestSybil object at 0x7f70d008c828>
def test_all_paths_with_base_directory(self):
sybil = Sybil([parse_for_x, parse_for_y],
path='./samples', pattern='*.txt')
> assert (self._evaluate_examples(self._all_examples(sybil), 42) ==
['X count was 4, as expected',
'Y count was 3, as expected',
'X count was 3 instead of 4',
'Y count was 3, as expected'])
E AssertionError: assert ['X count was... as expected'] == ['X count was ... as expected']
E At index 0 diff: 'X count was 3 instead of 4' != 'X count was 4, as expected'
E Full diff:
E - ['X count was 3 instead of 4',
E + ['X count was 4, as expected',
E 'Y count was 3, as expected',
E - 'X count was 4, as expected',
E + 'X count was 3 instead of 4',...
E
E ...Full output truncated (2 lines hidden), use '-vv' to show
tests/test_sybil.py:217: AssertionError
________________________________ test_namespace ________________________________
capsys = <_pytest.capture.CaptureFixture object at 0x7f70d00b9128>
def test_namespace(capsys):
sybil = Sybil([parse],
path='./samples', pattern='*.txt')
for document in sybil.all_documents():
for example in document:
print(split(example.document.path)[-1], example.line)
example.evaluate()
out, _ = capsys.readouterr()
> assert out.split('\n') == [
'sample1.txt 1',
'[0]',
'sample1.txt 3',
'[0, 14]',
'sample2.txt 1',
'[0]',
'sample2.txt 3',
'[0, 13]',
''
]
E AssertionError: assert ['sample2.txt...', '[0]', ...] == ['sample1.txt ...', '[0]', ...]
E At index 0 diff: 'sample2.txt 1' != 'sample1.txt 1'
E Full diff:
E - ['sample2.txt 1',
E ? ^
E + ['sample1.txt 1',
E ? ^
E + '[0]',...
E
E ...Full output truncated (12 lines hidden), use '-vv' to show
tests/test_sybil.py:246: AssertionError
===================== 4 failed, 44 passed in 0.61 seconds ======================
e.g. powered by: sybil.
I'd put it in my developer-facing docs.
The following put in a MyST Markdown file causes a LexingException
:
`````
this should lex
`````
With the following exception:
sybil.exceptions.LexingException: Could not match '(?<=\\n)```(:?\\n|\\Z)' in /home/dsel/test/test.md:
'`````\n'
Additional backticks are sometimes needed when you want to nest directives or handle literal code that has three backticks.
I'm am working on modernizing my public library, which involves switching over to pytest. I've already made this switch on other libraries, and Sybil has been fantastic for integrating my .rst
file based doctests. Great work @cjw296 !
However, with public
, functions and classes in my using.rst
file need to be defined in a module that ends up in sys.modules
and have __module__
attributes. This doesn't seem to be the case by default. What's especially weird is that functions __module__
attribute are None
, while class __module__
attributes are builtins
. I don't know if that's intentional or just a strange unexpected behavior.
I can't quite work out how to ensure that classes and functions in my using.rst
file get created and attached within a sys.modules
present module. Any clues/help would be greatly appreciated!
https://github.com/cjw296/sybil/blob/63f4d8251f9c8156a36b02da7609d841b3c3faff/sybil/document.py#L39 opens file without specifying an encoding; it means encoding is system-dependent. I think this is the problem which caused the test failure above: your file with a doctest has some specific encoding (e.g. utf-8), it doesn't vary based on machine.
I'm not sure how to fix that properly, but opening with utf8 encoding always could work.
In some docs I tried showing isinstance(x, Protocol)
but this fails, complaining that "__name__"
isn't in in the function's global keys. I believe this is a consequence of how Sybil sets up the doc tests, though I haven't been able to diagnose the specific issue. Other than this, Sybil has been great, thanks for the package!
Traceback
Example at /Users/.../docs/introduction.rst, line 171, column 1 did not evaluate as expected:
Exception raised:
Traceback (most recent call last):
File "/Users/.../lib/python3.10/doctest.py", line 1350, in __run
exec(compile(example.source, filename, "single",
File "<doctest /Users/.../docs/introduction.rst[0]>", line 1, in <module>
isinstance(v, NamedValue)
File "/Users/.../lib/python3.10/typing.py", line 1496, in __instancecheck__
not _allow_reckless_class_checks(depth=2)
File "/Users/.../lib/python3.10/typing.py", line 1473, in _allow_reckless_class_checks
return sys._getframe(depth).f_globals['__name__'] in ['abc', 'functools']
KeyError: '__name__'
MWE
.. code-block:: python
from typing import Protocol
class NamedValue(Protocol):
"""Interface for an object that has a name and a value."""
value: float
name: str
some text
.. code-block:: python
class NamedValueClass1:
def __init__(self, name: str, value: float):
self.name = name
self.value = value
v = NamedValueClass1("foo", 1.0)
::
>>> isinstance(v, NamedValue)
True
Not sure if you're aware, but Sybil causes two warnings on every test run for me:
.tox/py312-tests-optional/lib/python3.12/site-packages/sybil/document.py:3
/Users/hynek/FOSS/svcs/.tox/py312-tests-optional/lib/python3.12/site-packages/sybil/document.py:3: DeprecationWarning: ast.Str is deprecated and will be removed in Python 3.14; use ast.Constant instead
from ast import AsyncFunctionDef, FunctionDef, ClassDef, Module, Expr, Str, Constant
.tox/py312-tests-optional/lib/python3.12/site-packages/sybil/document.py:176
/Users/hynek/FOSS/svcs/.tox/py312-tests-optional/lib/python3.12/site-packages/sybil/document.py:176: DeprecationWarning: ast.Str is deprecated and will be removed in Python 3.14; use ast.Constant instead
isinstance(docstring, Str) or
Pytest 8 was just released and it seems that this breaks sybil:
../../.virtualenvs/typed-settings/lib/python3.12/site-packages/sybil/integration/pytest.py:51: in __init__
self.request_fixtures(sybil.fixtures)
../../.virtualenvs/typed-settings/lib/python3.12/site-packages/sybil/integration/pytest.py:56: in request_fixtures
closure = fm.getfixtureclosure(names, self)
E TypeError: FixtureManager.getfixtureclosure() missing 1 required positional argument: 'ignore_args'
During handling of the above exception, another exception occurred:
../../.virtualenvs/typed-settings/lib/python3.12/site-packages/sybil/integration/pytest.py:120: in collect
yield SybilItem.from_parent(self, sybil=self.sybil, example=example)
../../.virtualenvs/typed-settings/lib/python3.12/site-packages/_pytest/nodes.py:275: in from_parent
return cls._create(parent=parent, **kw)
../../.virtualenvs/typed-settings/lib/python3.12/site-packages/_pytest/nodes.py:167: in _create
return super().__call__(*k, **known_kw)
../../.virtualenvs/typed-settings/lib/python3.12/site-packages/sybil/integration/pytest.py:51: in __init__
self.request_fixtures(sybil.fixtures)
../../.virtualenvs/typed-settings/lib/python3.12/site-packages/sybil/integration/pytest.py:56: in request_fixtures
closure = fm.getfixtureclosure(names, self)
E TypeError: FixtureManager.getfixtureclosure() missing 1 required positional argument: 'ignore_args'
Found nothing related in the Changelogs and did not have time to dig deeper into it.
Maybe I can provide a fix later this day. :-)
While preparing a package for Debian I noticed that two tests fail:
capsys = <_pytest.capture.CaptureFixture object at 0x7f3e2f61b828>
def test_nose(capsys):
class ResultStoringMain(NoseMain):
def runTests(self):
self.testRunner = NoseRunner(stream=sys.stdout,
verbosity=self.config.verbosity,
config=self.config)
self.result = self.testRunner.run(self.test)
main = ResultStoringMain(
module=None,
argv=['x', join(example_dir, 'example_nose')]
)
> assert main.result.testsRun == 3
E assert 1 == 3
E + where 1 = <nose.result.TextTestResult run=1 errors=0 failures=0>.testsRun
E + where <nose.result.TextTestResult run=1 errors=0 failures=0> = <tests.test_doc_example.test_nose.<locals>.ResultStoringMain object at 0x7f3e2f61bd68>.result
tests/test_doc_example.py:50: AssertionError
capsys = <_pytest.capture.CaptureFixture object at 0x7f3e2f662128>
def test_nose(capsys):
class ResultStoringMain(NoseMain):
def runTests(self):
self.testRunner = NoseRunner(stream=sys.stdout,
verbosity=self.config.verbosity,
config=self.config)
self.result = self.testRunner.run(self.test)
main = ResultStoringMain(
module=None,
argv=['x', '-vs', join(functional_test_dir, 'nose')]
)
> assert main.result.testsRun == 9
E assert 2 == 9
E + where 2 = <nose.result.TextTestResult run=2 errors=0 failures=0>.testsRun
E + where <nose.result.TextTestResult run=2 errors=0 failures=0> = <tests.test_functional.test_nose.<locals>.ResultStoringMain object at 0x7f3e2f6620f0>.result
tests/test_functional.py:163: AssertionError
I'm not entirely sure why this works on Travis but fails on latest Debian unstable. How are those 3 and 9 test runs counted?
I am getting syntax errors from a Bash code block (.. code-block:: bash
). Could sybil automatically ignore non-Python snippets?
Thanks for making this great tool!
In https://sybil.readthedocs.io/en/latest/patterns.html#migrating-from-sphinx-ext-doctest it says to migrate testsetup
directives to invisible-code-block
. I would like a parser that makes testsetup
an alias for invisible-code-block
, so that I can keep my testsetup
blocks and stay compatible with sphinx.ext.doctest. Can such a parser be added to Sybil?
Hi, I've finally come around to try it!
Looks promising, but the following doesn't work:
```python
>>> import svc_reg
>>> import uuid
>>> reg = svc_reg.Registry()
>>> reg.register_factory(uuid.UUID, uuid.uuid4)
>>> reg.register_value(str, "Hello World")
```
Gives me:
Example at /Users/hynek/FOSS/svc-reg/README.md, line 41, column 1 did not evaluate as expected:
Expected:
```
Got nothing
README.md:41: SybilFailure
While the following does (empty last line):
```python
>>> import svc_reg
>>> import uuid
>>> reg = svc_reg.Registry()
>>> reg.register_factory(uuid.UUID, uuid.uuid4)
>>> reg.register_value(str, "Hello World")
```
This is not just about None results.
```python
>>> container = svc_reg.Container(reg)
>>> u = container.get(uuid.UUID)
>>> u
UUID('...')
>>> # Calling get() again returns the SAME UUID instance!
>>> # Good for DB connections, bad for UUIDs.
>>> u is container.get(uuid.UUID)
True
>>> container.get(str)
'Hello World'
```
gives me:
Example at /Users/hynek/FOSS/svc-reg/README.md, line 62, column 1 did not evaluate as expected:
Expected:
'Hello World'
```
Got:
'Hello World'
README.md:62: SybilFailure
Not sure if it's intended or not, but in 6.0 this no longer works:
Let's skips some stuff:
.. skip: next
.. code-block:: python
run.append(1)
Modifying tests/samples/skip.txt in this way gives the following error:
E ValueError: <Region start=70 end=124 evaluator=<sybil.evaluators.skip.Skipper object at 0x7f8515d8d490>><Parsed>('next', None)</Parsed></Region> from line 7, column 1 to line 11, column 1 overlaps <Region start=84 end=124 evaluator=<sybil.evaluators.python.PythonEvaluator object at 0x7f85152ff4d0>>
E directive: 'code-block'
E arguments: 'python'
E source: 'run.append(1)\n'
E options: {}
E <Parsed>'run.append(1)\n'</Parsed></Region> from line 8, column 1 to line 11, column 1
If you leave at least an empty line between directives it still works.
I recently found Sybil as a way to test my static documentation code with pytest. On some of the documentation pages, Matplotlib plt.show()
is invoked, and running pytest makes plots appear when each code block is executed.
I added .. skip: next
before the code blocks as shown below
Some documentation
.. skip: next
.. code-block:: python
import matplotlib.pyplot as plt
plt.plot([1, 2, 4, 9])
plt.show()
Commentary about above figure
However, plots are still opened when running pytest. Further, I tried the following code
.. skip: next
>>> print("Foo Bar")
which still throws the error
Expected nothing
Got:
Foo Bar
when running pytest.
Any ideas on why skip might be failing? Or am I using skip incorrectly?
Seeing the below stack. When I change to pytest <= 7.0.0
everything works again. This was with sybil 3.0.0.
.tox/py39-doctest/lib/python3.9/site-packages/_pytest/nodes.py:140: in _create
return super().__call__(*k, **kw)
E TypeError: __init__() got an unexpected keyword argument 'path'
During handling of the above exception, another exception occurred:
.tox/py39-doctest/lib/python3.9/site-packages/pluggy/_hooks.py:265: in __call__
return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
.tox/py39-doctest/lib/python3.9/site-packages/pluggy/_manager.py:80: in _hookexec
return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
.tox/py39-doctest/lib/python3.9/site-packages/sybil/integration/pytest.py:120: in pytest_collect_file
return SybilFile.from_parent(parent, fspath=path, sybil=sybil)
.tox/py39-doctest/lib/python3.9/site-packages/_pytest/nodes.py:633: in from_parent
return super().from_parent(parent=parent, fspath=fspath, path=path, **kw)
.tox/py39-doctest/lib/python3.9/site-packages/_pytest/nodes.py:264: in from_parent
return cls._create(parent=parent, **kw)
.tox/py39-doctest/lib/python3.9/site-packages/_pytest/nodes.py:146: in _create
warnings.warn(
E pytest.PytestDeprecationWarning: <class 'sybil.integration.pytest.SybilFile'> is not using a cooperative constructor and only takes {'fspath', 'sybil', 'parent'}.
E See https://docs.pytest.org/en/stable/deprecations.html#constructors-of-custom-pytest-node-subclasses-should-take-kwargs for more details.
Hello, thank you very much for this nifty library! In my use case, there are non-Python blocks that contain tabs but these tabs should just be ignored. My workaround is to let DocTestParser
take an allow_tabs
kwarg in my fork:
https://github.com/jacksonllee/sybil/commit/68944ab9881c8af70254317f36ae996e32faf25c
If there's any interest, I'd be more than happy to contribute a pull request so that it's available in the upstream source here as well.
(Found in Sybil 4.0.1)
Unescaped tokens in END_PATTERN_TEMPLATE = '(\n\\Z|\n[ \t]{{0,{len_prefix}}}(?=\\S))'
I think you meant: r'(\n\Z|\n[ \t]{{0,{len_prefix}}}(?=\S))'
?
if I have this:
"""
My comment
.. code-block:: python
test_my_code("Hello World")
"""
the PythonCodeBlockParser fails. For example:
.tox/local/lib/python3.8/site-packages/sybil/integration/pytest.py:104: in collect
self.document = self.sybil.parse(Path(self.fspath.strpath))
.tox/local/lib/python3.8/site-packages/sybil/sybil.py:144: in parse
return type_.parse(str(path), *self.parsers, encoding=self.encoding)
.tox/local/lib/python3.8/site-packages/sybil/document.py:201: in parse
for region in parser(docstring_document):
.tox/local/lib/python3.8/site-packages/sybil/parsers/abstract/codeblock.py:46: in __call__
for lexed in chain(*(lexer(document) for lexer in self.lexers)):
.tox/local/lib/python3.8/site-packages/sybil/parsers/abstract/lexers.py:59: in __call__
raise LexingException(
E sybil.parsers.abstract.lexers.LexingException: Could not match '(\n\\Z|\n[ \t]{0,8}(?=\\S))' in my_python.py (truncated)
Neither does this work because my IDE always elides extra whitespace:
"""
My comment
.. code-block:: python
test_my_code("Hello World")
"""
If I do this it will work:
"""
My comment
.. code-block:: python
test_my_code("Hello World")
\
"""
or this works too:
"""
My comment
.. code-block:: python
test_my_code("Hello World")
Any other rst comment text here.
"""
I'm creating a ReST document that uses code blocks to explain things in code via REPL-like syntax, like this:
.. code-block:: Python
class Rectangle:
pass
That is all we need to have a custom ``Rectangle`` object, as we can see below:
.. code-block:: Python
>>> rect = Rectangle()
>>> type(rect)
<class '__main__.Rectangle'>
When I run code like this through Sybil, I get overlap errors:
================================================================== ERRORS ===================================================================
_________________________________________________ ERROR collecting 1 1 what are classes.rst _________________________________________________
../.venv/lib/python3.9/site-packages/sybil/integration/pytest.py:101: in collect
self.document = self.sybil.parse(self.fspath.strpath)
../.venv/lib/python3.9/site-packages/sybil/sybil.py:106: in parse
return Document.parse(path, *self.parsers, encoding=self.encoding)
../.venv/lib/python3.9/site-packages/sybil/document.py:46: in parse
document.add(region)
../.venv/lib/python3.9/site-packages/sybil/document.py:89: in add
self.raise_overlap(region, next)
../.venv/lib/python3.9/site-packages/sybil/document.py:69: in raise_overlap
raise ValueError('{} overlaps {}'.format(*reprs))
E ValueError: <Region start=936 end=1039 <function evaluate_code_block at 0x1043e4700>> from line 40, column 1 to line 48, column 1 overlaps <Region start=960 end=982 <bound method DocTestParser.evaluate of <sybil.parsers.doctest.DocTestParser object at 0x1043ecbb0>>> from line 42, column 1 to line 43, column 1
I take it Sybil is trying to parse the >>>
indented code with a DocTestParser
and CodeBlockParser
.
I'm not sure how to resolve this. In my conftest.py
file, if I remove DocTestParser
from the parsers Sybil would use, I get SyntaxError
s:
E >>> rect = Rectangle()
E SyntaxError: invalid syntax
Do you have any ideas how I might solve this issue?
test
According to the sybil docs, you can skip exmamples with ; skip ...
.
According to the MyST parser docs, comments start with a %
.
Consequently, MySt does not hide ; skip ...
statements in the output.
I can work around the issue by subclassign the skip parser and setting my own regex that uses %
instead of ;
.
This issue occurs if the documentation file that is being parsed for code blocks contains \r\n
line separators. It results from the implementation of the builtin textwrap.dedent
function and the text that Sybil passes to this function.
The regex pattern used for end_pattern in the CodeBlockParser
finds the \n
but not the corresponding \r
which results in a trailing \r
.
The following illustrates how textwrap.dedent
behaves. The trailing \r
that exists when I initially define text
does not dedent the text -- which would result in an IndentationError
being raised by sybil.parsers.codeblock.compile_codeblock
>>> import textwrap
>>> text = ' from math import cos\r\n x=cos(0.1)\r\n\r'
>>> textwrap.dedent(text)
' from math import cos\r\n x=cos(0.1)\r\n\r'
>>> text = ' from math import cos\r\n x=cos(0.1)\r\n'
>>> textwrap.dedent(text)
'from math import cos\r\nx=cos(0.1)\r\n'
Here's an example for how to reproduce it with Sybil to raise an IndentationError
from sybil.document import Document
from sybil.example import Example
from sybil.parsers.codeblock import CodeBlockParser
text = 'This is my example:\r\n\r\n.. code-block:: python\r\n\r\n from math import cos\r\n x = cos(0.1)\r\n\r\nThat was my example.\r\n'
document = Document(text=text, path='whatever.rst')
region = list(CodeBlockParser()(document))[0]
region.evaluator(
Example(
document=document,
line=0,
column=0,
region=region,
namespace={}
)
)
A similar issue could exist when using dedent
for a capture, but I have not tested it.
The same issue was report as #4. But on Fedora. The RPM generation is failing as test_nose
are not passing.
_____________________________________________________________________ test_nose _____________________________________________________________________
capsys = <_pytest.capture.CaptureFixture object at 0x7f3671e321d0>
def test_nose(capsys):
class ResultStoringMain(NoseMain):
def runTests(self):
self.testRunner = NoseRunner(stream=sys.stdout,
verbosity=self.config.verbosity,
config=self.config)
self.result = self.testRunner.run(self.test)
main = ResultStoringMain(
module=None,
argv=['x', join(example_dir, 'example_nose')]
)
> assert main.result.testsRun == 3
E assert 1 == 3
E + where 1 = <nose.result.TextTestResult run=1 errors=0 failures=0>.testsRun
E + where <nose.result.TextTestResult run=1 errors=0 failures=0> = <tests.test_doc_example.test_nose.<locals>.ResultStoringMain object at 0x7f3671e32a90>.result
tests/test_doc_example.py:50: AssertionError
_____________________________________________________________________ test_nose _____________________________________________________________________
capsys = <_pytest.capture.CaptureFixture object at 0x7f3671a8e240>
def test_nose(capsys):
class ResultStoringMain(NoseMain):
def runTests(self):
self.testRunner = NoseRunner(stream=sys.stdout,
verbosity=self.config.verbosity,
config=self.config)
self.result = self.testRunner.run(self.test)
main = ResultStoringMain(
module=None,
argv=['x', '-vs', join(functional_test_dir, 'nose')]
)
> assert main.result.testsRun == 9
E assert 2 == 9
E + where 2 = <nose.result.TextTestResult run=2 errors=0 failures=0>.testsRun
E + where <nose.result.TextTestResult run=2 errors=0 failures=0> = <tests.test_functional.test_nose.<locals>.ResultStoringMain object at 0x7f3671a8e630>.result
tests/test_functional.py:163: AssertionError
PythonDocument.evaluator
in sybil/document.py
includes the line self.evaluator = None
.
mypy
has issues with this line, and I am trying to resolve those issues.
However, I cannot be confident that my attempts are valid, as I don't understand what this is meant to do, and the tests pass if this line is removed.
My request is that a test is added which fails if this line is removed, so that I can have more confidence.
I am not familiar with pytest. IMHO CI is testing the test coverage of tests.
See https://app.circleci.com/pipelines/github/simplistix/sybil/218/workflows/a3f16774-5da5-4ab6-b9f7-f97007b39ab3/jobs/1027/parallel-runs/0/steps/0-103
When I download a tarball from PyPi, the tests/ folder contains only test_*.py files, so when I try to run the tests, it fails on missing helpers.py. I can easily workaround the issue by downloading the GitHub tarball and run the tests from there. The building and installation work fine with the PyPI tarball.
Please consider tidying the PyPI tarball, either by completing the testsuite or dropping it entirely (not urgent at all). Thanks!
I seem to be hitting two new test failures:
============================================================== FAILURES ===============================================================
____________________________________________________________ test_unittest ____________________________________________________________
capsys = <_pytest.capture.CaptureFixture object at 0x7f60a0efae00>
def test_unittest(capsys: CaptureFixture[str]):
results = run_unittest(capsys, 'unittest')
out = results.out
out.then_find('sybil setup')
out.then_find('fail.rst,line:6,column:1 ... 0\nok')
out.then_find('fail.rst,line:8,column:1 ... 1\nFAIL')
out.then_find('fail.rst,line:10,column:1 ... 2\nERROR')
out.then_find('fail.rst,line:12,column:1 ... 3\nok')
> out.then_find('sybil teardown 4\nsybil setup')
tests/test_functional.py:98:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <tests.helpers.Finder object at 0x7f60a0ef8dc0>, substring = 'sybil teardown 4\nsybil setup'
def then_find(self, substring):
> assert substring in self.text[self.index:], self.text[self.index:]
E AssertionError: fail.rst,line:12,column:1 ... 3
E ok
E sybil teardown 4
E
E ======================================================================
E ERROR: /tmp/sybil/tests/functional/pytest/fail.rst,line:10,column:1
E ----------------------------------------------------------------------
E Traceback (most recent call last):
E File "/tmp/sybil/sybil/integration/unittest.py", line 18, in runTest
E self.example.evaluate()
E File "/tmp/sybil/sybil/example.py", line 64, in evaluate
E result = evaluator(self)
E File "/tmp/sybil/tests/functional/unittest/test_unittest.py", line 19, in check
E raise ValueError(message)
E ValueError: X count was 3 instead of 4
E
E ======================================================================
E FAIL: /tmp/sybil/tests/functional/pytest/fail.rst,line:8,column:1
E ----------------------------------------------------------------------
E Traceback (most recent call last):
E File "/tmp/sybil/sybil/integration/unittest.py", line 18, in runTest
E self.example.evaluate()
E File "/tmp/sybil/sybil/example.py", line 66, in evaluate
E raise SybilFailure(self, result)
E sybil.example.SybilFailure: Example at /tmp/sybil/tests/functional/pytest/fail.rst, line 8, column 1 did not evaluate as expected:
E Y count was 3 instead of 2
E
E ----------------------------------------------------------------------
E Ran 8 tests in 0.001s
E
E FAILED (failures=1, errors=1)
E
E assert 'sybil teardown 4\nsybil setup' in 'fail.rst,line:12,column:1 ... 3\nok\nsybil teardown 4\n\n============================================================...------------------------------------------------------------\nRan 8 tests in 0.001s\n\nFAILED (failures=1, errors=1)\n'
tests/helpers.py:67: AssertionError
___________________________________________________________ test_namespace ____________________________________________________________
capsys = <_pytest.capture.CaptureFixture object at 0x7f60987b5ba0>
def test_namespace(capsys):
sybil = Sybil([parse], path='./samples')
documents = [sybil.parse(p) for p in Path(sybil.path).glob('sample*.txt')]
actual = []
for document in documents:
for example in document:
print(split(example.path)[-1], example.line)
example.evaluate()
actual.append((
split(example.path)[-1],
example.line,
document.namespace['parsed'].copy(),
))
out, _ = capsys.readouterr()
> assert out.split('\n') == [
'sample1.txt 1',
'[0]',
'sample1.txt 3',
'[0, 14]',
'sample2.txt 1',
'[0]',
'sample2.txt 3',
'[0, 13]',
''
]
E AssertionError: assert ['sample2.txt 1',\n '[0]',\n 'sample2.txt 3',\n '[0, 13]',\n 'sample1.txt 1',\n '[0]',\n 'sample1.txt 3',\n '[0, 14]',\n ''] == ['sample1.txt 1',\n '[0]',\n 'sample1.txt 3',\n '[0, 14]',\n 'sample2.txt 1',\n '[0]',\n 'sample2.txt 3',\n '[0, 13]',\n '']
E At index 0 diff: 'sample2.txt 1' != 'sample1.txt 1'
E Full diff:
E [
E + 'sample2.txt 1',
E + '[0]',
E + 'sample2.txt 3',
E + '[0, 13]',
E 'sample1.txt 1',
E '[0]',
E 'sample1.txt 3',
E '[0, 14]',
E - 'sample2.txt 1',
E - '[0]',
E - 'sample2.txt 3',
E - '[0, 13]',
E '',
E ]
tests/test_sybil.py:246: AssertionError
======================================================= short test summary info =======================================================
FAILED tests/test_functional.py::test_unittest - AssertionError: fail.rst,line:12,column:1 ... 3
FAILED tests/test_sybil.py::test_namespace - AssertionError: assert ['sample2.txt 1',\n '[0]',\n 'sample2.txt 3',\n '[0, 13]',\n 'sa...
==================================================== 2 failed, 101 passed in 5.25s ====================================================
I don't know about the first failure but the second one seems to be related to filesystem file ordering being unpredictable, i.e.:
$ find -name sample*
./tests/samples
./tests/samples/sample2.txt
./tests/samples/sample1.txt
A quick bisect points to the following commit as the first bad commit:
commit 11496eb5761761b687ad4889b4173d3124caa844 (HEAD, refs/bisect/bad)
Author: Chris Withers <[email protected]>
Date: 2021-10-21 09:35:39 +0200
Fix source filtering.
- add exclude param for symmetry with pattern/patterns
- move all the testing to one place and ensure it's the same for all integration
I've reproduced this with Python 3.10.0 and 3.8.12.
Since upgrading pytest from 3.4.2 to 3.5.0, sybil's testsuite fails:
============================= test session starts ==============================
platform linux2 -- Python 2.7.14, pytest-3.5.0, py-1.5.3, pluggy-0.6.0 -- /nix/store/ivdcnkynqa01bq6n1srwi1kpjnawifkd-python-2.7.14/bin/python2.7
cachedir: .pytest_cache
rootdir: /build/sybil-1.0.7, inifile: setup.cfg
collecting ... collected 36 items
tests/test_capture.py::test_basic PASSED [ 2%]
tests/test_capture.py::test_directive_indent_beyond_block PASSED [ 5%]
tests/test_capture.py::test_directive_indent_equal_to_block PASSED [ 8%]
tests/test_codeblock.py::test_basic PASSED [ 11%]
tests/test_codeblock.py::test_future_imports PASSED [ 13%]
tests/test_doc_example.py::test_pytest PASSED [ 16%]
tests/test_doc_example.py::test_unittest PASSED [ 19%]
tests/test_doc_example.py::test_nose PASSED [ 22%]
tests/test_doctest.py::test_pass PASSED [ 25%]
tests/test_doctest.py::test_fail PASSED [ 27%]
tests/test_doctest.py::test_fail_with_options PASSED [ 30%]
tests/test_doctest.py::test_literals PASSED [ 33%]
tests/test_doctest.py::test_min_indent PASSED [ 36%]
tests/test_doctest.py::test_tabs PASSED [ 38%]
tests/test_functional.py::test_pytest FAILED [ 41%]
tests/test_functional.py::test_unittest PASSED [ 44%]
tests/test_functional.py::test_nose PASSED [ 47%]
tests/test_sybil.py::TestRegion::test_repr PASSED [ 50%]
tests/test_sybil.py::TestExample::test_repr PASSED [ 52%]
tests/test_sybil.py::TestExample::test_evaluate_okay PASSED [ 55%]
tests/test_sybil.py::TestExample::test_evaluate_not_okay PASSED [ 58%]
tests/test_sybil.py::TestExample::test_evaluate_raises_exception PASSED [ 61%]
tests/test_sybil.py::TestDocument::test_add PASSED [ 63%]
tests/test_sybil.py::TestDocument::test_add_no_overlap PASSED [ 66%]
tests/test_sybil.py::TestDocument::test_add_out_of_order PASSED [ 69%]
tests/test_sybil.py::TestDocument::test_add_adjacent PASSED [ 72%]
tests/test_sybil.py::TestDocument::test_add_before_start PASSED [ 75%]
tests/test_sybil.py::TestDocument::test_add_after_end PASSED [ 77%]
tests/test_sybil.py::TestDocument::test_add_overlaps_with_previous PASSED [ 80%]
tests/test_sybil.py::TestDocument::test_add_overlaps_with_next PASSED [ 83%]
tests/test_sybil.py::TestDocument::test_example_path PASSED [ 86%]
tests/test_sybil.py::TestDocument::test_example_line_and_column PASSED [ 88%]
tests/test_sybil.py::TestSybil::test_parse PASSED [ 91%]
tests/test_sybil.py::TestSybil::test_all_paths PASSED [ 94%]
tests/test_sybil.py::TestSybil::test_all_paths_with_base_directory PASSED [ 97%]
tests/test_sybil.py::test_namespace PASSED [100%]
=================================== FAILURES ===================================
_________________________________ test_pytest __________________________________
capsys = <_pytest.capture.CaptureFixture object at 0x7ffff1f9d7d0>
def test_pytest(capsys):
class CollectResults:
def pytest_sessionfinish(self, session):
self.session = session
results = CollectResults()
return_code = pytest_main(['-vs', join(functional_test_dir, 'pytest')],
plugins=[results])
assert return_code == 1
assert results.session.testsfailed == 4
assert results.session.testscollected == 10
out, err = capsys.readouterr()
# check we're trimming tracebacks:
index = out.find('sybil/example.py')
if index > -1: # pragma: no cover
raise AssertionError('\n'+out[index-500:index+500])
out = Finder(out)
out.then_find('fail.rst::line:1,column:1')
> out.then_find('fail.rst sybil setup function_fixture setup\n'
'class_fixture setup\n'
'module_fixture setup\n'
'session_fixture setup\n'
'x is currently: 0\n'
'FAILED class_fixture teardown\n'
'function_fixture teardown')
tests/test_functional.py:44:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <tests.test_functional.Finder object at 0x7ffff1ee2050>
substring = 'fail.rst sybil setup function_fixture setup\nclass_fixture setup\nmodule_fixture setup\nsession_fixture setup\nx is currently: 0\nFAILED class_fixture teardown\nfunction_fixture teardown'
def then_find(self, substring):
> assert substring in self.text[self.index:]
E AssertionError: assert 'fail.rst sybil setup function_fixture setup\nclass_fixture setup\nmodule_fixture setup\nsession_fixture setup\nx is currently: 0\nFAILED class_fixture teardown\nfunction_fixture teardown' in 'fail.rst::line:1,column:1 <- /build/sybil-1.0.7/tests/functional/pytest/fail.rst sybil setup session_fixture setup\nmo...tional/pytest/fail.rst:18: Exception\n====================== 4 failed, 6 passed in 0.05 seconds ======================\n'
tests/test_functional.py:19: AssertionError
===================== 1 failed, 35 passed in 0.41 seconds ======================
I'm trying to package your module as an rpm package. So I'm using the typical PEP517 based build, install and test cycle used on building packages from non-root account.
python3 -sBm build -w
Here is pytest output:
+ PYTHONPATH=/home/tkloczko/rpmbuild/BUILDROOT/python-sybil-3.0.0-2.fc35.x86_64/usr/lib64/python3.8/site-packages:/home/tkloczko/rpmbuild/BUILDROOT/python-sybil-3.0.0-2.fc35.x86_64/usr/lib/python3.8/site-packages
+ /usr/bin/pytest -ra -q
=========================================================================== test session starts ============================================================================
platform linux -- Python 3.8.12, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
rootdir: /home/tkloczko/rpmbuild/BUILD/sybil-3.0.0, configfile: setup.cfg
plugins: cov-3.0.0
collected 112 items
docs/example-skip.rst ......... [ 8%]
docs/example.rst .... [ 11%]
docs/parsers.rst .................. [ 27%]
tests/test_capture.py .... [ 31%]
tests/test_codeblock.py .......... [ 40%]
tests/test_doc_example.py .. [ 41%]
tests/test_doctest.py ........ [ 49%]
tests/test_functional.py .............................F.. [ 77%]
tests/test_helpers.py .. [ 79%]
tests/test_skip.py .... [ 83%]
tests/test_sybil.py ................... [100%]
================================================================================= FAILURES =================================================================================
___________________________________________________________________ test_modules_not_importable_unittest ___________________________________________________________________
tmpdir = local('/tmp/pytest-of-tkloczko/pytest-275/test_modules_not_importable_un0'), capsys = <_pytest.capture.CaptureFixture object at 0x7fd94b9d6670>
def test_modules_not_importable_unittest(tmpdir: local, capsys: CaptureFixture[str]):
# NB: no append to sys.path
results = clone_and_run_modules_tests(tmpdir, capsys, UNITTEST)
assert results.total == 5, results.out.text
assert results.failures == 0, results.out.text
assert results.errors == 5, results.out.text
a_py = tmpdir/'modules'/'a.py'
b_py = tmpdir/'modules'/'b.py'
out = results.out
out.then_find(f'ERROR: {a_py},line:3,column:1')
out.then_find(f"ImportError: 'a' not importable from {tmpdir/'modules'/'a.py'} as:")
out.then_find("ModuleNotFoundError: No module named 'a'")
> out.then_find(f'ERROR: {b_py},line:2,column:1')
tests/test_functional.py:329:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <tests.helpers.Finder object at 0x7fd94d764af0>, substring = 'ERROR: /tmp/pytest-of-tkloczko/pytest-275/test_modules_not_importable_un0/modules/b.py,line:2,column:1'
def then_find(self, substring):
> assert substring in self.text[self.index:], self.text[self.index:]
E AssertionError: ModuleNotFoundError: No module named 'a'
E
E ======================================================================
E ERROR: /tmp/pytest-of-tkloczko/pytest-275/test_modules_not_importable_un0/modules/a.py,line:7,column:1
E ----------------------------------------------------------------------
E Traceback (most recent call last):
E File "/home/tkloczko/rpmbuild/BUILD/sybil-3.0.0/sybil/integration/unittest.py", line 17, in runTest
E self.example.evaluate()
E File "/home/tkloczko/rpmbuild/BUILD/sybil-3.0.0/sybil/example.py", line 64, in evaluate
E result = evaluator(self)
E File "/home/tkloczko/rpmbuild/BUILD/sybil-3.0.0/sybil/document.py", line 142, in evaluator
E module = import_path(Path(example.path))
E File "/home/tkloczko/rpmbuild/BUILD/sybil-3.0.0/sybil/python.py", line 40, in import_path
E raise ImportError(
E ImportError: 'a' not importable from /tmp/pytest-of-tkloczko/pytest-275/test_modules_not_importable_un0/modules/a.py as:
E ModuleNotFoundError: No module named 'a'
E
E ----------------------------------------------------------------------
E Ran 5 tests in 0.006s
E
E FAILED (errors=5)
tests/helpers.py:67: AssertionError
========================================================================= short test summary info ==========================================================================
FAILED tests/test_functional.py::test_modules_not_importable_unittest - AssertionError: ModuleNotFoundError: No module named 'a'
====================================================================== 1 failed, 111 passed in 2.53s =======================================================================
pytest
is not defined as a requirement of sybil
, but sybil
depends on pytest
.
pip install sybil==5.0.3
import sybil
sybil.Sybil(parsers=[]).pytest()
This raises the error:
ModuleNotFoundError: No module named '_pytest'
This is not caught by the tests because pytest
is a test dependency.
I suggest that to fix this, move pytest
to the install requirements for sybil
.
FWIW a project like pip-check-reqs
would find this issue.
I am trying to switch from doctest
to code-block
with Sybil.
One issue I'm hitting is that I would like to have multiple separated tests within one document.
manuel
provides .. reset-globs
.
Is there a workaround to allow similar functionality with Sybil?
This took me a moment to figure out and I feel like the DX could be better.
If you have a file that doesn't have a final line feed like the following:
# Integrations
```{toctree}
:maxdepth: 1
flask
pyramid
custom
```
And there's no \n
behind the three backticks, you get an error like this:
sybil.parsers.abstract.lexers.LexingException: Could not match '(?<=\\n)```\\n' in /Users/hynek/FOSS/svcs/docs/integrations/index.md:
E ':maxdepth: 1\n\nflask\npyramid\ncustom\n```'
I know enough re to shoot myself into the foot, so I figured it out quite quick, but I think it would be good to have a nicer error message.
myvenv/python3.9/site-packages/sybil/integration/pytest.py:58: 193 warnings
myvenv/lib/python3.9/site-packages/sybil/integration/pytest.py:58: PytestDeprecationWarning: A private pytest class or function was used.
self._request = fixtures.FixtureRequest(self)
-- Docs: https://docs.pytest.org/en/stable/warnings.html
This has been going on for awhile. Doesn't break anything but it does normalize deviance having a huge number of warnings we are supposed to ignore.
Any ideas how we could fix this?
ReST/MyST directives can have options, e.g.:
.. code-block:: python
:caption: example.py
print("Hello, world!")
'``{code-block} python
:caption: example.py
print("Hello, world!")
'``
The options are already part of the directive regex (rest, myst (although myst use the ---
form and not the double colon form which seems to be recommended by MyST).
These options are not part of a capture group, though, and can thus not be used for more advanced stuff.
I have examples that show a config file, a python script that loads and prints the settings and a console code block that invokes the script and displays its output (similarly to doctests, but for bash). Here is an example: https://typed-settings.readthedocs.io/en/latest/#example
I use the :caption: filename.ext
option for directives to indicate that the contents of a code block should be written to that file instead of being executed (that option also has the nice side-effect of being displayed in the docs :)).
For this to work I needed to copy and adjust lexers and parsers from Sybil. This is generally okay for me, but maybe parts of that functionality (or all of it) might be a nice addition to Sybil.
Here is the current code for this: https://gitlab.com/sscherfke/typed-settings/-/blob/main/conftest.py
---
format).options = {"caption": "example.py"}
) and make them part of LexedRegion.lexemes
.options
property to Region
which is a shortcut to region.lexemes.get("options", {})
.CodeBlockParser
that can be used to trigger user defined functionality (e.g., depending on the directive's parameters). The current behavior is the default and fallback behavior.If you are interested in adding this to Sybil, I would start working on a PR.
It took me a while to realize that invisible-code-block
only works with code blocks.
As far as I can tell, it is currently not possible to write invisible code as context for a documentation example that has lines of output.
Spec is here: https://github.github.com/gfm/
Looks like both doctests and python source are supported in python
fenced code blocks:
import this
>>> "a doctest"
"a doctest"
the idiomatic way to configure pytest for sybil is to add a plugin to pytest
then a conftest would read:
import pytest
from sybil import Sybil
from sybil.parsers.codeblock import CodeBlockParser
from sybil.parsers.doctest import DocTestParser
pytest_plugins = "sybil.integration.pytest
@pytest.fixture()
def sybil_dir(tmpdir, monkeypatch):
# there are better ways to do temp directories, but it's a simple example:
path = tmpdir.mkdir('sybil')
monkeypatch.chdir(path)
return path
def pytest_sybil_configure():
return Sybil(
parsers=[
DocTestParser(),
CodeBlockParser(future_imports=['print_function']),
],
pattern='*.rst',
fixtures=['sybil_dir']
)
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.