simplistix / sybil Goto Github PK

View Code? Open in Web Editor NEW

69.0 69.0 13.0 386 KB

Automated testing for the examples in your documentation.

Home Page: https://sybil.readthedocs.io/en/latest/

License: Other

Python 99.60% Makefile 0.40%

sybil's People

Contributors

Stargazers

Watchers

Forkers

nicoddemus lamby thirtytwobits scoder keszybz rohberg mgorny wimglenn adamtheturtle sysfce2 hroncok fabaff

sybil's Issues

Parse content from docstrings?

I am looking sybil as a replacement for the built in doctest collector in pytest, so I can more easily extend the functionality with setup code blocks and namespaces, all the good things sybil supports.

The one thing I am not clear about is if sybil can parse blocks from docstrings inside the python source? Which is something the doctest collector can do.

Support for `testcode` and `testoutput` in `DocTestDirectiveParser`s

I'd love to have support for the testcode and testoutput directives from sphinx.ext.doctest since we already have a good amount of long-form documentation that uses them and like to be able to add some explanatory text before the output, which is harder to do with doctest blocks. You said in the docs adding more doctest features wouldn't be hard, so thought I'd ask! Thanks for your work!

Feature request: Add type hints

I see that there is an existing branch for this, https://github.com/simplistix/sybil/tree/typing. Let me know if help is welcome on that. It would be nice to not have to ignore_missing_imports = true for sybil for strict mypy to work on my projects.

Unexpected whitespace error

I used Sybil to test a simple docstring, but got an unexpected error.

conftest.py

from sybil import Sybil
from sybil.parsers.rest import DocTestParser

pytest_collect_file = Sybil(
    parsers=[DocTestParser()],
    patterns=['*.py'],
).pytest()

module.py

def f():
    """Do something.

    Example:
        >>> x = 3
        >>> x
        3
    """
    pass

Traceback

❯ pytest
================================================================================================ test session starts ================================================================================================
platform darwin -- Python 3.10.9, pytest-7.2.2, pluggy-1.0.0
rootdir: /Users/balaji/Documents/sybil-example, configfile: pytest.ini
collected 0 items / 1 error                                                                                                                                                                                         

====================================================================================================== ERRORS =======================================================================================================
____________________________________________________________________________________________ ERROR collecting module.py _____________________________________________________________________________________________
.venv/lib/python3.10/site-packages/sybil/integration/pytest.py:104: in collect
    self.document = self.sybil.parse(Path(self.fspath.strpath))
.venv/lib/python3.10/site-packages/sybil/sybil.py:143: in parse
    return type_.parse(str(path), *self.parsers, encoding=self.encoding)
.venv/lib/python3.10/site-packages/sybil/document.py:54: in parse
    for region in parser(document):
.venv/lib/python3.10/site-packages/sybil/parsers/abstract/doctest.py:41: in __call__
    source, options, want, exc_msg = self._parse_example(m, name, lineno)
/opt/homebrew/Cellar/[email protected]/3.10.9/Frameworks/Python.framework/Versions/3.10/lib/python3.10/doctest.py:721: in _parse_example
    self._check_prefix(want_lines, ' '*indent, name,
/opt/homebrew/Cellar/[email protected]/3.10.9/Frameworks/Python.framework/Versions/3.10/lib/python3.10/doctest.py:806: in _check_prefix
    raise ValueError('line %r of the docstring for %s has '
E   ValueError: line 8 of the docstring for /Users/balaji/Documents/sybil-example/module.py has inconsistent leading whitespace: '    """'
============================================================================================== short test summary info ==============================================================================================
ERROR module.py - ValueError: line 8 of the docstring for /Users/balaji/Documents/sybil-example/module.py has inconsistent leading whitespace: '    """'
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
================================================================================================= 1 error in 0.05s ==================================================================================================

Dependencies

❯ pip freeze
attrs==22.2.0
exceptiongroup==1.1.0
iniconfig==2.0.0
packaging==23.0
pluggy==1.0.0
pytest==7.2.2
sybil==4.0.1
tomli==2.0.1

Provide ways to skip tests

I spoke with Chris about this elsewhere, discovered another need.

Sometimes you have chunks of samples in docs that you don't want executed. They're un-runnable pseudocode. It would be nice to have a way to say "don't test these". Chris suggested a couple of variations of Sphinx directives.

Other times you want to skip when tox tests under certain interpreters. For example I have a snippet with a dataclass, in a package with optional dataclass support. I want the equivalent of:
@pytest.mark.skipif(sys.version_info < (3, 6), reason="requires python3.6 or higher")

Although it would be unique to pytest, it would be cool to have a Sphinx directive for skipif with the directive arguments matching the signature of skipif.

Use project link to point to https://sybil.readthedocs.io/en/latest/ ?

Hi!

it would be cool to have a quick way to go to https://sybil.readthedocs.io/en/latest/ from here. The project link is still unfilled and might make a great place for it.

Best, Sebastian

Update Example Documentation

The first example from the documentation does not include the CodeBlockParser(), so doesn't pick up code-block directives.
https://sybil.readthedocs.io/en/latest/use.html

Here's what I'm using:

from doctest import ELLIPSIS
from sybil import Sybil
from sybil.parsers.codeblock import CodeBlockParser
from sybil.parsers.doctest import DocTestParser
from sybil.parsers.skip import skip
from sybil.parsers.capture import parse_captures

pytest_collect_file = Sybil(
    parsers=[
        CodeBlockParser(),
        DocTestParser(optionflags=ELLIPSIS),
        parse_captures,
        skip,
    ],
    pattern="test_*.rst",
).pytest()

P.S. Nice library!

Recursive globs in pattern argument

I have .rst files I'd like tested, but they in child directories under the root. I'd like a pattern of something like '**/*.rst' using the improved glob support in Python post-3.5

If that's not feasible, can path optional be a sequence of places to look?

AttributeError: 'NoneType' object has no attribute 'end'

I am not sure if it is due to me using it wrong or simply a missing check, but i am getting this error:

.direnv/python-3.11.2/lib64/python3.11/site-packages/sybil/integration/pytest.py:104: in collect
    self.document = self.sybil.parse(Path(self.fspath.strpath))
.direnv/python-3.11.2/lib64/python3.11/site-packages/sybil/sybil.py:144: in parse
    return type_.parse(str(path), *self.parsers, encoding=self.encoding)
.direnv/python-3.11.2/lib64/python3.11/site-packages/sybil/document.py:196: in parse
    for start, end, text in cls.extract_docstrings(document.text):
.direnv/python-3.11.2/lib64/python3.11/site-packages/sybil/document.py:182: in extract_docstrings
    punc_size = punc.end() - punc.start()
E   AttributeError: 'NoneType' object has no attribute 'end'

While investigating, it seems to be that the result of the regex search is never checked before accessing it:

https://github.com/simplistix/sybil/blob/master/sybil/document.py#L183

Thus it should fail for any file that doesn't have the expected documentation type.

Failed tests against CPython-3.10

Several tests fail when building sybil-2.0.1 against python-3.10.0_beta2, attaching the output. Haven't checked if this is only a problem with tests or an actual issue with Sybil.
sybil-2.0.1-test.log

Skipping sections of docs when run under certain Python versions

I have a section of docs which use variable annotations: https://github.com/pauleveritt/wired/tree/dataclasses_tutorial/docs/tutorials/dcdi

The index.rst has a skip at the top. However, simply parsing the file under Python 3.5 causes a SyntaxError. I need to teach the Sybil collector to skip it.

I tried putting a conftest.py in that directory which sniffs the Python interpreter version and does an exclude. However, It seems to augment the Sybil instance, not replace it.

Namespace spanning examples

Hi!
Sybil is awesome! I would like to use Sybil for testing code examples in the documentation of plone.api https://github.com/plone/plone.api.
Plone needs a TestCase layer with an app instance to work on. I understand that Sybil can be initialized with a setup(namespace). But it's quite tricky to pass the app untouched from example to example. So after several approaches I ended up with the following suggestion: Allow the sybil.integration.unitttest.TestCase to be initialized not only with a single example, but optional also with a list of examples.

#40

What's your opinion?

Doesn't work with namespace packages

Reproducer:

# conftest.py
from sybil import Sybil
from sybil.parsers.rest import DocTestParser

sybil = Sybil(
    parsers=[DocTestParser()],
    patterns=["*.py"],
)
pytest_collect_file = sybil.pytest()

In pkg/foo.py (note: intentionally no pkg/__init__.py file present):

def func():
    """
    >>> 1 + 1
    2
    """

When letting doctest import the packages it works (although the test is executed twice, once by pytest doctest plugin and again by sybil)

$ pytest --doctest-modules -v
====================================== test session starts =======================================
platform darwin -- Python 3.11.4, pytest-7.4.0, pluggy-1.2.0 -- /private/tmp/s/.venv/bin/python
cachedir: .pytest_cache
rootdir: /private/tmp/s
collected 2 items                                                                                

pkg/foo.py::line:3,column:1 PASSED                                                         [ 50%]
pkg/foo.py::foo.func PASSED                                                                [100%]

======================================= 2 passed in 0.01s ========================================

However when disabling the doctest plugin, as the sybil docs advise to do, the imports fail:

$ pytest -p no:doctest
=========================================================== test session starts ===========================================================
platform darwin -- Python 3.11.4, pytest-7.4.0, pluggy-1.2.0
rootdir: /private/tmp/s
collected 1 item                                                                                                                          

pkg/foo.py F                                                                                                                        [100%]

================================================================ FAILURES =================================================================
_________________________________________________________ foo.py line=3 column=1 __________________________________________________________

self = <sybil.document.PythonDocStringDocument object at 0x10a329d10>
example = <Example path=/private/tmp/s/pkg/foo.py line=3 column=1 using <sybil.evaluators.doctest.DocTestEvaluator object at 0x10a346550>>

    def evaluator(self, example: Example) -> Optional[str]:
        """
        Imports the document's source file as a Python module when the first
        :class:`~sybil.example.Example` from it is evaluated.
        """
>       module = import_path(Path(self.path))

.venv/lib/python3.11/site-packages/sybil/document.py:153: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

path = PosixPath('/private/tmp/s/pkg/foo.py')

    def import_path(path: Path):
        container = path
        while True:
            container = container.parent
            if not (container / INIT_FILE).exists():
                break
        relative = path.relative_to(container)
        if relative.name == INIT_FILE:
            parts = tuple(relative.parts)[:-1]
        else:
            parts = tuple(relative.parts)[:-1]+(relative.stem,)
        module = '.'.join(parts)
        try:
            return importlib.import_module(module)
        except ImportError as e:
>           raise ImportError(
                f'{module!r} not importable from {path} as:\n{type(e).__name__}: {e}'
            ) from None
E           ImportError: 'foo' not importable from /private/tmp/s/pkg/foo.py as:
E           ModuleNotFoundError: No module named 'foo'

.venv/lib/python3.11/site-packages/sybil/python.py:40: ImportError
========================================================= short test summary info =========================================================
FAILED pkg/foo.py::line:3,column:1 - ImportError: 'foo' not importable from /private/tmp/s/pkg/foo.py as:
============================================================ 1 failed in 0.02s ============================================================

Doctest with MyST may be broken or incorrectly documented

I try to use Sybil with a Markdown file that contains normal Python code block and doctests.

Testing doctests does not work in several ways though and I am not sure whether I am doing something wrong, or whether there is a (documentation) issue in Sybil.

I am using:

Python 3.10.9
Sphinx 5.3.0 / 6.1.3
myst-parser 0.18.1
Sybil 4.0.0

Minimal example

`docs/index.md`

# Test

## 1

```
>>> print(2)
1
```

## 2

```python
>>> print(2)
2
```

## 3

```{code-block} python
>>> print(3)
3
```

## 4

```{doctest}
>>> print(4)
4
```

`docs/conf.py`

project = "Test"
author = "Monty"
release = "1.0.0"
version = "1.0"

extensions = [
    "myst_parser",
]
templates_path = ["_templates"]
exclude_patterns = ["_build", "Thumbs.db", ".DS_Store"]

`conftest.py`

from doctest import ELLIPSIS

from sybil import Sybil
from sybil.parsers.myst import DocTestDirectiveParser, PythonCodeBlockParser
from sybil.parsers.rest import DocTestParser


pytest_collect_file = Sybil(
    parsers=[
        DocTestParser(optionflags=ELLIPSIS),
        DocTestDirectiveParser(optionflags=ELLIPSIS),
        PythonCodeBlockParser(),
    ],
    patterns=["*.md"],
).pytest()

Problems

Sphinx does not recognice the {doctest} directive and will not show it, but the docs state otherwise
If I only use the PythonCodeBlockParser, only examples 2 and 3 are collected, but sucessfully tested
If I only use the DocTestDirectiveParser, only example 4 is collected, but successfully tested. But it leads to Sphinx errors (see 1.).
If I only use the DocTestParser, all four examples are collected, but doctest considers the closing three backticks as part of the example and fails with sth. like 1\n''' != 1`.
If I use all parsers (or at least DocTestParser and PythonCodeBlockParser (which I'd like to do in my real project, Sybil fails because of overlapping regions. I can "fix" this by removing the python/{code-block} python from examples 2 and 3, but then I get the same error as in point 4. again.

Test failures in 1.0.4 release

I am getting some test failures, and tried to understand since Travis is passing, but still didn't figure out. Any hints?

=================================== FAILURES ===================================
________________________________ test_unittest _________________________________

capsys = <_pytest.capture.CaptureFixture object at 0x7f70d00cfd68>

    def test_unittest(capsys):
        runner = TextTestRunner(verbosity=2, stream=sys.stdout)
        path = join(functional_test_dir, 'functional_unittest')
        main = unittest_main(
            exit=False, module=None, testRunner=runner,
            argv=['x', 'discover', '-v', '-t', path, '-s', path]
        )
        out, err = capsys.readouterr()
        assert err == ''
        out = Finder(out)
>       common_checks(out)

tests/test_functional.py:144:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/test_functional.py:120: in common_checks
    out.then_find('sybil teardown 4\nsybil setup')
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <tests.test_functional.Finder object at 0x7f70d00fbfd0>
substring = 'sybil teardown 4\nsybil setup'

    def then_find(self, substring):
>       assert substring in self.text[self.index:]
E       AssertionError: assert 'sybil teardown 4\nsybil setup' in 'fail.rst,line:12,column:1 ... 3\nok\nsybil teardown 4\n\n============================================================...------------------------------------------------------------\nRan 8 tests in 0.001s\n\nFAILED (failures=1, errors=1)\n'

tests/test_functional.py:19: AssertionError
__________________________________ test_nose ___________________________________

capsys = <_pytest.capture.CaptureFixture object at 0x7f70d00e4c18>

    def test_nose(capsys):
        class ResultStoringMain(NoseMain):
            def runTests(self):
                self.testRunner = NoseRunner(stream=sys.stdout,
                                             verbosity=self.config.verbosity,
                                             config=self.config)
                self.result = self.testRunner.run(self.test)

        main = ResultStoringMain(
            module=None,
            argv=['x', '-vs', join(functional_test_dir, 'nose')]
        )
        assert main.result.testsRun == 9
        assert len(main.result.failures) == 1
        assert len(main.result.errors) == 1

        out, err = capsys.readouterr()
        assert err == ''
        out = Finder(out)
>       common_checks(out)

tests/test_functional.py:170:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/test_functional.py:120: in common_checks
    out.then_find('sybil teardown 4\nsybil setup')
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <tests.test_functional.Finder object at 0x7f70d0085198>
substring = 'sybil teardown 4\nsybil setup'

    def then_find(self, substring):
>       assert substring in self.text[self.index:]
E       AssertionError: assert 'sybil teardown 4\nsybil setup' in 'fail.rst,line:12,column:1 ... 3\nok\nsybil teardown 4\nnose.test_other.test_it ... ok\n\n============================...------------------------------------------------------------\nRan 9 tests in 0.001s\n\nFAILED (errors=1, failures=1)\n'

tests/test_functional.py:19: AssertionError
_________________ TestSybil.test_all_paths_with_base_directory _________________

self = <tests.test_sybil.TestSybil object at 0x7f70d008c828>

    def test_all_paths_with_base_directory(self):
        sybil = Sybil([parse_for_x, parse_for_y],
                      path='./samples', pattern='*.txt')
>       assert (self._evaluate_examples(self._all_examples(sybil), 42) ==
                ['X count was 4, as expected',
                 'Y count was 3, as expected',
                 'X count was 3 instead of 4',
                 'Y count was 3, as expected'])
E       AssertionError: assert ['X count was... as expected'] == ['X count was ... as expected']
E         At index 0 diff: 'X count was 3 instead of 4' != 'X count was 4, as expected'
E         Full diff:
E         - ['X count was 3 instead of 4',
E         + ['X count was 4, as expected',
E         'Y count was 3, as expected',
E         -  'X count was 4, as expected',
E         +  'X count was 3 instead of 4',...
E
E         ...Full output truncated (2 lines hidden), use '-vv' to show

tests/test_sybil.py:217: AssertionError
________________________________ test_namespace ________________________________

capsys = <_pytest.capture.CaptureFixture object at 0x7f70d00b9128>

    def test_namespace(capsys):
        sybil = Sybil([parse],
                      path='./samples', pattern='*.txt')
        for document in sybil.all_documents():
            for example in document:
                print(split(example.document.path)[-1], example.line)
                example.evaluate()

        out, _ = capsys.readouterr()
>       assert out.split('\n') == [
            'sample1.txt 1',
            '[0]',
            'sample1.txt 3',
            '[0, 14]',
            'sample2.txt 1',
            '[0]',
            'sample2.txt 3',
            '[0, 13]',
            ''
        ]
E       AssertionError: assert ['sample2.txt...', '[0]', ...] == ['sample1.txt ...', '[0]', ...]
E         At index 0 diff: 'sample2.txt 1' != 'sample1.txt 1'
E         Full diff:
E         - ['sample2.txt 1',
E         ?         ^
E         + ['sample1.txt 1',
E         ?         ^
E         +  '[0]',...
E
E         ...Full output truncated (12 lines hidden), use '-vv' to show

tests/test_sybil.py:246: AssertionError
===================== 4 failed, 44 passed in 0.61 seconds ======================

Have a badge

e.g. powered by: sybil.
I'd put it in my developer-facing docs.

MyST lexer can't handle code fences with additional backticks

The following put in a MyST Markdown file causes a LexingException:

`````
this should lex
`````

With the following exception:

sybil.exceptions.LexingException: Could not match '(?<=\\n)```(:?\\n|\\Z)' in /home/dsel/test/test.md:
'`````\n'

Additional backticks are sometimes needed when you want to nest directives or handle literal code that has three backticks.

How can I control the module that classes and functions are created in?

I'm am working on modernizing my public library, which involves switching over to pytest. I've already made this switch on other libraries, and Sybil has been fantastic for integrating my .rst file based doctests. Great work @cjw296 !

However, with public, functions and classes in my using.rst file need to be defined in a module that ends up in sys.modules and have __module__ attributes. This doesn't seem to be the case by default. What's especially weird is that functions __module__ attribute are None, while class __module__ attributes are builtins. I don't know if that's intentional or just a strange unexpected behavior.

I can't quite work out how to ensure that classes and functions in my using.rst file get created and attached within a sys.modules present module. Any clues/help would be greatly appreciated!

UTF-8 documents don't work in Windows

Hey! See https://scrapinghub.visualstudio.com/scrapy/_build/results?buildId=5&view=logs&j=19c563d8-3dcd-57d1-cd5e-9dd946b0a29b&t=4292cd33-74e4-56d2-4fed-99bb1e6ffbb8&l=117:

https://github.com/cjw296/sybil/blob/63f4d8251f9c8156a36b02da7609d841b3c3faff/sybil/document.py#L39 opens file without specifying an encoding; it means encoding is system-dependent. I think this is the problem which caused the test failure above: your file with a doctest has some specific encoding (e.g. utf-8), it doesn't vary based on machine.

I'm not sure how to fix that properly, but opening with utf8 encoding always could work.

isinstance with `Protocol` fails in doctest

In some docs I tried showing isinstance(x, Protocol) but this fails, complaining that "__name__" isn't in in the function's global keys. I believe this is a consequence of how Sybil sets up the doc tests, though I haven't been able to diagnose the specific issue. Other than this, Sybil has been great, thanks for the package!

Traceback

Example at /Users/.../docs/introduction.rst, line 171, column 1 did not evaluate as expected:
Exception raised:
    Traceback (most recent call last):
      File "/Users/.../lib/python3.10/doctest.py", line 1350, in __run
        exec(compile(example.source, filename, "single",
      File "<doctest /Users/.../docs/introduction.rst[0]>", line 1, in <module>
        isinstance(v, NamedValue)
      File "/Users/.../lib/python3.10/typing.py", line 1496, in __instancecheck__
        not _allow_reckless_class_checks(depth=2)
      File "/Users/.../lib/python3.10/typing.py", line 1473, in _allow_reckless_class_checks
        return sys._getframe(depth).f_globals['__name__'] in ['abc', 'functools']
    KeyError: '__name__'

MWE

.. code-block:: python

    from typing import Protocol


    class NamedValue(Protocol):
        """Interface for an object that has a name and a value."""

        value: float
        name: str

some text

.. code-block:: python

    class NamedValueClass1:
        def __init__(self, name: str, value: float):
            self.name = name
            self.value = value


    v = NamedValueClass1("foo", 1.0)

::

    >>> isinstance(v, NamedValue)
    True

Warnings on Python 3.12 about ast.Str

Not sure if you're aware, but Sybil causes two warnings on every test run for me:

.tox/py312-tests-optional/lib/python3.12/site-packages/sybil/document.py:3
  /Users/hynek/FOSS/svcs/.tox/py312-tests-optional/lib/python3.12/site-packages/sybil/document.py:3: DeprecationWarning: ast.Str is deprecated and will be removed in Python 3.14; use ast.Constant instead
    from ast import AsyncFunctionDef, FunctionDef, ClassDef, Module, Expr, Str, Constant

.tox/py312-tests-optional/lib/python3.12/site-packages/sybil/document.py:176
  /Users/hynek/FOSS/svcs/.tox/py312-tests-optional/lib/python3.12/site-packages/sybil/document.py:176: DeprecationWarning: ast.Str is deprecated and will be removed in Python 3.14; use ast.Constant instead
    isinstance(docstring, Str) or

Pytest 8 error: FixtureManager.getfixtureclosure() missing 1 required positional argument: 'ignore_args'

Pytest 8 was just released and it seems that this breaks sybil:

../../.virtualenvs/typed-settings/lib/python3.12/site-packages/sybil/integration/pytest.py:51: in __init__
    self.request_fixtures(sybil.fixtures)
../../.virtualenvs/typed-settings/lib/python3.12/site-packages/sybil/integration/pytest.py:56: in request_fixtures
    closure = fm.getfixtureclosure(names, self)
E   TypeError: FixtureManager.getfixtureclosure() missing 1 required positional argument: 'ignore_args'

During handling of the above exception, another exception occurred:
../../.virtualenvs/typed-settings/lib/python3.12/site-packages/sybil/integration/pytest.py:120: in collect
    yield SybilItem.from_parent(self, sybil=self.sybil, example=example)
../../.virtualenvs/typed-settings/lib/python3.12/site-packages/_pytest/nodes.py:275: in from_parent
    return cls._create(parent=parent, **kw)
../../.virtualenvs/typed-settings/lib/python3.12/site-packages/_pytest/nodes.py:167: in _create
    return super().__call__(*k, **known_kw)
../../.virtualenvs/typed-settings/lib/python3.12/site-packages/sybil/integration/pytest.py:51: in __init__
    self.request_fixtures(sybil.fixtures)
../../.virtualenvs/typed-settings/lib/python3.12/site-packages/sybil/integration/pytest.py:56: in request_fixtures
    closure = fm.getfixtureclosure(names, self)
E   TypeError: FixtureManager.getfixtureclosure() missing 1 required positional argument: 'ignore_args'

Found nothing related in the Changelogs and did not have time to dig deeper into it.

Maybe I can provide a fix later this day. :-)

Tests fail in 1.0.5

While preparing a package for Debian I noticed that two tests fail:

capsys = <_pytest.capture.CaptureFixture object at 0x7f3e2f61b828>

    def test_nose(capsys):
        class ResultStoringMain(NoseMain):
            def runTests(self):
                self.testRunner = NoseRunner(stream=sys.stdout,
                                             verbosity=self.config.verbosity,
                                             config=self.config)
                self.result = self.testRunner.run(self.test)

        main = ResultStoringMain(
            module=None,
            argv=['x', join(example_dir, 'example_nose')]
        )
>       assert main.result.testsRun == 3
E       assert 1 == 3
E        +  where 1 = <nose.result.TextTestResult run=1 errors=0 failures=0>.testsRun
E        +    where <nose.result.TextTestResult run=1 errors=0 failures=0> = <tests.test_doc_example.test_nose.<locals>.ResultStoringMain object at 0x7f3e2f61bd68>.result

tests/test_doc_example.py:50: AssertionError

capsys = <_pytest.capture.CaptureFixture object at 0x7f3e2f662128>

    def test_nose(capsys):
        class ResultStoringMain(NoseMain):
            def runTests(self):
                self.testRunner = NoseRunner(stream=sys.stdout,
                                             verbosity=self.config.verbosity,
                                             config=self.config)
                self.result = self.testRunner.run(self.test)

        main = ResultStoringMain(
            module=None,
            argv=['x', '-vs', join(functional_test_dir, 'nose')]
        )
>       assert main.result.testsRun == 9
E       assert 2 == 9
E        +  where 2 = <nose.result.TextTestResult run=2 errors=0 failures=0>.testsRun
E        +    where <nose.result.TextTestResult run=2 errors=0 failures=0> = <tests.test_functional.test_nose.<locals>.ResultStoringMain object at 0x7f3e2f6620f0>.result

tests/test_functional.py:163: AssertionError

I'm not entirely sure why this works on Travis but fails on latest Debian unstable. How are those 3 and 9 test runs counted?

CodeBlockParser: Automatically skip non-Python snippets

I am getting syntax errors from a Bash code block (.. code-block:: bash). Could sybil automatically ignore non-Python snippets?

Allow test setup/cleanup

Thanks for making this great tool!
In https://sybil.readthedocs.io/en/latest/patterns.html#migrating-from-sphinx-ext-doctest it says to migrate testsetup directives to invisible-code-block. I would like a parser that makes testsetup an alias for invisible-code-block, so that I can keep my testsetup blocks and stay compatible with sphinx.ext.doctest. Can such a parser be added to Sybil?

Markdown forces an empty line at the end

Hi, I've finally come around to try it!

Looks promising, but the following doesn't work:

```python
>>> import svc_reg
>>> import uuid

>>> reg = svc_reg.Registry()

>>> reg.register_factory(uuid.UUID, uuid.uuid4)
>>> reg.register_value(str, "Hello World")
```

Gives me:

Example at /Users/hynek/FOSS/svc-reg/README.md, line 41, column 1 did not evaluate as expected:
Expected:
   ```
Got nothing

README.md:41: SybilFailure

While the following does (empty last line):

```python
>>> import svc_reg
>>> import uuid

>>> reg = svc_reg.Registry()

>>> reg.register_factory(uuid.UUID, uuid.uuid4)
>>> reg.register_value(str, "Hello World")

```

This is not just about None results.

```python
>>> container = svc_reg.Container(reg)

>>> u = container.get(uuid.UUID)
>>> u
UUID('...')
>>> # Calling get() again returns the SAME UUID instance!
>>> # Good for DB connections, bad for UUIDs.
>>> u is container.get(uuid.UUID)
True
>>> container.get(str)
'Hello World'
```

gives me:

Example at /Users/hynek/FOSS/svc-reg/README.md, line 62, column 1 did not evaluate as expected:
Expected:
    'Hello World'
    ```
Got:
    'Hello World'

README.md:62: SybilFailure

`skip: next` now seems to require an empty line after it

Not sure if it's intended or not, but in 6.0 this no longer works:

Let's skips some stuff:

.. skip: next
.. code-block:: python

  run.append(1)

Modifying tests/samples/skip.txt in this way gives the following error:

E       ValueError: <Region start=70 end=124 evaluator=<sybil.evaluators.skip.Skipper object at 0x7f8515d8d490>><Parsed>('next', None)</Parsed></Region> from line 7, column 1 to line 11, column 1 overlaps <Region start=84 end=124 evaluator=<sybil.evaluators.python.PythonEvaluator object at 0x7f85152ff4d0>>
E       directive: 'code-block'
E       arguments: 'python'
E       source: 'run.append(1)\n'
E       options: {}
E       <Parsed>'run.append(1)\n'</Parsed></Region> from line 8, column 1 to line 11, column 1

If you leave at least an empty line between directives it still works.

Skip comments do not skip next pieces of code

I recently found Sybil as a way to test my static documentation code with pytest. On some of the documentation pages, Matplotlib plt.show() is invoked, and running pytest makes plots appear when each code block is executed.

I added .. skip: next before the code blocks as shown below

Some documentation

.. skip: next

.. code-block:: python

    import matplotlib.pyplot as plt
    plt.plot([1, 2, 4, 9])
    plt.show()

Commentary about above figure

However, plots are still opened when running pytest. Further, I tried the following code

.. skip: next

>>> print("Foo Bar")

which still throws the error

Expected nothing
Got:
    Foo Bar

when running pytest.

Any ideas on why skip might be failing? Or am I using skip incorrectly?

Incompatibility with Pytest 7.0

https://docs.pytest.org/en/stable/deprecations.html#constructors-of-custom-pytest-node-subclasses-should-take-kwargs

Seeing the below stack. When I change to pytest <= 7.0.0 everything works again. This was with sybil 3.0.0.

.tox/py39-doctest/lib/python3.9/site-packages/_pytest/nodes.py:140: in _create
    return super().__call__(*k, **kw)
E   TypeError: __init__() got an unexpected keyword argument 'path'

During handling of the above exception, another exception occurred:
.tox/py39-doctest/lib/python3.9/site-packages/pluggy/_hooks.py:265: in __call__
    return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
.tox/py39-doctest/lib/python3.9/site-packages/pluggy/_manager.py:80: in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
.tox/py39-doctest/lib/python3.9/site-packages/sybil/integration/pytest.py:120: in pytest_collect_file
    return SybilFile.from_parent(parent, fspath=path, sybil=sybil)
.tox/py39-doctest/lib/python3.9/site-packages/_pytest/nodes.py:633: in from_parent
    return super().from_parent(parent=parent, fspath=fspath, path=path, **kw)
.tox/py39-doctest/lib/python3.9/site-packages/_pytest/nodes.py:264: in from_parent
    return cls._create(parent=parent, **kw)
.tox/py39-doctest/lib/python3.9/site-packages/_pytest/nodes.py:146: in _create
    warnings.warn(
E   pytest.PytestDeprecationWarning: <class 'sybil.integration.pytest.SybilFile'> is not using a cooperative constructor and only takes {'fspath', 'sybil', 'parent'}.
E   See https://docs.pytest.org/en/stable/deprecations.html#constructors-of-custom-pytest-node-subclasses-should-take-kwargs for more details.

Allow tabs in DocTestParser

Hello, thank you very much for this nifty library! In my use case, there are non-Python blocks that contain tabs but these tabs should just be ignored. My workaround is to let DocTestParser take an allow_tabs kwarg in my fork:

https://github.com/jacksonllee/sybil/commit/68944ab9881c8af70254317f36ae996e32faf25c

If there's any interest, I'd be more than happy to contribute a pull request so that it's available in the upstream source here as well.

Codeblock Lexer pattern doesn't work if codeblock ends with close comment (end of input to the lexer)

(Found in Sybil 4.0.1)

TLDR

Unescaped tokens in END_PATTERN_TEMPLATE = '(\n\\Z|\n[ \t]{{0,{len_prefix}}}(?=\\S))'

I think you meant: r'(\n\Z|\n[ \t]{{0,{len_prefix}}}(?=\S))'?

Details

if I have this:

    """
    My comment

    .. code-block:: python
        test_my_code("Hello World")
    """

the PythonCodeBlockParser fails. For example:

.tox/local/lib/python3.8/site-packages/sybil/integration/pytest.py:104: in collect
    self.document = self.sybil.parse(Path(self.fspath.strpath))
.tox/local/lib/python3.8/site-packages/sybil/sybil.py:144: in parse
    return type_.parse(str(path), *self.parsers, encoding=self.encoding)
.tox/local/lib/python3.8/site-packages/sybil/document.py:201: in parse
    for region in parser(docstring_document):
.tox/local/lib/python3.8/site-packages/sybil/parsers/abstract/codeblock.py:46: in __call__
    for lexed in chain(*(lexer(document) for lexer in self.lexers)):
.tox/local/lib/python3.8/site-packages/sybil/parsers/abstract/lexers.py:59: in __call__
    raise LexingException(
E   sybil.parsers.abstract.lexers.LexingException: Could not match '(\n\\Z|\n[ \t]{0,8}(?=\\S))' in my_python.py (truncated)

Neither does this work because my IDE always elides extra whitespace:

    """
    My comment

    .. code-block:: python
        test_my_code("Hello World")

    """

If I do this it will work:

    """
    My comment

    .. code-block:: python
        test_my_code("Hello World")
    \
    """

or this works too:

    """
    My comment

    .. code-block:: python
        test_my_code("Hello World")

    Any other rst comment text here.
    """

REPL functionality in code blocks

I'm creating a ReST document that uses code blocks to explain things in code via REPL-like syntax, like this:

.. code-block:: Python

	class Rectangle:
		pass


That is all we need to have a custom ``Rectangle`` object, as we can see below:


.. code-block:: Python

	>>> rect = Rectangle()
	>>> type(rect)
	<class '__main__.Rectangle'>

When I run code like this through Sybil, I get overlap errors:

================================================================== ERRORS ===================================================================
_________________________________________________ ERROR collecting 1 1 what are classes.rst _________________________________________________
../.venv/lib/python3.9/site-packages/sybil/integration/pytest.py:101: in collect
    self.document = self.sybil.parse(self.fspath.strpath)
../.venv/lib/python3.9/site-packages/sybil/sybil.py:106: in parse
    return Document.parse(path, *self.parsers, encoding=self.encoding)
../.venv/lib/python3.9/site-packages/sybil/document.py:46: in parse
    document.add(region)
../.venv/lib/python3.9/site-packages/sybil/document.py:89: in add
    self.raise_overlap(region, next)
../.venv/lib/python3.9/site-packages/sybil/document.py:69: in raise_overlap
    raise ValueError('{} overlaps {}'.format(*reprs))
E   ValueError: <Region start=936 end=1039 <function evaluate_code_block at 0x1043e4700>> from line 40, column 1 to line 48, column 1 overlaps <Region start=960 end=982 <bound method DocTestParser.evaluate of <sybil.parsers.doctest.DocTestParser object at 0x1043ecbb0>>> from line 42, column 1 to line 43, column 1

I take it Sybil is trying to parse the >>> indented code with a DocTestParser and CodeBlockParser.

I'm not sure how to resolve this. In my conftest.py file, if I remove DocTestParser from the parsers Sybil would use, I get SyntaxErrors:

E           >>> rect = Rectangle()
E       SyntaxError: invalid syntax

Do you have any ideas how I might solve this issue?

test

"skip" in MyST parser markdown documents

According to the sybil docs, you can skip exmamples with ; skip ....

According to the MyST parser docs, comments start with a %.

Consequently, MySt does not hide ; skip ... statements in the output.

I can work around the issue by subclassign the skip parser and setting my own regex that uses % instead of ;.

IndentationError results from trailing carriage return

This issue occurs if the documentation file that is being parsed for code blocks contains \r\n line separators. It results from the implementation of the builtin textwrap.dedent function and the text that Sybil passes to this function.

The regex pattern used for end_pattern in the CodeBlockParser finds the \n but not the corresponding \r which results in a trailing \r.

The following illustrates how textwrap.dedent behaves. The trailing \r that exists when I initially define text does not dedent the text -- which would result in an IndentationError being raised by sybil.parsers.codeblock.compile_codeblock

>>> import textwrap
>>> text = '    from math import cos\r\n    x=cos(0.1)\r\n\r'
>>> textwrap.dedent(text)
'    from math import cos\r\n    x=cos(0.1)\r\n\r'
>>> text = '    from math import cos\r\n    x=cos(0.1)\r\n'
>>> textwrap.dedent(text)
'from math import cos\r\nx=cos(0.1)\r\n'

Here's an example for how to reproduce it with Sybil to raise an IndentationError

from sybil.document import Document
from sybil.example import Example
from sybil.parsers.codeblock import CodeBlockParser

text = 'This is my example:\r\n\r\n.. code-block:: python\r\n\r\n    from math import cos\r\n    x = cos(0.1)\r\n\r\nThat was my example.\r\n'

document = Document(text=text, path='whatever.rst')

region = list(CodeBlockParser()(document))[0]
region.evaluator(
    Example(
        document=document,
        line=0,
        column=0,
        region=region,
        namespace={}
    )
)

A similar issue could exist when using dedent for a capture, but I have not tested it.

Tests are failing

The same issue was report as #4. But on Fedora. The RPM generation is failing as test_nose are not passing.

_____________________________________________________________________ test_nose _____________________________________________________________________

capsys = <_pytest.capture.CaptureFixture object at 0x7f3671e321d0>

    def test_nose(capsys):
        class ResultStoringMain(NoseMain):
            def runTests(self):
                self.testRunner = NoseRunner(stream=sys.stdout,
                                             verbosity=self.config.verbosity,
                                             config=self.config)
                self.result = self.testRunner.run(self.test)
    
        main = ResultStoringMain(
            module=None,
            argv=['x', join(example_dir, 'example_nose')]
        )
>       assert main.result.testsRun == 3
E       assert 1 == 3
E        +  where 1 = <nose.result.TextTestResult run=1 errors=0 failures=0>.testsRun
E        +    where <nose.result.TextTestResult run=1 errors=0 failures=0> = <tests.test_doc_example.test_nose.<locals>.ResultStoringMain object at 0x7f3671e32a90>.result

tests/test_doc_example.py:50: AssertionError
_____________________________________________________________________ test_nose _____________________________________________________________________

capsys = <_pytest.capture.CaptureFixture object at 0x7f3671a8e240>

    def test_nose(capsys):
        class ResultStoringMain(NoseMain):
            def runTests(self):
                self.testRunner = NoseRunner(stream=sys.stdout,
                                             verbosity=self.config.verbosity,
                                             config=self.config)
                self.result = self.testRunner.run(self.test)
    
        main = ResultStoringMain(
            module=None,
            argv=['x', '-vs', join(functional_test_dir, 'nose')]
        )
>       assert main.result.testsRun == 9
E       assert 2 == 9
E        +  where 2 = <nose.result.TextTestResult run=2 errors=0 failures=0>.testsRun
E        +    where <nose.result.TextTestResult run=2 errors=0 failures=0> = <tests.test_functional.test_nose.<locals>.ResultStoringMain object at 0x7f3671a8e630>.result

tests/test_functional.py:163: AssertionError

Request: Add test which fails if `self.evaluator = None` is removed

PythonDocument.evaluator in sybil/document.py includes the line self.evaluator = None.
mypy has issues with this line, and I am trying to resolve those issues.
However, I cannot be confident that my attempts are valid, as I don't understand what this is meant to do, and the tests pass if this line is removed.
My request is that a test is added which fails if this line is removed, so that I can have more confidence.

Question to test set up: Is it possible that CI is testing the coverage of tests?

I am not familiar with pytest. IMHO CI is testing the test coverage of tests.
See https://app.circleci.com/pipelines/github/simplistix/sybil/218/workflows/a3f16774-5da5-4ab6-b9f7-f97007b39ab3/jobs/1027/parallel-runs/0/steps/0-103

tests/helpers.py not in PyPI tarball

When I download a tarball from PyPi, the tests/ folder contains only test_*.py files, so when I try to run the tests, it fails on missing helpers.py. I can easily workaround the issue by downloading the GitHub tarball and run the tests from there. The building and installation work fine with the PyPI tarball.

Please consider tidying the PyPI tarball, either by completing the testsuite or dropping it entirely (not urgent at all). Thanks!

Test regressions since 11496eb5761761b (Fix source filtering)

I seem to be hitting two new test failures:

============================================================== FAILURES ===============================================================
____________________________________________________________ test_unittest ____________________________________________________________

capsys = <_pytest.capture.CaptureFixture object at 0x7f60a0efae00>

    def test_unittest(capsys: CaptureFixture[str]):
        results = run_unittest(capsys, 'unittest')
        out = results.out
        out.then_find('sybil setup')
        out.then_find('fail.rst,line:6,column:1 ... 0\nok')
        out.then_find('fail.rst,line:8,column:1 ... 1\nFAIL')
        out.then_find('fail.rst,line:10,column:1 ... 2\nERROR')
        out.then_find('fail.rst,line:12,column:1 ... 3\nok')
>       out.then_find('sybil teardown 4\nsybil setup')

tests/test_functional.py:98: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <tests.helpers.Finder object at 0x7f60a0ef8dc0>, substring = 'sybil teardown 4\nsybil setup'

    def then_find(self, substring):
>       assert substring in self.text[self.index:], self.text[self.index:]
E       AssertionError: fail.rst,line:12,column:1 ... 3
E         ok
E         sybil teardown 4
E         
E         ======================================================================
E         ERROR: /tmp/sybil/tests/functional/pytest/fail.rst,line:10,column:1
E         ----------------------------------------------------------------------
E         Traceback (most recent call last):
E           File "/tmp/sybil/sybil/integration/unittest.py", line 18, in runTest
E             self.example.evaluate()
E           File "/tmp/sybil/sybil/example.py", line 64, in evaluate
E             result = evaluator(self)
E           File "/tmp/sybil/tests/functional/unittest/test_unittest.py", line 19, in check
E             raise ValueError(message)
E         ValueError: X count was 3 instead of 4
E         
E         ======================================================================
E         FAIL: /tmp/sybil/tests/functional/pytest/fail.rst,line:8,column:1
E         ----------------------------------------------------------------------
E         Traceback (most recent call last):
E           File "/tmp/sybil/sybil/integration/unittest.py", line 18, in runTest
E             self.example.evaluate()
E           File "/tmp/sybil/sybil/example.py", line 66, in evaluate
E             raise SybilFailure(self, result)
E         sybil.example.SybilFailure: Example at /tmp/sybil/tests/functional/pytest/fail.rst, line 8, column 1 did not evaluate as expected:
E         Y count was 3 instead of 2
E         
E         ----------------------------------------------------------------------
E         Ran 8 tests in 0.001s
E         
E         FAILED (failures=1, errors=1)
E         
E       assert 'sybil teardown 4\nsybil setup' in 'fail.rst,line:12,column:1 ... 3\nok\nsybil teardown 4\n\n============================================================...------------------------------------------------------------\nRan 8 tests in 0.001s\n\nFAILED (failures=1, errors=1)\n'

tests/helpers.py:67: AssertionError
___________________________________________________________ test_namespace ____________________________________________________________

capsys = <_pytest.capture.CaptureFixture object at 0x7f60987b5ba0>

    def test_namespace(capsys):
        sybil = Sybil([parse], path='./samples')
        documents = [sybil.parse(p) for p in Path(sybil.path).glob('sample*.txt')]
        actual = []
        for document in documents:
            for example in document:
                print(split(example.path)[-1], example.line)
                example.evaluate()
                actual.append((
                    split(example.path)[-1],
                    example.line,
                    document.namespace['parsed'].copy(),
                ))
        out, _ = capsys.readouterr()
>       assert out.split('\n') == [
            'sample1.txt 1',
            '[0]',
            'sample1.txt 3',
            '[0, 14]',
            'sample2.txt 1',
            '[0]',
            'sample2.txt 3',
            '[0, 13]',
            ''
        ]
E       AssertionError: assert ['sample2.txt 1',\n '[0]',\n 'sample2.txt 3',\n '[0, 13]',\n 'sample1.txt 1',\n '[0]',\n 'sample1.txt 3',\n '[0, 14]',\n ''] == ['sample1.txt 1',\n '[0]',\n 'sample1.txt 3',\n '[0, 14]',\n 'sample2.txt 1',\n '[0]',\n 'sample2.txt 3',\n '[0, 13]',\n '']
E         At index 0 diff: 'sample2.txt 1' != 'sample1.txt 1'
E         Full diff:
E           [
E         +  'sample2.txt 1',
E         +  '[0]',
E         +  'sample2.txt 3',
E         +  '[0, 13]',
E            'sample1.txt 1',
E            '[0]',
E            'sample1.txt 3',
E            '[0, 14]',
E         -  'sample2.txt 1',
E         -  '[0]',
E         -  'sample2.txt 3',
E         -  '[0, 13]',
E            '',
E           ]

tests/test_sybil.py:246: AssertionError
======================================================= short test summary info =======================================================
FAILED tests/test_functional.py::test_unittest - AssertionError: fail.rst,line:12,column:1 ... 3
FAILED tests/test_sybil.py::test_namespace - AssertionError: assert ['sample2.txt 1',\n '[0]',\n 'sample2.txt 3',\n '[0, 13]',\n 'sa...
==================================================== 2 failed, 101 passed in 5.25s ====================================================

I don't know about the first failure but the second one seems to be related to filesystem file ordering being unpredictable, i.e.:

$ find -name sample*
./tests/samples
./tests/samples/sample2.txt
./tests/samples/sample1.txt

A quick bisect points to the following commit as the first bad commit:

commit 11496eb5761761b687ad4889b4173d3124caa844 (HEAD, refs/bisect/bad)
Author: Chris Withers <[email protected]>
Date:   2021-10-21 09:35:39 +0200

    Fix source filtering.
    
    - add exclude param for symmetry with pattern/patterns
    - move all the testing to one place and ensure it's the same for all integration

I've reproduced this with Python 3.10.0 and 3.8.12.

Tests broken by pytest 3.5.0

Since upgrading pytest from 3.4.2 to 3.5.0, sybil's testsuite fails:

============================= test session starts ==============================
platform linux2 -- Python 2.7.14, pytest-3.5.0, py-1.5.3, pluggy-0.6.0 -- /nix/store/ivdcnkynqa01bq6n1srwi1kpjnawifkd-python-2.7.14/bin/python2.7
cachedir: .pytest_cache
rootdir: /build/sybil-1.0.7, inifile: setup.cfg
collecting ... collected 36 items

tests/test_capture.py::test_basic PASSED                                 [  2%]
tests/test_capture.py::test_directive_indent_beyond_block PASSED         [  5%]
tests/test_capture.py::test_directive_indent_equal_to_block PASSED       [  8%]
tests/test_codeblock.py::test_basic PASSED                               [ 11%]
tests/test_codeblock.py::test_future_imports PASSED                      [ 13%]
tests/test_doc_example.py::test_pytest PASSED                            [ 16%]
tests/test_doc_example.py::test_unittest PASSED                          [ 19%]
tests/test_doc_example.py::test_nose PASSED                              [ 22%]
tests/test_doctest.py::test_pass PASSED                                  [ 25%]
tests/test_doctest.py::test_fail PASSED                                  [ 27%]
tests/test_doctest.py::test_fail_with_options PASSED                     [ 30%]
tests/test_doctest.py::test_literals PASSED                              [ 33%]
tests/test_doctest.py::test_min_indent PASSED                            [ 36%]
tests/test_doctest.py::test_tabs PASSED                                  [ 38%]
tests/test_functional.py::test_pytest FAILED                             [ 41%]
tests/test_functional.py::test_unittest PASSED                           [ 44%]
tests/test_functional.py::test_nose PASSED                               [ 47%]
tests/test_sybil.py::TestRegion::test_repr PASSED                        [ 50%]
tests/test_sybil.py::TestExample::test_repr PASSED                       [ 52%]
tests/test_sybil.py::TestExample::test_evaluate_okay PASSED              [ 55%]
tests/test_sybil.py::TestExample::test_evaluate_not_okay PASSED          [ 58%]
tests/test_sybil.py::TestExample::test_evaluate_raises_exception PASSED  [ 61%]
tests/test_sybil.py::TestDocument::test_add PASSED                       [ 63%]
tests/test_sybil.py::TestDocument::test_add_no_overlap PASSED            [ 66%]
tests/test_sybil.py::TestDocument::test_add_out_of_order PASSED          [ 69%]
tests/test_sybil.py::TestDocument::test_add_adjacent PASSED              [ 72%]
tests/test_sybil.py::TestDocument::test_add_before_start PASSED          [ 75%]
tests/test_sybil.py::TestDocument::test_add_after_end PASSED             [ 77%]
tests/test_sybil.py::TestDocument::test_add_overlaps_with_previous PASSED [ 80%]
tests/test_sybil.py::TestDocument::test_add_overlaps_with_next PASSED    [ 83%]
tests/test_sybil.py::TestDocument::test_example_path PASSED              [ 86%]
tests/test_sybil.py::TestDocument::test_example_line_and_column PASSED   [ 88%]
tests/test_sybil.py::TestSybil::test_parse PASSED                        [ 91%]
tests/test_sybil.py::TestSybil::test_all_paths PASSED                    [ 94%]
tests/test_sybil.py::TestSybil::test_all_paths_with_base_directory PASSED [ 97%]
tests/test_sybil.py::test_namespace PASSED                               [100%]

=================================== FAILURES ===================================
_________________________________ test_pytest __________________________________

capsys = <_pytest.capture.CaptureFixture object at 0x7ffff1f9d7d0>

    def test_pytest(capsys):

        class CollectResults:
            def pytest_sessionfinish(self, session):
                self.session = session

        results = CollectResults()
        return_code = pytest_main(['-vs', join(functional_test_dir, 'pytest')],
                                  plugins=[results])
        assert return_code == 1
        assert results.session.testsfailed == 4
        assert results.session.testscollected == 10

        out, err = capsys.readouterr()
        # check we're trimming tracebacks:
        index = out.find('sybil/example.py')
        if index > -1:  # pragma: no cover
            raise AssertionError('\n'+out[index-500:index+500])

        out = Finder(out)
        out.then_find('fail.rst::line:1,column:1')
>       out.then_find('fail.rst sybil setup function_fixture setup\n'
                      'class_fixture setup\n'
                      'module_fixture setup\n'
                      'session_fixture setup\n'
                      'x is currently: 0\n'
                      'FAILED class_fixture teardown\n'
                      'function_fixture teardown')

tests/test_functional.py:44:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <tests.test_functional.Finder object at 0x7ffff1ee2050>
substring = 'fail.rst sybil setup function_fixture setup\nclass_fixture setup\nmodule_fixture setup\nsession_fixture setup\nx is currently: 0\nFAILED class_fixture teardown\nfunction_fixture teardown'

    def then_find(self, substring):
>       assert substring in self.text[self.index:]
E       AssertionError: assert 'fail.rst sybil setup function_fixture setup\nclass_fixture setup\nmodule_fixture setup\nsession_fixture setup\nx is currently: 0\nFAILED class_fixture teardown\nfunction_fixture teardown' in 'fail.rst::line:1,column:1 <- /build/sybil-1.0.7/tests/functional/pytest/fail.rst sybil setup session_fixture setup\nmo...tional/pytest/fail.rst:18: Exception\n====================== 4 failed, 6 passed in 0.05 seconds ======================\n'

tests/test_functional.py:19: AssertionError
===================== 1 failed, 35 passed in 0.41 seconds ======================

3.0.0: pytest is failing in `tests/test_functional.py::test_modules_not_importable_unittest`unit

I'm trying to package your module as an rpm package. So I'm using the typical PEP517 based build, install and test cycle used on building packages from non-root account.

python3 -sBm build -w
install .whl file in </install/prefix>
run pytest with PYTHONPATH pointing to sitearch and sitelib inside </install/prefix>

Here is pytest output:

+ PYTHONPATH=/home/tkloczko/rpmbuild/BUILDROOT/python-sybil-3.0.0-2.fc35.x86_64/usr/lib64/python3.8/site-packages:/home/tkloczko/rpmbuild/BUILDROOT/python-sybil-3.0.0-2.fc35.x86_64/usr/lib/python3.8/site-packages
+ /usr/bin/pytest -ra -q
=========================================================================== test session starts ============================================================================
platform linux -- Python 3.8.12, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
rootdir: /home/tkloczko/rpmbuild/BUILD/sybil-3.0.0, configfile: setup.cfg
plugins: cov-3.0.0
collected 112 items

docs/example-skip.rst .........                                                                                                                                      [  8%]
docs/example.rst ....                                                                                                                                                [ 11%]
docs/parsers.rst ..................                                                                                                                                  [ 27%]
tests/test_capture.py ....                                                                                                                                           [ 31%]
tests/test_codeblock.py ..........                                                                                                                                   [ 40%]
tests/test_doc_example.py ..                                                                                                                                         [ 41%]
tests/test_doctest.py ........                                                                                                                                       [ 49%]
tests/test_functional.py .............................F..                                                                                                            [ 77%]
tests/test_helpers.py ..                                                                                                                                             [ 79%]
tests/test_skip.py ....                                                                                                                                              [ 83%]
tests/test_sybil.py ...................                                                                                                                              [100%]

================================================================================= FAILURES =================================================================================
___________________________________________________________________ test_modules_not_importable_unittest ___________________________________________________________________

tmpdir = local('/tmp/pytest-of-tkloczko/pytest-275/test_modules_not_importable_un0'), capsys = <_pytest.capture.CaptureFixture object at 0x7fd94b9d6670>

    def test_modules_not_importable_unittest(tmpdir: local, capsys: CaptureFixture[str]):
        # NB: no append to sys.path
        results = clone_and_run_modules_tests(tmpdir, capsys, UNITTEST)
        assert results.total == 5, results.out.text
        assert results.failures == 0, results.out.text
        assert results.errors == 5, results.out.text
        a_py = tmpdir/'modules'/'a.py'
        b_py = tmpdir/'modules'/'b.py'
        out = results.out
        out.then_find(f'ERROR: {a_py},line:3,column:1')
        out.then_find(f"ImportError: 'a' not importable from {tmpdir/'modules'/'a.py'} as:")
        out.then_find("ModuleNotFoundError: No module named 'a'")
>       out.then_find(f'ERROR: {b_py},line:2,column:1')

tests/test_functional.py:329:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <tests.helpers.Finder object at 0x7fd94d764af0>, substring = 'ERROR: /tmp/pytest-of-tkloczko/pytest-275/test_modules_not_importable_un0/modules/b.py,line:2,column:1'

    def then_find(self, substring):
>       assert substring in self.text[self.index:], self.text[self.index:]
E       AssertionError: ModuleNotFoundError: No module named 'a'
E
E       ======================================================================
E       ERROR: /tmp/pytest-of-tkloczko/pytest-275/test_modules_not_importable_un0/modules/a.py,line:7,column:1
E       ----------------------------------------------------------------------
E       Traceback (most recent call last):
E         File "/home/tkloczko/rpmbuild/BUILD/sybil-3.0.0/sybil/integration/unittest.py", line 17, in runTest
E           self.example.evaluate()
E         File "/home/tkloczko/rpmbuild/BUILD/sybil-3.0.0/sybil/example.py", line 64, in evaluate
E           result = evaluator(self)
E         File "/home/tkloczko/rpmbuild/BUILD/sybil-3.0.0/sybil/document.py", line 142, in evaluator
E           module = import_path(Path(example.path))
E         File "/home/tkloczko/rpmbuild/BUILD/sybil-3.0.0/sybil/python.py", line 40, in import_path
E           raise ImportError(
E       ImportError: 'a' not importable from /tmp/pytest-of-tkloczko/pytest-275/test_modules_not_importable_un0/modules/a.py as:
E       ModuleNotFoundError: No module named 'a'
E
E       ----------------------------------------------------------------------
E       Ran 5 tests in 0.006s
E
E       FAILED (errors=5)

tests/helpers.py:67: AssertionError
========================================================================= short test summary info ==========================================================================
FAILED tests/test_functional.py::test_modules_not_importable_unittest - AssertionError: ModuleNotFoundError: No module named 'a'
====================================================================== 1 failed, 111 passed in 2.53s =======================================================================

ModuleNotFoundError if pytest is not installed

pytest is not defined as a requirement of sybil, but sybil depends on pytest.

Create a new Python environment
pip install sybil==5.0.3
Run the following code:

import sybil
sybil.Sybil(parsers=[]).pytest()

This raises the error:

ModuleNotFoundError: No module named '_pytest'

This is not caught by the tests because pytest is a test dependency.

I suggest that to fix this, move pytest to the install requirements for sybil.

FWIW a project like pip-check-reqs would find this issue.

Provide a way to isolate tests within one document

I am trying to switch from doctest to code-block with Sybil.
One issue I'm hitting is that I would like to have multiple separated tests within one document.

manuel provides .. reset-globs.
Is there a workaround to allow similar functionality with Sybil?

Unhelpful error with missing final linefeed

This took me a moment to figure out and I feel like the DX could be better.

If you have a file that doesn't have a final line feed like the following:

# Integrations

```{toctree}
:maxdepth: 1

flask
pyramid
custom
```

And there's no \n behind the three backticks, you get an error like this:

sybil.parsers.abstract.lexers.LexingException: Could not match '(?<=\\n)```\\n' in /Users/hynek/FOSS/svcs/docs/integrations/index.md:
E   ':maxdepth: 1\n\nflask\npyramid\ncustom\n```'

I know enough re to shoot myself into the foot, so I figured it out quite quick, but I think it would be good to have a nicer error message.

Numerours PytestDeprecationWarning raised when using Sybil

myvenv/python3.9/site-packages/sybil/integration/pytest.py:58: 193 warnings
  myvenv/lib/python3.9/site-packages/sybil/integration/pytest.py:58: PytestDeprecationWarning: A private pytest class or function was used.
    self._request = fixtures.FixtureRequest(self)

-- Docs: https://docs.pytest.org/en/stable/warnings.html

This has been going on for awhile. Doesn't break anything but it does normalize deviance having a huge number of warnings we are supposed to ignore.

Any ideas how we could fix this?

Parse and export directive options, too

Status quo

ReST/MyST directives can have options, e.g.:

.. code-block:: python
   :caption: example.py

   print("Hello, world!")

'``{code-block} python
:caption: example.py

print("Hello, world!")
'``

The options are already part of the directive regex (rest, myst (although myst use the --- form and not the double colon form which seems to be recommended by MyST).

These options are not part of a capture group, though, and can thus not be used for more advanced stuff.

Use case

I have examples that show a config file, a python script that loads and prints the settings and a console code block that invokes the script and displays its output (similarly to doctests, but for bash). Here is an example: https://typed-settings.readthedocs.io/en/latest/#example

I use the :caption: filename.ext option for directives to indicate that the contents of a code block should be written to that file instead of being executed (that option also has the nice side-effect of being displayed in the docs :)).

For this to work I needed to copy and adjust lexers and parsers from Sybil. This is generally okay for me, but maybe parts of that functionality (or all of it) might be a nice addition to Sybil.

Here is the current code for this: https://gitlab.com/sscherfke/typed-settings/-/blob/main/conftest.py

Proposed changes

Modify the code-block regexes to also capture the directive parameters/options.
Modify the MyST code-block regex to use the double-colon format (because that is what seems to be recommended) (in addition or instead of the --- format).
Modify the lexer(s) to process the options string and convert it to a dictionary (e.g. ,options = {"caption": "example.py"}) and make them part of LexedRegion.lexemes.
Maybe add an options property to Region which is a shortcut to region.lexemes.get("options", {}).
Add a hook (or something) to CodeBlockParser that can be used to trigger user defined functionality (e.g., depending on the directive's parameters). The current behavior is the default and fallback behavior.

If you are interested in adding this to Sybil, I would start working on a PR.

Support an alternative to invisible-code-block for doctests

It took me a while to realize that invisible-code-block only works with code blocks.

As far as I can tell, it is currently not possible to write invisible code as context for a documentation example that has lines of output.

Support GitHub Flavored Markdown (GFM)

Spec is here: https://github.github.com/gfm/

Looks like both doctests and python source are supported in python fenced code blocks:

import this

>>> "a doctest"
"a doctest"

provide the integration as pytest plugin instance instead of generating a function with a hidden instance

the idiomatic way to configure pytest for sybil is to add a plugin to pytest

then a conftest would read:

import pytest
from sybil import Sybil
from sybil.parsers.codeblock import CodeBlockParser
from sybil.parsers.doctest import DocTestParser

pytest_plugins = "sybil.integration.pytest

@pytest.fixture()
def sybil_dir(tmpdir, monkeypatch):
    # there are better ways to do temp directories, but it's a simple example:
    path = tmpdir.mkdir('sybil')
    monkeypatch.chdir(path)
    return path

def pytest_sybil_configure():
  return Sybil(
     parsers=[
        DocTestParser(),
        CodeBlockParser(future_imports=['print_function']),
      ],
      pattern='*.rst',
      fixtures=['sybil_dir']
    )