`doctest` cannot be used to test markdown files. about cpython

[sobolevn]

I would not say that Markdown is not supported, it is. I use Markdown and doctests in multiple projects. Here's just one example: https://github.com/wemake-services/inspect313

But, yes. You have to keep in mind that triple backticks are treated as an output. So, you have to add an empty line before the backticks: https://github.com/wemake-services/inspect313/blob/1ed3f77059199aa17db4be0fe713315bda87d101/README.md#L46-L48

[MusicalNinjaDad]

Yes, you can workaround by adding a blank line at the end but that adds an easy source of error, looks ugly in the code and I've run into cases where the block gets rendered with blank line. Sorry, I didn't include this point in the original description - I felt it was already quite long for such a small change.

[oscarbenjamin]

There are also downstream issues in pytest (pytest-dev/pytest#7374) and pytest-doctestplus (scientific-python/pytest-doctestplus#221). This behaviour is inherited by those libraries as well since they depend on the stdlib doctest module for collecting the doctest fragments. The SymPy in-house doctest runner removes these (https://github.com/sympy/sympy/blob/d91b8ad6d36a59a879cc70e5f4b379da5fdd46ce/sympy/testing/runtests.py#L1827-L1831).

It looks like doctest was not designed for markdown files. It doesn't know anything about markdown backticks and just uses a regex starting with PS1 and ending with either PS1 or a blank line:

cpython/Lib/doctest.py

Lines 608 to 624 in 027fa2e

	# This regular expression is used to find doctest examples in a
	# string. It defines three groups: `source` is the source code
	# (including leading indentation and prompts); `indent` is the
	# indentation of the first (PS1) line of the source code; and
	# `want` is the expected output (including leading indentation).
	_EXAMPLE_RE = re.compile(r'''
	# Source consists of a PS1 line followed by zero or more PS2 lines.
	(?P<source>
	(?:^(?P<indent> [ ]*) >>> .*) # PS1 line
	(?:\n [ ]* \.\.\. .*)*) # PS2 lines
	\n?
	# Want consists of any non-blank lines that do not start with PS1.
	(?P<want> (?:(?![ ]*$) # Not a blank line
	(?![ ]*>>>) # Not a line starting with PS1
	.+$\n? # But any other line
	)*)
	''', re.MULTILINE | re.VERBOSE)

[MusicalNinjaDad]

It looks like this is worthy of a discussion, as it's clearly not something that will "just be OK" but may be of use to the community nonetheless. I'll open one in the morning when I have sufficient time at the keyboard.
@oscarbenjamin - what do you think of the proposed change to adjust the regex to:

    _EXAMPLE_RE = re.compile(r'''
        # Source consists of a PS1 line followed by zero or more PS2 lines.
        (?P<source>
            (?:^(?P<indent> [ ]*) >>>    .*)    # PS1 line
            (?:\n           [ ]*  \.\.\. .*)*)  # PS2 lines
        \n?
        # Want consists of any non-blank lines that do not start with PS1.
        (?P<want> (?:(?![ ]*$)    # Not a blank line
                     (?![ ]*```)  # Not end of a code block
                     (?![ ]*>>>)  # Not a line starting with PS1
                     .+$\n?       # But any other line
                  )*)
        ''', re.MULTILINE | re.VERBOSE)

[oscarbenjamin]

what do you think of the proposed change to adjust the regex to:

The obvious failure case would be if the test output includes triple backticks. That doesn't seem like a big deal to me but I might be wrong.

I imagine that this is best handled outside of the stdlib in the first instance though. See scientific-python/pytest-doctestplus#247.