pytest-asyncio-0.23.2 loads packages more eagerly about pytest-asyncio

[seifertm]

@sanderr @xmatthias @bnavigator
Pytest-asyncio v0.23.4a0 was just released. I encourage you to try the pre-release and I'd love to hear if it fixes your problems.

For example, I tested the bcrypt test suite based on the instructions that @mgorny provided in #738 and the tests pass with v0.23.4a0, whereas the crashed with a different 0.23 version.

[xmatthias]

We're seeing something similar (i suspect) - though due to an even odder case.

Our tests work fine up to pytest-asyncio-0.23.x ... at which point, ONLY the mac runs are failing (they do fail with Fatal Python error: Segmentation fault).
Based on my tests, this is related to the torch module (uninstalling torch completely will fix the issue).

Now torch (or pytorch) is a problematic package to depend on on it's own...
it's doing some funky stuff when imported (it messes with logging, for one - which can be triggered by simply running import torch).

We do have an auto-use fixture to completely mock the torch module on macos to work around this problem - but (i suspect) due to the eager loading of pytest-asyncio, it's loading torch BEFORE actually running the auto-use fixture, therefore importing torch - which causes the random segfaults.

I could also confirm this by messing with the installed torch file (simply placing a raise ValueError() in the __init__.py file confirmed that pytest-asyncio==0.21.1 doesn't import this file, while pytest-asyncio==0.23.2 does during the collection phase (see logs below).

It's unclear to me why pytest-asyncio does need to mess with the importing logic and would import packages that

• don't have a direct import in any test file
• are not imported in a nested way
• are not imported when using an older version of pytest-asyncio

Now torch is imported in some files within the project - which are only referenced through a custom resolver responsible to load the package (similar to the original author of this issue) - but these files are not touched or imported directly anywyere.

In my eyes/understanding (correct me if i'm wrong), pytest-asyncio is a plugin to provide async support for tests, not to modify the test sequence, or help find additional tests - so what's the exact reason to have to modify the importing logic?

Exception details of "importing torch" error

added raise ValueError("IMported torch!") at the bottom of /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/torch/__init__.py on my CI runner

bash-3.2$ pytest --random-order-seed=469386 
=========================================================== test session starts ============================================================
platform darwin -- Python 3.11.7, pytest-7.4.3, pluggy-1.3.0
Using --random-order-bucket=module
Using --random-order-seed=469386

rootdir: /Users/runner/work/freqtrade/freqtrade
configfile: pyproject.toml
plugins: random-order-1.1.0, time-machine-2.13.0, cov-4.1.0, asyncio-0.23.2, mock-3.12.0, anyio-3.7.1, xdist-3.5.0
asyncio: mode=Mode.AUTO
collected 0 items                                                                                                                          
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/_pytest/main.py", line 271, in wrap_s
ession
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>                          ^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/_pytest/main.py", line 324, in _main
INTERNALERROR>     config.hook.pytest_collection(session=session)
INTERNALERROR>   File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/pluggy/_hooks.py", line 493, in __cal
l__
INTERNALERROR>     return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/pluggy/_manager.py", line 115, in _ho
okexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/pluggy/_callers.py", line 152, in _mu
lticall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/pluggy/_result.py", line 114, in get_
result
INTERNALERROR>     raise exc.with_traceback(exc.__traceback__)
INTERNALERROR>   File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/pluggy/_callers.py", line 77, in _mul
ticall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>           ^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/_pytest/main.py", line 335, in pytest
_collection
INTERNALERROR>     session.perform_collect()
INTERNALERROR>   File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/_pytest/main.py", line 675, in perfor
m_collect
INTERNALERROR>     self.items.extend(self.genitems(node))
INTERNALERROR>   File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/_pytest/main.py", line 842, in genite
ms
INTERNALERROR>     rep = collect_one_node(node)
INTERNALERROR>           ^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/_pytest/runner.py", line 546, in coll
ect_one_node
INTERNALERROR>     ihook.pytest_collectstart(collector=collector)
INTERNALERROR>   File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/pluggy/_hooks.py", line 493, in __cal
l__
INTERNALERROR>     return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/pluggy/_manager.py", line 115, in _ho
okexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/pluggy/_callers.py", line 113, in _mu
lticall
INTERNALERROR>     raise exception.with_traceback(exception.__traceback__)
INTERNALERROR>   File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/pluggy/_callers.py", line 77, in _mul
ticall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>           ^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/pytest_asyncio/plugin.py", line 612, 
in pytest_collectstart
INTERNALERROR>     pyobject = collector.obj
INTERNALERROR>                ^^^^^^^^^^^^^
INTERNALERROR>   File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/_pytest/python.py", line 310, in obj
INTERNALERROR>     self._obj = obj = self._getobj()
INTERNALERROR>                       ^^^^^^^^^^^^^^
INTERNALERROR>   File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/_pytest/python.py", line 528, in _get
obj
INTERNALERROR>     return self._importtestmodule()
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/_pytest/python.py", line 617, in _imp
orttestmodule
INTERNALERROR>     mod = import_path(self.path, mode=importmode, root=self.config.rootpath)
INTERNALERROR>           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/_pytest/pathlib.py", line 567, in imp
ort_path
INTERNALERROR>     importlib.import_module(module_name)
INTERNALERROR>   File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/importlib/__init__.py", line 126, in import_module
INTERNALERROR>     return _bootstrap._gcd_import(name[level:], package, level)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
INTERNALERROR>   File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
INTERNALERROR>   File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
INTERNALERROR>   File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
INTERNALERROR>   File "<frozen importlib._bootstrap_external>", line 940, in exec_module
INTERNALERROR>   File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
INTERNALERROR>   File "/Users/runner/work/freqtrade/freqtrade/freqtrade/freqai/tensorboard/__init__.py", line 3, in <module>
INTERNALERROR>     from freqtrade.freqai.tensorboard.tensorboard import TensorBoardCallback, TensorboardLogger
INTERNALERROR>   File "/Users/runner/work/freqtrade/freqtrade/freqtrade/freqai/tensorboard/tensorboard.py", line 5, in <module>
INTERNALERROR>     from torch.utils.tensorboard import SummaryWriter
INTERNALERROR>   File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/torch/__init__.py", line 1836, in <mo
dule>
INTERNALERROR>     raise ValueError("IMported torch!")
INTERNALERROR> ValueError: IMported torch!

[bnavigator]

We also see this on a minimal Pandas installation. Pandas has lots of optional dependencies but imports fine with the minimal requirements. With older pytest-asyncio the test suite would collect fine and skip tests not applicable. But with the new version, the optional Pandas submodules cause an early pytest internalerror.

[    4s] + python3.11 -c 'import pandas; print(pandas.__path__); print(pandas.show_versions())'
[    8s] /usr/lib/python3.11/site-packages/_distutils_hack/__init__.py:33: UserWarning: Setuptools is replacing distutils.
[    8s]   warnings.warn("Setuptools is replacing distutils.")
[    8s] ['/home/abuild/rpmbuild/BUILD/pandas-2.1.4/pandas']
[    8s] 
[    8s] INSTALLED VERSIONS
[    8s] ------------------
[    8s] commit              : a671b5a8bf5dd13fb19f0e88edc679bc9e15c673
[    8s] python              : 3.11.6.final.0
[    8s] python-bits         : 64
[    8s] OS                  : Linux
[    8s] OS-release          : 6.6.6-1-default
[    8s] Version             : #1 SMP PREEMPT_DYNAMIC Mon Dec 11 09:46:39 UTC 2023 (a946a9f)
[    8s] machine             : x86_64
[    8s] processor           : x86_64
[    8s] byteorder           : little
[    8s] LC_ALL              : en_US.UTF-8
[    8s] LANG                : en_US.UTF-8
[    8s] LOCALE              : en_US.UTF-8
[    8s] 
[    8s] pandas              : 2.1.4
[    8s] numpy               : 1.26.2
[    8s] pytz                : 2023.3.post1
[    8s] dateutil            : 2.8.2
[    8s] setuptools          : 69.0.3
[    8s] pip                 : 23.3.2
[    8s] Cython              : 0.29.36
[    8s] pytest              : 7.4.4
[    8s] hypothesis          : 6.92.1
[    8s] sphinx              : None
[    8s] blosc               : None
[    8s] feather             : None
[    8s] xlsxwriter          : None
[    8s] lxml.etree          : None
[    8s] html5lib            : None
[    8s] pymysql             : None
[    8s] psycopg2            : None
[    8s] jinja2              : None
[    8s] IPython             : None
[    8s] pandas_datareader   : None
[    8s] bs4                 : None
[    8s] bottleneck          : None
[    8s] dataframe-api-compat: None
[    8s] fastparquet         : None
[    8s] fsspec              : None
[    8s] gcsfs               : None
[    8s] matplotlib          : None
[    8s] numba               : None
[    8s] numexpr             : None
[    8s] odfpy               : None
[    8s] openpyxl            : None
[    8s] pandas_gbq          : None
[    8s] pyarrow             : None
[    8s] pyreadstat          : None
[    8s] pyxlsb              : None
[    8s] s3fs                : None
[    8s] scipy               : None
[    8s] sqlalchemy          : None
[    8s] tables              : None
[    8s] tabulate            : None
[    8s] xarray              : None
[    8s] xlrd                : None
[    8s] zstandard           : None
[    8s] tzdata              : None
[    8s] qtpy                : None
[    8s] pyqt5               : None
[    8s] None
[    8s] + xvfb-run pytest-3.11 -vvv -rsfE -n0 -m 'not (network or clipboard or single_cpu)' -k 'not (test_pivot_number_of_levels_larger_than_int32 or psycopg2_engine or psycopg2_conn or pymysql_engine or pymysql_conn or test_git_version)' pandas
[   12s] ============================= test session starts ==============================
[   12s] platform linux -- Python 3.11.6, pytest-7.4.4, pluggy-1.3.0 -- /usr/bin/python3.11
[   12s] cachedir: .pytest_cache
[   12s] hypothesis profile 'ci' -> deadline=None, suppress_health_check=[HealthCheck.too_slow, HealthCheck.differing_executors], database=DirectoryBasedExampleDatabase(PosixPath('/home/abuild/rpmbuild/BUILD/pandas-2.1.4/.hypothesis/examples'))
[   12s] rootdir: /home/abuild/rpmbuild/BUILD/pandas-2.1.4/pandas
[   12s] configfile: pyproject.toml
[   12s] plugins: asyncio-0.23.3, xdist-3.5.0, hypothesis-6.92.1
[   12s] asyncio: mode=Mode.STRICT
[   12s] collecting ... collected 0 items
[   12s] INTERNALERROR> Traceback (most recent call last):
[   12s] INTERNALERROR>   File "/usr/lib/python3.11/site-packages/_pytest/python.py", line 617, in _importtestmodule
[   12s] INTERNALERROR>     mod = import_path(self.path, mode=importmode, root=self.config.rootpath)
[   12s] INTERNALERROR>           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[   12s] INTERNALERROR>   File "/usr/lib/python3.11/site-packages/_pytest/pathlib.py", line 567, in import_path
[   12s] INTERNALERROR>     importlib.import_module(module_name)
[   12s] INTERNALERROR>   File "/usr/lib64/python3.11/importlib/__init__.py", line 126, in import_module
[   12s] INTERNALERROR>     return _bootstrap._gcd_import(name[level:], package, level)
[   12s] INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[   12s] INTERNALERROR>   File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
[   12s] INTERNALERROR>   File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
[   12s] INTERNALERROR>   File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
[   12s] INTERNALERROR>   File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
[   12s] INTERNALERROR>   File "<frozen importlib._bootstrap_external>", line 940, in exec_module
[   12s] INTERNALERROR>   File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
[   12s] INTERNALERROR>   File "/home/abuild/rpmbuild/BUILD/pandas-2.1.4/pandas/core/_numba/kernels/__init__.py", line 1, in <module>
[   12s] INTERNALERROR>     from pandas.core._numba.kernels.mean_ import (
[   12s] INTERNALERROR>   File "/home/abuild/rpmbuild/BUILD/pandas-2.1.4/pandas/core/_numba/kernels/mean_.py", line 13, in <module>
[   12s] INTERNALERROR>     import numba
[   12s] INTERNALERROR> ModuleNotFoundError: No module named 'numba'
[   12s] INTERNALERROR> 
[   12s] INTERNALERROR> The above exception was the direct cause of the following exception:
[   12s] INTERNALERROR> 
[   12s] INTERNALERROR> Traceback (most recent call last):
[   12s] INTERNALERROR>   File "/usr/lib/python3.11/site-packages/_pytest/main.py", line 271, in wrap_session
[   12s] INTERNALERROR>     session.exitstatus = doit(config, session) or 0
[   12s] INTERNALERROR>                          ^^^^^^^^^^^^^^^^^^^^^
[   12s] INTERNALERROR>   File "/usr/lib/python3.11/site-packages/_pytest/main.py", line 324, in _main
[   12s] INTERNALERROR>     config.hook.pytest_collection(session=session)
[   12s] INTERNALERROR>   File "/usr/lib/python3.11/site-packages/pluggy/_hooks.py", line 493, in __call__
[   12s] INTERNALERROR>     return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
[   12s] INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[   12s] INTERNALERROR>   File "/usr/lib/python3.11/site-packages/pluggy/_manager.py", line 115, in _hookexec
[   12s] INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
[   12s] INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[   12s] INTERNALERROR>   File "/usr/lib/python3.11/site-packages/pluggy/_callers.py", line 152, in _multicall
[   12s] INTERNALERROR>     return outcome.get_result()
[   12s] INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^
[   12s] INTERNALERROR>   File "/usr/lib/python3.11/site-packages/pluggy/_result.py", line 114, in get_result
[   12s] INTERNALERROR>     raise exc.with_traceback(exc.__traceback__)
[   12s] INTERNALERROR>   File "/usr/lib/python3.11/site-packages/pluggy/_callers.py", line 77, in _multicall
[   12s] INTERNALERROR>     res = hook_impl.function(*args)
[   12s] INTERNALERROR>           ^^^^^^^^^^^^^^^^^^^^^^^^^
[   12s] INTERNALERROR>   File "/usr/lib/python3.11/site-packages/_pytest/main.py", line 335, in pytest_collection
[   12s] INTERNALERROR>     session.perform_collect()
[   12s] INTERNALERROR>   File "/usr/lib/python3.11/site-packages/_pytest/main.py", line 675, in perform_collect
[   12s] INTERNALERROR>     self.items.extend(self.genitems(node))
[   12s] INTERNALERROR>   File "/usr/lib/python3.11/site-packages/_pytest/main.py", line 842, in genitems
[   12s] INTERNALERROR>     rep = collect_one_node(node)
[   12s] INTERNALERROR>           ^^^^^^^^^^^^^^^^^^^^^^
[   12s] INTERNALERROR>   File "/usr/lib/python3.11/site-packages/_pytest/runner.py", line 546, in collect_one_node
[   12s] INTERNALERROR>     ihook.pytest_collectstart(collector=collector)
[   12s] INTERNALERROR>   File "/usr/lib/python3.11/site-packages/pluggy/_hooks.py", line 493, in __call__
[   12s] INTERNALERROR>     return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
[   12s] INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[   12s] INTERNALERROR>   File "/usr/lib/python3.11/site-packages/pluggy/_manager.py", line 115, in _hookexec
[   12s] INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
[   12s] INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[   12s] INTERNALERROR>   File "/usr/lib/python3.11/site-packages/pluggy/_callers.py", line 113, in _multicall
[   12s] INTERNALERROR>     raise exception.with_traceback(exception.__traceback__)
[   12s] INTERNALERROR>   File "/usr/lib/python3.11/site-packages/pluggy/_callers.py", line 77, in _multicall
[   12s] INTERNALERROR>     res = hook_impl.function(*args)
[   12s] INTERNALERROR>           ^^^^^^^^^^^^^^^^^^^^^^^^^
[   12s] INTERNALERROR>   File "/usr/lib/python3.11/site-packages/pytest_asyncio/plugin.py", line 626, in pytest_collectstart
[   12s] INTERNALERROR>     pyobject = collector.obj
[   12s] INTERNALERROR>                ^^^^^^^^^^^^^
[   12s] INTERNALERROR>   File "/usr/lib/python3.11/site-packages/_pytest/python.py", line 310, in obj
[   12s] INTERNALERROR>     self._obj = obj = self._getobj()
[   12s] INTERNALERROR>                       ^^^^^^^^^^^^^^
[   12s] INTERNALERROR>   File "/usr/lib/python3.11/site-packages/_pytest/python.py", line 528, in _getobj
[   12s] INTERNALERROR>     return self._importtestmodule()
[   12s] INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^
[   12s] INTERNALERROR>   File "/usr/lib/python3.11/site-packages/_pytest/python.py", line 642, in _importtestmodule
[   12s] INTERNALERROR>     raise self.CollectError(
[   12s] INTERNALERROR> _pytest.nodes.Collector.CollectError: ImportError while importing test module '/home/abuild/rpmbuild/BUILD/pandas-2.1.4/pandas/core/_numba/kernels/__init__.py'.
[   12s] INTERNALERROR> Hint: make sure your test modules/packages have valid Python names.
[   12s] INTERNALERROR> Traceback:
[   12s] INTERNALERROR> /usr/lib/python3.11/site-packages/_pytest/python.py:617: in _importtestmodule
[   12s] INTERNALERROR>     mod = import_path(self.path, mode=importmode, root=self.config.rootpath)
[   12s] INTERNALERROR> /usr/lib/python3.11/site-packages/_pytest/pathlib.py:567: in import_path
[   12s] INTERNALERROR>     importlib.import_module(module_name)
[   12s] INTERNALERROR> /usr/lib64/python3.11/importlib/__init__.py:126: in import_module
[   12s] INTERNALERROR>     return _bootstrap._gcd_import(name[level:], package, level)
[   12s] INTERNALERROR> <frozen importlib._bootstrap>:1204: in _gcd_import
[   12s] INTERNALERROR>     ???
[   12s] INTERNALERROR> <frozen importlib._bootstrap>:1176: in _find_and_load
[   12s] INTERNALERROR>     ???
[   12s] INTERNALERROR> <frozen importlib._bootstrap>:1147: in _find_and_load_unlocked
[   12s] INTERNALERROR>     ???
[   12s] INTERNALERROR> <frozen importlib._bootstrap>:690: in _load_unlocked
[   12s] INTERNALERROR>     ???
[   12s] INTERNALERROR> <frozen importlib._bootstrap_external>:940: in exec_module
[   12s] INTERNALERROR>     ???
[   12s] INTERNALERROR> <frozen importlib._bootstrap>:241: in _call_with_frames_removed
[   12s] INTERNALERROR>     ???
[   12s] INTERNALERROR> pandas/core/_numba/kernels/__init__.py:1: in <module>
[   12s] INTERNALERROR>     from pandas.core._numba.kernels.mean_ import (
[   12s] INTERNALERROR> pandas/core/_numba/kernels/mean_.py:13: in <module>
[   12s] INTERNALERROR>     import numba
[   12s] INTERNALERROR> E   ModuleNotFoundError: No module named 'numba'
[   12s] 
[   12s] ============================ no tests ran in 0.58s =============================

[sanderr]

I have to confess that I haven't given it a lot of thought yet, but at least for my scenario I guess it would be possible (albeit through a bit of a hack) to not use __init__.py files as testing artifacts, or to modify their load-time behavior when loaded in an unexpected context.

I agree there's no urgent need to upgrade for now, but I also don't want to lag behind indefinitely. Therefore, if this turns out to be considered a low-priority issue, I might consider trying to work around it from our side, rather than to keep the pinned version. This would allow us to keep moving forward now, rather than if/when a need to upgrade does pop up.

Don't get me wrong, I'd much prefer it if it the old behavior would be restored, but that doesn't necessarily mean that the maintainer(s?) feel the same, or that it fits their agenda, hence the question.

[blink1073]

v23.4.a1 is failing for a different reason: https://github.com/blink1073/ipykernel/actions/runs/7480301750/job/20359431173?pr=1

[mgorny]

I'm very sorry but I've found another regression.

git clone https://github.com/pydata/pydata-sphinx-theme/
cd pydata-sphinx-theme
pip install .[test]
python -m pytest  # → collected 66 items / 1 skipped
pip install pytest-asyncio==0.23.4a2
python -m pytest  # → collected 0 items / 1 skipped 

[seifertm]

I'm very sorry but I've found another regression.

git clone https://github.com/pydata/pydata-sphinx-theme/
cd pydata-sphinx-theme
pip install .[test]
python -m pytest  # → collected 66 items / 1 skipped
pip install pytest-asyncio==0.23.4a2
python -m pytest  # → collected 0 items / 1 skipped 

Good catch! I made yet another change to the test collection that seems to fix the issue. Rather than another pre-release, I did a proper release of v0.23.4, because pytest 8 has been released and older pytest-asyncio version don't have proper upper bounds on the pytest version.

[sanderr]

@seifertm Thanks for your efforts in triaging this. I see you added this ticket to the v0.23 milestone. Does that mean you aim to address it sometime soon? I'm just trying to determine whether we should give it some more time, or start looking for a longer term workaround.

[xmatthias]

@sanderr i'm not entirely sure what a longer-term workaround would look like (?)

I've had a look around, but it doesn't seem that there's alternative plugins that provide the same functionality.
For the moment, i think we're fine to just pin the dependency to a non-broken version - there's no (known) security enhancements that would force us upgrade - at least not for the moment ..

[seifertm]

It's unclear to me why pytest-asyncio does need to mess with the importing logic and would import packages that
* don't have a direct import in any test file
* are not imported in a nested way
* are not imported when using an older version of pytest-asyncio

@xmatthias The concept of scoped event loops essentially requires generating pytest fixtures dynamically, depending on the structure of the test suite. This is a feature that's currently not supported in pytest. The current workaround, which is also used by pytest internally for calling things such as setUp in unittest suites, is to programmatically add fixture definitions to a module during collection time. See the following code example:
https://github.com/pytest-dev/pytest/blob/23906106968eb95afbd61adfbc7bbb795fc9aaa9/src/_pytest/python.py#L564-L577

Accessing a the obj attribute of a pytest.Module collector triggers a module import. Accessing pytest.Package.obj imports the package's __init__.py. That means when pytest-asyncio tries to attach a scoped event loop fixture to a package, that package is imported.

Starting from pytest v8, pytest.Package no longer has an obj attribute. We could try attaching the package-scoped event loop fixture to a "virtual" module that doesn't have a corresponding file on the file system.

@sanderr In my opinion, a pytest plugin should not change the way tests are collected and the old behaviour should be restored. I'm planning to do this in a patch release for v0.23. I don't think you should have to look into changes of your test suite, because of this issue.

I set aside some time for pytest-asyncio this weekend, but I cannot promise I will resolve this issue then. I'm sorry the 0.23 releases are causing so many problems, but currently my support of pytest-asyncio happens on a best-effort basis in my free time.

As for alternative testing packages, there's anyio, which comes with a pytest-plugin. I have no experience how much effort it is to migrate to anyio, though.

[sanderr]

Thanks for the update. I understand that you can not provide any guarantees with respect to the timeline but it's good to have rough idea of how and when this will move forward. That's all I was after at this point. Thank you for looking into this, and for the clear communication.

[bnavigator]

I can confirm that the pandas test suite now sucessfully collects the tests in the minimal environment:

Pandas environment and pytest header

[   53s] + python3.11 -c 'import pandas; print(pandas.__path__); print(pandas.show_versions())'
[   55s] /usr/lib/python3.11/site-packages/_distutils_hack/__init__.py:33: UserWarning: Setuptools is replacing distutils.
[   55s]   warnings.warn("Setuptools is replacing distutils.")
[   55s] ['/home/abuild/rpmbuild/BUILD/pandas-2.1.4/pandas']
[   55s] 
[   55s] INSTALLED VERSIONS
[   55s] ------------------
[   55s] commit              : a671b5a8bf5dd13fb19f0e88edc679bc9e15c673
[   55s] python              : 3.11.7.final.0
[   55s] python-bits         : 64
[   55s] OS                  : Linux
[   55s] OS-release          : 6.6.9-1-default
[   55s] Version             : #1 SMP PREEMPT_DYNAMIC Tue Jan  2 07:19:30 UTC 2024 (61d1d44)
[   55s] machine             : x86_64
[   55s] processor           : x86_64
[   55s] byteorder           : little
[   55s] LC_ALL              : en_US.UTF-8
[   55s] LANG                : en_US.UTF-8
[   55s] LOCALE              : en_US.UTF-8
[   55s] 
[   55s] pandas              : 2.1.4
[   55s] numpy               : 1.26.2
[   55s] pytz                : 2023.3.post1
[   55s] dateutil            : 2.8.2
[   55s] setuptools          : 69.0.3
[   55s] pip                 : 23.3.2
[   55s] Cython              : 0.29.36
[   55s] pytest              : 7.4.4
[   55s] hypothesis          : 6.92.2
[   55s] sphinx              : None
[   55s] blosc               : None
[   55s] feather             : None
[   55s] xlsxwriter          : None
[   55s] lxml.etree          : None
[   55s] html5lib            : None
[   55s] pymysql             : None
[   55s] psycopg2            : None
[   55s] jinja2              : None
[   55s] IPython             : None
[   55s] pandas_datareader   : None
[   55s] bs4                 : None
[   55s] bottleneck          : None
[   55s] dataframe-api-compat: None
[   55s] fastparquet         : None
[   55s] fsspec              : None
[   55s] gcsfs               : None
[   55s] matplotlib          : None
[   55s] numba               : None
[   55s] numexpr             : None
[   55s] odfpy               : None
[   55s] openpyxl            : None
[   55s] pandas_gbq          : None
[   55s] pyarrow             : None
[   55s] pyreadstat          : None
[   55s] pyxlsb              : None
[   55s] s3fs                : None
[   55s] scipy               : None
[   55s] sqlalchemy          : None
[   55s] tables              : None
[   55s] tabulate            : None
[   55s] xarray              : None
[   55s] xlrd                : None
[   55s] zstandard           : None
[   55s] tzdata              : None
[   55s] qtpy                : None
[   55s] pyqt5               : None
[   55s] None
[   56s] + xvfb-run pytest-3.11 -vvv -rsfE -n0 -m 'not (network or clipboard or single_cpu)' -k 'not (test_pivot_number_of_levels_larger_than_int32 or psycopg2_engine or psycopg2_conn or pymysql_engine or pymysql_conn or test_git_version)' pandas
[   59s] ============================= test session starts ==============================
[   59s] platform linux -- Python 3.11.7, pytest-7.4.4, pluggy-1.3.0 -- /usr/bin/python3.11
[   59s] cachedir: .pytest_cache
[   59s] hypothesis profile 'ci' -> deadline=None, suppress_health_check=[HealthCheck.too_slow, HealthCheck.differing_executors], database=DirectoryBasedExampleDatabase(PosixPath('/home/abuild/rpmbuild/BUILD/pandas-2.1.4/.hypothesis/examples'))
[   59s] rootdir: /home/abuild/rpmbuild/BUILD/pandas-2.1.4/pandas
[   59s] configfile: pyproject.toml
[   59s] plugins: asyncio-0.23.4a0, xdist-3.5.0, hypothesis-6.92.2
[   59s] asyncio: mode=Mode.STRICT
[  128s] collecting ... collected 203562 items / 2692 deselected / 59 skipped / 200870 selected
[  128s] 

[blink1073]

Here's what I'm seeing with 0.23.4a0:

https://github.com/blink1073/ipykernel/actions/runs/7469345207/job/20326312710?pr=1

[xmatthias]

Hi @seifertm,

Thanks for this - the test that was failing (in my case, the MacOS version of our CI) is working fine now (previous errors: https://github.com/freqtrade/freqtrade/actions/runs/7442875401).

With the new alpha version - i'm now getting a (new) failure on Windows which i don't quite understand - but is most likely windows specific, and based on your idea posted above, most likely caused by the new virtual module that's being added.

Link to the Actions run: https://github.com/xmatthias/freqtrade/actions/runs/7471096789/job/20330820415

Error on Windows

============================= test session starts =============================
platform win32 -- Python 3.[11](https://github.com/xmatthias/freqtrade/actions/runs/7471096789/job/20330820415#step:6:12).7, pytest-7.4.4, pluggy-1.3.0
Using --random-order-bucket=module
Using --random-order-seed=78313

rootdir: D:\a\freqtrade\freqtrade
configfile: pyproject.toml
plugins: anyio-4.2.0, asyncio-0.23.4a0, cov-4.1.0, mock-3.[12](https://github.com/xmatthias/freqtrade/actions/runs/7471096789/job/20330820415#step:6:13).0, random-order-1.1.0, xdist-3.5.0, time-machine-2.[13](https://github.com/xmatthias/freqtrade/actions/runs/7471096789/job/20330820415#step:6:14).0
asyncio: mode=Mode.AUTO
collected 3319 items / 1 error / 250 deselected / 3069 selected

=================================== ERRORS ====================================
_____________________ ERROR collecting tests/__init__.py ______________________
C:\hostedtoolcache\windows\Python\3.11.7\x64\Lib\site-packages\pytest_asyncio\plugin.py:716: in _patched_collect
    fixturemanager.parsefactories(virtual_module.obj, nodeid=pkg_nodeid)
C:\hostedtoolcache\windows\Python\3.11.7\x64\Lib\site-packages\_pytest\python.py:310: in obj
    self._obj = obj = self._getobj()
C:\hostedtoolcache\windows\Python\3.11.7\x64\Lib\site-packages\_pytest\python.py:528: in _getobj
    return self._importtestmodule()
C:\hostedtoolcache\windows\Python\3.11.7\x64\Lib\site-packages\_pytest\python.py:617: in _importtestmodule
    mod = import_path(self.path, mode=importmode, root=self.config.rootpath)
C:\hostedtoolcache\windows\Python\3.11.7\x64\Lib\site-packages\_pytest\pathlib.py:567: in import_path
    importlib.import_module(module_name)
C:\hostedtoolcache\windows\Python\3.11.7\x64\Lib\importlib\__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
<frozen importlib._bootstrap>:1204: in _gcd_import
    ???
<frozen importlib._bootstrap>:1176: in _find_and_load
    ???
<frozen importlib._bootstrap>:1[14](https://github.com/xmatthias/freqtrade/actions/runs/7471096789/job/20330820415#step:6:15)7: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:690: in _load_unlocked
    ???
<frozen importlib._bootstrap_external>:936: in exec_module
    ???
<frozen importlib._bootstrap_external>:1073: in get_code
    ???
<frozen importlib._bootstrap_external>:1130: in get_data
    ???
E   PermissionError: [Errno 13] Permission denied: 'D:\\a\\freqtrade\\freqtrade\\tests\\pytest_asyncio_virtual_module_oq5zt_ka.py'
=========================== short test summary info ===========================
ERROR tests/__init__.py - PermissionError: [Errno 13] Permission denied: 'D:\\a\\freqtrade\\freqtrade\\tests\\pytest_asyncio_virtual_module_oq5zt_ka.py'
!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
====================== [25](https://github.com/xmatthias/freqtrade/actions/runs/7471096789/job/20330820415#step:6:26)0 deselected, 1 error in 6.91s =======================

edit seems to be a similar error / identical than the one posted by @blink1073 above ...

[sanderr]

Thanks. I can confirm that the issue I observed on our test suite is fixed with version 0.23.4a0.

[seifertm]

Thank you for your feedback!

There are still problems when running on Windows due to platform specific behavior of tempfile.NamedTemporaryFile. Therefore, I enabled testing on Windows in the pytest-asyncio CI, where I could reproduce the issue. I just merged #746 which seems to fix it.

@xmatthias @blink1073 I tagged the pre-release v0.23.4.a1 which you can use for testing. If you can verify that it also works for you, I'll tag v0.23.4

[xmatthias]

@seifertm thanks a lot for your efforts!

the v0.23.4.a1 release seems to work fine now! 👍

https://github.com/xmatthias/freqtrade/actions/runs/7479730385/job/20357544175

[mgorny]

After upgrading from 0.23.3 to 0.23.4a1, I'm seeing random packages failing with pytest-xdist with errors like:

Different tests were collected between gw2 and gw7.

Example log (from setuptools): dev-python:setuptools-69.0.3:20240111-191012.log

[seifertm]

I created another pre-release (v0.23.4a2) which no longer creates temporary files during test runs. The dynamic package-scoped fixtures are now attached to the first test module in each package.

This should eliminate any issues with test suites that rely on a specific set of collected items (I assume this is the issue for setuptools).

I tested the pre-release against ipythonkernel and bcrypt successfully. One other user also confirmed that the code fixes their INTERNALERROR.

I'll tag a proper patch release by the end of the week, unless I receive any further reports regarding this issue.

[mgorny]

Thanks a lot. I've just tested it with known suspects, and I'll be running it doing version bumps for the next days. I'll let you know if I hit something else.

[xmatthias]

The 2nd pre-release appears to work fine on all OS's, from python 3.9 to python 3.12. 👍

[Mark90]

Hello, I think there's still a problem as of pytest-asyncio==0.23.5. When I have pytest configured to run doctests and ignore any modules that raise import errors, like so:

[tool:pytest]
addopts=--doctest-modules --doctest-ignore-import-errors

then this raises an ImportError during test collection. When I downgrade to pytest-asyncio==0.21.0 this is not the case.

(Sorry if this is not entirely relevant, I didn't read the entire thread)