Coverage calculation has degraded? about jupyterlab-lsp

[bollwyvl]

I'll have a look. We'll likely want to tighten up some of the test/lint deps (e.g. pytest-flake8, consider dropping pytest-cov and using coverage run -m directly).

It brings up the larger question of the packaging infrastructure: whatever we pick, we should get everything we can into pyproject.toml, even if we just stay on setuptools for the time being. Very much not interested in the layer cake of hatch-jupyter-builder, but we could re-examine the previous proposal to adopt a higher-order automation system that would better encapsulate the cross-language and -package entropy.

[bollwyvl]

Oh, and while I'm working on coverage: I'll definitely plan to fire up the jupyter server (and therefore the serverextension) under coverage during robot tests.

Out of scope on the PR that fixes this issue is to get typescript coverage from the as-executed acceptance tests with istanbul and nyc: this takes some rather intense surgery to a build, but is definitely worth it for getting the maximum value out of starting up those real browsers.

coverage.py (and eventually nyc) can output lcov data, so a final sweep-up job can combine all of those into a big picture of all .py (and eventually .ts) stuff, emitted as a job summary without relying on third-party integrations (ick)... getting diff from main is a little trickier (and better now with the updated actions/upload-artifact, but also doable.

[krassowski]

@bollwyvl do you have anything locally?

For reference:

Failing runs:

============================= test session starts ==============================
platform linux -- Python 3.8.18, pytest-7.4.4, pluggy-1.4.0 -- /usr/share/miniconda3/envs/test/bin/python
cachedir: .pytest_cache
rootdir: /home/runner/work/jupyterlab-lsp/jupyterlab-lsp
plugins: asyncio-0.23.4, flake8-1.1.1, cov-4.1.0, github-actions-annotate-failures-0.2.0, anyio-4.2.0
asyncio: mode=strict
collecting ... collected 103 items

[...]

---------- coverage: platform linux, python 3.8.18-final-0 -----------
Name                                                                                                             Stmts   Miss  Cover   Missing
----------------------------------------------------------------------------------------------------------------------------------------------
/usr/share/miniconda3/envs/test/lib/python3.8/site-packages/jupyter_lsp/handlers.py                                 68     22    68%   13-17, 22-23, 62-70, 75-77, 80-84, 87-88, 91-92
/usr/share/miniconda3/envs/test/lib/python3.8/site-packages/jupyter_lsp/manager.py                                 123     22    82%   192, 196-206, 209-222, 225-235, 238-248
/usr/share/miniconda3/envs/test/lib/python3.8/site-packages/jupyter_lsp/paths.py                                    22      5    77%   41-45
/usr/share/miniconda3/envs/test/lib/python3.8/site-packages/jupyter_lsp/serverextension.py                          25      1    96%   82
/usr/share/miniconda3/envs/test/lib/python3.8/site-packages/jupyter_lsp/session.py                                  94     38    60%   80-93, 101-102, 104-105, 107-108, 111, 118-121, 125-126, 129, 133, 143-144, 148, 154, 159-164, 167, 170, 176-179
/usr/share/miniconda3/envs/test/lib/python3.8/site-packages/jupyter_lsp/specs/bash_language_server.py               14      1    93%   30
/usr/share/miniconda3/envs/test/lib/python3.8/site-packages/jupyter_lsp/stdio.py                                    96     60    38%   42-44, 47-48, 64, 70-72, 78, 82-95, 121-145, 149-172, 177-178, 188-197, 201-202
/usr/share/miniconda3/envs/test/lib/python3.8/site-packages/jupyter_lsp/tests/conftest.py                           76     76     0%   1-156
/usr/share/miniconda3/envs/test/lib/python3.8/site-packages/jupyter_lsp/tests/listener.py                            2      2     0%   1-2
/usr/share/miniconda3/envs/test/lib/python3.8/site-packages/jupyter_lsp/tests/test_auth.py                          81     81     0%   2-117
/usr/share/miniconda3/envs/test/lib/python3.8/site-packages/jupyter_lsp/tests/test_bad_spec.py                       7      7     0%   1-19
/usr/share/miniconda3/envs/test/lib/python3.8/site-packages/jupyter_lsp/tests/test_conf_d.py                         6      6     0%   1-6
/usr/share/miniconda3/envs/test/lib/python3.8/site-packages/jupyter_lsp/tests/test_detect.py                        24     24     0%   1-41
/usr/share/miniconda3/envs/test/lib/python3.8/site-packages/jupyter_lsp/tests/test_extension.py                     26     26     0%   1-49
/usr/share/miniconda3/envs/test/lib/python3.8/site-packages/jupyter_lsp/tests/test_listener.py                      51     51     0%   1-118
/usr/share/miniconda3/envs/test/lib/python3.8/site-packages/jupyter_lsp/tests/test_paths.py                         31     31     0%   1-109
/usr/share/miniconda3/envs/test/lib/python3.8/site-packages/jupyter_lsp/tests/test_session.py                       57     57     0%   1-102
/usr/share/miniconda3/envs/test/lib/python3.8/site-packages/jupyter_lsp/tests/test_stdio.py                         33     33     0%   1-88
/usr/share/miniconda3/envs/test/lib/python3.8/site-packages/jupyter_lsp/tests/test_virtual_documents_shadow.py     [135](https://github.com/jupyter-lsp/jupyterlab-lsp/actions/runs/7814645646/job/21316502635?pr=1051#step:18:136)    135     0%   1-266
/usr/share/miniconda3/envs/test/lib/python3.8/site-packages/jupyter_lsp/trait_types.py                              22      9    59%   16, 30-39
/usr/share/miniconda3/envs/test/lib/python3.8/site-packages/jupyter_lsp/types.py                                   113     18    84%   97-98, 118-123, 128, 176-194, 225-226, 239
/usr/share/miniconda3/envs/test/lib/python3.8/site-packages/jupyter_lsp/virtual_documents_shadow.py                118     81    31%   19-24, 32, 35, 38, 43-49, 53-54, 58-64, 68-70, 73-81, 91-96, 128-209
python_packages/jupyter_lsp/jupyter_lsp/handlers.py                                                                 68     18    74%   13-17, 22-23, 57-70, 74-77
python_packages/jupyter_lsp/jupyter_lsp/serverextension.py                                                          25      1    96%   82
python_packages/jupyter_lsp/jupyter_lsp/tests/test_auth.py                                                          81      7    91%   37-38, 81-84, 112
python_packages/jupyter_lsp/jupyter_lsp/tests/test_paths.py                                                         31      1    97%   56
----------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                                                                                             2908    813    72%

68 files skipped due to complete coverage.

FAIL Required test coverage of 100% not reached. Total coverage: 72.04%

=================== 95 passed, 8 skipped in 85.74s (0:01:25) ===================

Passing runs:

============================= test session starts ==============================
platform linux -- Python 3.8.18, pytest-7.4.4, pluggy-1.3.0 -- /usr/share/miniconda3/envs/test/bin/python
cachedir: .pytest_cache
rootdir: /home/runner/work/jupyterlab-lsp/jupyterlab-lsp
plugins: flake8-1.1.1, github-actions-annotate-failures-0.2.0, asyncio-0.23.3, anyio-4.2.0, cov-4.1.0
asyncio: mode=strict
collecting ... collected 73 items

[...]

---------- coverage: platform linux, python 3.8.18-final-0 -----------
Name    Stmts   Miss  Cover   Missing
-------------------------------------
TOTAL    1308      0   100%

46 files skipped due to complete coverage.

Required test coverage of 100% reached. Total coverage: 100.00%

=================== 68 passed, 5 skipped in 84.22s (0:01:24) ===================

[krassowski]

This is pretty tough - I cannot reproduce this locally, and trying out different things on CI (using omit= */site-packages/*, */tests/*, uninstalling jupyter-lsp copy coming from conda, changing how python paths injected) does not help.

[bollwyvl]

We're certainly going to not hit 100% without a lot of #pragma, due to the auth stuff. I have something in progress locally... Not in a pushable state as yet, and will probably take a bit of fighting with CI once I do.