niclasvaneyk / pytest-gitlab-code-quality

Collects warnings while testing and generates a GitLab Code Quality Report.

GitLab Code Quality Report Provider for Pytest Warnings

Pytest plugin that generates a GitLab Code Quality Report file from the warnings emitted when running the tests.

Terminal	GitLab

If you run GitLab Premium or Ultimate, you should even see the warnings right next to the code in the diff view of merge requests. The official documentation contains more information and screenshots.

Getting Started

Install the plugin using a package manager of your choice

pip install pytest-gitlab-code-quality

then specify the output location for the report using the --gitlab-code-quality-report option

pytest --gitlab-code-quality-report=pytest-warnings.json

In GitLab CI, this will look similar to this:

# .gitlab-ci.yml
pytest:
  stage: test
  image: python
  script:
    - pip install -r requirements.txt # Or however you install your dependencies
    - python -m pytest --gitlab-code-quality-report=pytest-warnings.json

  # The three lines below are required in order for the warnings to show up!
  artifacts:
    reports:
      codequality: pytest-warnings.json

Motivation

Some warnings are only surfaced during runtime, so static analyzers do not always catch them. The screenshots at the top of this document contain the example of a questionably configured SQLAlchemy model. Tests are a cheap way to surface such issues.

While you may run the tests locally and see these warnings there, you also might overlook them or don't know whether they were introduced by your changes or were already present before. Either way, I think it makes sense to explicitly surface and track them during code review instead of burying them in the CI logs that nobody looks at when the tests pass. And that is exactly why this plugin was created.