asottile-archive / azure-pipeline-templates Goto Github PK

there are some corner cases it doesnt work

Hi,

I was trying to use the pre-commit azure pipeline and it is broken now I guess.
Azure pipeline complains about the way parameters are defined in it.

I changed it to this below and it works. Hope if all look good, kindly change.

parameters:
  - name: ruby
    type: string
    default: '>= 2.4'
  - name: python
    type: string
    default: '3.9'

jobs:
- job: precommit
  pool:
    vmImage: ubuntu-latest
  variables:
    PRE_COMMIT_HOME: $(Pipeline.Workspace)/pre-commit-cache

  steps:
  - task: UsePythonVersion@0
    inputs:
      versionSpec: ${{parameters.python}}
  - task: UseRubyVersion@0
    inputs:
      versionSpec: ${{parameters.ruby}}

  - script: |
      echo "##vso[task.setvariable variable=PY]$(python -VV)"
      echo "##vso[task.setvariable variable=RB]$(ruby -v)"
    displayName: set version variables

  - task: CacheBeta@0
    inputs:
      key: pre-commit | "4" | .pre-commit-config.yaml | "$(PY)" | "$(RB)"
      path: $(PRE_COMMIT_HOME)

  - script: python -m pip install --upgrade pre-commit
    displayName: install pre-commit
  - script: pre-commit run --all-files --show-diff-on-failure
    displayName: run pre-commit

Hi.

I'm trying to move flask-smorest CI from Travis to Azure (marshmallow-code/flask-smorest#195) using azure-pipeline-templates (@sloria's fork, actually, to get the PyPI release template). Thanks for sharing those templates.

My concern is the coverage report.

AFAIU, when using the coverage feature of job--python-tox, the coverage that is displayed is the one of the job that finishes last. This is fine if all jobs have the same coverage, but wrong if jobs don't pass through all branches (e.g. testing conditional code against different versions of a dependency, or different Python versions).

I've been searching the web for solutions to this. Apparently Azure doesn't do a great job with this. Some suggest to use a third-party app called ReportGenerator.

I have the feeling it should be possible to keep the coverage report for each job using pipeline artifacts, then combining them in a later stage using coverage combine.

A CI pipeline could look like

- lint
  flake8
- tests
  pytest py36-dep-v1 + publish coverage as pipeline artifact
  pytest py36-dep-v2 + publish coverage as pipeline artifact
  pytest py39-dep-v1 + publish coverage as pipeline artifact
  pytest py39-dep-v2 + publish coverage as pipeline artifact
- coverage
  download coverage infos as pipeline artifacts + combine + check threshold (`fail-under`) + publish code coverage result
- release
  send to pypi

Is my understanding correct about current issue?

Is this something you would like to address? Does the proposal above make sense?

I've spent hours struggling with Azure and I feel like I'm still scratching the surface.

Out of curiosity, I tried building my fork of pyupgrade on my own Azure Pipelines account but it didn't work:

Repository asottile references endpoint asottile-github which does not exist or is not authorized for use

https://dev.azure.com/hugovk/hugovk/_build/results?buildId=126&view=results

Probably not surprising, as you've documented that the user needs to find the name of their GitHub service connection.

https://github.com/asottile/azure-pipeline-templates#usage

(I couldn't actually find "Project Settings => Service connections in the Azure Devops dashboard for your project", but that's not the main point of this particular issue.)

So it looks like I'd have to put my own value in my forked azure-pipelines.yml, but that would be a pain when wanting to make a PR back upstream.

This is definitely a downside when comparing against Travis which is easy to enable and use for a fork (eg. travis sync && travis enable CLI, or click a couple of button in the UI).

I wonder if there's anything to be done about this? Perhaps the endpoint could be included from another YAML file that's Git ignored? Is it something that should be reported to Azure Pipelines?

This is not urgent in any way! Just thought you might be interested to hear :)