kernel-patches / vmtest Goto Github PK

Consider the following veristat input files:

two-ok:

file_name,prog_name,verdict,total_states
file-a,a,success,12
file-b,b,success,67

add-success:

file_name,prog_name,verdict,total_states
file-a,a,success,12
file-b,b,success,67
file-b,c,success,32

veristat produces the following log (veristat -e file,prog,verdict,states -o csv -C two-ok add-success > $tmp):

file_name,prog_name,verdict_base,verdict_comp,verdict_diff,total_states_base,total_states_comp,total_states_diff
file-a,a,success,success,MATCH,12,12,+0 (+0.00%)
file-b,b,success,success,MATCH,67,67,+0 (+0.00%)
file-b,c,N/A,success,N/A,N/A,32,N/A

And veristat-compare.py fails with the log below:

Traceback (most recent call last):
  File "/home/eddy/work/vmtest/.github/scripts/veristat-compare.py", line 256, in <module>
    sys.exit(main(args.filename, summary_filename))
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/eddy/work/vmtest/.github/scripts/veristat-compare.py", line 233, in main
    veristat_results = parse_table(csv_file)
                       ^^^^^^^^^^^^^^^^^^^^^
  File "/home/eddy/work/vmtest/.github/scripts/veristat-compare.py", line 173, in parse_table
    diff = get_state_diff(record[VeristatFields.TOTAL_STATES_DIFF])
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/eddy/work/vmtest/.github/scripts/veristat-compare.py", line 139, in get_state_diff
    if percentage_diff := matches.group("percentage_diff"):
                          ^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'group'

This causes CI failures, e.g. here:
https://github.com/kernel-patches/bpf/actions/runs/6567849918/job/17845229398#step:35:111

Same as libbpf/ci#47 but for vmtest repo

We started to use 3-column format in https://github.com/kernel-patches/bpf/blob/bpf-next/tools/testing/selftests/bpf/DENYLIST.s390x, where the 3rd column identify where the problem is stemming from.

It would be useful to have the format

1. captured as a header of the file
2. used in the existing files so the next person that updates the file know what to put in.

#154 added netcnt to the DENYLIST because it started failing, as far as I can tell, running the VM with kvm enabled.

The symptom is that it seems 1 packet too many is sent. Troubleshooting will probably be easier with tcpdump as we could spot what ping believe it is sending.

Basically, continue work started in #135

Context: I'm reusing some of the CI infra for a Linux fork on my own GitHub account.

Following commit fa07d13, the description of the GitHub workflow has the following snippet to allow the CI to run on the vmtest repository.

      - name: Get bpf-next source
        if: ${{ github.repository != 'kernel-patches/bpf' }}
        uses: actions/checkout@v2
        with:
          repository: kernel-patches/bpf
          ref: 'bpf-next'
          path: 'linux'
      - name: Move linux source in place
        if: ${{ github.repository != 'kernel-patches/bpf' }}
        run: rsync --exclude .git -av linux/. . && rm -rf linux

This makes the Action download kernel sources before compiling and running the checks, if github.repository != 'kernel-patches/bpf'.

It is rather easy to copy all files from vmtest to another Linux repository and to run the same CI checks from another GitHub account by creating a dedicated PR (e.g. qmonnet/linux#4, the branch was manually created, without the Patchwork parts of the CI), except that the above check will overwrite the Linux sources (and the modifications to test). It's not hard to remove this check before copying the file and running the action, but it makes the operation less straightforward.

Would it be possible to adjust this check? One solution could be to have if: ${{ github.repository == 'kernel-patches/vmtest' }}, although the same issue would arise again if someone tried to run the checks on e.g. a fork of vmtest. Alternatively, would it be possible to check whether the repository seems to contain the Linux sources? E.g. by looking at a few known files maybe?