kernelci / test-definitions

Test definitions work with and without LAVA

Test Definitions

A set of testing scripts designed to work with LAVA. Also contains test-runner script that allows execution outside LAVA.

More details in docs

test-definitions's Issues

kselftest log and results appear to be mangled

As a follow-up from kernelci/kernelci-core#445, there appears to be some issues with the kselftest log being mangled in the LAVA logs. Also, the output appear to be parsed by sed rather than a proper TAP13 parser. These 2 things may be related, if the mangled log output is a side effect of parsing results with sed.

A better approach would seem to be capturing the TAP13 results and parse it with something such as tappy. Then ideally, the output associated with each test case should be output before telling LAVA the test case name and pass/fail status.

False positives in kselftests-rtc (arm64 - kevin)

Seen in some builds but not all. At least in stable-rc/linux-4.19.y: https://linux.kernelci.org/test/plan/id/637dd59d221061bf322abd0b/

Saved log:
kselftest-rtc-rk3399-gru-kevin.html.gz

The job definition yaml is missing a few important parameters:
Example: https://lava.collabora.dev/scheduler/job/7971458

in job definition:

- from: git
  name: kselftest-rtc
  parameters:
    BOARD: rk3399-gru-kevin
    BRANCH: stable-rc
    SKIPFILE: skipfile-lkft.yaml
    TESTPROG_URL: None
    TST_CASENAME: null
    TST_CMDFILES: rtc
  path: automated/linux/kselftest/kselftest.yaml
  repository: https://github.com/kernelci/test-definitions.git
  revision: kernelci.org

TESTPROG_URL and TST_CASENAME aren't getting filled, but I don't know who is supposed to fill in the kselftests_url and kselftest_tests parameters in the job template: https://github.com/kernelci/kernelci-core/blob/main/config/lava/kselftest/kselftest.jinja2

Anyway, this doesn't sit well with the https://github.com/kernelci/test-definitions/blob/main/automated/linux/kselftest/kselftest.sh script. The script will use a default URL if no URL is supplied (

test-definitions/automated/linux/kselftest/kselftest.sh

Line 110 in 2ce1f65

TESTPROG_URL=http://testdata.validation.linaro.org/tests/kselftest/"${TESTPROG}"

) but here it's getting "None" instead, so the actual kselftests aren't even downloaded or run (https://lava.collabora.dev/scheduler/job/7971458#L1802)

I don't know why LAVA considers this a success, though.

Results are showing some skipped tests that aren't related to the collection

Since inclusion of -t and -c options in kselftest framework it's a bit weird show on results unrelated tests to the collection causing some noise on reports. It would be a good improvement to have for collection and tests cases not show this tests from skip list.

E.g.
https://qa-reports.linaro.org/lkft/linux-next-master/build/next-20211111/testrun/6396555/suite/kselftest-seccomp/tests/
https://linux.kernelci.org/test/plan/id/618ca944db6db902573358dc/