canonical / checkbox Goto Github PK

Checkbox

Home Page: https://checkbox.readthedocs.io

License: GNU General Public License v3.0

Shell 6.24% Python 53.27% Vim Script 0.05% Makefile 0.08% CSS 0.10% HTML 29.48% Perl 9.58% QML 0.16% ActionScript 0.01% C++ 0.23% C 0.80% M4 0.02%

checkbox's Introduction

What is Checkbox?

Checkbox is a testing framework used to validate device compatibility with Ubuntu Linux. It’s the testing tool developed for the purposes of the Ubuntu Certification program.

You can use Checkbox to check if your system is behaving correctly, or you can develop your own set of tests to check your needs.

Checkbox generates test reports in different formats (HTML, JSON, JUnit, plain text) that can be used to easily share the results of a test session.

For more information, check the documentation (installation instructions).

Getting started

To get started with Checkbox, follow the Checkbox tutorial (installation instructions).

Check the contributing guide to learn how to:

setup a test environment
run Checkbox and its providers
run the associated tests
share your contributions with everyone.

Content of this repository

Checkbox is composed of a number of parts:

checkbox-core-snap: snapcraft recipe to build the Checkbox core snap which contains the Checkbox runtime and the public providers (i.e. the checkbox[16|18|20|22] snaps in the Snap store)
checkbox-ng: the core application
checkbox-snap: snapcraft recipe to build the Checkbox test runner (i.e. the checkbox snap in the Snap store)
checkbox-support: Python scripts and helper modules (for instance information parsers for different Linux utilities) used by Checkbox and its providers
metabox: application to help test and validate Checkbox in different configurations using Linux containers or virtual machines

providers: the main providers (base¹, resource) along with other public providers²:

└── providers
    ├── base
    ├── certification-client
    ├── certification-server
    ├── docker
    ├── genio
    ├── gpgpu
    ├── iiotg
    ├── resource
    ├── sru
    └── tpm2

formerly known as "Checkbox provider" or plainbox-provider-checkbox ↩
beside Checkbox framework bundled providers, providers hosted elsewhere can also be used (e.g. providers specific to private projects). ↩

checkbox's People

Contributors

Stargazers

Watchers

Forkers

pieqq isabella232 weizhenwu medicalwei yphus clairlin53 binli pseudocc baconyao xanthein eugene-yujinwu os369510 rickwu666666 zongminl seankingyang dannf liaou3 kevinyehk hector-cao hanhsuan variabledeclared rpjday stanley31huang dbruno74 xnox rmartin013 cyruslien kaichuan-hsieh patliuu mckees andychi117 saviq tang-mm kissiel gabrielchencc alexandredosim hugh712 artur-at-work alanqq0624 cheeyanglee mthaddon 0xff07 waveform80

checkbox's Issues

LP1982291: [Checkbox SNAP] Checkbox crash while running thunderbolt3/insert test

This issue was migrated from https://bugs.launchpad.net/plainbox-provider-checkbox/+bug/1982291

Summary

Status	Created on	Heat	Importance	Security related
New	2022-07-20 05:06:35	6	Critical	False

Description

[Summary]
Checkbox crash while running the thunderbolt3/insert test

[Steps to reproduce]

Install checkbox20 and checkbox snaps
Issue command to perform thunderbolt3/insert test
checkbox.checkbox-cli run com.canonical.certification::after-suspend-thunderbolt3/insert

[Expected result]
thunderbolt3/insert test passed

[Actual result]
thunderbolt3/insert test failed, and the thunderbolt3/storage and thunderbolt3/remove tests shall has the same problem.

[Failure rate]

[Checkbox job com.canonical.certification::after-suspend-thunderbolt3/insert output]

stdout

INSERT NOW

stderr

Traceback (most recent call last):
File "/snap/checkbox-iiotg/21/checkbox-runtime/usr/lib/python3/dist-packages/dbus/bus.py", line 177, in activate_name_owner
return self.get_name_owner(bus_name)
File "/snap/checkbox-iiotg/21/checkbox-runtime/usr/lib/python3/dist-packages/dbus/bus.py", line 361, in get_name_owner
return self.call_blocking(BUS_DAEMON_NAME, BUS_DAEMON_PATH,
File "/snap/checkbox-iiotg/21/checkbox-runtime/usr/lib/python3/dist-packages/dbus/connection.py", line 652, in call_blocking
reply_message = self.send_message_with_reply_and_block(
dbus.exceptions.DBusException: org.freedesktop.DBus.Error.NameHasNoOwner: Could not get owner of name 'org.freedesktop.UDisks': no such name

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/tmp/nest-pgp_x2ar.30d6185b436b5daa6a2fc33c1a0806378add146da5fd3b86324e64b4940af1fb/removable_storage_watcher.py", line 940, in
sys.exit(main())
File "/tmp/nest-pgp_x2ar.30d6185b436b5daa6a2fc33c1a0806378add146da5fd3b86324e64b4940af1fb/removable_storage_watcher.py", line 934, in main
return listener.check(args.timeout)
File "/tmp/nest-pgp_x2ar.30d6185b436b5daa6a2fc33c1a0806378add146da5fd3b86324e64b4940af1fb/removable_storage_watcher.py", line 104, in check
self._starting_devices = self.get_existing_devices()
File "/tmp/nest-pgp_x2ar.30d6185b436b5daa6a2fc33c1a0806378add146da5fd3b86324e64b4940af1fb/removable_storage_watcher.py", line 226, in get_existing_devices
ud_manager_obj = self._bus.get_object("org.freedesktop.UDisks",
File "/snap/checkbox-iiotg/21/checkbox-runtime/usr/lib/python3/dist-packages/dbus/bus.py", line 241, in get_object
return self.ProxyObjectClass(self, bus_name, object_path,
File "/snap/checkbox-iiotg/21/checkbox-runtime/usr/lib/python3/dist-packages/dbus/proxies.py", line 250, in init
self._named_service = conn.activate_name_owner(bus_name)
File "/snap/checkbox-iiotg/21/checkbox-runtime/usr/lib/python3/dist-packages/dbus/bus.py", line 182, in activate_name_owner
self.start_service_by_name(bus_name)
File "/snap/checkbox-iiotg/21/checkbox-runtime/usr/lib/python3/dist-packages/dbus/bus.py", line 277, in start_service_by_name
return (True, self.call_blocking(BUS_DAEMON_NAME, BUS_DAEMON_PATH,
File "/snap/checkbox-iiotg/21/checkbox-runtime/usr/lib/python3/dist-packages/dbus/connection.py", line 652, in call_blocking
reply_message = self.send_message_with_reply_and_block(
dbus.exceptions.DBusException: org.freedesktop.DBus.Error.ServiceUnknown: The name org.freedesktop.UDisks was not provided by any .service files

[Additional information]
CID:
SKU:
system-manufacturer: AAEON
system-product-name: UPX-TGL01
bios-version: 5.19
CPU: Intel(R) Celeron(R) 6305E @ 1.80GHz (2x)
GPU: 00:02.0 VGA compatible controller [0300]: Intel Corporation Device [8086:9a78] (rev 01)
kernel-version: 5.15.0-1010-intel-iotg

[Stage]
Issue reported and logs collected at a later stage

Attachments

checkbox-session.tgz
snap_list.log
acpidump.log
sosreport-ubuntu-2022-07-20-rfdjbgz.tar.gz

Tags:
['checkbox', 'test-case']

LP1954766: fingerprint/verify-no-match is stuck

This issue was migrated from https://bugs.launchpad.net/plainbox-provider-checkbox/+bug/1954766

Summary

Status	Created on	Heat	Importance	Security related
Confirmed	2021-12-14 11:56:14	6	High	False

Description

[Summary]
fingerprint/verify-no-match is stuck when verifying fingerprint

[Steps to reproduce]

Run fingerprint/enroll to enroll the fingerprint and the case is passed
Run fingerprint/verify-no-match afterwards

[Actual result]
test case got stuck when verifying fingerprint

[I/O logs]
Using device /net/reactivated/Fprint/Device/0
Listing enrolled fingers:
- #0: right-index-finger
Verify started!
Verifying: right-index-finger

#after re-run
Using device /net/reactivated/Fprint/Device/0
failed to claim device: GDBus.Error:net.reactivated.Fprint.Error.AlreadyInUse: Device was already claimed

[Additional information]
SKU: CB16P-DVT1-C1
Image: canonical-oem-somerville-focal-amd64-20200502-85+fossa-minccino+X185
system-manufacturer: Dell Inc.
system-product-name: Precision 7670
bios-version: 88.64.1
CPU: Genuine Intel(R) 0000 (24x)
GPU: 0000:00:02.0 VGA compatible controller [0300]: Intel Corporation Device [8086:4688] (rev 0c)
0000:01:00.0 VGA compatible controller [0300]: NVIDIA Corporation Device [10de:24b9] (rev a1)
kernel-version: 5.14.0-1010-oem

Attachments

checkbox-session.tgz
snap_list.log
acpidump.log
sosreport-16-C1-Full-2021-12-14-zwfabaa.tar.gz

Tags:
['cbox-17', 'hwe-fingerprint']

LP1845160: should include the test case that hook thunderbolt device before system power on.

This issue was migrated from https://bugs.launchpad.net/plainbox-provider-checkbox/+bug/1845160

Summary

Status	Created on	Heat	Importance	Security related
Confirmed	2019-09-24 08:48:51	10	High	False

Description

to cover the case like this issue https://bugs.launchpad.net/ubuntu/+source/bolt/+bug/1844650
that user 1st thunderbolt device pluging in happens in power off mode.

Attachments

No attachments

Tags:
['ce-qa-concern', 'oem-priority']

LP1958482: The test job led/mute and led/microphone-mute does not included in the test plan led-cert-manual

This issue was migrated from https://bugs.launchpad.net/plainbox-provider-checkbox/+bug/1958482

Summary

Status	Created on	Heat	Importance	Security related
New	2022-01-20 04:14:52	6	Critical	False

Description

The test job 'com.canonical.certification::led/mute' and 'com.canonical.certification::led/microphone-mute' does not include in the test plan
led-cert-manual, but the test job after-suspend-led/mute and after-suspend-led/microphone-mute was include in the after-suspend-led-cert-full. For those test plan include the led-cert-manual and after-suspend-led-cert-full, it will not run led/mute and led/microphone-mute before suspend, but run twice after suspend.

I thought the led/mute and led/microphone-mute here is just missed and should be added into the led-cert-manual.

Attachments

microphone_mute_led.png

Tags:
['cbox-6']

LP1905650: checkbox may resume a session as "root", causing problems with some jobs

This issue was migrated from https://bugs.launchpad.net/plainbox-provider-checkbox/+bug/1905650

Summary

Status	Created on	Heat	Importance	Security related
Triaged	2020-11-26 02:42:11	14	High	False

Description

[Summary]

This issue was identified while running OEM tests on 202010-28292 device, but it happens with other devices as well.

Some jobs fail with unexpected errors. For instance, audio/2_playback_displayport_* [1] fails with:

subprocess.CalledProcessError: Command '['pactl', 'info']' returned non-zero exit status 1.
Connection failure: Connection refused
pa_context_connect() failed: Connection refused

It turns out it's because the command pactl info is run as root. (try running sudo pactl info on your own device and you will have the same problem)

It seems that, when Checkbox automatically resumes a session after rebooting, it resumes the session as root.

I noticed this because if I open another tab in the gnome-terminal window that has been opened by Checkbox, the user is root.

[Steps to reproduce]

Run a session that involves an audio job (that calls pactl at some point) after a reboot

[Expected outcome]

The jobs run normally and pass

[Actual outcome]

The jobs spit some weird output and/or fail (because they are run as root when they should be run as a normal user)

[Additional info]

Checkbox is used in local mode (not remote)

checkbox-ng: 1.11.0
checkbox-support: 0.48.0
com.canonical.ce:oem: 1.0
certification-client: 0.43.0
plainbox-provider-checkbox: 0.55.0
plainbox-provider-resource-generic: 0.46.0
plainbox-provider-tpm2: 1.14.0

[1] https://certification.canonical.com/hardware/202010-28292/submission/189495/test/87427/result/15223112/

Attachments

No attachments

Tags:
[]

CPU Frequency Test fails on AMD Genoa and Jammy

https://certification.canonical.com/hardware/202211-30840/submission/289256/test/110216/result/29829602/

[CpuFreqTest Results]
{0: {400000: ['474%', 'Fail', 1897149], 3709000: ['76%', 'Fail', 2822385]},
1: {400000: ['474%', 'Fail', 1897149], 3709000: ['76%', 'Fail', 2822403]},
2: {400000: ['474%', 'Fail', 1897149], 3709000: ['76%', 'Fail', 2822394]},
3: {400000: ['474%', 'Fail', 1897149], 3709000: ['76%', 'Fail', 2822432]},
4: {400000: ['474%', 'Fail', 1897149], 3709000: ['76%', 'Fail', 2822428]},
5: {400000: ['474%', 'Fail', 1897149], 3709000: ['76%', 'Fail', 2822413]},...

LP1829970: [Drift-3] [Ironhide-T-3] Test case 'graphics/x_driver_version_PCI_ID_0xXXXX' is always failed on Nvidia driver 430.

This issue was migrated from https://bugs.launchpad.net/plainbox-provider-checkbox/+bug/1829970

Summary

Status	Created on	Heat	Importance	Security related
Confirmed	2019-05-22 06:32:43	6	High	False

Description

Test machine: Drift-3
Image: sutton-mccarthy-bionic-amd64-20190520-115
CPU: Intel(R) Core(TM) i7-8565U CPU @ 1.80GHz
GPU: VGA compatible controller [0300]: Intel Corporation Device [8086:3ea0] (prog-if 00 [VGA controller]) + 3D controller [0302]: NVIDIA Corporation Device [10de:1d13] (rev a1)
BIOS: N2IET62W (1.40)

Steps:

Install image on Drift-3.
Install checkbox and execute it.
Checked test case 'graphics/1_driver_version_PCI_ID_0x3ea0'.
Checked test case 'graphics/2_driver_version_PCI_ID_0x1d13'.
Checked test case 'Manual test of suspend function'
Checked test case 'Test X driver/version after suspend for Intel Corporation PCI ID 0x3ea0'
Checked test case 'Test X driver/version after suspend for NVIDIA Corporation PCI ID 0x1d13'
Test those cases and verify the results.

Expected result:
All test cases should be pass.

Actual result:
Test cases are failed always
'graphics/1_driver_version_PCI_ID_0x3ea0';
'graphics/2_driver_version_PCI_ID_0x1d13';
'Test X driver/version after suspend for Intel Corporation PCI ID 0x3ea0';
'Test X driver/version after suspend for NVIDIA Corporation PCI ID 0x1d13'.

--------------[ Running job 1 / 2. Estimated time left: 0:00:01 ]---------------
---------[ Test X driver/version for NVIDIA Corporation PCI ID 0x1d13 ]---------
ID: com.canonical.certification::graphics/2_driver_version_PCI_ID_0x1d13
Category: com.canonical.plainbox::graphics
... 8< -------------------------------------------------------------------------
------------- VIDEO DRIVER INFORMATION -------------
------------- HYBRID GRAPHICS CHECK ----------------
Graphics Chipset: Intel (8086:3ea0)
Graphics Chipset: NVIDIA (10de:1d13 (rev a1))
Loaded DDX Drivers: modesetting, fbdev, vesa
Hybrid Graphics: yes
ERROR: No video driver loaded! Possibly in failsafe mode!
------------------------------------------------------------------------- >8 ---
Outcome: job failed
--------------[ Running job 2 / 2. Estimated time left: 0:00:00 ]---------------
---------[ Test X driver/version for Intel Corporation PCI ID 0x3ea0 ]----------
ID: com.canonical.certification::graphics/1_driver_version_PCI_ID_0x3ea0
Category: com.canonical.plainbox::graphics
... 8< -------------------------------------------------------------------------
ERROR: No video driver loaded! Possibly in failsafe mode!
------------- VIDEO DRIVER INFORMATION -------------
------------- HYBRID GRAPHICS CHECK ----------------
Graphics Chipset: Intel (8086:3ea0)
Graphics Chipset: NVIDIA (10de:1d13 (rev a1))
Loaded DDX Drivers: modesetting, fbdev, vesa
Hybrid Graphics: yes
------------------------------------------------------------------------- >8 ---
Outcome: job failed

Notes:
Run those test cases on a laptop with nvidia-418

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
-------------[ Running job 31 / 32. Estimated time left: unknown ]--------------
---------[ Test X driver/version for Intel Corporation PCI ID 0x3e9b ]----------
ID: com.canonical.certification::graphics/1_driver_version_PCI_ID_0x3e9b
Category: com.canonical.plainbox::graphics
... 8< -------------------------------------------------------------------------
------------- VIDEO DRIVER INFORMATION -------------
Video Driver: nvidia
Driver Version: 418.56

------------- HYBRID GRAPHICS CHECK ----------------
Graphics Chipset: Intel (8086:3e9b)
Graphics Chipset: NVIDIA (10de:1cbd)
Loaded DDX Drivers: nouveau, modesetting, fbdev, vesa
Hybrid Graphics: yes
------------------------------------------------------------------------- >8 ---
Outcome: job passed
-------------[ Running job 32 / 32. Estimated time left: unknown ]--------------
---------[ Test X driver/version for NVIDIA Corporation PCI ID 0x1cbd ]---------
ID: com.canonical.certification::graphics/2_driver_version_PCI_ID_0x1cbd
Category: com.canonical.plainbox::graphics
... 8< -------------------------------------------------------------------------
------------- VIDEO DRIVER INFORMATION -------------
Video Driver: nvidia
Driver Version: 418.56

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

Attachments

nvidia-dpkg.list
nvidia-bug-report.log.gz
system-logs.tar.gz

Tags:
['affect-submission', 'ce-qa-concern']

LP1910451: ethernet/multi_nic_device{index}_{interface} test does not fail on 5GBase-T controller connected to 1Gb network

This issue was migrated from https://bugs.launchpad.net/plainbox-provider-checkbox/+bug/1910451

Summary

Status	Created on	Heat	Importance	Security related
Confirmed	2021-01-06 21:43:49	6	High	False

Description

Network devices must be connected to a network segment that is at or above their max line rate. As you know, a 10Gb controller has to be connected to a network of at least 10Gb/s capability (or faster).

There's a new crop of low-end NICs out there that are 2.5 and 5Gb ethernet.

https://certification.canonical.com/hardware/202012-28475/submission/191584/test/64025/result/15449384/

Settings for eno1:
Supported ports: [ TP ]
Supported link modes: 100baseT/Full
1000baseT/Full
2500baseT/Full
5000baseT/Full
Supported pause frame use: Symmetric
Supports auto-negotiation: Yes
Supported FEC modes: Not reported
Advertised link modes: 100baseT/Full
1000baseT/Full
2500baseT/Full
5000baseT/Full
Advertised pause frame use: Symmetric
Advertised auto-negotiation: Yes
Advertised FEC modes: Not reported
Speed: 1000Mb/s
Duplex: Full
Port: Twisted Pair
PHYAD: 0
Transceiver: internal
Auto-negotiation: on
MDI-X: Unknown
Supports Wake-on: g
Wake-on: d
Link detected: yes

This advertises a 5Gb max speed, but was connected to a 1Gb network, and still passed the network stress test.

What SHOULD have happened is that the network test should have failed because 1GbE is too slow for a 5GbE NIC.

This NIC would need to be connected to a 10Gb Lan segment (I don't know if 5Gb switches exist) in order to be tested at the proper speed.

Need to figure out why the network test created a false positive here instead of failing this test case.

Attachments

No attachments

Tags:
['blocks-hwcert-server']

LP1712301: Need S3 stress test for discrete graphic

This issue was migrated from https://bugs.launchpad.net/plainbox-provider-checkbox/+bug/1712301

Summary

Status	Created on	Heat	Importance	Security related
Confirmed	2017-08-22 09:59:39	14	High	False

Description

We do have S3 single test for discrete graphic (Nvidia graphic) but no stress test. It would be better to have S3 stress for Nvidia and AMD dGPUs as it is cert blocker.

How to switch graphic
To switch to Nvidia card:

sudo prime-select nvidia
reboot system
prime-select query to check which card is being used right now

To switch to Intel card:
sudo prime-select intel

For amdgpu-pro stack, it is also switchable (amdgpu is not). We need S3 stress test for amdgpu-pro stack as well. Checkbox already has a way to identify amdgpu and amdgpu-pro.

To switch graphic mode on amdgpu-pro stack
sudo /opt/amdgpu-pro/bin/amdgpu-pro-px --mode performance/powersaving

The idea is to have S3 stress test for discrete graphic in stress test plan and can be completed without interaction.
The critical part is system needs reboot after switching graphic mode (performance-discrete and power saving-UMA).

Attachments

/usr/share/plainbox-provider-checkbox/units/graphics/test-plan.pxu

Tags:
['ce-qa-concern']

Displayport job be skipped since gnome-randr cannot get DP information

This issue was migrated from https://bugs.launchpad.net/plainbox-provider-checkbox/+bug/1982288

Summary

Status	Created on	Heat	Importance	Security related
Confirmed	2022-07-20 03:13:19	6	Critical	False

Description

[Summary]
There are two output port, HDMI and Displayport, on "All In One" DUT.
But "gnome-randr" cannot get the information of Displayport, cause the checkbox cases related to Displayport are skipped automatically.
- See: https://certification.canonical.com/hardware/202207-30419/submission/272693/test/168578/result/27221500/

If Windowing System is Wayland, checkbox grabs the info of monitor via "gnome-randr" command.
commit: https://git.launchpad.net/plainbox-provider-resource/commit/?id=8901ac96bb0b52d54c9501240b07b6a95783ced9

[Steps to reproduce]

Boot into OS
Run 'gnome-randr' command to get the information of physical monitors

[Expected result]
It should give HDMI and DP information

[Actual result]
Only HDMI information

gnome-randr's output on DAKAR-DGPU-EVT-C6

supports-mirroring: true
layout-mode: physical
supports-changing-layout-mode: false
global-scale-required: false
renderer: "native"
legacy-ui-scaling-factor: 1

logical monitor 0:
x: 0, y: 0, scale: 1, rotation: normal, primary: yes
associated physical monitors:
 HDMI-1 DEL OptiPlex AIO 0x44474c78

HDMI-1 DEL OptiPlex AIO 0x44474c78
  [email protected]	1920x1080 	62.04*+   	[x1.00+, x2.00]
                  1920x1080@60	1920x1080 	60.00     	[x1.00+, x2.00]
  [email protected]	1680x1050 	59.88     	[x1.00+, x2.00]
   [email protected]	1440x900  	59.90     	[x1.00+]
  [email protected]	1280x1024 	60.02     	[x1.00+]
                   1280x720@60	1280x720  	60.00     	[x1.00+]
   [email protected]	1024x768  	60.00     	[x1.00+]
    [email protected]	800x600   	60.32     	[x1.00+]
is-builtin: false
display-name: "Dell Inc. 23\""

[Failure rate]
always

[Additional Info]
CID: 202207-30419, 202207-30420
SKU: DAKAR-DGPU-EVT-C6, DAKAR-UMA-EVT-C4
Image: dell-bto-jammy-jellyfish-chansey-X21-20220704-2.iso

Attachments

HDMI and Displayport output port
DAKAR-UMA-EVT-C2_only_DP_output
output.tar.xz

Tags:
['jammy-test']

LP1832688: Improve lid tests

This issue was migrated from https://bugs.launchpad.net/plainbox-provider-checkbox/+bug/1832688

Summary

Status	Created on	Heat	Importance	Security related
Confirmed	2019-06-13 08:16:09	6	High	False

Description

LP:868678 lid_open and lid_close tests need to be re-written
LP:1103925 lid_open test does not make sense
LP:1832683 Test case lid_open was done immediately
LP:1832678 Automated power-management/lid test

Following the idea to automated power-management/lid test (suspend), I am thinking we could merge lid(suspend), lid_close and lid_open into one or improve the tests.

The idea is tester doing the lid close/open one time, then checkbox can verify the status of suspend/resume, lid_close/open from dmesg/kernel log and /proc/acpi/button/lid/*/state automatically

Attachments

No attachments

Tags:
['ce-qa-concern']

LP1948384: Checkbox test fails with encrypted partition

This issue was migrated from https://bugs.launchpad.net/plainbox-provider-checkbox/+bug/1948384

Summary

Status	Created on	Heat	Importance	Security related
Confirmed	2021-10-22 06:22:23	28	Critical	False

Description

Running UC20 automated tests on a system that has FDE enabled.

Test: com.canonical.certification::disk/storage_device_sda requires it to mount the data partition and try to write a file to it.

The test fails with the following output:

Testing disk sda
identified as a disk...
test will be run on partition /dev/sda5

mount /dev/sda5 /tmp/tmpszi24803
mount: /tmp/tmpszi24803: unknown filesystem type 'crypto_LUKS'.
Traceback (most recent call last):
File "/tmp/nest-u4v36e2x.1cbb951e7a49372b643651a046b06db20660ce01a1e0085d00cc869994722df7/storage_test.py", line 152, in
main()
File "/tmp/nest-u4v36e2x.1cbb951e7a49372b643651a046b06db20660ce01a1e0085d00cc869994722df7/storage_test.py", line 148, in main
disk_test(udev_name)
File "/tmp/nest-u4v36e2x.1cbb951e7a49372b643651a046b06db20660ce01a1e0085d00cc869994722df7/storage_test.py", line 134, in disk_test
mount(part_to_test, mount_dir)
File "/tmp/nest-u4v36e2x.1cbb951e7a49372b643651a046b06db20660ce01a1e0085d00cc869994722df7/storage_test.py", line 50, in mount
sp.check_call(cmd, shell=True)
File "/snap/cdts/186/checkbox-runtime/usr/lib/python3.8/subprocess.py", line 364, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'mount /dev/sda5 /tmp/tmpszi24803' returned non-zero exit status 32.

/dev/sda5 is the encrypted partition.

This problem was first reported in lp:1938995 but will affect any project using a DUT with FDE enabled.

Attachments

No attachments

Tags:
['cbox-4', 'checkbox', 'test-case']

LP1905252: Re-run ignored when resuming session

This issue was migrated from https://bugs.launchpad.net/checkbox-ng/+bug/1905252

Summary

Status	Created on	Heat	Importance	Security related
Triaged	2020-11-23 08:55:45	156	High	False

Description

[Summary]

Checkbox was running and waiting for user input at job com.canonical.certification::audio/2_playback_displayport_PCI_ID_0x1f95. It's a user-verify job, so the job had ran, was pre-filled at "failed" because of erroneous output (see lp:1905251), and was waiting for user input.

The GNOME term where it was running was closed, and when the session was resumed, even though user selected "run the job again", Checkbox moved on to the next job in the list.

Please use Checkbox session attached to lp:1905251 if you want to try to reproduce this issue.

=========================[ Resume Incomplete Session ]==========================
There is 1 incomplete session that might be resumed
Do you want to resume session 'session_title-2020-11-23T03.50.12'?
r => resume this session
n => next session
c => create new session
d => delete old sessions
[rncd]: r
Previous session run tried to execute job: com.canonical.certification::audio/2_playback_displayport_PCI_ID_0x1f95
What do you want to do with that job?
s => skip that job
p => mark it as passed and continue
f => mark it as failed and continue
r => run it again
[spfr]: r
Reports will be saved to: /home/u/.local/share/checkbox-ng
-------------[ Running job 1 / 143. Estimated time left: unknown ]--------------
[ Display connected via DisplayPort using an USB Type-C port for NVIDIA Corporation PCI ID 0x1f95 ]
ID: com.canonical.certification::monitor/2_type-c_displayport_PCI_ID_0x1f95
Category: com.canonical.plainbox::monitor

[Additional information]
CID: 202010-28292
Manifest: canonical-oem-somerville-focal-amd64-20200502-85+fossa-cubone-rkl+X59
system-manufacturer: Dell Inc.
system-product-name: OptiPlex 7490 AIO
bios-version: 0.4.4
CPU: Genuine Intel(R) 0000 @ 1.80GHz (16x)
GPU: 00:02.0 VGA compatible controller [0300]: Intel Corporation Device [8086:4c8a] (rev 01)
02:00.0 VGA compatible controller [0300]: NVIDIA Corporation Device [10de:1f95] (rev a1)
kernel-version: 5.10.0-1003-oem

Attachments

checkbox-session.tgz
snap_list.log
acpidump.log
sosreport-u-OptiPlex-7490-AIO-2020-11-23-fmwbwid.tar.gz

Tags:
['checkbox', 'test-case']

LP1988030: Wrong sink option when executing audio/playback_headphones

This issue was migrated from https://bugs.launchpad.net/plainbox-provider-checkbox/+bug/1988030

Summary

Status	Created on	Heat	Importance	Security related
Confirmed	2022-08-29 09:05:12	6	Critical	False

Description

[Summary]
The Sink will be changed during audio/playback_headphones testing

[Steps to reproduce]

Connect monitor via DP cable
Boot into OS
Run "checkbox-cli run com.canonical.certification::audio/playback_headphones"
Follow the steps to plug a headset and choose headphone on popup dialog
Check the Sink in Settings -> Sound, it should turn to the "Headphones - Build in-Audio" from "Speakers - Build in-Audio"
Press Enter to perform case
The Sink becomes "HDMI/Displayport-HDA NVDIA" and no sound from headset (issue happened)

[Expected result]
Sink should keep as "Headphones - Build in-Audio" and can hear sound from headset

[Actual result]
Sink is changed to "HDMI/Displayport-HDA NVDIA" during testing and back automatically.
See comment 2 (Video record)

[Failure rate]
Always

[Additional Info]
SKU ：KRPL-CFF-EVT-C1 (Desktop)
BIOS Version : 0.4.18
Image/Manifest : CFF X19
CPU : Raptor Lake 8+16 65W LGA ES2
GPU : T400 dGFx

Attachments

sosreport.7z
Issue VP with dGFx.webm
pactl_list.log
sosreport-ubuntu-wrongsinkoption-2022-11-03-xwgfxsy.tar.xz

Tags:
['ce-qa-concern']

LP1929543: audio/1_playback_hdmi_* always fail: ERROR:Failed to find status for device: hdmi

This issue was migrated from https://bugs.launchpad.net/plainbox-provider-checkbox/+bug/1929543

Summary

Status	Created on	Heat	Importance	Security related
New	2021-05-25 11:03:35	6	High	False

Description

[Summary]

[Steps to reproduce]
Run test com.canonical.certification::audio/1_playback_hdmi_PCI_ID_0x9a49

[Expected result]
Test pass

[Actual result]
Test always fail, but can hear sound from monitor and video & music can be played normally

INFO:===========================[ Saving audio settings ]============================
INFO:==========================[ HDMI / DisplayPort ports ]==========================
INFO:Card #0 Port: PortWithProfile(name='HDMI3', label='HDMI3/DP3 Output', priority=700, latency_offset=0, availability='not available', properties=[], profile_list=['HiFi'])
INFO:Card #0 Port: PortWithProfile(name='HDMI2', label='HDMI2/DP2 Output', priority=600, latency_offset=0, availability='not available', properties=[], profile_list=['HiFi'])
INFO:Card #0 Port: PortWithProfile(name='HDMI1', label='HDMI1/DP1 Output', priority=500, latency_offset=0, availability='available', properties=[Property(name='device.product.name', value='DELL P2415Q')], profile_list=['HiFi'])
Traceback (most recent call last):
  File "/tmp/nest-d9c8j5_5.a5412f3f1f04af24efab9aa1494248a48b23d88e8a751ef10c79d5e6985235d5/audio_settings.py", line 9, in
    sys.exit(main())
  File "/usr/lib/python3/dist-packages/checkbox_support/scripts/audio_settings.py", line 383, in main
    set_profile_hdmi()
  File "/usr/lib/python3/dist-packages/checkbox_support/scripts/audio_settings.py", line 138, in set_profile_hdmi
    card, profile = _guess_hdmi_profile(pactl_list)
  File "/usr/lib/python3/dist-packages/checkbox_support/scripts/audio_settings.py", line 123, in _guess_hdmi_profile
    profile = min([p for p in port.profile_list if ('stereo' in p) or ('Hdmi' in p)], key=len)
ValueError: min() arg is an empty sequence
ERROR:Failed to find status for device: hdmi
Attempting to initialize Gstreamer pipeline: audiotestsrc wave=sine freq=512 ! audioconvert ! audioresample ! autoaudiosink
Pipeline initialized, now starting playback.
INFO:==========================[ Restoring audio settings ]==========================

[Additional information]
CID: 202012-28492
CPU: 11th Gen Intel(R) Core(TM) i7-1185G7 @ 3.00GHz (8x)
GPU: 00:02.0 VGA compatible controller [0300]: Intel Corporation Device [8086:9a49] (rev 01)
kernel-version: 5.10.0-1028-oem

[Stage]
Issue reported and logs collected right after it happened

Attachments

checkbox-session.tgz
snap_list.log
acpidump.log
sosreport-u-Latitude-9420-2021-05-25-lplnbks.tar.gz

Tags:
['cbox-10', 'checkbox', 'test-case']

Checkbox-ng remote takes WAY too long to run on Jammy

This issue was migrated from https://bugs.launchpad.net/checkbox-ng/+bug/1980288

Summary

Status	Created on	Heat	Importance	Security related
Triaged	2022-06-29 22:07:54	6	Critical	False

Description

I've spent weeks now doing repeated testing to get the data needed to highlight this.

When running SRU tests against Jammy, it takes upwards of 2 days per run, whereas for any other Ubuntu release the test takes only a few hours.

This is not strictly an OS issue, as running the SRU tests locally using Jammy takes significantly less time than running it via checkbox remote (as done with testflinger). Because of this, I'm practically unable to do SRU testing on Jammy, as each SRU cycle potentially means testing 5 different kernels on each machine.

I have a spreadsheet with collated timing data for every current LTS kernel using the same SRU tests. I can share that if necessary, but the tl;dr of that spreadsheet is:

        Avg Time

Bionic 11:23:49
Bionic HWE 9:45:47
Focal 9:37:33
Focal HWE 10:46:38
Jammy TF 48:17:45
Jammy Local 17:23:34

As you can see, Bionic and Focal run to completion in less than 12 hours on average.

Jammy TF is Jammy run via Testflinger using checkbox-remote. in EVERY instance, it hits the global timeout because of something going on that is specific to checkbox-remote and Jammy.

Jammy Local is instances where I did a deployment of jammy, then logged into the machines directly and ran the same SRU suite by hand in a screen session. That average DOES take longer on jammy, so a component of this is related to Jammy and I'll start tracking this down later... but for now, the fact that checkbox-remote exacerbates this to the point where testing is never completed is a huge problem.

I logged times across 27 machines to get this data. Some systems I didn't get times for due to other issues with the machines and MAAS during deployment, but those are all deployment issues, not issues once deployed and running tests.

Attachments

doubletusk-jammy-egx-2022-08-19_1209.log

Tags:
[]

LP1927663: [Checkbox deb] the checkbox session won't resume automatically while running power-automated test plan

This issue was migrated from https://bugs.launchpad.net/plainbox-provider-checkbox/+bug/1927663

Summary

Status	Created on	Heat	Importance	Security related
Confirmed	2021-05-07 02:45:38	10	Critical	False

Description

[Summary]
The checkbox session won't resume automatically while running power-automated test plan

[Steps to reproduce]

Power on system and boot into OS.
checkbox-cli run com.canonical.certification::power-automated

[Expected result]
checkbox session will be resume automatically after restart system.

[Actual result]
checkbox session won't resume automatically after restart system.

[Additional information]
kernel-version: 5.10.0-1023-oem
ubuntu-version: 20.04

dpkg -l | grep checkbox
ii checkbox-ng 1.13.0~~ppa~~ubuntu20.04.1 all CheckBoxNG test runner
ii plainbox-provider-checkbox 0.58.0~~ppa~~ubuntu20.04.1 amd64 CheckBox provider for PlainBox
ii python3-checkbox-ng 1.13.0~~ppa~~ubuntu20.04.1 all CheckBoxNG test runner (Python 3 library)
ii python3-checkbox-support 0.50.0~~ppa~~ubuntu20.04.1 all collection of Python modules used by PlainBox providers

Attachments

checkbox-session.tgz
snap_list.log
acpidump.log
sosreport-u-ThinkStation-P520c-2021-05-06-cvighup.tar.gz

Tags:
['cbox-1', 'checkbox-session-resume', 'go-to-jira']

LP1956293: "audio/valid-sof-firmware-sig" Should not be tested with the machine does not use Intel DMIC to PCH or Soundwire feature

This issue was migrated from https://bugs.launchpad.net/plainbox-provider-checkbox/+bug/1956293

Summary

Status	Created on	Heat	Importance	Security related
New	2022-01-04 06:44:48	6	Critical	False

Description

Not all machines use Intel DMIC to PCH or Soundwire Feature. When this test is run on those machines, it always fail. The test case needs to add more resource check to make sure it only run on the machine which uses DMIC or Soundwire.

plugin: shell
category_id: com.canonical.plainbox::audio
id: audio/valid-sof-firmware-sig
requires:
device.category == 'AUDIO'
package.name == 'linux-firmware'
command:
fail=0
for fw in /lib/firmware/intel/sof/sof-*.ri;
do
     # shellcheck disable=SC2016
     if xxd "${fw}" | grep -q '$CPD'; then
         if ! sof_ri_info.py --valid "${fw}";
         then
             fail=1
         fi
     fi
done
exit $fail
_description: Valid sof firmware signature

Attachments

No attachments

Tags:
['cbox-5']

LP1923016: [ODM bug] the system stays at Boot Logo after the graphics/{index}_switch_card_{product_slug} is run

This issue was migrated from https://bugs.launchpad.net/plainbox-provider-checkbox/+bug/1923016

Summary

Status	Created on	Heat	Importance	Security related
New	2021-04-08 09:17:10	12	High	False

Description

Description:
Checkbox-ng would adjust the graphic mode from "Performance" to "Power Saving" during client-cert-20-04 testing, and then the screen would stays at Boot logo.

User need to re-connect the monitor to integrated GPU to switch the graphic mode to "Performance" mode to continue rest of testing.

My question is shall we test integrated GPU while a discrete GPU installed for a Desktop/Tower server/Rack server? If not, we should not list the test cases for integrated GPU.

Steps to reproduce:

Connect an external monitor to dGPU.
Power on system and boot into OS.
Make sure the graphic profile is in performance mode.
stanley@stanley-ThinkPad-T490:~$ prime-select query
nvidia
Run client-cert-20-04 test plan.
Note: the screen would stays at Boot logo after checkbox-ng switch the graphic mode to "power saving" mode.

Test environment:
A system w/ iGPU and nVidia GPU installed.

Attachments

No attachments

Tags:
['cbox-9', 'oem-priority']

LP1943007: Checkbox Suspend tests should respect supported HW

This issue was migrated from https://bugs.launchpad.net/plainbox-provider-checkbox/+bug/1943007

Summary

Status	Created on	Heat	Importance	Security related
Triaged	2021-09-08 08:51:20	6	High	False

Description

Running cdts.odm-certification

It will ask if specific HW features are available, such as TPM, Thunderbolt, WWAN, Bluetooth, etc.

While this allows Checkbox to skip that feature area, the respective tests in Suspend still get run.

This means even if I tell Checkbox that the DUT does not support Thunderbolt, it still runs the Thunderbolt related tests after Suspend. These then fail, meaning the tester has to clean up the test submission.

Attachments

No attachments

Tags:
['cbox-14', 'checkbox', 'test-case']

LP1954375: the summary() in bin/pm_test.py missing some test cases condition when QA runs a bigger test plan

This issue was migrated from https://bugs.launchpad.net/plainbox-provider-checkbox/+bug/1954375

Summary

Status	Created on	Heat	Importance	Security related
Confirmed	2021-12-10 06:14:27	8	High	False

Description

When a checkbox session contains many pm_test calls[1], then it will append the "Completed S3 cycle(s) \n" and "Completed s2idle cycle(s) \n" to the same path (e.g. --log=/var/tmp/checkbox-ng/sessions/session_title-2021-12-06T10.25.57.session/session-share/fwts).

When the second (or later) pm_test be called, it will count previous fwts logs.

                with open(fwts_log_path, 'rt') as f:
                    magic_line_s3 = 'Completed S3 cycle(s) \n'
                    magic_line_s2idle = 'Completed s2idle cycle(s) \n'
                    lines = f.readlines()
                    count_s3 = lines.count(magic_line_s3)
                    count_s2idle = lines.count(magic_line_s2idle)
                    count = count_s3 + count_s2idle
                    if count != total_suspends_expected:
                        problems = (
                            "Found {} occurrences of S3/S2idle."
                            " Expected {}".format(
                                count, total_suspends_expected))

In the case that Stella met, it shows:

[Actual result]
Stress_suspend_30_cycles_with_reboots test failed
Log:Found 126 occurrences of S3/S2idle. Expected 90

The potential solution is to create fwts-cmd.log per test case or check the timestamp[2] whether inside the range.

Feel free to find the detail logs on https://bugs.launchpad.net/stella/+bug/1953597.

[1] https://bugs.launchpad.net/stella/+bug/1953597
[2] "This test run on 06/12/21 at 22:21:08 on host Linux u 5.14.0-1007-oem #7-Ubuntu"

Attachments

No attachments

Tags:
['cbox-16', 'oem-priority', 'originate-from-1953597', 'stella']

LP1917701: KVM/LXD tests do not run on checkbox-snappy

This issue was migrated from https://bugs.launchpad.net/plainbox-provider-resource/+bug/1917701

Summary

Status	Created on	Heat	Importance	Security related
Confirmed	2021-03-04 04:10:22	8	High	False

Description

KVM/LXD tests do not run on checkbox-snappy with Ubuntu Core 20

[steps]
$ Install Ubuntu Core 20
$ snap install checkbox-snappy --devmode --channel=20/stable
$ checkbox-snappy.checkbox-cli # make sure KVM and LXD are selected

[expected]
KVM/LXD tested and passed

[actual]
Tests do not run with

job cannot be started: resource expression "package.name == 'qemu-system'" evaluates to false
job cannot be started: resource expression "executable.name == 'lxc'" evaluates to false

https://kernel.ubuntu.com/~ikepanhc/uc20-checkbox/submission_2021-03-03T14.39.06.645877.html

[details]
$ snap list
Name Version Rev Tracking Publisher Notes
checkbox-snappy 20.16 2108 20/stable ce-certification-qa devmode
checkbox20 1.20 428 latest/stable ce-certification-qa -
core20 20210216 958 latest/edge canonical✓ base
pc 20-0.4 115 20/edge canonical✓ gadget
pc-kernel 5.4.0-66.74.1 722 20/edge canonical✓ kernel
snapd 2.49+git549.g07e33f2 11343 latest/edge canonical✓ snapd

Attachments

No attachments

Tags:
[]

LP1916429: disk/storage_device_nvme0n1 failed

This issue was migrated from https://bugs.launchpad.net/plainbox-provider-checkbox/+bug/1916429

Summary

Status	Created on	Heat	Importance	Security related
New	2021-02-22 04:05:39	8	High	False

Description

[Summary]
disk/storage_device_nvme0n1 failed
certification_status: unspecified

When the Windows disk does not be clean, this test case will fail.

[I/O log]
Testing disk nvme0n1
identified as a disk...
test will be run on partition /dev/nvme0n1p3

mount /dev/nvme0n1p3 /tmp/tmplt1a96u3
Windows is hibernated, refused to mount.
The disk contains an unclean file system (0, 0).
Metadata kept in Windows cache, refused to mount.
Falling back to read-only mount because the NTFS partition is in an
unsafe state. Please resume and shutdown Windows fully (no hibernation
or fast restarting.)
Could not mount read-write, trying read-only
Performed mount /dev/nvme0n1p3 at /tmp/tmplt1a96u3
169262.265625MB of free space avaialble
Forcing memory setting to 8000MB
bonnie++ -d /tmp/tmplt1a96u3 -u root -r 8000
Using uid:0, gid:0.
Can't open file ./Bonnie.44911
umount /tmp/tmplt1a96u3
Traceback (most recent call last):
File "/tmp/nest-q45t9ok5.31c87e8f098638c0afa1eb324ec91d6eadc1613fa55a5682526f26226262a817/storage_test.py", line 146, in
main()
File "/tmp/nest-q45t9ok5.31c87e8f098638c0afa1eb324ec91d6eadc1613fa55a5682526f26226262a817/storage_test.py", line 142, in main
disk_test(udev_name)
File "/tmp/nest-q45t9ok5.31c87e8f098638c0afa1eb324ec91d6eadc1613fa55a5682526f26226262a817/storage_test.py", line 131, in disk_test
run_bonnie(mount_dir)
File "/tmp/nest-q45t9ok5.31c87e8f098638c0afa1eb324ec91d6eadc1613fa55a5682526f26226262a817/storage_test.py", line 88, in run_bonnie
sp.check_call(cmd, shell=True)
File "/usr/lib/python3.8/subprocess.py", line 364, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'bonnie++ -d /tmp/tmplt1a96u3 -u root -r 8000' returned non-zero exit status 1.

[Reproduce Steps]

sudo checkbox-cli run com.canonical.certification::disk/storage_device_nvme0n1

[Results]
failed

Attachments

No attachments

Tags:
['oem-priority', 'originate-from-1909268', 'somerville']

checkbox remote needs a way to resume a test session

This issue was migrated from https://bugs.launchpad.net/checkbox-ng/+bug/1936477

Summary

Status	Created on	Heat	Importance	Security related
Confirmed	2021-07-16 08:29:07	28	Critical	False

Description

It is far too easy for the CDTS master to lose connection with the SUT. When this happens, you end up back at the main selection screen, and the previous test session is completely lost.

Even trying to select tests to rerun them can cause the session to be lost.

Running Checkbox on the SUT locally can detect interrupted sessions and allow you to resume.

Steps to reproduce

Install checkbox on the host and the device under test (DUT).
Make sure the checkbox service is launched on the DUT:

systemctl status checkbox-ng.service
● checkbox-ng.service - Checkbox Remote Service
     Loaded: loaded (/lib/systemd/system/checkbox-ng.service; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2021-11-10 10:01:37 CST; 5 days ago
   Main PID: 1455 (checkbox-cli)
      Tasks: 1 (limit: 19051)
     Memory: 28.1M
     CGroup: /system.slice/checkbox-ng.service
             └─1455 /usr/bin/python3 /usr/bin/checkbox-cli service

(Please note: the service name might be different if you use CDTS or Checkbox as a snap. For CDTS for instance, it's snap.cdts.service.service)

On the host, connect to the DUT:

$ checkbox-cli remote <DUT_IP>

Select a test plan and start the test run (for instance, an automated test plan or a stress test plan such as client-cert-iot-server-20-04-[automated|stress]).
While testing is ongoing, restart your host computer.
After restart, try connecting back to the DUT:

$ checkbox-cli remote <DUT_IP>

Expected result

Checkbox prompts to ask if you want to resume the session.

Actual result

Checkbox starts afresh, showing you the list of available test plans in order to start a new test run.

Attachments

sessions.tar.xz
202110-29555-sessions.tar.gz

Tags:
['checkbox', 'checkbox-session-resume']

LP1982450: No Gtk window while executing camera/display_video0 job

This issue was migrated from https://bugs.launchpad.net/plainbox-provider-checkbox/+bug/1982450

Summary

Status	Created on	Heat	Importance	Security related
Confirmed	2022-07-21 05:56:25	6	Critical	False

Description

[Summary]
While running display_video0 case, the Gtk window doesn't show up

[Steps to reproduce]

Boot into OS
Run 'checkbox-cli'
Choose 'com.canonical.certification::client-cert-desktop-22-04-manual' test plan
Run Camera tests

[Expected result]
The Gtk Window should appear on the screen while testing display_video0

[Actual result]

No Gtk window
Camera works well via Cheese App or Webcamtests.com website.

Video record: https://drive.google.com/file/d/1870WQaPAGyWebEGQhqtBTH2QBL-hYeN_/view?usp=sharing

[Failure rate]
Always

[Checkbox job com.canonical.certification::camera/display_video0 output]

stderr

../src/intel/isl/isl.c:2216: FINISHME: ../src/intel/isl/isl.c:isl_surf_supports_ccs: CCS for 3D textures is disabled, but a workaround is available.

[Additional information]
CID: 202207-30419, 202207-30416
SKU: DKR-DGPU-EVT-C6
Image: canonical-oem-somerville-jammy-amd64-20220504-33+jellyfish-chansey+X21
system-manufacturer: Dell Inc.
system-product-name: OptiPlex AIO Plus 7410
bios-version: 0.4.16
CPU: Genuine Intel(R) 0000 (32x)
GPU: 0000:00:02.0 VGA compatible controller [0300]: Intel Corporation Device [8086:a780] (rev 0c)
0000:03:00.0 Display controller [0380]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:743f] (rev c3)
kernel-version: 5.17.0-1012-oem

[Stage]
Issue reported and logs collected at a later stage

Attachments

checkbox-session.tgz
snap_list.log
acpidump.log
sosreport-u-OptiPlex-AIO-Plus-7410-2022-07-21-ctyxbqx.tar.gz

Tags:
['oem-camera']

LP1858573: Need a job for screen auto-rotation in 2-in-1

This issue was migrated from https://bugs.launchpad.net/plainbox-provider-checkbox/+bug/1858573

Summary

Status	Created on	Heat	Importance	Security related
Confirmed	2020-01-07 02:57:07	6	High	False

Description

In 2-in-1 section of certification coverage guide, it said

Convertible Laptops (aka 2-in-1s)
The system must support display rotation (0°, 90°, 180°, 270°)
The desktop environment must rotate automatically when sensors detect a rotation

We need a test case for auto-rotation. It should only apply to 2-in-1.
It could be a user-interact job.

Attachments

No attachments

Tags:
[]

LP1940492: the graphics/2_valid_opengl_renderer_AMD-GPU test failed

This issue was migrated from https://bugs.launchpad.net/plainbox-provider-checkbox/+bug/1940492

Summary

Status	Created on	Heat	Importance	Security related
New	2021-08-19 03:07:49	6	High	False

Description

[Summary]
This test case would try to validate OpenGL renderer is AMD GPU with DRI_RPIME=1 argument.
But in desktop platform, we would connect a monitor to AMD GPU before power on the system.
At this moment the primary GPU provider is AMD GPU, that means the default renderer is AMD GPU, and the OpenGL renderer is Intel GPU with DRI_PRIME=1.

--------------[ Running job 2 / 2. Estimated time left: unknown ]---------------
------------[ Check the OpenGL renderer (AMD GPU and DRI_PRIME=1) ]-------------
ID: com.canonical.certification::graphics/2_valid_opengl_renderer_Navi_10__Radeon_Pro_W5700_
Category: com.canonical.plainbox::graphics
... 8< -------------------------------------------------------------------------
OpenGL renderer string: Mesa Intel(R) Graphics (RKL GT1)
ERROR: renderer is Intel when DRI_PRIME=1

[Steps to reproduce]

Connect a monitor to AMD GPU.
Power on system and boot into OS.
Install checkbox and provider
Run valid_opengl_renderer with below command.
Cmd> checkbox-cli run com.canonical.certification::graphics_card com.canonical.certification::graphics/2_valid_opengl_renderer_Navi_10__Radeon_Pro_W5700_

[Expected result]
The valid_opengl_renderer test passed

[Actual result]
The valid_opengl_renderer test failed

[Failure rate]
10/10

[Additional information]
CID: 202010-28295
SKU: RDT-RKL-DVT2.1-C1
Image: canonical-oem-somerville-focal-amd64-20200502-85+fossa-proposed-release+X124
bios-version: 1.2.0
GPU: 00:02.0 VGA compatible controller [0300]: Intel Corporation Device [8086:4c8a] (rev 04)
03:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:7312]
kernel-version: 5.10.0-1043-oem

[Stage]
Issue reported and logs collected right after it happened

Attachments

checkbox-session.tgz
snap_list.log
acpidump.log
sosreport-u-Precision-3650-Tower-2021-08-18-diwskum.tar.gz
display_logs.tar

Tags:
['cbox-13', 'hwe-graphics', 'ihv-amd']

duplicate

Add Genoa CPUID strings to cpuid.py

0xa10f11 needs to be added to cpuid.py to identify new AMD Genoa CPUs

LP1831824: Missing after suspend amdgpu glxgears and valid_opengl_renderer test

This issue was migrated from https://bugs.launchpad.net/plainbox-provider-checkbox/+bug/1831824

Summary

Status	Created on	Heat	Importance	Security related
New	2019-06-06 02:20:47	6	High	False

Description

Because of "suspend/{index}glxgears_after_suspend{product_slug}" depends on "suspend/{index}suspend_after_switch_to_card{product_slug}", for amdgpu the test "suspend/{index}glxgears_after_suspend{product_slug}" won't run when "suspend/{index}suspend_after_switch_to_card{product_slug}" failed.

Since the "suspend/{index}suspend_after_switch_to_card{product_slug}" will fail when the graphic is amdgpu, but we do need to confirm amdgpu still can run glxgears without problem after suspend.

The tests affect with amdgpu are:
graphics/{index}valid_opengl_renderer{product_slug}
graphics/{index}switch_card{product_slug}_xenial
graphics/{index}auto_switch_card{product_slug}

Also, we found graphics/{index}valid_opengl_renderer{product_slug} not in any after suspend test plan too.

Attachments

No attachments

Tags:
['ce-qa-concern']

LP1908490: com.canonical.certification::suspend/{index}_gl_support_after_suspend_{product_slug} fails if the desktop contains two GPUs and the output is from add-on graphic card

This issue was migrated from https://bugs.launchpad.net/plainbox-provider-checkbox/+bug/1908490

Summary

Status	Created on	Heat	Importance	Security related
Incomplete	2020-12-17 06:41:13	10	High	False

Description

Steps to reproduce:
checkbox-cli run com.canonical.certification::device com.canonical.certification::graphics_card com.canonical.certification::suspend/2_gl_support_after_suspend_Oland__Radeon_HD_8570___R7_240_340___Radeon_520_OEM__auto
Description:
It could get the positive result when

systemctl suspend
/usr/lib/nux/unity_support_test -p 2>&1

OpenGL vendor string:   X.Org
OpenGL renderer string: AMD OLAND (DRM 2.50.0, 5.6.0-1036-oem, LLVM 10.0.0)
OpenGL version string:  4.5 (Compatibility Profile) Mesa 20.0.8

Not software rendered:    yes
Not blacklisted:          yes
GLX fbconfig:             yes
GLX texture from pixmap:  yes
GL npot or rect textures: yes
GL vertex program:        yes
GL fragment program:      yes
GL vertex buffer object:  yes
GL framebuffer object:    yes
GL version is 1.4+:       yes

Unity 3D supported:       yes

But if I ran the command from "2_gl_support_after_suspend_Oland__Radeon_HD_8570___R7_240_340___Radeon_520_OEM__auto" the result will be

--------------[ Running job 6 / 6. Estimated time left: unknown ]---------------
[ Test OpenGL support for Oland [Radeon HD 8570 / R7 240/340 / Radeon 520 OEM] after suspend ]
ID: com.canonical.certification::suspend/2_gl_support_after_suspend_Oland__Radeon_HD_8570___R7_240_340___Radeon_520_OEM__auto
Category: com.canonical.plainbox::suspend
... 8< -------------------------------------------------------------------------
Setting up PRIME GPU offloading for AMD discrete GPU
OpenGL vendor string:   Intel
OpenGL renderer string: Mesa Intel(R) UHD Graphics 630 (CML GT2)
OpenGL version string:  4.6 (Compatibility Profile) Mesa 20.0.8

Not software rendered:    yes
Not blacklisted:          yes
GLX fbconfig:             yes
GLX texture from pixmap:  no
GL npot or rect textures: yes
GL vertex program:        yes
GL fragment program:      yes
GL vertex buffer object:  yes
GL framebuffer object:    yes
GL version is 1.4+:       yes

Unity 3D supported:       no
------------------------------------------------------------------------- >8 ---
Outcome: job failed

BTW, it's a cert blocker issue. Set the importance to Critical.

Attachments

No attachments

Tags:
['oem-priority', 'originate-from-1908254', 'originate-from-1924705', 'originate-from-1924706', 'originate-from-1924709', 'originate-from-1924710', 'stella', 'sutton']

LP1897557: suspend/memory_after_suspend_auto give incorrect alert

This issue was migrated from https://bugs.launchpad.net/plainbox-provider-checkbox/+bug/1897557

Summary

Status	Created on	Heat	Importance	Security related
New	2020-09-28 11:30:57	8	High	False

Description

HWE team treats the alarm from the current test case as a false alarm.
So, it needs to be revised.

The false alarm io log:

2c2
< total: 16537169920
---
> total: 16537174016

refer to https://bugs.launchpad.net/somerville/+bug/1895788/comments/4
comment from HWE

"
From kernel doc. https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/filesystems/proc.rst

MemTotal
Total usable RAM (i.e. physical RAM minus a few reserved
bits and the kernel binary code)

The MemTotal is the total usable memory to the kernel, not total physical RAM size, so it may change.
I can't find a real case about why suspend affects this value, but if this value does not keep changing every time(a memory leak?), we may ignore this failed.
"

"
If it's purpose is to check memory leak, then try suspend 3 times and check if the usable memory keeps decreasing is more reasonable.
"

Attachments

No attachments

Tags:
['oem-priority', 'originate-from-1895788', 'somerville']

LP1875830: ipdt/PCH-x86_64 fails on devices with secure boot

This issue was migrated from https://bugs.launchpad.net/plainbox-provider-checkbox/+bug/1875830

Summary

Status	Created on	Heat	Importance	Security related
Confirmed	2020-04-29 08:35:44	14	High	False

Description

On systems with secure boot enabled, ipdt/PCH-x86_64 always fails with the following errors:

--- Platform Controller Hub Test ---
...
Version: 1.0.9.64b.L
...

pcilib: No permission to access I/O ports (you probably have to be root).

Running checkbox with root does not solve issue.
Saw issue with checkbox18 snap rev #974. The issue exists in earlier versions of checkboxes and has been seen across various oem projects that requires secure boot enabled, see bug #1809421 and bug #1806568

Attachments

No attachments

Tags:
['ce-qa-concern']

LP1898024: remote get_result stack trace

This issue was migrated from https://bugs.launchpad.net/checkbox-ng/+bug/1898024

Summary

Status	Created on	Heat	Importance	Security related
Confirmed	2020-10-01 09:50:32	6	High	False

Description

---------------------------[ Running job 158 / 195 ]----------------------------
--------------------------[ Attach C-States test log ]--------------------------
ID: com.canonical.certification::cpu/cstates_results.log
Category: CPU tests
--------------------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/bin/checkbox-cli", line 11, in <module>
    load_entry_point('checkbox-ng==1.10.0.dev0', 'console_scripts', 'checkbox-cli')()
  File "/usr/lib/python3/dist-packages/checkbox_ng/launcher/checkbox_cli.py", line 123, in main
    subcmd.invoked(ctx)
  File "/usr/lib/python3/dist-packages/checkbox_ng/launcher/master.py", line 155, in invoked
    self.connect_and_run(ctx.args.host, port)
  File "/usr/lib/python3/dist-packages/checkbox_ng/launcher/master.py", line 219, in connect_and_run
    keep_running = {
  File "/usr/lib/python3/dist-packages/checkbox_ng/launcher/master.py", line 435, in resume_interacting
    self.run_jobs()
  File "/usr/lib/python3/dist-packages/checkbox_ng/launcher/master.py", line 420, in run_jobs
    self._run_jobs(jobs_repr, total_num)
  File "/usr/lib/python3/dist-packages/checkbox_ng/launcher/master.py", line 607, in _run_jobs
    self.wait_for_job()
  File "/usr/lib/python3/dist-packages/checkbox_ng/launcher/master.py", line 461, in wait_for_job
    self.finish_job()
  File "/usr/lib/python3/dist-packages/checkbox_ng/launcher/master.py", line 466, in finish_job
    job_result = self.sa.finish_job(result)
  File "/usr/lib/python3/dist-packages/plainbox/vendor/rpyc/core/netref.py", line 253, in __call__
    return syncreq(_self, consts.HANDLE_CALL, args, kwargs)
  File "/usr/lib/python3/dist-packages/plainbox/vendor/rpyc/core/netref.py", line 76, in syncreq
    return conn.sync_request(handler, proxy, *args)
  File "/usr/lib/python3/dist-packages/plainbox/vendor/rpyc/core/protocol.py", line 469, in sync_request
    return self.async_request(handler, *args, timeout=timeout).value
  File "/usr/lib/python3/dist-packages/plainbox/vendor/rpyc/core/async_.py", line 102, in value
    raise self._obj
AttributeError: 'NoneType' object has no attribute 'get_result'

========= Remote Traceback (1) =========
Traceback (most recent call last):
  File "/snap/checkbox-snappy/2037/checkbox-runtime/lib/python3.8/site-packages/plainbox/vendor/rpyc/core/protocol.py", line 320, in _dispatch_request
    res = self._HANDLERS[handler](self, *args)
  File "/snap/checkbox-snappy/2037/checkbox-runtime/lib/python3.8/site-packages/plainbox/vendor/rpyc/core/protocol.py", line 593, in _handle_call
    return obj(*args, **dict(kwargs))
  File "/snap/checkbox-snappy/2037/checkbox-runtime/lib/python3.8/site-packages/plainbox/impl/session/remote_assistant.py", line 551, in finish_job
    result = self._be.wait().get_result()
AttributeError: 'NoneType' object has no attribute 'get_result'

+ EXITCODE=1

LP1906216: need a way to switch back to default mode after graphics/1_auto_switch_card_* or graphics/2_auto_switch_card_*

This issue was migrated from https://bugs.launchpad.net/plainbox-provider-checkbox/+bug/1906216

Summary

Status	Created on	Heat	Importance	Security related
New	2020-11-30 05:46:32	6	High	False

Description

We are counting on graphics/1_auto_switch_card_* switching to dGPU rendering and graphics/2_auto_switch_card_* to switch back to Intel gpu rendering [1].

But there's no way to switch back to default mode in the current design and we used to assume the default mode is powersaving(Intel rendering) mode so some existed plan[2]just switch back to powersaving(Intel gpu) mode in the end.

This operation cause problem on the following 2 exceptions:

Some platforms are asked to be shipped with default performance(Nvidia rendering) mode.
After Focal, some platform is asked to be shipped with default on-demand(runtime selecting rendering gpu[3]) mode.

In these 2 cases with current test plans, it will switch to intel in the end and it could impact the following tests.

We might need to redesign the graphic testing based on the current new dGPU using cases:

On Intel only(UMA):

default pwersaving(Intel rendering) mode.

On Intel + Nvidia:

default pwersaving(Intel rendering) mode by prime-select intel.
default performance(Nvidia rendering) mode by prime-select nvidia.
default pwersaving(runtime selecting rendering gpu [3]) mode by prime-select on-demand.

On Nvidia only:

default pwersaving(Intel rendering) mode by prime-select nvidia.
default pwersaving(runtime selecting rendering gpu [3]) mode by prime-select on-demand.

On Intel + AMD:

default pwersaving(runtime selecting rendering gpu by DRI_PRIME=1) mode.

On AMD only:

default pwersaving(runtime selecting rendering gpu by DRI_PRIME=1) mode.

[1] https://git.launchpad.net/plainbox-provider-checkbox/tree/units/graphics/test-plan.pxu
[2] https://git.launchpad.net/plainbox-provider-certification-client/tree/units/client-cert-auto.pxu
[3] https://docs.google.com/document/d/12zbZfGI_a4azR7a5UvVIib6gDyiK43ZFJOBlt0LPrEU/edit#

Attachments

No attachments

Tags:
['oem-priority']

LP1974205: Case memory/memory_stress_ng be terminated after performing for a while

This issue was migrated from https://bugs.launchpad.net/plainbox-provider-checkbox/+bug/1974205

Summary

Status	Created on	Heat	Importance	Security related
Confirmed	2022-05-19 15:14:17	172	Critical	False

Description

[Summary]
The terminal which runs memory/memory_stress_ng job will be closed (It seems that system be reboot) and the process of checkbox-cli be terminated after executing for a while.

Issue can also be observed on Cinnamon Bay platform. It will cause system reboot after running this case for a while.

[Steps to reproduce]

Boot into OS
Run "checkbox-cli run com.canonical.certification::memory/memory_stress_ng"

[Expected result]
checkbox-cli won't be terminated.

[Actual result]
checkbox-cli will be terminated.

[Failure rate]
10/10

[Additional information]
CID: 202112-29802
SKU: TRBA-DVT2-C4
Base Image: dell-bto-jammy-jellyfish-tentacool-X07-20220331-4.iso
Product Name: XPS 13 9320
BIOS Version: 0.2.13
kernel-version: 5.15.0-23-generic

CID: 202203-30134
SKU: CB16T-DVT2-C1
Image: canonical-oem-somerville-jammy-amd64-20220504-33+jellyfish-minccino+X11
system-manufacturer: Dell Inc.
system-product-name: Precision 7670
bios-version: 1.3.1
CPU: 12th Gen Intel(R) Core(TM) i5-12600HX (16x)
GPU: 00:02.0 VGA compatible controller [0300]: Intel Corporation Device [8086:4688] (rev 0c)
kernel-version: 5.17.0-9004-oem

Attachments

sosreport-V-high-memory-2022-06-28-zjcpqwv.tar.xz
checkbox-ng.zip
sosreport-V-high-memory-2022-06-28-zjcpqwv.tar.xz

Tags:
['jammy-test', 'oem-priority', 'originate-from-1974073', 'originate-from-1981168', 'originate-from-1981180', 'originate-from-1982914', 'originate-from-1983006', 'originate-from-1983068', 'originate-from-1989083', 'somerville', 'stella', 'sutton']

LP1967454: [Checkbox SNAP] the checkbox session won't resume automatically after running reboot/power-off test on a desktop image

This issue was migrated from https://bugs.launchpad.net/plainbox-provider-checkbox/+bug/1967454

Summary

Status	Created on	Heat	Importance	Security related
Confirmed	2022-04-01 04:03:56	6	Critical	False

Description

[Summary]
the checkbox session won't resume automatically after running reboot/power-off test

I did some investigation on this issue, the reason why checkbox session won't resume automatically it's because of the command is incorrect in autostart config file. As you can see from following logs, it try to resume pm_test script by /snap/bin/checkbox.pm-test. But there's no such file in /snap/bin folder.

u@u-ThinkStation-P350:~$ cat pm_test.py.desktop

[Desktop Entry]
Name=poweroff test
Comment=Verify poweroff works properly
Exec=sudo /snap/bin/checkbox.pm-test -r 0 -w 60 --hardware-delay 30 --pm-delay 5 --min-pm-time 58 --max-pm-time 300 --append --total 1 --start 1648781293 --pm-timestamp 1648781293 --silent --log-level=debug --log-dir=/var/tmp/checkbox-ng/sessions/remote-2022-04-01T02.47.34.session/session-share --suspends-before-reboot=0 --checkbox-respawn-cmd=/var/tmp/checkbox-ng/sessions/remote-2022-04-01T02.47.34.session/session-share/__respawn_checkbox poweroff
Type=Application
X-GNOME-Autostart-enabled=true
Hidden=false

u@u-ThinkStation-P350:~$ ls /snap/bin/
checkbox.checkbox-cli checkbox.sru qabro qabro.sosreport
checkbox.client-cert-iot-server checkbox.test-runner qabro.amixer snap-store
checkbox.configure chromium qabro.lspci snap-store.ubuntu-software
checkbox.shell chromium.chromedriver qabro.lsusb snap-store.ubuntu-software-local-file

[Steps to reproduce]

Install checkbox20 snap and checkbox snap
Perform "Power Management reboot and power off tests (automated)" test via checkbox.checkbox-cli
cmd> checkbox.checkbox-cli run com.canonical.certification::power-management-reboot-poweroff-cert-automated
checkbox session won't resume automatically after boot into OS.

[Expected result]
checkbox session will be resume automatically after restart system.

[Actual result]
checkbox session won't resume automatically after restart system.

[Failure rate]
3/3

[Stage]
Issue reported and logs collected at a later stage

Attachments

checkbox-session.tgz
snap_list.log
acpidump.log
sosreport-u-ThinkStation-P350-2022-04-01-vsjejro.tar.gz

Tags:
['checkbox-session-resume', 'hwe-powercycle']

LP1947701: Create a lsb dictionary

This issue was migrated from https://bugs.launchpad.net/checkbox-ng/+bug/1947701

Summary

Status	Created on	Heat	Importance	Security related
New	2021-10-19 09:41:01	6	High	False

Description

========== Request ==========

A new template parameter dictionary[1] that holds lsb_release information[2]

========== Rationale ==========

[Current situation]
There are times when we would need to write cases that require dynamically using and/or displaying lsb info. Example: ubuntucore/os-fail-boot-{{description}}[3]

Using the above ubuntucore/os-fail-boot-{{description}} as an example, as shown in its job unit[3], to achieve the goal of "dynamically getting the core snap name", one would need to:

Use a template job unit
Include a dynamic variable in the job ID
Run the template-resource job first prior to running the actual job during runtime

Templating these jobs is inconvenient and also a bit unnecessary, as there would only be one {{description}} and therefore only one possible job created based on this template (e.g. "ubuntucore/os-fail-boot-Ubuntu Core 20"). Also if something goes wrong while collecting lsb info, there is high possibility the job gets silently skipped (i.e. actual job not created at all).

[Solution]
A lsb dictionary that fixes the inconveniences by allowing the creation of these jobs to be simplified, and be more like when snappy/snap-search calls __checkbox_env__.get("TEST_SNAP", "test-snapd-tools")[4]:

No need for the job unit to be a template
No need for dynamic job ID
Job does not get skipped if something goes wrong
No need to run a template-resource job first during checkbox runtime

Using the same ubuntucore/os-fail-boot-{{description}} as an example, the job can now be rewritten into

using a static job ID called ubuntucore/os-fail-boot
identifying the core snap (core18, core20 etc.) by calling something like __lsb__.get("release",<default>) and appending that value to the end of the string "core"

====================================
[1] https://checkbox.readthedocs.io/en/latest/units/template.html?highlight=template#template-engine-additional-features

[2] https://git.launchpad.net/plainbox-provider-resource/tree/jobs/resource.pxu#n85 ,
job run output: https://pastebin.canonical.com/p/49hJhdqvxh/

[3] https://git.launchpad.net/plainbox-provider-checkbox/tree/units/ubuntucore/jobs.pxu#n58

[4] https://git.launchpad.net/plainbox-provider-checkbox/tree/units/snapd/snapd.pxu#n11

Attachments

No attachments

Tags:
[]

LP1980375: "suspend/audio_before_suspend" will always fail when run all suspend tests together

This issue was migrated from https://bugs.launchpad.net/plainbox-provider-checkbox/+bug/1980375

Summary

Status	Created on	Heat	Importance	Security related
Confirmed	2022-06-30 14:43:30	6	Critical	False

Description

[Summary]
"suspend/audio_before_suspend" will always fail when run all suspend tests together
[Steps to reproduce]

Execute checkbox-cli
De-select all the test plan.
Select "Suspend tests" Plan
Run the test.
[Expected result]
"suspend/audio_before_suspend" test should pass.
[Actual result]
"suspend/audio_before suspend" test fail
[Failure rate]
100%
[Checkbox job com.canonical.certification::suspend/audio_before_suspend output]

stderr

Connection failure: Connection refused
pa_context_connect() failed: Connection refused
Traceback (most recent call last):
  File "/tmp/nest-_fdq8pin.3361f09f54625130ca8353599d2d0b04ec843608119aba04070f3a0948f151e7/audio_settings.py", line 9, in
    sys.exit(main())
  File "/usr/lib/python3/dist-packages/checkbox_support/scripts/audio_settings.py", line 388, in main
    store_audio_settings(settings_file)
  File "/usr/lib/python3/dist-packages/checkbox_support/scripts/audio_settings.py", line 247, in store_audio_settings
    audio_settings = get_audio_settings(type)
  File "/usr/lib/python3/dist-packages/checkbox_support/scripts/audio_settings.py", line 207, in get_audio_settings
    pactl_status = check_output(["pactl", "info"],
  File "/usr/lib/python3.10/subprocess.py", line 420, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/lib/python3.10/subprocess.py", line 524, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['pactl', 'info']' returned non-zero exit status 1.

[Additional information]
CID:
SKU:
Image: canonical-oem-somerville-jammy-amd64-20220504-33+jellyfish+X19
system-manufacturer: Dell Inc.
system-product-name: Inspiron 16 5620
bios-version: 0.2.12
CPU: 12th Gen Intel(R) Core(TM) i7-1280P (20x)
GPU: 0000:00:02.0 VGA compatible controller [0300]: Intel Corporation Device [8086:46a6] (rev 08)
0000:01:00.0 3D controller [0302]: NVIDIA Corporation Device [10de:25a6] (rev a1)
kernel-version: 5.17.0-1011-oem

[Stage]
Issue reported and logs collected at a later stage

[Checkbox Version]
checkbox-ng: 1.18.0rc1
checkbox-support: 0.55.0rc2
certification-client: 0.50.0rc1
plainbox-provider-checkbox: 0.65.0rc2
plainbox-provider-resource-generic: 0.51.0rc2
plainbox-provider-tpm2: 1.18.0rc1

Attachments

checkbox-session.tgz
snap_list.log
acpidump.log
sosreport-u-Inspiron-16-5620-2022-06-30-rhdjsnq.tar.gz

Tags:
['ce-qa-concern', 'jammy-test']

LP1976100: CPU Stress test was deleted from bionic suitte

This issue was migrated from https://bugs.launchpad.net/plainbox-provider-certification-server/+bug/1976100

Summary

Status	Created on	Heat	Importance	Security related
In Progress	2022-05-27 16:41:04	6	Critical	False

Description

in commit 9cda18c I accidentally removed stress/cpu_stress_ng_test from server-full-18.04 while doing a cleanup and moving many things to nested testplans.

This restores that test to the testplan.

Attachments

No attachments

Tags:
['servcert-380']

LP1953118: graphics/_gl_support_ fail: because binary unity_support_test is missing

This issue was migrated from https://bugs.launchpad.net/plainbox-provider-checkbox/+bug/1953118

Summary

Status	Created on	Heat	Importance	Security related
New	2021-12-03 06:57:33	6	High	False

Description

Checkbox snaps:
Name Version Rev Tracking Publisher Notes
cdts 0.9.dev 232 latest/edge/… ce-certification-qa classic
checkbox20 1.24 717 latest/stable ce-certification-qa -

graphics/1_gl_support_PCI_ID_0x4571 fails with error:

bash: /usr/lib/nux/unity_support_test: No such file or directory

See:

https://certification.canonical.com/hardware/202109-29496/submission/238308/test-results/fail/?term=gl_support

This is due to commit 184487acc3442d45afd0d619fa7d30faf681d171 in plainbox-provider-checkbox:

https://git.launchpad.net/plainbox-provider-checkbox/commit/?id=184487acc3442d45afd0d619fa7d30faf681d171

Attachments

No attachments

Tags:
['cbox-15']

memory stress-ng (and cpu) sometimes go a bit OOM crazy

While doing validation testing of Kinetic on s390x the memory-stress-ng test kept killing the session. SOMETHING has changed that has made OOMKiller more aggressive. After debugging a bit one thing we did discover (thanks to Frank and Po-hsu Lin) is that the kernel regression test for stress-ng attempts to mitigate the potential damage done by the oomkiller. This is defined in the script here:

https://git.launchpad.net/~canonical-kernel-team/+git/autotest-client-tests/tree/ubuntu_stress_smoke_test/ubuntu_stress_smoke_test.sh

We should adjust the stress-ng script to mitigate the effects of OOM killer in line with what Kernel does in their stress test script.

LP1979698: media card insert case fail in classic image

This issue was migrated from https://bugs.launchpad.net/plainbox-provider-checkbox/+bug/1979698

Summary

Status	Created on	Heat	Importance	Security related
Incomplete	2022-06-24 00:53:15	14	Critical	False

Description

[Summary]
SDHC card insert case fail in classic image. This issue seems to be a common issue for classic image. I encounter to it in few different project's classic image, and only classic image not UC. However, media card is functional if manually test it.

[Step]
1.Boot into system
2.Run test case of "mediacard/sdhc-insert"

[Expected result]
The case should passed after the case executed and insert the sd card.

[Actual result]
Case fail with some error message.

INSERT NOW

/tmp/nest-05oocov8.e74aa7c94e92195ff3a8518e1c6ed3ff68137ed0fd3d05185135588bf6e0cb39/removable_storage_watcher.py:500: PyGIDeprecationWarning: GObject.timeout_add_seconds is deprecated; use GLib.timeout_add_seconds instead
GObject.timeout_add_seconds(timeout, self._on_timeout_expired)
You have failed to perform the required manipulation in time

[Failure rate]
2/2

[Additional information]
checkbox20 1.28 976

Attachments

usb_insert_fail.log
usb-debug.log
removable-storage-debug-20.04-classic-snap-OK.log
journalctl.log

Tags:
[]

Crash report dialog be shown caused by graphics_driver.py

Summary

Crash report dialog will be shown while running driver_version_after_suspend_xxx case.
Please see the attachment.

Description

[Steps to reproduce]

Boot into OS
Run driver_version_after_suspend_xxx case

[Expected result]
The crash report dialog won't be shown

[Actual result]
The crash report dialog be shown which is caused by graphics_driver.py
See attachment

[Failure rate]
Always

[Additional Info]
OS: 22.04
Windowing System: Wayland

Attachment

LP1811931: [Missing test case] suspend/record_playback_after_suspend_auto not in after-suspend-reference-cert-blockers nested test plan

This issue was migrated from https://bugs.launchpad.net/plainbox-provider-checkbox/+bug/1811931

Summary

Status	Created on	Heat	Importance	Security related
Confirmed	2019-01-16 06:40:22	154	High	False

Description

The suspend/record_playback_after_suspend_auto not in any nested test plan, it should be in whitelist, please add it to the after-suspend-reference-cert-blockers.

Attachments

No attachments

Tags:
['ce-qa-concern']

dupl

This issue was migrated from https://bugs.launchpad.net/plainbox-provider-checkbox/+bug/1988030

Summary

The Sink will be changed during audio/playback_headphones testing

Description

[Steps to reproduce]

Connect monitor via DP cable
Boot into OS
Run "checkbox-cli run com.canonical.certification::audio/playback_headphones"
Follow the steps to plug a headset and choose headphone on popup dialog
Check the Sink in Settings -> Sound, it should turn to the "Headphones - Build in-Audio" from "Speakers - Build in-Audio"
Press Enter to perform case
The Sink becomes "HDMI/Displayport-HDA NVDIA" and no sound from headset (issue happened)

[Expected result]
Sink should keep as "Headphones - Build in-Audio" and can hear sound from headset

[Actual result]
Sink is changed to "HDMI/Displayport-HDA NVDIA" during testing and back automatically.
See comment 2 (Video record)

[Failure rate]
Always

[Additional Info]
SKU ：KRPL-CFF-EVT-C1 (Desktop)
BIOS Version : 0.4.18
Image/Manifest : CFF X19
CPU : Raptor Lake 8+16 65W LGA ES2
GPU : T400 dGFx

Attachment

sink_will_be_changed_while_running_case.webm

LP1928195: [Nvidia + NVLINK] Checkbox would switch prime profile to intel during graphic testing

This issue was migrated from https://bugs.launchpad.net/plainbox-provider-checkbox/+bug/1928195

Summary

Status	Created on	Heat	Importance	Security related
New	2021-05-12 10:17:13	6	Critical	False

Description

Description:
The checkbox would switch PRIME profile to intel when there're only two nVidia discrete GPU cards on the system.

I have investigated the source code about graphic resource, and found we didn't check is integrated GPU is installed or not.

Script block

switch_cmds['nvidia'] = ('prime-select nvidia', 'prime-select intel')
...
record['switch_to_cmd'] = switch_cmds[record['driver']][0]
if index == 2 and len(video_devices) == 2:
    # we're at GPU number 2 and there are two, so here we assume
    # that video_devices[0] is the built-in one
    video_devices[0]['switch_to_cmd'] = (switch_cmds[record['driver']][1])

Steps to reproduce:

Install two nVidia GPU w/ NVLINK. (The machine has not integrated GPU).
Install OS and checkbox.
Run client-cert-20-04 test plan.
Note: the screen would become blank after graphics/1_auto_switch_card_.* job.

Expected result:
The checkbox should not switch to intel profile.

Observed result:
The checkbox issue a command to switch PRIME profile to intel. So the screen become blank due to no integrated GPU on the machine.

Test Environment:

A workstation w/o integrated GPU.
Two nVidia discrete GPU are installed.

Attachments

No attachments

Tags:
['cbox-3']

LP1811928: [Missing test case] audio/alsa_record_playback_automated not in audio-cert-blockers nested test plan

This issue was migrated from https://bugs.launchpad.net/plainbox-provider-checkbox/+bug/1811928

Summary

Status	Created on	Heat	Importance	Security related
Confirmed	2019-01-16 06:19:53	154	High	False

Description

The audio/alsa_record_playback_automated not in any nested test plan, it should be in whitelist, please add it to the audio-cert-blockers.

Attachments

No attachments

Tags:
['ce-qa-concern']

LP1898867: Ctrl-c to kill remote job traceback

This issue was migrated from https://bugs.launchpad.net/checkbox-ng/+bug/1898867

Summary

Status	Created on	Heat	Importance	Security related
Confirmed	2020-10-07 11:57:20	6	Critical	False

Description

-----------------------------[ Running job 2 / 2 ]------------------------------
---------------------[ Stress of CPUs (very long runtime) ]---------------------
ID: com.canonical.certification::stress/cpu_stress_ng_test
Category: Stress tests

STRESS_NG_CPU_TIME env var is not found, stress_ng cpu running time is default value
stress-ng: info: [1858] dispatching hogs: 4 af-alg, 4 bsearch, 4 context, 4 cpu, 4 crypt, 4 hsearch, 4 longjmp, 4 lsearch, 4 matrix, 4 qsort, 4 str, 4 stream, 4 tsearch, 4 vecmath, 4 wcs
stress-ng: info: [1859] stress-ng-af-alg: 23 cryptographic algorithms found in /proc/crypto
stress-ng: info: [1859] stress-ng-af-alg: 106 cryptographic algorithms max (with defconfigs)
stress-ng: info: [1913] stress-ng-stream: using built-in defaults as unable to determine cache details
stress-ng: info: [1913] stress-ng-stream: stressor loosely based on a variant of the STREAM benchmark code
stress-ng: info: [1913] stress-ng-stream: do NOT submit any of these results to the STREAM benchmark results
stress-ng: info: [1913] stress-ng-stream: Using CPU cache size of 2048K
^CTraceback (most recent call last):
File "/usr/local/bin/checkbox-cli", line 11, in
load_entry_point('checkbox-ng', 'console_scripts', 'checkbox-cli')()
File "/home/sylvain/lp/checkbox-ng/checkbox_ng/launcher/checkbox_cli.py", line 123, in main
subcmd.invoked(ctx)
File "/home/sylvain/lp/checkbox-ng/checkbox_ng/launcher/master.py", line 155, in invoked
self.connect_and_run(ctx.args.host, port)
File "/home/sylvain/lp/checkbox-ng/checkbox_ng/launcher/master.py", line 179, in connect_and_run
keep_running = self._handle_interrupt()
File "/home/sylvain/lp/checkbox-ng/checkbox_ng/launcher/master.py", line 373, in _handle_interrupt
self._sa.send_signal(signal.SIGKILL.value)
File "/home/sylvain/lp/checkbox-ng/plainbox/vendor/rpyc/core/netref.py", line 253, in call
return syncreq(self, consts.HANDLE_CALL, args, kwargs)
File "/home/sylvain/lp/checkbox-ng/plainbox/vendor/rpyc/core/netref.py", line 76, in syncreq
return conn.sync_request(handler, proxy, *args)
File "/home/sylvain/lp/checkbox-ng/plainbox/vendor/rpyc/core/protocol.py", line 469, in sync_request
return self.async_request(handler, *args, timeout=timeout).value
File "/home/sylvain/lp/checkbox-ng/plainbox/vendor/rpyc/core/async.py", line 102, in value
raise self._obj
TypeError: unsupported operand type(s) for +: 'NoneType' and 'bytes'

========= Remote Traceback (1) =========
Traceback (most recent call last):
File "/snap/checkbox20/current/lib/python3.8/site-packages/plainbox/vendor/rpyc/core/protocol.py", line 320, in _dispatch_request
res = self._HANDLERS[handler](self, *args)
File "/snap/checkbox20/current/lib/python3.8/site-packages/plainbox/vendor/rpyc/core/protocol.py", line 593, in _handle_call
return obj(*args, **dict(kwargs))
File "/snap/checkbox20/current/lib/python3.8/site-packages/plainbox/impl/session/remote_assistant.py", line 703, in send_signal
self._sa.send_signal(signal, target_user)
File "/snap/checkbox20/current/lib/python3.8/site-packages/plainbox/impl/session/assistant.py", line 1744, in send_signal
self._runner.send_signal(signal, target_user)
File "/snap/checkbox20/current/lib/python3.8/site-packages/plainbox/impl/execution.py", line 377, in send_signal
os.write(in_w, self._password_provider() + b'\n')
TypeError: unsupported operand type(s) for +: 'NoneType' and 'bytes'

Attachments

No attachments

Tags:
[]

LP1995454: Graphics/_driver_version_ failed

This issue was migrated from https://bugs.launchpad.net/plainbox-provider-checkbox/+bug/1995454

Summary

Status	Created on	Heat	Importance	Security related
Confirmed	2022-11-02 07:30:18	6	Critical	False

Description

[Summary]
Run Checkbox Graphic test item the following test were fail:
1.Graphics/1_driver_version_PCI_ID_0x4688
2.Graphics/2_driver_version_GA107BM_GeForce_RTX_3050_Ti_Mobie

[Graphics/1_driver_version_PCI_ID_0x4688 Output]
error: list of process IDs must follow -p

Usage:
ps [options]

For more details see ps(1).
Traceback (most recent call last):
File "/tmp/nest-ynqwi8rq.1531dd90874d443ab9a3fbf7a63c668f7a0ab3881a634524bc7811bd83eed721/graphics_driver.py", line 399, in
sys.exit(main())
File "/tmp/nest-ynqwi8rq.1531dd90874d443ab9a3fbf7a63c668f7a0ab3881a634524bc7811bd83eed721/graphics_driver.py", line 373, in main
xorg_owner = check_output("ps -o user= -p $(pidof Xorg)",
File "/usr/lib/python3.10/subprocess.py", line 420, in check_output
return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
File "/usr/lib/python3.10/subprocess.py", line 524, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command 'ps -o user= -p $(pidof Xorg)' returned non-zero exit status 1.

[Graphics/2_driver_version_GA107BM_GeForce_RTX_3050_Ti_Mobie Output]
error: list of process IDs must follow -p

Usage:
ps [options]

For more details see ps(1).
Traceback (most recent call last):
File "/tmp/nest-ritxmv9m.6fe1458016043765a23331f4a48413f7f2bbb8cb9332037b39fb229ff44db35e/graphics_driver.py", line 399, in
sys.exit(main())
File "/tmp/nest-ritxmv9m.6fe1458016043765a23331f4a48413f7f2bbb8cb9332037b39fb229ff44db35e/graphics_driver.py", line 373, in main
xorg_owner = check_output("ps -o user= -p $(pidof Xorg)",
File "/usr/lib/python3.10/subprocess.py", line 420, in check_output
return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
File "/usr/lib/python3.10/subprocess.py", line 524, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command 'ps -o user= -p $(pidof Xorg)' returned non-zero exit status 1.

[Addtional Information]
Test Vault ID: 100505
image/manifest:dell-bto-jammy-jellyfish-gardevoir-X44-20220928-7.iso
Product Name:Galio
Bios Version:0.2.15
Kernel Version:6.0.0.1004-oem
CPU:Raptor Lake HX ES
GPU:Nvidia GN20-P1
Sku:All
Failure rate:2/2 units,4/4 times.

Attachments

Graphic.7z

Tags:
[]

canonical / checkbox Goto Github PK

checkbox's Introduction

What is Checkbox?

Getting started

Content of this repository

Footnotes

checkbox's People

Contributors

Stargazers

Watchers

Forkers

checkbox's Issues

Summary

Description

stdout

stderr

Attachments

Summary

Description

Attachments

Summary

Description

Attachments

Summary

Description

Attachments

Summary

Description

Attachments

Summary

Description

Attachments

Summary

Description

Attachments

Summary

Description

Attachments

Summary

Description

Attachments

Summary

Description

Attachments

Summary

Description

Attachments

Summary

Description

Attachments

Summary

Description

Attachments

Summary

Description

Attachments

Summary

Description

Attachments

Summary

Description

Attachments

Summary

Description

Attachments

Summary

Description

Attachments

Summary

Description

Attachments

Summary

Description

Attachments

Summary

Description

Attachments

Summary

Description

Attachments