canonical / checkbox-kivu

Checkbox provider for the Kivu project

Welcome to the Checkbox Kivu project!

This repository contains the Checkbox Kivu Provider (Kivu-specific test cases and test plans for Checkbox) as well as everything that is required to build the checkbox-kivu-classic snap in the snapstore.

Checkbox Kivu Provider

Located in the checkbox-provider-kivu directory, it contains:

• the test cases (also called "jobs" in the Checkbox jargon) and test plans to be run by Checkbox (in the units directory)
• the scripts required by some of the test cases (in the bin directory)
• the data (sample video, HTML pages) required by some of the test cases (in the data directory)
• unit tests for the scripts (in the tests directory)

Requirements

• Ubuntu Jammy (22.04)
• Chromium browser
• Supported hardware platforms:
  • Intel platforms with recent GPU (>= Broadwell)
  • AMD platform with Radeon GPU
• At least 1 active display (typically, a laptop with closed lid will not work)

Installation

Install the Checkbox runtime and the Kivu provider snaps:

sudo snap install checkbox22
sudo snap install checkbox-kivu-classic --classic

Make sure that the provider service is running and active:

systemctl status snap.checkbox-kivu-classic.remote-slave.service

Install dependencies

Some test need dependencies, so in order to run all tests, you might way to install those dependencies. A helper script is available to install them:

checkbox-kivu-classic.install-full-deps

Automated Run

To run the full test plan:

checkbox-kivu-classic.test-runner-automated

To run the hardware decoding tests:

checkbox-kivu-classic.test-runner-automated-hwdec

Interactive Run

Interactive run allows you to select test plan and specific tests to run.

checkbox.checkbox-cli remote 127.0.0.1

You will be presented with a long list of available test plans. You can filter this out using the / key to search for Kivu. Select the Kivu test plan, and follow the instructions on
screen to start the test run.

Develop the Checkbox Kivu provider

Since snaps are immutable, it is not possible to modify the content of the scripts or the test cases. Fortunately, Checkbox provides a functionality to side-load a provider on the DUT.

Therefore, if you want to edit a job definition, a script or a test plan, run the following commands on the DUT:

cd $HOME
git clone [email protected]:canonical/checkbox-kivu.git
mkdir /var/tmp/checkbox-providers
cp -r $HOME/checkbox-kivu/checkbox-provider-kivu /var/tmp/checkbox-providers/

You can then modify the content of the provider in /var/tmp/checkbox-providers/checkbox-provider-kivu/, and it's this version that will be used when you run the tests.

Please refer to the Checkbox documentation on side-loading providers for more information.

checkbox-kivu's Issues

Add job in checkbox-kivu to test webcam stream encoding in Chromium

KIVU-73

Capture powerstat data on jobs with and without VAAPI enabled.

Comparing powerstat data (with and without vaapi) might be a good option to identify any power regressions. Majority of the power regressions would be captured by CPU usage comparison (see #10), but power regressions due to device failures (eg. something going wrong in audio/camera devices) may not be captured by this flow.

GPU load usage script ignore 0.0 values

When parsing the intel_gpu_top output, the script (compare_intel_gpu_top_json.py & parse_intel_gpu_top_json.py) ignores values 0.0

Lacks test for overlay planes (MPO optimization.)

Checkbox should be able to test if overlay planes can be used by chromium.

Or a simpler test: can overlay planes be used at all?
(For Weston: Yes. For Mutter: No.)

Also, see this slide deck from Intel.

The test could either incorporate gfxi or obtain info from /sys/kernel/debug/dri/ like Intel does.

This is KIVU-97

Lacks test for chromium flags

We should check if all chromium flags are present that are required for Vaapi use.

Capture CPU usage with and without VAAPI enabled on Chromium

Similar to the GPU encoding/decoding tests on Chromium, we should have tests that capture CPU usage and store it into the test results for later analysis/comparison.

Test job passes after a core dump of chromium.

Non hw-acc chromium test dumps core.

What is even worse: checkbox marks it as a PASS.

ID: com.canonical.certification::kivu/chromium_h264_decoding_vaapi_disabled
Category: Intel Kivu HW Accel tests
--------------------------------------------------------------------------------
XDG_SESSION_TYPE wayland
[90501:90501:0307/090157.088378:WARNING:chrome_main_delegate.cc(618)] This is Chrome version 111.0.5563.19 (not a warning)
[90501:90501:0307/090157.116723:WARNING:wayland_object.cc(152)] Binding to gtk_shell1 version 4 but version 5 is available.
[90501:90501:0307/090157.116741:WARNING:wayland_object.cc(152)] Binding to zwp_pointer_gestures_v1 version 1 but version 3 is available.
[90501:90501:0307/090157.116759:WARNING:wayland_object.cc(152)] Binding to zwp_linux_dmabuf_v1 version 3 but version 4 is available.
[90501:90501:0307/090157.288310:WARNING:account_consistency_mode_manager.cc(70)] Desktop Identity Consistency cannot be enabled as no OAuth client ID and client secret have been configured.
[90501:90501:0307/090157.302627:WARNING:wayland_surface.cc(151)] Server doesn't support zcr_alpha_compositing_v1.
[90501:90501:0307/090157.302635:WARNING:wayland_surface.cc(166)] Server doesn't support overlay_prioritizer.
[90501:90501:0307/090157.302638:WARNING:wayland_surface.cc(180)] Server doesn't support surface_augmenter.
[90501:90501:0307/090157.302639:WARNING:wayland_surface.cc(195)] Server doesn't support wp_content_type_v1
[90501:90501:0307/090157.302641:WARNING:wayland_surface.cc(214)] Server doesn't support zcr_color_management_surface.
[90501:90501:0307/090157.302770:WARNING:cursor_loader.cc(123)] Failed to load a platform cursor of type kNull
[90627:90627:0307/090157.308757:ERROR:gpu_init.cc(525)] Passthrough is not supported, GL is egl, ANGLE is 
[90627:90627:0307/090157.310966:WARNING:sandbox_linux.cc(393)] InitializeSandbox() called with multiple threads in process gpu-process.
[90501:90501:0307/090157.355657:WARNING:bluez_dbus_manager.cc(247)] Floss manager not present, cannot set Floss enable/disable.
[90665:14:0307/090157.472664:INFO:decoder.cc(41)] DecryptingVideoDecoder
[90665:14:0307/090157.472794:INFO:decoder.cc(32)] VpxVideoDecoder
[90665:14:0307/090157.472845:INFO:decoder.cc(44)] Dav1dVideoDecoder
[90665:14:0307/090157.476986:INFO:decoder.cc(29)] FFmpegVideoDecoder
[90665:14:0307/090157.477041:INFO:decoder.cc(29)] FFmpegVideoDecoder
[0307/090157.502363:ERROR:ptracer.cc(567)] ptrace: Input/output error (5)
[0307/090157.502513:ERROR:ptracer.cc(567)] ptrace: Input/output error (5)
[0307/090157.521070:ERROR:elf_dynamic_array_reader.h(64)] tag not found
timeout: the monitored command dumped core
Test finished
--------------------------------------------------------------------------------
Outcome: job passed

Lacks test for Hardware-Compositing / Software-Compositing in chromium.

To catch regression like this we would like to monitor the output of the chrome://gpu URL.

We should check for any software fallbacks, especially when it comes to compositing.

It will be especially helpful to run these on non-Intel platforms, in the scope of the KIVU project, as we want chromium to be as efficient as possible on Intel, but without major regressions on other GPUs.

NOTE: The chrome://gpu is special, in the sense that you cannot launch it from the command line, nor can you link to it, as it will be blocked. We may have to resort to injecting keystrokes into the browser to get this data from chromium.

Lacks test for onecopy.

We should have tests for all the optimization recipes.
Including the onecopy strategy.

Cannot lock device after running tests

The disabling of the screen saver is never undone by the test scripts, which gives my machine pretty intense insomnia. At some point, we should run:

/usr/bin/gsettings set org.gnome.desktop.lockdown disable-lock-screen 'false'

I haven't dug into what else some of these scripts are doing, but it seems like a good idea to do an audit to make sure that system changes are undone after the test is completed.

Lacks test for non-embedded videos.

When testing chromium video playback, a video inside an HTML webpage is used.

We also need to test playing a video file (mp4) directly without first going through html, so that the video is truly full screen, without browser decoration space, from webpage.

This is because I've found the browser to take different paths for that, and the direct playback causes a crash, whereas the embedded on does not.

It's better to test both scenarios.

I will make a PR for this.

Tested with checkbox version: 1.31 and HEAD from checkbox-kivu repo.

Add job in checkbox-kivu to test webcam stream encoding using gstreamer

KIVU-72

wl-paste hangs when screen saver is active

When the screen saver is active, the wl-paste hangs
This prevents the kivu/chromium_hardware_compositing from working