linuxkit / rtf

Regression testing framework

LinuxKit

LinuxKit, a toolkit for building custom minimal, immutable Linux distributions.

• Secure defaults without compromising usability
• Everything is replaceable and customisable
• Immutable infrastructure applied to building Linux distributions
• Completely stateless, but persistent storage can be attached
• Easy tooling, with easy iteration
• Built with containers, for running containers
• Designed to create reproducible builds [WIP]
• Designed for building and running clustered applications, including but not limited to container orchestration such as Docker or Kubernetes
• Designed from the experience of building Docker Editions, but redesigned as a general-purpose toolkit
• Designed to be managed by external tooling, such as Infrakit (renamed to deploykit which has been archived in 2019) or similar tools
• Includes a set of longer-term collaborative projects in various stages of development to innovate on kernel and userspace changes, particularly around security

LinuxKit currently supports the x86_64, arm64, and s390x architectures on a variety of platforms, both as virtual machines and baremetal (see below for details).

Subprojects

• LinuxKit kubernetes aims to build minimal and immutable Kubernetes images. (previously projects/kubernetes in this repository).
• LinuxKit LCOW LinuxKit images and utilities for Microsoft's Linux Containers on Windows.
• linux A copy of the Linux stable tree with branches LinuxKit kernels.
• virtsock A go library and test utilities for virtio and Hyper-V sockets.
• rtf A regression test framework used for the LinuxKit CI tests (and other projects).
• homebrew Homebrew packages for the linuxkit tool.

Getting Started

Build the linuxkit tool

LinuxKit uses the linuxkit tool for building, pushing and running VM images.

Simple build instructions: use make to build. This will build the tool in bin/. Add this to your PATH or copy it to somewhere in your PATH eg sudo cp bin/* /usr/local/bin/. Or you can use sudo make install.

If you already have go installed you can use go install github.com/linuxkit/linuxkit/src/cmd/linuxkit@latest to install the linuxkit tool.

On MacOS there is a brew tap available. Detailed instructions are at linuxkit/homebrew-linuxkit, the short summary is

brew tap linuxkit/linuxkit
brew install --HEAD linuxkit

Build requirements from source using a container

• GNU make
• Docker
• optionally qemu

For a local build using make local

• go
• make
• go get -u golang.org/x/lint/golint
• go get -u github.com/gordonklaus/ineffassign

Building images

Once you have built the tool, use

linuxkit build linuxkit.yml

to build the example configuration. You can also specify different output formats, eg linuxkit build --format raw-bios linuxkit.yml to output a raw BIOS bootable disk image, or linuxkit build --format iso-efi linuxkit.yml to output an EFI bootable ISO image. See linuxkit build -help for more information.

Booting and Testing

You can use linuxkit run <name> or linuxkit run <name>.<format> to execute the image you created with linuxkit build <name>.yml. This will use a suitable backend for your platform or you can choose one, for example VMWare. See linuxkit run --help.

Currently supported platforms are:

• Local hypervisors
  • Virtualization.Framework (macOS) [x86_64, arm64]
  • HyperKit (macOS) [x86_64]
  • Hyper-V (Windows) [x86_64]
  • qemu (macOS, Linux, Windows) [x86_64, arm64, s390x]
  • VMware (macOS, Windows) [x86_64]
• Cloud based platforms:
  • Amazon Web Services [x86_64]
  • Google Cloud [x86_64]
  • Microsoft Azure [x86_64]
  • OpenStack [x86_64]
  • Scaleway [x86_64]
• Baremetal:
  • packet.net [x86_64, arm64]
  • Raspberry Pi Model 3b [arm64]

Running the Tests

The test suite uses rtf To install this you should use make bin/rtf && make install. You will also need to install expect on your system as some tests use it.

To run the test suite:

cd test
rtf -v run -x

This will run the tests and put the results in a the _results directory!

Run control is handled using labels and with pattern matching. To run add a label you may use:

rtf -v -l slow run -x

To run tests that match the pattern linuxkit.examples you would use the following command:

rtf -v run -x linuxkit.examples

Building your own customised image

To customise, copy or modify the linuxkit.yml to your own file.yml or use one of the examples and then run linuxkit build file.yml to generate its specified output. You can run the output with linuxkit run file.

The yaml file specifies a kernel and base init system, a set of containers that are built into the generated image and started at boot time. You can specify the type of artifact to build eg linuxkit build -format vhd linuxkit.yml.

If you want to build your own packages, see this document.

Yaml Specification

The yaml format specifies the image to be built:

• kernel specifies a kernel Docker image, containing a kernel and a filesystem tarball, eg containing modules. The example kernels are built from kernel/
• init is the base init process Docker image, which is unpacked as the base system, containing init, containerd, runc and a few tools. Built from pkg/init/
• onboot are the system containers, executed sequentially in order. They should terminate quickly when done.
• services is the system services, which normally run for the whole time the system is up
• files are additional files to add to the image

For a more detailed overview of the options see yaml documentation

Architecture and security

There is an overview of the architecture covering how the system works.

There is an overview of the security considerations and direction covering the security design of the system.

Roadmap

This project was extensively reworked from the code we are shipping in Docker Editions, and the result is not yet production quality. The plan is to return to production quality during Q3 2017, and rebase the Docker Editions on this open source project during this quarter. We plan to start making stable releases on this timescale.

This is an open project without fixed judgements, open to the community to set the direction. The guiding principles are:

• Security informs design
• Infrastructure as code: immutable, manageable with code
• Sensible, secure, and well-tested defaults
• An open, pluggable platform for diverse use cases
• Easy to use and participate in the project
• Built with containers, for portability and reproducibility
• Run with system containers, for isolation and extensibility
• A base for robust products

Development reports

There are monthly development reports summarising the work carried out each month.

Adopters

We maintain an incomplete list of adopters. Please open a PR if you are using LinuxKit in production or in your project, or both.

FAQ

See FAQ.

Released under the Apache 2.0 license.

rtf's Issues

Add support for creating a `json` output and additional tooling

When running tests either repeatedly or the same tests on different system it is often useful to compare results. Currently, we store the test results in a csv file which is a ok, but a bit tedious to post-process in particular when trying to compare different results.

The proposal is to stash all the tests results as well as the system information and test meta data (such as start/end times etc) in a single json file with a reasonably well defined schema.

Once we have that we should be able to add simple tooling to read/compare multiple json files and generate, for example, a summary csv file from it.

Add ability to run a single test or a group

With the original python based implementation one could run a single test or group of tests. Very useful when debugging and it would be nice if we could do:

rtf run foo.bar.baz

Honour ordering for mixed tests/subgroups

if you have something like:

cases/010_foo/000_bar/test.sh
cases/010_foo/010_baz/000_foobar/test.sh
cases/010_foo/020_foo/test.sh

currently foo.baz.foobar get's executed before foo.bar and foo.foo. It would be good to honour the order when traversing into subgroups

panic with: fatal error: concurrent map iteration and map write

Seen in https://linuxkit.datakit.ci/log/saved/linuxkit-build-of-c6b7b3c00af3e8619b15d43be52c53c9e8db5a46-default-ci-pr/1b0a95c509e2145e4a0550902ca71535517c00d7 which was testing linuxkit/linuxkit#2224 (a docs only change).

Add Cross Builds to Makefile

Currently the windows and linux builds aren't tested.
I have a feeling that windows won't work at all!
It'd be good to get this tested soonish...

switch from circleci to actions

For consistency we should test in actions.

Add CI

it's currently very easy to break things right now...

Support powershell test scripts

For windows only tests it would be nice to be able to write test scripts, group initialisation scripts, and pre/post test scripts in powershell.

This issue is to track this.

I actually started working on it and my WIP branch is here: https://github.com/rn/rtf/tree/ps1

Panic when post-test.sh fails

Having a failure in post-test.sh involved a panic.

panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xc0000005 code=0x0 addr=0x20 pc=0x537ca8]

goroutine 1 [running]:
github.com/linuxkit/rtf/local.(*Test).Run(0xc0421456c0, 0x1, 0x862cd4, 0x5, 0xc0421b3c00, 0x65, 0xa74520, 0xc042002e20, 0x86204d, 0x3, ...)
	C:/Users/docker/go/src/github.com/linuxkit/rtf/local/test.go:154 +0xa58
github.com/linuxkit/rtf/local.(*Group).Run(0xc04211fab0, 0x1, 0x862cd4, 0x5, 0xc0421b3c00, 0x65, 0xa74520, 0xc042002e20, 0x86204d, 0x3, ...)

I suspect this return : https://github.com/linuxkit/rtf/blob/master/local/script.go#L143
res.TestResult can be different than Pass and have nil error.

recent changes broke LinuxKit tests

rtf revision c7b1f074f12ab2116509f69ffa6331aec104e244

$ pwd
~/src/linuxkit/linuxkit/test
$ rtf list
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x1a8 pc=0x10fa332]

goroutine 1 [running]:
github.com/linuxkit/rtf/local.(*Test).LabelString(0x0, 0x1, 0x1)
        /go/src/github.com/linuxkit/rtf/local/test.go:65 +0x22
github.com/linuxkit/rtf/cmd.list(0x165c880, 0x167f2c0, 0x0, 0x0, 0x0, 0x0)
        /go/src/github.com/linuxkit/rtf/cmd/list.go:66 +0x47e
github.com/linuxkit/rtf/vendor/github.com/spf13/cobra.(*Command).execute(0x165c880, 0x167f2c0, 0x0, 0x0, 0x165c880, 0x167f2c0)
        /go/src/github.com/linuxkit/rtf/vendor/github.com/spf13/cobra/command.go:621 +0x3f8
github.com/linuxkit/rtf/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0x165cac0, 0xc42006c058, 0x0, 0xc420109f48)
        /go/src/github.com/linuxkit/rtf/vendor/github.com/spf13/cobra/command.go:700 +0x339
github.com/linuxkit/rtf/vendor/github.com/spf13/cobra.(*Command).Execute(0x165cac0, 0x1, 0x1)
        /go/src/github.com/linuxkit/rtf/vendor/github.com/spf13/cobra/command.go:659 +0x2b
github.com/linuxkit/rtf/cmd.Execute()
        /go/src/github.com/linuxkit/rtf/cmd/root.go:42 +0x31
main.main()
        /go/src/github.com/linuxkit/rtf/main.go:20 +0x20

Probably one of my refactoring changes.

Add `junit` support

junit is a "standard" XML based file format to report unit test results and there seems to be some tooling support for it as well (at least in Jenkins). Would be good to add junit support to rtf to take advantage of this tooling and to allow for better integration with Jenkins based CI systems.

Here is a link to some documentation I found: http://llg.cubic.org/docs/junit/ and here is a link to the XSD schema: https://github.com/windyroad/JUnit-Schema

Labels don't seem to work

For some linuxkit tests I tried adding a labels such as:

# LABELS: darwin,linux

with the intention that they would only be run on macOS (darwin) and Linux.

I've added this to all of the linuxkit.kernel tests but when run on macOS:

# rtf list  linuxkit.kernel
STATE           TEST                                    LABELS
[...]
SKIP     linuxkit.kernel.config_4.4.x            darwin, linux
SKIP     linuxkit.kernel.config_4.9.x            darwin, linux
SKIP     linuxkit.kernel.config_4.11.x           darwin, linux
SKIP     linuxkit.kernel.kmod_4.9.x              darwin, linux
[...]

tests are skipped. However:

rtf list -l linux linuxkit.kernel
STATE           TEST                                    LABELS
[...]
RUN      linuxkit.kernel.config_4.4.x            linux, darwin
RUN      linuxkit.kernel.config_4.9.x            darwin, linux
RUN      linuxkit.kernel.config_4.11.x           darwin, linux
RUN      linuxkit.kernel.kmod_4.9.x              linux, darwin
[...]

But:

rtf list -l darwin linuxkit.kernel
STATE           TEST                                    LABELS
[...]
SKIP     linuxkit.kernel.config_4.4.x            linux, darwin
SKIP     linuxkit.kernel.config_4.9.x            darwin, linux
SKIP     linuxkit.kernel.config_4.11.x           darwin, linux
SKIP     linuxkit.kernel.kmod_4.9.x              darwin, linux
[...]

It would also be good if leading or trailing whitespaces would be stripped, so that one can write:

# LABELS: darwin, linux