osbuild / cloud-image-val

Multi-cloud image validation tool.

OSBuild

Build-Pipelines for Operating System Artifacts

OSBuild is a pipeline-based build system for operating system artifacts. It defines a universal pipeline description and a build system to execute them, producing artifacts like operating system images, working towards an image build pipeline that is more comprehensible, reproducible, and extendable.

See the osbuild(1) man-page for details on how to run osbuild, the definition of the pipeline description, and more.

Project

• Website: https://www.osbuild.org
• Bug Tracker: https://github.com/osbuild/osbuild/issues
• Matrix: #image-builder on fedoraproject.org
• Mailing List: [email protected]
• Changelog: https://github.com/osbuild/osbuild/releases

Principles

1. OSBuild stages are never broken, only deprecated. The same manifest should always produce the same output.
2. OSBuild stages should be explicit whenever possible instead of e.g. relying on the state of the tree.
3. Pipelines are independent, so the tree is expected to be empty at the beginning of each.
4. Manifests are expected to be machine-generated, so OSBuild has no convenience functions to support manually created manifests.
5. The build environment is confined against accidental misuse, but this should not be considered a security boundary.
6. OSBuild may only use Python language features supported by the oldest target distribution.

Contributing

Please refer to the developer guide to learn about our workflow, code style and more.

Requirements

The requirements for this project are:

• bubblewrap >= 0.4.0
• python >= 3.6

Additionally, the built-in stages require:

• bash >= 5.0
• coreutils >= 8.31
• curl >= 7.68
• qemu-img >= 4.2.0
• rpm >= 4.15
• tar >= 1.32
• util-linux >= 235
• skopeo

At build-time, the following software is required:

• python-docutils >= 0.13
• pkg-config >= 0.29

Testing requires additional software:

• pytest

Build

Osbuild is a python script so it is not compiled. To verify changes made to the code use included makefile rules:

• make lint to run linter on top of the code
• make test-all to run base set of tests
• sudo make test-run to run extended set of tests (takes long time)

Installation

Installing osbuild requires to not only install the osbuild module, but also additional artifacts such as tools (i.e: osbuild-mpp) sources, stages, schemas and SELinux policies.

For this reason, doing an installation from source is not trivial and the easier way to install it is to create the set of RPMs that contain all these components.

This can be done with the rpm make target, i.e:

make rpm

A set of RPMs will be created in the ./rpmbuild/RPMS/noarch/ directory and can be installed in the system using the distribution package manager, i.e:

sudo dnf install ./rpmbuild/RPMS/noarch/*.rpm

Repository

• web: https://github.com/osbuild/osbuild
• https: https://github.com/osbuild/osbuild.git
• ssh: [email protected]:osbuild/osbuild.git

License

• Apache-2.0
• See LICENSE file for details.

cloud-image-val's Issues

add redhat-cloud-client-configuration to pkg wanted list from RHEL-8.7 and RHEL-9.1 AMIs

Reference ticket:COMPOSER-1804.

add new coverage for bz2117700

kernel-debug, kernel-debug-devel matching current kernel version are available in repo

Here is ami-val key steps:
run_cmd(test_instance, "sudo yum -y install install kernel-debug-devel-$(uname -r)", timeout=1200)
run_cmd(test_instance, "sudo yum -y install install kernel-debug-$(uname -r)", timeout=1200)

float() use should be replaced with version.parse()

float values are not safe for comparisons, especially equality. They should not be used for version (or really, anything other than math). This codebase is already using packaging.version's parse for some of the version handling. The floats should be converted to do this as well.

Failing tests due to string quoting changes in systemd unit files

In osbuild/images#567 we made a non-functional change to the way we write systemd unit files. The Environment option in the osbuild stage now supports multiple entries. At the same time, we made it so that key=value assignments are quoted, which makes it easier to work with values with spaces.
From the systemd-exec man page:

ENVIRONMENT
       Environment=
           Sets environment variables for executed processes. Each line is unquoted using the rules described in "Quoting" section in
           systemd.syntax(7) and becomes a list of variable assignments. If you need to assign a value containing spaces or the equals sign to a
           variable, put quotes around the whole assignment.
...

           Example:

               Environment="VAR1=word1 word2" VAR2=word3 "VAR3=$word 5 6"

I think this change made CIV start failing to validate because it expects the unquoted strings in some of the cloud systemd units like here:

cloud-image-val/test_suite/generic/test_generic.py

Line 716 in 32b5842

'expect_config': 'Environment=NM_CLOUD_SETUP_AZURE=yes'

The line now is Environment="NM_CLOUD_SETUP_AZURE=yes" instead of Environment=NM_CLOUD_SETUP_AZURE=yes.

Terraform AWS keypair creation improvement

We discovered that sometimes the AWS keypairs are not created in time while an instance for the same region is being created and it fails due to "nonexistent keypair".

Areas of improvement are:

• Remove ".pub" in line 94
• Modify Terraform controller to handle the Terraform configuration and create keypairs first
• Remove "cloud" attribute from Configurator constructor. It already comes from resources.json
• Add "get_cloud_provider_from_resources_json" function to configurator