ubi-agni / ros-builder-action

Github action to build .deb packages from ROS sources

github action to build .deb packages from ROS sources

This repository provides actions and resusable workflows helping to build Debian packages from ROS package sources.

For convenience, the built packages from ros-one.repos are deployed to https://ros.packages.techfak.net.
If you want any other packages to be included in the repository, please open an issue or pull request.

Interactive build workflow

For building a ROS distro it is recommended to use the high-level interactive.yaml workflow. This manually triggered workflow builds all packages from the given source(s) and provides the resulting .debs as an artifact. Optionally, the .debs can be deployed to a github repo. However, using github for this purpose is not recommended (slow and size limitations). Better use a dedicated repository server like aptly or reprepro. The reprepro-import.sh script helps with the latter.

What to build?

ROS_SOURCES specifies a (space-separated) list of inputs. Usually these should be .repos or .rosinstall files suitable for vcs import. However, an URL pointing to a release .tar.gz or a github repository is also supported. In the latter case, use the following scheme: github:organization/repo#version. Other URLs are interpreted as inputs for vcs import.

A subset of packages can be selected via colcon package selection options specified in COLCON_PKG_SELECTION. For example:

• To rebuild a specific package and all its downstream dependencies:
  --packages-above-and-dependencies <pkg ...>
• To rebuild a single package (only use if ABI/API hasn't changed):
  --packages-select <pkg ...>

Where to start building from?

Building a complete ROS distro from scratch takes a lot of time, often more than the 6 hours allowed by default GHA runners. For this reason, it is possible to continue a build either from a previous build (downloading an existing debs artifact) or from an existing repository. For the former, set the input DOWNLOAD_DEBS=true, for the latter add the deploy repository to EXTRA_DEB_SOURCES. Note that this variable supports shell variable expansion, i.e. you can use $DEB_DISTRO and/or $ROS_DISTRO to generically specify the deploy repository across different builds. In both cases, set the variable/input SKIP_EXISTING=true to actually skip building already existing packages.

In the interactive.yaml workflow, a combination of these options is offered:

• Tick option Continue from previous build to draw on https://ros.packages.techfak.net (or untick to trigger a clean build from sratch).
• Tick option Skip already built packages? to skip existing packages (or untick to enforce a rebuild).
• Simply rerun the workflow to automatically continue building a previous build that was cancelled due to a timeout.

Reusable workflows build.yaml + deploy.yaml

These workflows are intended for reuse by an external repository to build a custom list of ROS packages. The build workflow stores created .debs as an artifact that can be manually downloaded or automatically uploaded to a repository server (via the deploy workflow) subsequently.

A simple usage example looks like this:

jobs:
  build:
    uses: ubi-agni/ros-builder-action/.github/workflows/build.yaml@main
    with:
      ROS_SOURCES: '*.repos'

  deploy:
    needs: build
    if: always()
    uses: ubi-agni/ros-builder-action/.github/workflows/deploy.yaml@main
    with:
      DEPLOY_URL: ${{ vars.DEPLOY_URL || 'self' }}
      TOKEN: ${{ secrets.GITHUB_TOKEN }}                  # used for own repo
      SSH_PRIVATE_KEY: ${{ secrets.DEPLOY_PRIVATE_KEY }}  # used for other repo

More complex usage examples can be found in interactive.yaml or splitted.yaml.

Environment variables

The build process is controlled by several environment variables. Usually, those environment variables are inititialized from inputs, repository variables, or the given default value - in that order.

variable	type	default	semantics
ROS_DISTRO	string	one	ROS distribution codename to compile for
DEB_DISTRO	string	jammy	The Debian/Ubuntu distribution codename to compile for.
ROS_SOURCES	string	*.repos	ROS sources to compile
COLCON_PKG_SELECTION	string		colcon package selectio argument(s)
	Workflow control
SKIP_EXISTING	boolean	false	Skip (re)building packages already existing in the repository
DOWNLOAD_DEBS	boolean	false	Continue building from previous debs artifact?
BUILD_TIMEOUT	number	340	Cancel build after this time, before github will do (minutes)
CONTINUE_ON_ERROR	boolean	false	Continue building even if some packages already failed
	Package source options
EXTRA_DEB_SOURCES	string		Extra debian sources to use in host and sbuild chroot
INSTALL_GPG_KEYS	string		code to run for installing GPG keys (for use with EXTRA_DEB_SOURCES)
EXTRA_ROSDEP_SOURCES	string		path to a rosdep-compatible yaml file specifying custom dependency mappings
	Package source options
EXTRA_SBUILD_CONFIG	string		lines to add to ~/.sbuildrc
EXTRA_SBUILD_OPTS	string		options to pass to sbuild on commandline
DEB_BUILD_OPTIONS	string	nocheck	options used debian/rules
	Deployment variables
DEBS_PATH	string	~/debs	path to store generated .debs in
DEPLOY_URL	string		repository URL for deployment
DEPLOY_FILES	string	deb	list of file types to deploy: deb, ddeb, dsc, changes
DEPLOY_FILE_SIZE_LIMIT	string		size limit for files (100M on github)
TOKEN	secret	secrets.GITHUB_TOKEN	github token for pushing to own repo
SSH_PRIVATE_KEY	secret		ssh private key for pushing to an external repo
	Debugging options
VERBOSE	string		verboseness for all (value true) or selected steps (bloom sbuild apt ccache)
TRACE	boolean	false	Enable function tracing
DEBUG_BASH	boolean	false	Enable bash debugging (set -x) and tracing

Where to deploy?

The built packages can be deployed to a flat Debian repository, for example on a github branch. The easiest way to do so is to specify DEPLOY_URL=self, causing deployment to the active github repository into the branch $DEB_DISTRO-$ROS-DISTRO. However, you can also specify another github repo.

ros-builder-action's Issues

Further improvements

Speed

• Use eatmydata?
• Use layered chroot: base (ro) - populated (rw) - sbuild (volatile)
  This will allow to enable populated as default and fallback to base on failure automatically.
  For example, gazebo requiring ogre 9.0 conflicts with rviz using ogre 12.x
• Skip a package if its dependencies are not available
• Replace vcs --shallow with partial clone

TODO

• Check and fix security using eval: Only expand vars, but don't allow subshell execution?
  https://www.baeldung.com/linux/bash-safe-use-eval
• Fix ici_filter usage
  • Output of ici_apt_install is not filtered?!
  • Use ici_quiet for bloom-generate and sbuild? Optionally, create full output again.
• Replace deprecated apt-key.
  https://askubuntu.com/questions/1286545/what-commands-exactly-should-replace-the-deprecated-apt-key
  https://wiki.debian.org/SecureApt#How_to_find_and_add_a_key

Features

• Implement CONTENT_MODE=newer: only keep newest package files
• Allow selection of (additional) files to deploy: DEPLOY_FILES=".dsc *.ddeb changes"
• Maintain the database of actually build sources (with commit sha) in a sources.repos file on the deploy repository.
  • Optionally? use that version when rebuilding downstream packages in the repo
  • python helper to add, query, etc that database.
• Merge remote.yaml and local.yaml into ros-$ROS_DISTRO.yaml?
• Create docker image for use in CI builds (e.g. MoveIt, MTC)
• Deploy to reprepro
  • Setup a TF-hosted webserver for reprepro
    • Generate keys for https and to sign .debs: https://letsencrypt.org/de/getting-started
• Support rosdistro package index for ROS_SOURCES
  This should allow building existing ROS distros on custom debian distros

Ideas for further improvement

Speed

• Use unpacked chroot instead of tarball in .cache/sbuild. Extracting the tarball for each package build is costly.
  https://manpages.debian.org/testing/schroot/schroot.conf.5.de.html
  https://manpages.debian.org/testing/mmdebstrap/mmdebstrap.1.en.html
  https://manpages.ubuntu.com/manpages/jammy/en/man1/sbuild.1.html
  https://wiki.ubuntu.com/SimpleSbuild
  https://wiki.debian.org/sbuild#Using_unshare_with_mmdebstrap_.28no_root_needed.29
• Don't build packages within sandboxes, but keep previously installed packages in chroot?
  This will hide dependency issues but might be unproblematic for a mere API rebuild of the whole distro.
• GHA sometimes fails due to disk errors.
  Enable resuming from where we left. Don't rebuild a package, if its .deb already exists.
• Continuation of workflow
  • Reuse existing debs artifact. Skip all packages already built.
  • Install signal trap in generic.sh to output an error message on failure, cancellation, or timeout.
  • Having continuation in place, we can get rid off the splitted workflow. Just need to re-run.
• Use colcon package selection features, e.g. --packages-above-and-dependencies, to select a subset of packages to rebuild after an update. Provide an update workflow for this.
  • Ensure rebuilding downstream packages.
  • Optionally disable downstream rebuilding if the user knows that ABI will not break.
• Cache the chroot environment for reuse
  We cannot use the environment settings as hash for cache key.

Features

• Use repository variables
  • This allows to quickly change the config of a workflow w/o inputs or commits
  • Make inputs optional and retrieve defaults from those (repo) environment variables
  • We cannot easily pass a callers environment to the callee:

    To reuse variables in multiple workflows, set them at the organization, repository, or environment levels and reference them using the vars context.
    Any environment variables set in an env context defined at the workflow level in the caller workflow are not propagated to the called workflow.
    For more information see "Variables" and "Contexts."

TODO

• Restore ici_step semantics: steps call unique functions and allow for a BEFORE_* and AFTER_* hook.
• Make deploy a reusable workflow

Fix ici_filter

ici_filter should filter stdout for a given string. However, on error, the full output should be shown.
While the original implementation of industrial_ci realizes the former, the latter fails.

My implementation (8c72e0f) works locally but fails on GHA (showing everything all the time, not filtering at all). For this reason I reverted to the original ici implementation (02ff659) for now. Need to further debug...

Maybe GHA redirects stdout + stderr of the main process?

Ubuntu 24.04 Noble Numbat

Is there a chance 24.04 gets supported at https://ros.packages.techfak.net/?

teleop_twist_keyboard package missing

Any chance the teleop_twist_keyboard package (https://github.com/ros-teleop/teleop_twist_keyboard.git) can be added to the build list?

Cloning it manually into a workspace and building works, it’s really a minor inconvenience not to have it available as an installable package. I think it’s a fairly important base functionality package, it would be nice to have it.

rosbash can't be installed

It seems like that dependencies for rosbash (python3-catkin-pkg) are broken on 22.04.