hnez / forrest

Run GitHub self-hosted runners in qemu VMs

Forrest - A GitHub Action Runner Runner

                                    ┏━━━━━━━━━━━━━━━┓
                                    ┃      Run      ┃
                                    ┃    Forrest    ┃
                                    ┃      Run      ┃
                                    ┗━━━┯━━━━━━━┯━━━┛

Now you wouldn’t believe me if I told you, but I could run like the wind blows.
From that day on, if I was goin’ somewhere, I was runnin’!

Forrest takes a single host computer and uses it to run multiple virtual machines with GitHub action runners in them.

The Gist

The virtual machine disk images used are ephermal and removed after a run completes, but may optionally be persisted to enable pre-generation of images or to speed up builds.

The following diagram illustrates the possible evolution of an image:

src    machine    run
 ╻
 ┣━━━━━━━━━━━━━━━━━┓
 ┃                 ┠─╴Start of a build job for the main branch of the repository.
 ┃                 ┃  The disk image is copied using a reflink copy to create a
 ┃                 ┃  fork of the image that the job can read and write to without
 ┃                 ┃  affecting the src image.
 ┃                 ┃
 ┃                 ┠─╴A virtual machine is started that uses the forked image
 ┃                 ┃  as disk image.
 ┋                 ┋
 ┃                 ┠─╴The job succeeds and provides the correct PERSISTENCE_TOKEN
 ┃                 ┃  for this repository from its GitHub secrets.
 ┃        ┏━━━━━━━━┛
 ┃        ┠──────────╴The disk image left behind by the job becomes the new
 ┃        ┃           base image for this machine.
 ┃        ┃
 ┃        ┣━━━━━━━━┓
 ┃        ┃        ┠─╴Start of a build job for a pull request.
 ┃        ┃        ┃  The job can use everything the previous run left behind to
 ┃        ┃        ┃  speed up the build process.
 ┋        ┋        ┋
 ┃        ┃        ┠─╴The job succeeds but can not provide the PERSISTENCE_TOKEN,
 ┃        ┃        ┃  because secrets are not available to runs on pull requests.
 ┃        ┃        ┃
 ┃        ┃        ┞─╴The disk image for this job is removed.
 ┃        ┃
 ┋        ┋

The different stages an image can be in are:

• src - A base image, these can for example be provided by a Linux Distribution. It is also possible to use a machine image from another machine as a base image, but we will get to that later in the documentation.
• machine - A base image for a run of a specific machine type. We will get to what a machine type is later in the documentation as well, but for now it just means that later runs will use this image as a base instead of src.
• run - The ephermal virtual machine disk image used in a run. The image may be persisted at the end of a run, but could also just be thrown away.

Documentation

The documentation is split into multiple files:

1. Acquiring operating system images suitable for use with Forrest:

   • Debian
   • Arch Linux

2. Registering a GitHub App for Forrest

3. Writing a Forrest Config File

4. Configuring nginx as Reverse Proxy

5. Writing Workflow Jobs using Forrest

6. Debugging Machines

---

                                    ┏━━━━━━━━━━━━━━┓
                                    ┃     Stop     ┃
                                    ┃    Forrest   ┃
                                    ┃     Stop     ┃
                                    ┗━━━┯━━━━━━┯━━━┛

forrest's Issues

Add GitLab support

At the surface the GitLab runner and the GitHub runner look quite different, but there may be a way to add GitLab support to Forrest without too many hacks and without sacrificing features like Docker support and without having to run the runner client outside of the VMs (if these two details were not relevant to us we could build something using the custom executor).

The GitLab runner polls for jobs to run by sending POST requests to the (undocumented) /api/v4/jobs/request GitLab API endpoint:

$ gitlab-runner run-single --url "http://localhost:8080" --token "glrt-GNma-AAAAAAAAAAAAAAA" --executor "shell"
…

$ nc -l 8080
POST /api/v4/jobs/request HTTP/1.1
Host: localhost:8080
User-Agent: gitlab-runner development version (HEAD; go1.22.6; linux/arm)
Content-Length: 847
Accept: application/json
Content-Type: application/json
Private-Token: glrt-GNma-AAAAAAAAAAAAAAA
Accept-Encoding: gzip

{"info":{"name":"gitlab-runner","version":"development version","revision":"HEAD","platform":"linux","architecture":"arm","executor":"shell","shell":"bash","features":{"variables":true,"image":false,"services":false,"artifacts":true,"cache":true,"fallback_cache_keys":true,"shared":true,"upload_multiple_artifacts":true,"upload_raw_artifacts":true,"session":true,"terminal":true,"refspecs":true,"masking":true,"proxy":false,"raw_variables":true,"artifacts_exclude":true,"multi_build_steps":true,"trace_reset":true,"trace_checksum":true,"trace_size":true,"vault_secrets":true,"cancelable":true,"return_exit_code":true,"service_variables":false,"service_multiple_aliases":false,"image_executor_opts":false,"service_executor_opts":false,"cancel_gracefully":true},"config":{"gpus":""}},"token":"glrt-GNma-AAAAAAAAAAAAAAA","system_id":"s_f7b6304c4e01"}

When no job is pending the API will respond with 204 No Content, but if a job is pending it will assign the job to the runner and will send back a json-formatted job definition. The definition also contains a job-token that will be used for all further communication.

This means we can:

1. Pretend to be a GitLab runner and poll the API endpoint mentioned above (when we have the capacity to run another job of the configured type).
2. Spawn a VM if a job is available.
3. Provide a HTTP server that will respond to POST requests to /api/v4/jobs/request with the job definition in 1). It will also have to proxy some other API requests to the actual GitLab instance, since job status updates are sent that way (git clone etc. are however not. The correct URLs are included in the job definition).
   This proxy can run either inside the VM or as part of the Forrest process.

I've stumbled upon this approach thanks to the collabora/gitlab-runner-rs project, which allows building custom GitLab runners like the collabora/lava-gitlab-runner. Most of the features there are however out of scope for us, since we actually want to use the official GitLab runner inside of the VMs.

Nested templates do not work in the config

When writing a config file one may be tempted to use nested templates like these, where one snippet builds on top of another:

machine_snippets:
  cfg-template: &cfg-template
    setup_template:
      path: /etc/forrest/templates/generic
      parameters:
        RUNNER_VERSION: "2.318.0"
        RUNNER_HASH: "28ed88e4cedf0fc93201a901e392a70463dbd0213f2ce9d57a4ab495027f3e2f"

  os-debian: &os-debian
    << : *cfg-template
    base_image: /srv/forrest/images/debian-12-generic-amd64.raw

  debian-small: &debian-small
    << : *os-debian
    cpus: 4
    disk: 16G
    ram: 4G

This does however not work due to faulty handling of nested merges in the yaml crate used.
See sebastienrousseau/serde_yml#14 and sebastienrousseau/serde_yml#15 for more information.

notify systemd earlier

When you have many repositories in your config, you can reach the systemd startup timeout of 1,5 minutes. It's probably better to fetch at most the installation tokens and log success/failure for each.

support a single label

When deciding which runner to use in runs-on, using a single string would allow us to avoid using fromJSON, which currently makes the expression hard to read.

Use `HashMap::entry()`

This code snippet:

forrest/src/machines/manager.rs

Lines 140 to 143 in 6793258

	match demand.get_mut(triplet) {
	Some(0) | None => machine.kill(),
	Some(count) => *count -= 1,
	}

and others like it could benefit from using HashMap::entry() and its Entry::and_modify() method.

reduce amount of scanned old runs

It should be enough to scan the last day only.

status page

It would be nice to have a simple http status page with an overview of running jobs.

uses systemd machine slices

Perhaps this can be done with systemd-run easily. It would allow us to see the resource use per build job.