traderepublic / cilicon

🛠️ Self-Hosted ephemeral macOS CI on Apple Silicon

Self-Hosted macOS CI on Apple Silicon

About • Getting Started • Ideas for the Future • Join Us

💥 What's new in 2.0?

We're excited to announce a new major update to Cilicon! Here's a summary of what's new:

• While Cilicon 1.0 relied on a user-defined Login Item script in the VM, its new version now includes an SSH client and directly executes commands on the VM.
• Cilicon has partially adopted the tart image format and can automatically convert 1.0 images to it.
• The integrated OCI client can download pre-built CI images that have been created with/for tart. We recommend their macos-sonoma-xcode images.

Migrating from 1.0

• The config file schema has changed slightly. In most cases renaming the vmBundlePath property to source should suffice.
• When Cilicon detects a 1.0 image it will offer you to automatically convert it to the new format for you.
• When converting an image from the 1.0 format, you must enable SSH and set the respective credentials in the config (or use the default admin:admin).

🔁 About Cilicon

Cilicon is a macOS App that leverages Apple's Virtualization Framework to create, provision and run ephemeral CI VMs with near-native performance. It currently supports Github Actions, Buildkite Agent, GitLab Runner and arbitrary scripts. Depending on your setup, should be able to get up and running with your self-hosted CI in minutes 🚀

Cilicon operates in a very simple cycle described below:

The Cilicon Cycle

Running a sample job via GitHub Actions (2x playback)

🚀 Getting Started

To get started, download the latest release here.

✨ Choosing a Source

Cilicon uses the tart container format and comes with an integrated OCI client to fetch images from the internet.

It's recommended to use publicly hosted images, however if you need to create or customize your own image, you may use tart.

⚠️ Important

• When choosing an OCI hosted image, make sure to prepend the oci:// scheme to the url. Cilicon will otherwise assume a local filesystem path.
• Don't use the latest tag when choosing an image version. Instead pick the specific version of Xcode you would like to have installed (e.g. 14.3).
• Images downloaded via OCI will reside in the ~/.tart folder which should be cleared of unused images periodically.
• Images with newer versions of macOS may be published with the same version of Xcode installed. In case you want to upgrade, you may need to manually delete the outdated image and start Cilicon again.

⚙️ Configuration

Cilicon expects a cilicon.yml file to be present in the Host OS's home directory. For more information on all available settings see Config.swift.

GitHub Actions

To use the GitHub Actions provisioner you will need to create and install a new GitHub App with Self-hosted runners Read & Write permissions on the organization level and download the private key file to be referenced in the configuration file.

source: oci://ghcr.io/cirruslabs/macos-sonoma-xcode:15.3
provisioner:
  type: github
  config:
    appId: <APP_ID>
    organization: <ORGANIZATION_SLUG>
    privateKeyPath: ~/github.pem

GitLab Runner

To use the GitLab Runner provisioner you will need to create a runner with an authentication token.

Minimal example:

source: oci://ghcr.io/cirruslabs/macos-sonoma-xcode:15.3
provisioner:
  type: gitlab
  config:
    gitlabURL: <GITLAB_INSTANCE_URL>
    runnerToken: <RUNNER_TOKEN>

Full configuration:

source: oci://ghcr.io/cirruslabs/macos-sonoma-xcode:15.3
provisioner:
  type: gitlab
  config:
    gitlabURL: <GITLAB_INSTANCE_URL>
    runnerToken: <RUNNER_TOKEN>
    executor: <EXECUTOR> # defaults to 'shell'
    maxNumberOfBuilds: <MAX_BUILDS> # defaults to '1'
    downloadLatest: <DOWNLOAD_LATEST> # defaults to 'true'
    downloadURL: <DOWNLOAD_URL> # defaults to GitLab official S3 bucket
    configToml: > # Advanced config as custom config.toml file to be appended to the basic config and copied to the runner.
      <CONFIG_TOML>

Buildkite Agent

To use the Buildkite Agent provisioner, simply set your agent token in the provisioner config.

source: oci://ghcr.io/cirruslabs/macos-sonoma-xcode:15.3
provisioner:
  type: buildkite
  config:
    agentToken: <AGENT_TOKEN>

Script

If you want to run a script (e.g. to start a runner that's not natively supported), you may use the script provisioner.

source: oci://ghcr.io/cirruslabs/macos-sonoma-xcode:15.3
provisioner:
  type: script
  config:
     run: |
     	echo "Hello World"
        sleep 10

🔨 Setting Up the Host OS

It is recommended to use Cilicon on a macOS device fully dedicated to the task, ideally one that is freshly restored.

If you don't want to host your machine locally, OakHost provides great value dedicated Macs and are kindly offering 10% off the first two months for new customers with the code CILICON10 (Disclaimer: We do not receive any form of compensation from sharing this code).

• Transfer Cilicon.app, cilicon.yml as well as any other files referenced by your config (e.g. Local image, GitHub private key etc.) to your Host OS.
• Add Cilicon.app as a launch item
• Set up Automatic Login
• Disable automatic software updates
• Disable any concept of screen lock, battery saving etc.

🔮 Ideas for the Future

Support for more Provisioners

We use GitHub Actions for our iOS builds at Trade Republic but would love to see Cilicon being used for other CI services as well. Implementing support for more services should be easy by building on top of the Provisioner protocol.

Running 2 VMs in parallel

Xcode builds often don't use all of the compute resources available. Therefore running 2 VMs im parallel (more are not possible due to a limitation of the Virtualization framework) would be a welcome addition.

Monitoring

A logging or monitoring concept would greatly improve identifying and troubleshooting any potential issues and provide the ability to notify the team in real time.

👩‍💻 Join Us!

At Trade Republic, we are on a mission to democratize wealth. We set up millions of Europeans for wealth with fast, easy, and free access to capital markets. With over one million customers we are one of the largest savings platforms in Europe, with users holding over €6 billion on our platform. Join us to build the FinTech of the future.

Disclaimer: Trade Republic is not affiliated with Cirrus Labs or their tart product

cilicon's Issues

Can't run a Cilicon via Gitlab runner (Citadel.SSHClient.CommandFailed error 1.)

Hi! I've tried to start workin with cilicon and get an error that I can't figure out how to fix.
After launc Cilicon i get this error:
Configuring GitLab Runner...�
The operation couldn’t be completed. (Citadel.SSHClient.CommandFailed error 1.)

here is my cilicon.yml

source: oci://ghcr.io/cirruslabs/macos-sonoma-xcode:15.3
provisioner:
  type: gitlab
  config:
    gitlabURL: https://gitlab.com
    runnerToken: <runner_token>

Could somebody tell me what I'm doing wrong ?

Support for GitLab Runners

Unfortunately we due to the major changes in Cilicon 2.0 we had to break/remove support for GitLab runners.
Hoping that someone who uses GitLab could contribute the provisioner.
@ast3150 perhaps you could have a look?

Is there a good reason the GitLab runner script does a lot of things with `sudo`?

I notice the GitLab start.command runs a bunch of commands with sudo. This doesn't seem strictly necessary, it could just install the runner binary in ~/bin and run it from there, or something like that.

Is there a good reason why it's doing this with elevated privileges, or would you accept a patch that changes this behaviour so sudo is only needed for the shutdown at the end. (Which also has a bug in it… see #19 )

Sonoma images not working

I was trying the sonoma xcode 15.01. image ( https://github.com/cirruslabs/macos-image-templates/pkgs/container/macos-sonoma-xcode/144771897?tag=15.0.1) but it seems they have application/vnd.cirruslabs.tart.disk.v2 for the image in their name and the VMManager is specifically filtering for application/vnd.cirruslabs.tart.disk.v1.

Im not sure why this is the case.

Release to homebrew

Is there any plan to release to homebrew ?

Unclear failure with GitHub token

On launch, the VM loops:

I don't see any other log or obvious issue.

GitLab Runner add additional registration attribute in Provisoner

Hey folks, GitLab Runners will always register and run with the setting untagged jobs set to true.
It would be nice if the configuration block could contain the attribute run_untaggedin the cilicon.yml to set this value during the GitLab Runner registration process [1] against the GitLab Server API[2].

Here is a draft example how the config block could look like:

provisioner:
  type: gitlab
  config:
     name: "cilicon-runner"
     url: "https://<gitlab-url>"
     registrationToken: "<registration-token>"
     run_untagged: false
     tagList: "macos-cilicon"

[1] https://github.com/traderepublic/Cilicon/blob/main/Cilicon/Provisioner/GitLab%20Runner/GitLabService.swift#L32-L34
[2] https://docs.gitlab.com/ee/api/runners.html#register-a-new-runner

Better errors

hey! i was looking to create a github action worker. i've made it through a few of the setup steps, but the error messages from the runner are really hard to decipher. is there a version i can run that has more verbose error messages/stack traces so i can understand what's wrong with my configuration?

also, i saw the medium post that was written about the release of this, are there any other resources that walk through setting up a github app + configuration for a runner using Cilicon?

FAQ to GitHub App creation

Your guide provides following step:

To use the Github Actions provisioner you will need to create and install a new Github App with Self-hosted runners Read & Write permissions on the organization level and provide your config with the respective information.

While trying to create such an app I stumble upon 'Callback URL' field. It's required, but I can't fully understand what URL to provide.

VPN / Host Network

First of all, thanks for building Cilicon!

Is there currently a way to share the host network?
We have connected the host machines to our VPN but the Vm seems to not share the host network and therefore isn't able to reach our Gitlab instance.

Feature request: CLI interface

Would be very useful to have command line interface for tool which allows to select for example cilicon.yml filename and it's path. This makes easier to integrate tool to deployment systems, such as salt-stack.

VM Startup silently fails on Virtualization error

The setupAndRunVirtualMachine() function in VMManager silently fails if an error in the Virtualization framework occurrs.

This is because the try syntax in a Task does not require the code to actually handle the error. (see https://forums.swift.org/t/task-initializer-with-throwing-closure-swallows-error/56066 )

Cilicon/Cilicon/VMManager.swift

Lines 94 to 97 in df4f3dc

	Task { @MainActor in
	vmState = .running(virtualMachine)
	try await virtualMachine.start()
	}

My recommended solution is to run the entire method on @MainActor and just remove the Task altogether.

This can help users debug issues such as this, because important context is provided for debugging:

Self-hosted runners for non-organization repositories

I was hoping to be able to use an M1 Mini I have with Github Actions instead of turning to something like BuildKite, and I notice that although Github only supports runner groups for organizations, self-hosted runners are technically supported for personal repositories without an organization.

I tried (just to see if by some chance it would work) setting the organization slug to my Github username, but no dice. Is there any way support for personal repositories could be added for adding self-hosted runners?

VirtualMachineView not responding to keyboard input after VM restart

The VirtualMachineView stops responding to keyboard input after the first reboot. While it shouldn't be an issue, as interactions should not be required outside of editor mode, it would be nice to fix it.

Using advanced configurations for GitLab Runners (config.toml)

Thanks to #38 it's now possible to define more advanced configuration options for GitLab Runners 🎉.
However I had some difficulties to get it to work:

1. Right now #38 isn't released while the Readme is already updated which is a little bit misleading (took me quite some time to figure out, why my configuration was ignored by the runner).
2. In the Readme it's not written that you have to pass your options as a multiline-argument omitting [[runners]] because this part is already present (see below). Instead it sounds a bit like you have to pass the path to a config.toml file.
   Relevant code in GitLabRunnerProvisioner.swift:

let copyConfigTomlCommand = """
mkdir -p ~/.gitlab-runner
rm -rf ~/.gitlab-runner/config.toml
cat <<'EOF' >> ~/.gitlab-runner/config.toml
[[runners]]
  url = "\(config.gitlabURL)"
  token = "\(config.runnerToken)"
  executor = "\(config.executor)"
  limit = \(config.maxNumberOfBuilds)
\(config.configToml ?? "")
EOF
exit 1
"""

Especially 2. should be addressed imho 🙂 ...

Migrating from v1.1.0 to v2.0 does lead to NIOConnectionError

I'm trying to convert from v1.1.0 to v2.0 but I wasn't able to because the vm wouldn't start. I changed the yml file vmBundlePath to source, it tries to convert the VM to the new standard so I clicked yes, but then it is not able to connect and I get this error all the time:

I'm running this on a M2 Pro Mac mini. Ventura 13.2.1.

Crashing on startup

Hey.

I have a problem with Cilicon crashing on startup. It's probably related to a misconfiguration on my ec2-builder, but the crash makes it hard to tell what goes on.

I've attached it to this ticket, any idea where it's stopping? I've had a very similar config run fine on a local mac. This is with v2.1.0 from github releases.

crash.txt

edit I've built and uploaded cilicon to get symbols, and this is a bit more useful:

crash-sym.txt

Looks like it's crashing in the lease parser because /var/db/dhcpd_leases can't be found. On my builder I see

sudo ls /var/db/dhcpclient/leases
en4.plist       en7.plist

but not the /var/db/dhcpd_leases file.

Couldn't resolve host | Message queue listen OAuth token

macOS 13.2.1
Cilicon 1.1.0

5 of 6 machines from our CI stopped working without any changes being made by our team today.

I've switched machines to editor mode and had enough time to compare network configs of working and non-working machines. There are differences.

Working config:

Non-working config:

All non-working configs have empty Router, 255.255.0.0``Mask and IPs starting from 169

What's interesting is that if I wait for a few minutes the VM will finally get valid network settings.

I've tried switching from DHCP to manual config and that helps with network problem, but I immediately do face another issue — something related to token:

I've found this exact exception in runner sources, but I couldn't figure out what's wrong and how to fix it.

I believe that token is valid, because one of CI machines is still working fine, using exact same token.
Anyway I've generated and provided new token with no luck.

I know that I'm using an outdated version of macOS and Cilicon — I've just had no time to install the update.

So at the end I have this questions:

1. What could have happened to network settings?
2. What is the problem with token?
3. What can I do to fix issues?

Improve download logic for OCI images

Cilicon currently uses URLSession.bytes to download the OCI images.

Problems

This approach has some problems:

• Downloading files serially is rather slow and could be sped up by downloading chunks in parallel and combining the files at the end
• The memory usage is high because downloads are held in-memory
• If the download is interrupted at any moment, all chunks must be re-downloaded

Possible solutions

I've investigated some approaches how this could be handled differently. So far the most promising approach I've seen would involve switching to the AsyncHTTPClient library which offers FileDownloadDelegate. This would address the high memory load during downloads by streaming the download directly to the disc using SwiftNIO for non-blocking I/O.

Secondly, the downloads could be split into different files, called chunks. This would mean the downloader can check whether a certain chunk already exists on-disk before starting the download, which would mean that an interrupted download could be continued relatively easily by only re-downloading any chunks which are not yet completed.

Support for GIthub Actions Jitconfig API

First, let me start by saying thank you for this project! I've searched far and wide for a simple solution to this very problem, and had my first successful run using a self-hosted solution with a Mac Mini for my project

Since the newest release supports targeting the /repos endpoint and individual's projects rather than organizations, it might make sense to support the JIT config setup for Github Actions Runners. I ended up using a custom script provision using this endpoint along with ./run.sh --jitconfig {jitconfig} option.

So basically the command would look like:

jitconfig=$(curl -L \
  -X POST \
  -H "Accept: application/vnd.github+json" \
  -H "Authorization: Bearer <TOKEN with repo scope>" \
  -H "X-GitHub-Api-Version: 2022-11-28" \
  https://api.github.com/repos/OWNER/REPO/actions/runners/generate-jitconfig \
  -d '{"name":"{NAME_PARAM}-'$(uuidgen)'","runner_group_id":{RUNNER_GROUP},"labels":LABELS_ARRAY,"work_folder":{WORK_FOLDER}}' \
  | jq -r '.encoded_jit_config') && ~/actions-runner/run.sh --jitconfig $jitconfig

You already have access to most of these with the config options as they are currently. According to the docs, it also works with GitHub apps, so maybe it could even replace the current implementation (not confident on that).

I can try my hand at doing it myself, but wanted to get your opinion on its inclusion first.

Thanks again!

Skip download gitlab-runner after each restart to make it more stable and avoid throttling

After each Cilicon cycle the gitlab-runner is downloaded from the internet. When downloading from default location: https://gitlab-runner-downloads.s3.amazonaws.com/latest/binaries/gitlab-runner-darwin-arm64 one quickly become throttled which makes the startup cycle to increase drastic +30 minutes

I have two ideas to solve the issue:
A. Put the file on company owned s3 bucket.
B. Store binary file on the host-os and mount it in after the startup of the vm. This would make the default setup of the Cilicon, more stable.

Do we have any idea of how to start implementing option B?
I suppose this is the issue for all the runners and not explicit gitlab.
I can try to implement it but currently don't know where to start, unless I go with option A.

The `runnerName` in the config file will not change the generated RUNNER_NAME

In the cilicon.yml config file I specify the runner name, but that will change nothing in the generated RUNNER_NAME file.

Use dotfile for cilicon.yml config

Placing a cilicon.yml in the home directory is unconventional for macOS. A more common approach is .cilicon.yml. Or use NSUserDefaults, which reads from ~/Library/Preferences

GitLab Runner Advanced Configuration

Add support for passing an advanced config to GitLab Runner provisioner

Motivation

Some features of the GitLab Runner are only available through an advanced config. Critically, this includes external runner caches. This is a desirable feature because it can vastly improve the build times for ephemeral build runners.

Solution

Introduce a new config parameter configToml in cilicon.yml, an optional multi-line argument. Users can pass any custom runner configuration here, including a cache configuration.

Change the runner command from gitlab-runner run-single to gitlab-runner run. This is required because run-single does not take a configuration file. Consequently, the configuration parameters are passed via the config.toml file to the runner, instead of being passed as parameters to the shell command.

Cilicon runner don't restart after gitlab job is finished in Cilicon version 2.2.0

Issue:

Cilicon 2.2.0 vm don't restart when gitlab job is done as shown in GIF.
Followed setup in Readme.md

Background:

From the cilion cycle the vm is stopped and removed after a job is done.

In the instructions it is not mentioned any setting required to restart Cilion VM. From the additional swift settings it seems numberOfRunsUntilHostReboot is the most likley setting to make it restart.

Settings

Cilicon version: v2.2.0

Mac version

ProductName:		macOS
ProductVersion:		14.5
BuildVersion:		23F79

cat ~/cilicon.yml

source: oci://ghcr.io/cirruslabs/macos-sonoma-xcode:15.3
provisioner:
  type: gitlab
  config:
    gitlabURL: https://gitlab.com
    runnerToken: <replace with gitlab-ci-cd-token>
    maxNumberOfBuilds: 1
numberOfRunsUntilHostReboot: 1

Running:

Configuring GitLab Runner...�
�Successfully configured GitLab Runner�
�Downloading GitLab Runner Binary from Source�
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  3 61.5M    3 2330k    0     0  1944k      0  0:00:32  0:00:01  0:00:31 1943k
 26 61.5M   26 16.1M    0     0  7747k      0  0:00:08  0:00:02  0:00:06 7746k
 43 61.5M   43 26.5M    0     0  8614k      0  0:00:07  0:00:03  0:00:04 8613k
 60 61.5M   60 36.9M    0     0  9153k      0  0:00:06  0:00:04  0:00:02 9153k
 78 61.5M   78 48.0M    0     0  9578k      0  0:00:06  0:00:05  0:00:01 9989k
 96 61.5M   96 59.4M    0     0  9896k      0  0:00:06  0:00:06 --:--:-- 11.5M
100 61.5M  100 61.5M    0     0  9921k      0  0:00:06  0:00:06 --:--:-- 10.7M
�Downloaded GitLab Runner Binary from Source successfully�
�Starting GitLab Runner...�
Runtime platform                                  �  arch�=arm64 os�=darwin pid�=660 revision�=44feccdf version�=17.0.0
Starting multi-runner from /Users/admin/.gitlab-runner/config.toml...�  builds�=0 max_builds�=0
�WARNING: Running in user-mode.                    � 
�WARNING: Use sudo for system-mode:                � 
�WARNING: $ sudo gitlab-runner...               

.
.
.

Submitting job to coordinator...accepted, but not yet completed�  bytesize=707934 checksum=checksum code�=202 job=[job-number] job-status=running runner�=[runner-id] update-interval=1s

Submitting job to coordinator...ok.      bytesize=707934 checksum=[checksum] code=200 job=[job-number] job-status=success runner=[runner-id] update-interval=0s

Removed job from processing list.        builds=0 job=[job-number] max_builds=1 project=[gitlab-project]  repo_url�=[gitlab-repo-url]  time_in_queue_seconds=0

improve error message that doesn't help at all; dataCorrupted

I except that my cilicon.yml config was invalid somehow, but failure message doesn't help at all to identify issue;

Propose to give at least filename or even line number which is invalid. This would help for learning curve.

Runner does not automatically start up again when vm reboots on v2

I got the vm up and running now. When the runner task is done it shutdown and starts up again as expected but then it will not start the github action runner script again so it's stuck booted up but not available to run anything.

Am I missing a setting that resolves this?

Nested virtualisation on M1/M2

I don't think this is an issue with Cilicon

Unless I've misunderstood the docs, it doesn't appear to be possible to create a vm with the Apple Hypervisor that has nested virtualisation enabled.
What this means is that things like android studio booting an emulator will fail, and provide errors like

HVF error: HV_UNSUPPORTED
qemu-system-aarch64: failed to initialize HVF: Invalid argument

Which google will rightly point you at code signing qemu and some other bits, until if you're like me you've spent hours falling down reddit+stackoverflow rabbit holes before it dawns on you whats going on, and what might be needed for it to be possible

🤦

Anyway, leaving this here to pay it forward hopefully with enough words that google finds it.

Please close issue unless you can see anyway to do nested virtualisation, vmware fusion and parallels appear to have something from google, but I'm not sure what workaround they've got for it.

Now to go build a amd64 box for android e2e testing :(

Thanks!

"home" could not be copied

After running for the first time.

Tart License Violation

Hey team and @Marcocanc, it was hard to not recognize some of Tart's code in #46. For example, LayerV2Downloader seems like a copy/paste of Tart's DiskV2#pull:

https://github.com/cirruslabs/tart/blob/f4bc02d1751dd5ad447619515e7d73a9efc2d7e6/Sources/tart/OCI/Layerizer/DiskV2.swift#L39-L121

Here is a visual diff that shows similarities. All the variable names are the same, structure is the same, just some comments removed and different kind of error handling:

On behalf of Tart authors we are glad you found Tart code useful but unfortunately such usage is agains Tart's License.

Is it possible to start multiple instances of VMs at the same time?

We have a bunch of Mac Studios for CI in our office at this moment but we can use it only for run a single job at a time right now due to flaky Xcode with parallel executions.

Can Cilicon start multiple VMs at the same time, which run independent from each others? This'd help us to ultilise the Mac Studios' resources better by parallelising multiple jobs in multiple VMS