hackworthltd / nix-buildkite-plugin

A Nix plugin for Buildkite.

nix-buildkite-plugin

nix-buildkite-plugin is a Buildkite plugin that can take a Nix expression that describes a set of builds and transforms them into separate Buildkite jobs. nix-buildkite-plugin evaluates Nix to create derivations and then analyses these derivations to find a topological ordering that will ensure steps have the correct dependencies between them.

Note: this project is a fork of Circuithub's `nix-buildkite-buildkite-plugin``, and we would like to thank them for their work, without which this project wouldn't be possible!

Getting Started

jobs.nix

First, create a jobs.nix file in your repository. This file will contain a tree of all builds that you are interested in. We create this tree using nested attrsets that eventually have leaves that are derivations.

For this example, we'll start by building the nix-buildkite project. Our jobs.nix file is:

(import nix/flake-compat.nix).defaultNix.ciJobs

.buildkite/pipeline.yml

Next, add a .buildkite/pipeline.yml file with the following contents:

steps:
  - command: nix-buildkite
    label: ":nixos: :buildkite:"
    plugins:
      - hackworthltd/nix#v1.0.0:
          file: jobs.nix

Add Your Pipeline

The final step is to add your pipeline to Buildkite. See https://buildkite.com/docs/pipelines/defining-steps#getting-started for details on how to do this. Once you have a pipeline created, make sure that the only step declared in the pipeline configuration in Buildkite's UI is:

steps:
  - command: buildkite-agent pipeline upload
    label: ":pipeline:"

Post-build hooks

The plugin accepts an optional post-build-hook argument, whose value is the name (or path) of an executable that's compatible with Nix's post-build hook semantics:

steps:
  - command: nix-buildkite
    label: ":nixos: :buildkite:"
    plugins:
      - hackworthltd/nix#v1.0.0:
          file: jobs.nix
          post-build-hook: /etc/nix/upload-to-cache.sh

When specified, the plugin will run this hook after its nix-instantiate phase, and after each individual job that it creates. This option is useful when you want to take advantage of Nix's post-build hook feature (e.g., to upload the derivations created by the pipeline), but you don't want to enable a system-wide post-build hook. For example, you might only want to upload some pipelines' outputs to your binary cache, or you might want to upload different pipelines' outputs to different binary caches.

Note that in order to use this feature, you'll need to add the user that the Buildkite agent runs as to nix.trustedUsers, as only trusted users can run post-build hooks.

Sit Back and Enjoy!

That's it! Following these steps should give you a working pipeline that builds nix-buildkite.

nix-buildkite-plugin's Issues

Support for `system` -> tag mappings

We want to be able to, e.g., tag aarch64-darwin jobs with os: macos and x86_64-linux jobs with os: nixos, etc.

If I understand correctly, currently we do the following:

1. All derivations emitted by the plugin are tagged with whatever tags the origin pipeline uses, unless overridden by the BUILDKITE_PLUGIN_NIX_AGENT_TAGS env var. In either case, every emitted derivation has the same tags.
2. Buildkite will then schedule one job per derivation on an agent whose tags match those associated with the derivation.

Because of this static mapping of tags to derivations, it is incumbent upon the admin to ensure that any derivations whose architecture differs from that of the agent’s host can be built on remote builders. For this reason, the easiest thing to do is to set up an x86_64-linux agent that catches all of the nix-buildkite-plugin builds and farms out the non-x86_64-linux ones to the correct remote builder.

By tagging each derivation with its architecture-specific tag(s), Buildkite could route the job directly to a Nix builder that can build the derivation natively and locally, without needing to round-trip through an x86_64-linux host. Not only would this result in lower latency for the builds, it would also free up x86_64-linux Buildkite agents that appear to Buildkite as “busy,” but are really just waiting for the remote builder to finish.

(One downside to this system is that it would mean every Buildkite builder would need a way to push the derivations it builds to the shared binary cache. We’re already doing that in our CI setup, though.)