buildkite / charts

Buildkite Helm Charts repository

⚠️ Deprecation Notice

We've since iterated on other solutions for using Buildkite with k8s. Please check out github.com/buildkite/agent-stack-k8s for a supported solution. We'll continue to accept PRs for this repository, but won't be doing any active maintenance.

Buildkite Helm Charts Repository

The official Buildkite Helm Charts repository.

Getting Started

Install Helm

Get the latest Helm release.

Add Buildkite Helm chart repository:

helm repo add buildkite https://buildkite.github.io/charts/
helm repo update

Install chart

To install the Agent chart with the release name bk-agent:

helm install --name bk-agent --namespace buildkite buildkite/agent --set agent.token="BUILDKITE_AGENT_TOKEN"

Check Agent chart readme for more customisation options.

** You’ve now got Buildkite Agents running on your Kubernetes cluster! 🎉 **

Contributing to Buildkite Charts

Fork the repo, make changes and test it by installing the chart to see it is working. :)

On success make a pull request (PR).

Upon successful review, someone will give the PR a LGTM in the review thread.

Thanks ❤️

• A massive thanks to Rimas Mocevicius for authoring Buildkite Agent chart.

Copyright

Copyright (c) 2020 Buildkite Pty Ltd. See LICENSE for details.

charts's Issues

Improve build efficiency via support for pod anti-affinity

Is this a request for help?: No

---

FEATURE REQUEST?:

Support for pod anti-affinity rules.

We deploy multiple instances of this chart into our CI cluster, however occasionally we run into issues where one node is loaded more than others, due to an uneven distribution of pods. This can sometimes be rectified by redeploying the Deployment, but ideally we would be able to specify anti-affinity rules. We would likely preferentially distribute based on hostname where possible in our setup.

Ideally, we would have it automatically reschedule pods if agents aren't being used and there is capacity, although this probably requires a custom scheduler which would be more complicated.

Another aspect of this problem is the Buildkite scheduler would ideally consider the node an agent is running on, but that's outside the scope of the Helm chart.

We are currently on Kubernetes 1.20.x.

health checks

sometimes the docker socket can get unmounted for reasons; adding a health check can be beneficial:

          livenessProbe:
            initialDelaySeconds: 15
            timeoutSeconds: 1
            exec:
              command:
                - sudo
                - docker
                - ps

See Also

• http://kubernetes.io/docs/user-guide/liveness/

Allow use of other secrets providers

Is this a request for help?: No

---

Is this a BUG REPORT or FEATURE REQUEST? (choose one): Feature Request

Currently, the chart insists that the .agent.token is provided, and it creates a Secret. We would like to use a CRD to provide our own secret, such as external-secrets.io.

The Chart should be updated to optionally reference an existing secret, and not depend on the in-house secret.

Allow adding extra rules to the clusterrole

Is this a request for help?: Yes

---

Version of Helm and Kubernetes: N/A

Which chart: buildkite

What happened: Not enough or different clusterrole rules required

What you expected to happen: I have a use case that requires different clusterrole rules. I not that you can disable the creation of any RBAC in this chart. Is your intention that if a user needs something different they should create the role and binding separate to this chart? Or would you support breaking out the definition of rules to a helm variable so that users can define their own?

How to reproduce it (as minimally and precisely as possible): N/A

Anything else we need to know: N/A

Deprecated?

Hey

Getting the following when i install the helm package?

NAME:   bk-agent
LAST DEPLOYED: Sat May 11 19:31:49 2019
NAMESPACE: buildkite
STATUS: DEPLOYED

RESOURCES:
==> v1/Secret
NAME                TYPE    DATA  AGE
bk-agent-buildkite  Opaque  2     0s

==> v1/ServiceAccount
NAME                SECRETS  AGE
bk-agent-buildkite  1        0s

==> v1beta1/Deployment
NAME                DESIRED  CURRENT  UP-TO-DATE  AVAILABLE  AGE
bk-agent-buildkite  4        0        0           0          0s


NOTES:
#### THIS CHART IS DEPRECATED! ####

Is this deprecated? Should I be installing/running this in a different way?

Make mounting docker socket optional

Is this a request for help?: no

---

Is this a BUG REPORT or FEATURE REQUEST? (choose one): FEATURE REQUEST

Mounting in the docker socket from the host is quite a large security risk to accept, especially if deploying this chart in a critical cluster like a production cluster. I'd like to make this feature configurable, with a default of enabled for backwards compatibility (if desired) so that the risk can be mitigated for users who wish.
An example use case is running a fleet of Buildkite agents on vanilla hosts (eg using the EC2 CloudFormation template) which have access to docker for building and pushing to a registry (and can also keep warm caches). Then using a fleet of agents deployed directly into a Kubernetes cluster which use images from the previous fleet to deploy apps. The agents running inside the cluster don't need access to docker since that is done in a previous step and hence would benefit from closing off the risk of mounting the hosts docker socket.

Version of Helm and Kubernetes: n/a

Which chart: agent

What happened: mounts host docker socket by default

What you expected to happen: optionally mount host docker socket

How to reproduce it (as minimally and precisely as possible): apply the chart as-is

Anything else we need to know: I'm willing to make a PR if you would accept such a change 😄

Agent version needs to be bumped to 3.25

Chart.yaml is defaulted at 3.22.1.

Regression when using dind.enabled for volumes mounts on GKE

Is this a request for help?:
Yes

Is this a BUG REPORT or FEATURE REQUEST? (choose one):
BUG REPORT, likely reopen/duplicate of #66

Version of Helm and Kubernetes:

15:48 $ helm version
version.BuildInfo{Version:"v3.2.4", GitCommit:"0ad800ef43d3b826f31a5ad8dfbb4fe05d143688", GitTreeState:"clean", GoVersion:"go1.14.4"}

Cluster version: 1.15.12-gke.2

Which chart:

15:51 $ helm history buildkite-agent
REVISION        UPDATED                         STATUS          CHART           APP VERSION     DESCRIPTION
1               Tue Aug 18 14:39:08 2020        deployed        agent-0.4.4     3.22.1          Install complete

What happened:

It seems like since #70 got merged that any volume mounted with docker run from /var/buildkite/lib is only exposing empty directories. This is similar problem as the one that was flagged in #66 but it seems like it regressed after #70 got merged.

Manually updating the deployment to set mountPath back to /var/buildkite/lib instead of /var/buildkite/shared seems to fix the issue even-though I'm not sure why/how things work.

This can be tested when used alongside https://github.com/buildkite-plugins/docker-buildkite-plugin. The mounted checked out volume from /var/buildkite/lib is empty. Inspecting it on host properly shows the underlying folders and the checked out code, but not from within the container.

What you expected to happen:

To be able to interact with checked out code in steps defined using https://github.com/buildkite-plugins/docker-buildkite-plugin.

How to reproduce it (as minimally and precisely as possible):

helm install buildkite-agent buildkite/agent --set agent.token="..." --set privateSshKey="..." --set dind.enabled=true

trigger a manual build with something like:

steps:
  - command:
      - "ls -la /workdir"
    plugins:
      - docker#v3.5.0:
          image: "alpine:latest"
          mount-buildkite-agent: false
          volumes:
            - /var/run/docker.sock:/var/run/docker.sock

Notice that the workdir folder is empty.

Anything else we need to know:

It seems important that a build ran at least once so the /var/buildkite/lib folder exists in the container.
I tried to reproduce the volume mount the plugin does directly into the agent with:

kubectl exec -it buildkite-agent-agent-... -c agent -- bash

then

bash-4.4# docker run -v /var/buildkite:/workdir alpine ls -la /workdir
total 12
drwxrwxrwx    3 root     root          4096 Aug 19 00:07 .
drwxr-xr-x    1 root     root          4096 Aug 19 00:14 ..
drwxr-xr-x    3 root     root          4096 Aug 19 00:07 builds

and I can see the folder tree, but they all appear empty even-though they are populated on host!

how is the GCP service key used?

This is a question.

I noticed that the service key provided as a value to the chart is put into /etc/service_key.

From here, is it automatically used bykubectl, helm and gcloud?

Support Pod annotations

Is this a request for help?: No

---

Is this a BUG REPORT or FEATURE REQUEST? (choose one): Feature request

Add support to add extra annotations on the pod. The specific use case I am after is to support kiam so the pod that is deployed can access some AWS resources. Eg for pulling config or secrets to perform a deploy of an app.

I think this could be added just like the extraEnv value.

This would be used instead of a Buildkite agent hook that exported raw IAM credentials.

Ability to use PVC for caching

We are currently using https://github.com/gencer/cache-buildkite-plugin, which is a great way to cache dependencies between steps. One room for improvement would be to have a shared read/write PVC which we could use to cache the deps easily on different agents.

I'm not sure how exactly this would work with this chart, but figured I would pitch the idea. Does this seem like a reasonable use case?

what extra does this do over just using the deployment provided in the official tutorial?

See https://buildkite.com/docs/agent/v3/gcloud#running-the-agent-on-google-container-engine

What does this provide over that?

make gh-pages branch to work as Helm v2 repo

Buildkite should provide Docker-in-Docker functionalities

This ticket is related to the following issues happening when using Buildkite on Kubernetes:

• buildkite-plugins/docker-buildkite-plugin#61
• buildkite-plugins/docker-buildkite-plugin#140
• buildkite/agent#729

The following configuration works successfully on Google Kubernetes Engine:

apiVersion: v1
kind: Pod
metadata:
  name: buildkite-agent 
  namespace: buildkite
spec:
  containers:
    - name: agent
      image: buildkite/agent:3.17.0
      imagePullPolicy: Always
      env: 
      - name: BUILDKITE_AGENT_TOKEN
        valueFrom:
          secretKeyRef:
            key: agent-token
            name: buildkite-agent-agent
      - name: BUILDKITE_AGENT_META_DATA
        value: role=agent
      - name: BUILDKITE_BUILD_PATH
        value: /var/buildkite/builds
      - name: DOCKER_HOST
        value: tcp://localhost:2375
      volumeMounts:
        - name: shared-volume
          mountPath: /var/buildkite
    - name: dind
      image: docker:19.03-dind
      securityContext:
        privileged: true
      env:
        - name: DOCKER_TLS_CERTDIR
          value: ""
      volumeMounts:
        - name: docker-graph-storage
          mountPath: /var/lib/docker
        - name: shared-volume
          mountPath: /var/buildkite
  volumes:
    - name: docker-graph-storage
      emptyDir: {}
    - name: shared-volume
      emptyDir: {}

With the following pipeline:

steps:
  - commands:
      - "docker build ."
    plugins:
      - docker#v3.3.0:
          image: "docker:latest"
          always-pull: true
          volumes:
            - "/var/run/docker.sock:/var/run/docker.sock"

I spent around 16 hours trying to figure out how to work around GKE security limitations and how to make Buildkite work with Docker-out-of-Docker, but I ended up reaching the same conclusion I did for Drone CI: Docker-in-Docker is more suitable for the Kubernetes architecture especially for Google Kubernetes Engine architecture.

The following article also explains really well why Docker-in-Docker (dind) is considered more functional in a Kubernetes environment:
https://applatix.com/case-docker-docker-kubernetes-part-2/

The main highlight from the article is the following:

The Pod will create a container that will run outside of the Pod. By running the container using DooD, you lose out on the following for the spawned container:

• Pod Networking - Cannot access the container using Pod IP.
• Pod Lifecycle - On Pod termination, this container will keep running especially if the container was started with -d flag.
• Pod Cleanup - Graph storage will not be cleanup after pod terminates.
• Scheduling and Resource Utilization - Cpu and Memory requested by Pod, will only be for the Pod and not the container spawned from the Pod. Also, limits on CPU and memory set for the Pod will not be inherited by the spawned container.

Aside from that, as far as I know the following tools are all using dind instead of dood when deployed to a k8s environment:

• Gitlab CI
• Drone CI
• Jenkins CI
• GoCD CI

It also looks like AWS is no longer working with DooD without additional configuration (i.e. Buildkite won't work out-of-the-box on ECS without the --enable-docker-bridge flag):
https://support.cloudbees.com/hc/en-us/articles/360028151031-Docker-outside-of-Docker-no-longer-works-in-EKS

This is also the only configuration I am aware of that fully runs on GKE without breaking down when the docker container is used.

Handle deployments of multiple agents better

Is this a request for help?:

No.

---

Is this a BUG REPORT or FEATURE REQUEST? (choose one):

FEATURE REQUEST

Provide a nicer way to deploy multiple agents with different configuration into the same namespace where the app label and selector can be set to unique names but without producing awkwardly named pods while doing so.
I'm raising this feature request as I plan to raise a pull request and want something to reference as to why I'm creating it.

Version of Helm and Kubernetes:

Not Applicable. But Helm v3.11.2 and Kubernetes 1.26.3.

Which chart:

buildkite/agent

What happened:

I wish to be able to deploy multiple agents with a slightly different configuration into the same namespace in our cluster.
For example I use helmfile and I define deployments like so:

releases:
  - name: buildkite-agent
    chart: buildkite/agent
    namespace: buildkite
    values:
      - helm/vars/values.yaml.gotmpl

  - name: buildkite-large-agent
    chart: buildkite/agent
    namespace: buildkite
    values:
      - helm/vars/values.yaml.gotmpl
      - helm/vars/large.yaml.gotmpl

Where helm/vars/values.yaml.gotmpl contains the values for both of the agents and helm/vars/large.yaml.gotmpl overrides some values (like the queue name in agent.tags) for the buildkite-large-agent deployment only.

If I do that with the version of the chart as it is now, then both Deployment resources end up with a label of app: agent and the selector will match against that.
The two deployments will have no way of distinguishing which pods belong to which deployment.
In the case of us using KEDA to scale these agents it can't because it produces HPAs that end up with errors like:

  Warning  AmbiguousSelector             4m39s (x26 over 23m)  horizontal-pod-autoscaler  pods by selector app=agent are controlled by multiple HPAs: [buildkite/keda-hpa-buildkite-large-agent buildkite/keda-hpa-buildkite-agent]

I can set nameOverride to make this app label unique, however it combined with the .Release.Name variable leads to some awkwardly named agents.
E.g. if I set nameOverride: large for the 2nd agent I end up with a Deployment name like buildkite-large-agent-large for example.

What you expected to happen:

I'd like a way to have more control over the app label and selector that gets generated while still having control over the names of the deployments.
I plan to submit a PR with the following change:

diff --git a/stable/agent/templates/_helpers.tpl b/stable/agent/templates/_helpers.tpl
index 559791e..724c5b2 100644
--- a/stable/agent/templates/_helpers.tpl
+++ b/stable/agent/templates/_helpers.tpl
@@ -11,9 +11,17 @@ Create a default fully qualified app name.
 We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).
 */}}
 {{- define "fullname" -}}
+{{- if .Values.fullnameOverride -}}
+{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" -}}
+{{- else -}}
 {{- $name := default .Chart.Name .Values.nameOverride -}}
+{{- if contains $name .Release.Name -}}
+{{- .Release.Name | trunc 63 | trimSuffix "-" -}}
+{{- else -}}
 {{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" -}}
 {{- end -}}
+{{- end -}}
+{{- end -}}
 
 {{/*
 Set the Deployment API version based on the version of Kubernetes the chart is being deployed into.

With the code change above I can set nameOverride: buildkite-large-agent in helm/vars/large.yaml.gotmpl to match the release name and it'll produce resources called buildkite-large-agent with the label and selector set to the same.
E.g. part of the output from Helm

buildkite, buildkite-large-agent, Deployment (apps) has been added:
+ # Source: agent/templates/deployment.yaml
+ apiVersion: apps/v1
+ kind: Deployment
+ metadata:
+   name: buildkite-large-agent
+   labels:
+     app: buildkite-large-agent
+     chart: agent-0.6.2
+     release: buildkite-large-agent
+     heritage: Helm
+ spec:
+   replicas: 1
+   selector:
+     matchLabels:
+       app: buildkite-large-agent
+   template:

In the code above I've also catered for a fullnameOverride variable to completely override the names of resources independently of .Release.Name and .nameOverride if you like since that is a common pattern I see in Helm charts.

How to reproduce it (as minimally and precisely as possible):

I've shown a way to produce it with helmfile.

Anything else we need to know:

I plan to raise a PR to address this.

Allow not creating service accounts (serviceAccount.create=false)

Currently the chart forces you to allow it to create the service account, which is something we have a general rule not to allow vendored charts to do (our practice is to pre-create the namespaces, service accounts, and cluster role bindings using a common Helm chart).

The Buildkite agent Helm chart should allow disabling creation of the service account, and allowing specification of the pre-created one.

Buildkite agent deployment yaml breaks upon setting certain values in values.yaml

Is this a request for help?:
no

Is this a BUG REPORT or FEATURE REQUEST? (choose one):
Bug REport

Version of Helm and Kubernetes:
v3.8.1 (helm) and v0.25.18 (k8s)

Which chart:
agent

What happened:
i set both dockerconfigjson and privateSshKey and the chart would not install. Upon investigation the deployment uses the same checksum annotation for multiple downstream secrets. If more than one secret is set, more than one checksum is set with the same name and this is not allowed by the api server. As a result the helm chart errors with marshalling errors. see

charts/stable/agent/templates/deployment.yaml

Line 22 in 293ec74

checksum/binarystore: {{ include (print $.Template.BasePath "/secret.yaml") . | sha256sum }}

What you expected to happen:
I expected it to install.

How to reproduce it (as minimally and precisely as possible):
set more than one key that creates a secret and the chart will not install

Anything else we need to know:
I created a PR for this and tested it.

deploy pipeline is not working

The deploy pipeline doesn't get triggered, release was tagged.
cc @toolmantim

How to set registryCreds.dockerConfig

Hey

How do i set registryCreds.dockerConfig? Doing "(cat docker.config)" results in

Error: failed parsing --set data: key "}" has no value (cannot end with ,)

Cannot perform docker run with volume mounts inside step

---

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT

Version of Helm and Kubernetes:
Helm 3.2.1
Kubernetes 1.15.11 (AWS EKS)

Which chart:
CHART: agent-0.3.14
APP VERSION: 3.17.0

What happened:
When running a step that includes a Docker run with mounted volumes, the volumes fail to be mounted.

When running the below example, the job output shows this:

trap 'kill -- $$' INT TERM QUIT; touch foobar
--
  | docker run \
  | -w /code \
  | -v `pwd`:/code \
  | --entrypoint=/bin/sh \
  | gardendev/garden-aws:v0.11.14 -c "ls -al"
  |  
  |  
  | total 0
  | drwxr-xr-x    2 root     root             6 May 21 22:26 .
  | drwxr-xr-x    1 root     root            18 May 22 01:44 ..

What you expected to happen:

I expect there to 3 entries in the ls output, but foobar is missing.

How to reproduce it (as minimally and precisely as possible):

steps:
  - label: "Perform a docker run with mounted volume"
    command: |
      touch foobar
      docker run \
        -w /code \
        -v `pwd`:/code \
        --entrypoint=/bin/sh \
        gardendev/garden-aws:v0.11.14 -c "ls -al"

Anything else we need to know:

We are using a very simple configuration for the chart, setting only agent.meta, privateSshKey and agent.token.

We also tried enabling DinD (via setting dind.enabled to true), but that did not resolve the issue.

what extra does this do over just using the deployment provided in the official tutorial?

See https://buildkite.com/docs/agent/v3/gcloud#running-the-agent-on-google-container-engine

What does this provide over that?

Remove unnecessary RBAC permissions from chart

When enabling rbac the chart grants a lot of permissions to the buildkite agent: https://github.com/buildkite/charts/blob/master/stable/agent/templates/clusterrole.yaml

What are those permissions used for by the agent? I didn't see any mention of RBAC in the official docs and I'm wondering if this is even necessary? Does the buildkite agent ever call kubernetes APIs?

Integration with TravisCI is not working

@toolmantim can you take a look to it, I do not have permissions to change integration settings

expose gcloud && kubectl

This might be something for the buildkite agent image, but currently to create a full builder that can perform redeploys as a first step in my pipeline I need to do:

steps:
  - name: ":ubuntu: setup"
    command: >
      export CLOUD_SDK_REPO="cloud-sdk-$(lsb_release -c -s)";
      echo "deb http://packages.cloud.google.com/apt $CLOUD_SDK_REPO main" \
        | sudo tee /etc/apt/sources.list.d/google-cloud-sdk.list;
      curl https://packages.cloud.google.com/apt/doc/apt-key.gpg \
        | sudo apt-key add -;
      sudo apt-get update;
      sudo apt-get -y install google-cloud-sdk;
      sed -i -e 's/true/false/' \
        /usr/lib/google-cloud-sdk/lib/googlecloudsdk/core/config.json;
      gcloud components install -q kubectl;
      PATH="/usr/lib/google-cloud-sdk/bin:$PATH";
      kubectl get pods;

And in the template, to inherit config from the compute node I had to mount the gcloud config:

spec:
  replicas: 2
  template:
    metadata:
      labels:
        app: buildkite-agent-builder-private-git
        heritage: helm
    spec:
      containers:
        - name: buildkite-agent
          image: buildkite/agent:ubuntu-docker
          volumeMounts:
            - name: var-lib-docker
              mountPath: /var/lib/docker
            - name: home-anon-config-gcloud
              mountPath: /home/anon/.config/gcloud
      volumes:
        - name: var-lib-docker
          emptyDir: {}
        - name: home-anon-config-gcloud
          hostPath:
            path: /home/anon/.config/gcloud

I think it would make sense to add this to the buildkite images / charts in some way so people who want to perform deploys don't need to add this as boilerplate. Perhaps we need a new type that includes this? - idk. Thanks! ✨

Question: Automatically horizontally scale when building

Is this a request for help?: Yes

---

I'm trying to make my agents horizontally scale when all agents are busy. Lets say capping it at 10 workers and looks somewhat like this:

No pipelines triggered (agents => 1)
Incoming Work triggers pipelines
Agent_1 gets assigned and starts building
Automatically create Agent_2
Agent_2 gets assigned some work
Automatically create Agent_3
Agent_3 waits for job
Agent_1 finishes (gets killed)
Agent_2 finished (gets killed)
Remaining agents (1): Agent_3

Add better secret support

Is this a request for help?: No

---

Is this a BUG REPORT or FEATURE REQUEST? (choose one): Feature Request

Add support for different secrets like S3 credentials for non AWS platforms and document ways to retrieve them in pipelines (normal steps and docker plugin steps).

If I understand the current code correctly, only a handful of custom settings are currently stored as k8s secrets.
Other secrets can only be added as ENV vars.
Ideally you could add all kind of different vars as secrets to k8s.

Additionally, it would be helpful for beginners to document, how to utilize the k8s secret store in pipelines.

create Helm v2 charts