Improve build efficiency via support for pod anti-affinity about charts

[triarius]

Hi @chendo, unfortunately we've stopped active development on this project. There's a newer, fully supported way to run buildkite-agent on k8s in github.com/buildkite/agent-stack-k8s. You should be able to specify anti-affinity rules in the Buildkite Pipeline YAML for your steps through it.

You're more than welcome to contribute PRs to this repo, however.

[chendo]

@triarius I see. I've not had a chance to play with agent-stack-k8s just yet, but it doesn't seem to have anything out of the box that address the efficiency issues. If a Job is being created for every step, wouldn't it need to:

• Checkout from git from scratch
• Boot a fresh DIND instance
• Pull all the images on a fresh DIND instance (or build from scratch)
• Run the step

This seems rather inefficient. I suspect it would be doable to shoehorn in some kind of improvements, but I feel like the agent-stack should have built-in primitives to help with this stuff.

[triarius]

That's a good call out. The approach we took in agent-stack-k8s is that building container images is a service that will have to be provided by some other means, and then maintaining a build cache is the responsibility of that service. But it's up to the user on what to choose for that, we don't want to make this a primitive for Buildkite Pipeline YAML as we think different users would want to take different approaches, including ones that don't involve docker at all.