Distributed cache about infra

[kim]

This is actually by design: build caches tend to be huge, and constantly transferring them via cloud storage is orders of magnitude slower than the local disk. On a physical machine, cache volumes are created from ZFS snapshots, so what you see as a distinct agent is benefitting from the local cache. Additionally, we can provision agents warm by just sending the ZFS snapshots. I’m not terribly excited about adding more than the bare minimum of branching logic for “trusted” vs “untrusted” builds. Neither am I about granting more privileges than absolutely necessary. Also note that buildkite steps are completely independent. Their idea is to rather have a script executing things in order than to split your pipeline into many steps. There are cases imaginable where you do want to share artifacts between steps, but I think that is more relevant to release pipelines. There is a way to download the artifacts via the agent binary, but that requires the binary to be in your Docker image. Also it needs the agent token, which is a security concern. I’ll need to look more into their API, perhaps we could have more granular permissions there.

[geigerzaehler]

Yes, it is nice that the local cache is so fast and remote caches are going to be quite slow. However, the three-in-four chance of a cache miss might lead to slower build times on average compared to remote caching. I’ll gather some build time data from the registry to compare these options.

There also may be other options to increase the cache hit likelihood. Maybe we can define affinity for projects to agents. Or maybe we can asynchronously distribute the master build cache.

[kim]

I’ll gather some build time data

Yes please. I think that if your caches are so volatile that you end up with a lot of work on a cache miss -- even when the cache state starts from master -- something is fishy about your caching strategy.

Maybe we can define affinity for projects to agents.

Not sure. Affinity of branch builds to where they were scheduled previously might be nice, but that assumes mostly-idle agents. I don't think we can influence this anyway -- the scheduling decision is made on buildkite's servers.

Or maybe we can asynchronously distribute the master build cache.

The master build cache is agent-scoped because there is no guarantee that builds are strictly sequential. We could lift this restriction if we'd adopt a bors-style model.

[kim]

[the buildkite-agent binary] needs the agent token, which is a security concern

Actually, that seems less of an issue than I thought. See here for what an agent can do. One can mess with the current pipeline (which is fine), start a nested agent (which is silly), or run a job (which is also silly). Wdyt?

[geigerzaehler]

[the buildkite-agent binary] needs the agent token, which is a security concern

Actually, that seems less of an issue than I thought. See here for what an agent can do. One can mess with the current pipeline (which is fine), start a nested agent (which is silly), or run a job (which is also silly). Wdyt?

Are you suggesting using buildkite artifact for caching? This is something we could investigate.

[geigerzaehler]

This has been solved for the master cache volume. Because of snapshotting this also reduces the issue for branch builds.