ly0 / bb-remote-execution

Tools for Buildbarn to allow remote execution of build actions

Buildbarn Remote Execution

This repository provides tools that can be used in combination with the Buildbarn storage daemon to add support for remote execution, allowing you to create a build farm that can be called into using tools such as Bazel, BuildStream and recc.

This repository provides three programs:

• bb_scheduler: A service that receives requests from bb_storage to queue build actions that need to be run.
• bb_worker: A service that requests build actions from bb_scheduler and orchestrates their execution. This includes downloading the build action's input files and uploading its output files.
• bb_runner: A service that executes the command associated with the build action.

Most setups will run a single instance of bb_scheduler and a large number of pairs of bb_worker/bb_runner processes. Older versions of Buildbarn integrated the functionality of bb_worker and bb_runner into a single process. These processes were decomposed to accomplish the following:

• To make it possible to use privilege separation. Privilege separation is used to prevent build actions from overwriting input files. This allows bb_worker to cache these files across build actions, exposing it to the build action through hardlinking.
• To make execution pluggable. bb_worker communicates with bb_runner using a simple gRPC-based protocol. One could, for example, implement a custom runner process that executes build actions using QEMU user-mode emulation.
• To work around a race condition that effectively prevents multi-threaded processes from writing executables to disk and spawning them. Through this decomposition, bb_worker writes executables to disk, while bb_runner spawns them.

This repository provides container images for each of these components. For bb_runner, it provides two container images based on Google RBE's Debian 8 and Ubuntu 16.04 images. The advantage of using these container images is that the Bazel project provides ready-to-use toolchain definitions for them.

Please refer to the Buildbarn Deployments repository for examples on how to set up these tools.