philips-labs / garo

Github Actions Runners Orchestrator

Github actions runners orchestrator

GitHub allows developers to run GitHub Actions workflows on your own runners. This tool allows you to deploy self hosted runners for your repositories.

Documentation

Important notes

GitHub recommends that you do NOT use self-hosted runners with public repositories, for security reasons.

Prerequisites

This project makes use of CMake to automate a few tasks in development. Therefore it is recommended to install CMake. Furthermore the Cmake tasks depend on Graphviz to generate the diagrams.

platform	install	url
Windows	choco install -y cmake	cmake-3.16.2-win64-x64.msi
MacOSX	brew install cmake	cmake-3.16.2-Darwin-x86_64.dmg
Windows	choco install -y graphviz	graphviz-2.38.msi
MacOSX	brew install graphviz	graphviz-2.42.2.tar.gz
Windows	choco install -y protoc
MacOSX	brew install protobuf

To get an overview of the available make targets simply run the following:

$ make
clean-binaries       Cleans binaries
clean-diagrams       Cleans plantuml.jar and generated diagrams
clean                Clean binaries and diagrams
compile-agent        Compile garo-agent
compile-server       Compile garo-server
compile              Compile garo-agent and garo-server
coverage-html        Show coverage in browser
coverage-out         Show coverage in cli
diagrams             Generate diagrams in SVG and PNG format
download             Fetches go.mod dependencies via go mod download
install-protoc       Installs protoc
install-tools        Installs tools from tools.go
png-diagrams         Generate diagrams in PNG format
proto-gen            Generate protocol buffer implementations
svg-diagrams         Generate diagrams in SVG format
test-cover           Run tests and coverage
test                 Run tests

Run

Github Testing implementation (To Be Removed) can be run as following.

export GARO_GH_TOKEN=MYDUMMYPERSONALGHTOKEN
go build .
./garo my-gh-organization my-repository

Server

The Server currently exposes a http server where agents can fetch a configuration. The configuration specifics and remainder of features still has to be implemented.

$ bin/garo-server
2020-02-27T14:45:03.921+0100    DEBUG    server/api.go:75       Registering route  {"method": "GET", "route": "/"}
2020-02-27T14:45:03.921+0100    DEBUG    server/api.go:75       Registering route  {"method": "GET", "route": "/ping"}
2020-02-27T14:45:03.921+0100    DEBUG    server/api.go:75       Registering route  {"method": "OPTIONS", "route": "/twirp/philips.garo.garo.AgentConfigurationService/*"}
2020-02-27T14:45:03.921+0100    DEBUG    server/api.go:75       Registering route  {"method": "TRACE", "route": "/twirp/philips.garo.garo.AgentConfigurationService/*"}
2020-02-27T14:45:03.921+0100    DEBUG    server/api.go:75       Registering route  {"method": "POST", "route": "/twirp/philips.garo.garo.AgentConfigurationService/*"}
2020-02-27T14:45:03.921+0100    DEBUG    server/api.go:75       Registering route  {"method": "PUT", "route": "/twirp/philips.garo.garo.AgentConfigurationService/*"}
2020-02-27T14:45:03.921+0100    DEBUG    server/api.go:75       Registering route  {"method": "GET", "route": "/twirp/philips.garo.garo.AgentConfigurationService/*"}
2020-02-27T14:45:03.921+0100    DEBUG    server/api.go:75       Registering route  {"method": "CONNECT", "route": "/twirp/philips.garo.garo.AgentConfigurationService/*"}
2020-02-27T14:45:03.921+0100    DEBUG    server/api.go:75       Registering route  {"method": "HEAD", "route": "/twirp/philips.garo.garo.AgentConfigurationService/*"}
2020-02-27T14:45:03.921+0100    DEBUG    server/api.go:75       Registering route  {"method": "DELETE", "route": "/twirp/philips.garo.garo.AgentConfigurationService/*"}
2020-02-27T14:45:03.921+0100    DEBUG    server/api.go:75       Registering route  {"method": "PATCH", "route": "/twirp/philips.garo.garo.AgentConfigurationService/*"}
2020-02-27T14:45:03.921+0100    INFO     garo-server/root.go:63 Server is ready to handle requests  {"addr": ":8080"}
^C
2020-02-27T14:45:15.097+0100    INFO     server/server.go:81    Server is shutting down  {"reason": "interrupt"}
2020-02-27T14:45:15.097+0100    INFO     server/server.go:90    Server stopped

Agent

The Agent currently connects to the server to fetch a configuration. The configuration specifics and the remainder of features still has to be implemented.

bin/garo-agent

Test

export GARO_GH_TOKEN=MYDUMMYPERSONALGHTOKEN
make test

To run the tests from VSCode you will have to copy the .env.example and fill out your personal Github Token.

cp .env.example .env

To view the code coverage you can make use of make coverage-out or make coverage-html.

Update diagrams

To update the diagrams you simply edit the plantuml files followed by running the according make target.

make clean-diagrams diagrams

NOTE: due to make caching ensure to run the clean-diagrams target to regenerate existing png and svg files.

garo's Issues

Garo-agent should fetch failed jobs from garo-server

Garo-agent should be able to fetch failed jobs from garo-server

for every repository it should deploy a runner.
for every failed workflow it should rerun the particular build.

Optionally the runner should be removed after a grace period for cost savings.

Implement webhook endpoint for garo-server

Garo server needs a /webhook endpoint which can be consumed by github.com

The webhook endpoint should be protected by a configurable secret to prevent abuse of this API.

IP whitelisting can be arranged via a LB / security group

The check_run webhook should enable us to check if a build failed due to no available runner. If this occurs Garo-server needs to track this so the agents can retrieve for which repositories a runner has to be deployed and as well which runs need to be retriggered.

Implement the actions/workflows endpoints

Add a file github/actions_workflows.go to implement these Apis.

https://developer.github.com/v3/actions/workflows/

Implement actions/workflow_jobs endpoints

https://developer.github.com/v3/actions/workflow_jobs/

Official implementation progres.
google/go-github#1399 (comment)

Implement runner provisioning for a given repository

By updating the protobuf message additional configuration parameters can be added.

By running make gen-proto the RPC client will be regenerated.

In agent/agent.go the logic can be implemented to iterate over the given set of repositories to implement the provisioning of a runner for a given repository.

This can be done via the os/exec package which can be used to invoke terraform cli or via https://terratest.gruntwork.io/.

The typical flow to provision a runner is:

• Create Runner registration token
• Provision the runner using terraform
• Keep track on the provisioned runner to be able to cleanup later (if runner doesn't clean itself) or if grace period expires to manage costs https://developer.github.com/v3/actions/self_hosted_runners/#remove-a-self-hosted-runner

Background info:
Best is if the Runner will auto unregister itself using https://developer.github.com/v3/actions/self_hosted_runners/#example-using-remove-token
This can be done by trapping some signals
https://github.com/joakimhellum/docker-actions-runner/blob/master/ubuntu-18.04/entrypoint.sh

Implement actions/workflow_runs endpoints

https://developer.github.com/v3/actions/workflow_runs/

Official implementation progres.
google/go-github#1399 (comment)