twitter / scoot

Scoot is a distributed task runner, supporting both a proprietary API and Bazel's Remote Execution.

Scoot

Scoot is a distributed task runner. The core concept of Scoot is the Snapshot, which represents an immutable filesystem state. Scoot allows for execution of commands against input Snapshots to create new output Snapshots.

Scoot Components

Scheduler

The Scheduler receives and distributes jobs to workers, and maintains state. It is also responsible for serving the Cloud Scoot API for clients.

Worker

The Worker (workerserver) receives information about jobs and runs them, and is responsible for all Snapshot-related functionality.

Client APIs

• Cloud Scoot API
• Worker API

Jobs and Tasks

Scoot work is broken down into independent jobs, which can consist of one or more tasks, which are executed sequentially.

Go Version

Scoot is intended to work with go modules. If this repository is on your GOPATH, please set GO111MODULE=on when working with go tools in this repository, and preferably use the Go distribution pinned in the .github/workflows/ci.yml file.

Example

Setup a scheduler and worker nodes locally:

go run ./setup-cloud-scoot/main.go --strategy local.local

Run a series of randomly generated tests against the local scheduler and workers:

go run ./scheduler/client/scootcl/main.go smoke_test

Scoot Integration Tests

Scoot has a few tests that exercise varying levels of common usages and workflows.

Smoketest

Invokes a scoot client directly to run jobs against a local cluster and waits for the scheduled jobs to complete.

(./scheduler/client/cli/smoketest.go)

Recoverytest

Invokes a scoot client directly to run jobs against a local cluster, kills the cluster, attempts to spin up a new one, and waits for the originally scheduled jobs to complete.

(./binaries/recoverytest/main.go)

Integration

Invokes a scoot and scoot-snapshot-db client via CLI to run a job against a local cluster and waits for the job to complete

(./binaries/scoot-integration/main.go)

Scoot Thrift Code

Thrift Prerequisites Install the Thrift tool and golang thrift repository locally using the following section.

Generating thrift files See documentation in thrift definition files for specific generation instructions, or make thrift.

Development Installation Instructions

Install 3rd party tools:

Thrift (version == 0.9.3)

macOS:

brew install [email protected]

Thrift for go:

go modules will take care of this

Install/Access Scoot Executables and libraries

Install binaries to $GOPATH/bin:

make install

scoot's Issues

Bazel client setup and scaling instructions?

I can not find any info how to configure bazel client to use Scoot. Also, I don’t see any info how to scale the service, for example, using Kubernetes.

Bazel API output directories have inconsistent prefixes

I ran this operation which expects output directory dir:

DEBUG 2018-02-28T13:49:24Z: process_execution::remote: Executing remotely request: action {command_digest {hash: "a4124e7e29e8f625473652fca900c6b8a89960316784ff31b54f784860278904" size_bytes: 71} input_root_digest {hash: "6f9a33e23a007c58c1e83f29bcf52979c797aba200cec0058695ff72d181e6ed" size_bytes: 85} output_directories: "dir"} (command: arguments: "/bin/bash" arguments: "-c" arguments: "/usr/bin/mkdir -p dir ; /usr/bin/echo 'foo' > dir/file")

This returned me:

DEBUG 2018-02-28T13:49:29Z: process_execution::remote: Got (nested) execute response: result {output_directories {path: "/dir" tree_digest {hash: "ebbc315a499c5aa2dc8f972a535486e0097a871279b275ec6cf3326ff1cb9fcc" size_bytes: 77}} stdout_digest {hash: "644a7d0ab7451b1fecc58a742b45fe720d5d6a9ea92c29d0e9e63f80f591e0ee" size_bytes: 2219} stderr_digest {hash: "644a7d0ab7451b1fecc58a742b45fe720d5d6a9ea92c29d0e9e63f80f591e0ee" size_bytes: 2219}} status {}

the output_directories entry is a directory containing dir:

$ ../fs/fs_util/target/debug/fs_util --local-store-path=${HOME}/tmp/store --server-address=localhost:50001 directory cat-proto --output-format=text ebbc315a499c5aa2dc8f972a535486e0097a871279b275ec6cf3326ff1cb9fcc 77
directories {name: "dir" digest {hash: "a8eeda526245a7d32d88e2f6cdf790a96990c2a77b575c12009e11975c62e35a" size_bytes: 78}}

I believe that this is one level too much nesting. If path=/dir then the root hash should be a directory containing file, not containing dir/file.

Fail to run

One line summary of the issue here.

Expected behavior

Run success.

Actual behavior

Encounter error as:

# github.com/twitter/scoot/common/dialer                                                                                                   
common/dialer/dialer.go:40:12: assignment mismatch: 1 variable but d.transportFactory.GetTransport returns 2 values
# github.com/twitter/scoot/scootapi/gen-go/scoot
scootapi/gen-go/scoot/cloudscoot.go:96:20: not enough arguments in call to oprot.Flush
        have ()                   
        want (context.Context)                                                                                                             
scootapi/gen-go/scoot/cloudscoot.go:120:15: assignment mismatch: 2 variables but error8.Read returns 1 values
scootapi/gen-go/scoot/cloudscoot.go:180:20: not enough arguments in call to oprot.Flush
        have ()                   
        want (context.Context)                                       
scootapi/gen-go/scoot/cloudscoot.go:204:16: assignment mismatch: 2 variables but error10.Read returns 1 values
scootapi/gen-go/scoot/cloudscoot.go:264:20: not enough arguments in call to oprot.Flush
        have ()                                                      
        want (context.Context)                                       
scootapi/gen-go/scoot/cloudscoot.go:288:16: assignment mismatch: 2 variables but error12.Read returns 1 values
scootapi/gen-go/scoot/cloudscoot.go:348:20: not enough arguments in call to oprot.Flush
        have ()                                                      
        want (context.Context)                                       
scootapi/gen-go/scoot/cloudscoot.go:372:16: assignment mismatch: 2 variables but error14.Read returns 1 values
scootapi/gen-go/scoot/cloudscoot.go:431:20: not enough arguments in call to oprot.Flush
        have ()                   
        want (context.Context)
scootapi/gen-go/scoot/cloudscoot.go:455:16: assignment mismatch: 2 variables but error16.Read returns 1 values
scootapi/gen-go/scoot/cloudscoot.go:455:16: too many errors  

Steps to reproduce the behavior

go run ./binaries/setup-cloud-scoot/main.go --strategy local.local

CLA check on PRs looks broken, status is not updating

I signed the cla already, but the check on my PR #484 still says "license/cla Expected — Waiting for status to be reported".

explicitly report missing dependencies in the environment, e.g. `nc`

I observed failure

INFO[0000] repo.Repository.Run, [rev-parse --show-toplevel]  sourceLine="repo/repo.go:63"
INFO[0000] repo.Repository.Run complete. Err: <nil>      sourceLine="repo/repo.go:72"
INFO[0000] git.NewRepository: /tmp/scoot-tmp-024806936/gitdb-repo-963465879, top: /tmp/scoot-tmp-024806936/gitdb-repo-963465879  sourceLine="repo/repo.go:132"
INFO[0010] Killing 1 cmds                                sourceLine="setup/cmds.go:75"
INFO[0010] SIGINT: 32304 /root/go/bin/apiserver          sourceLine="setup/cmds.go:91"
INFO[0010] Cmd 32304 (/root/go/bin/apiserver) finished   sourceLine="setup/cmds.go:149"
INFO[0010] All completed                                 sourceLine="setup/cmds.go:100"
FATA[0010] port 11100 is not up after 5s                 sourceLine="setup-cloud-scoot/main.go:53"
exit status 1

Looking at the code, it became apparent I am missing nc. I installed that and it fixed the problem.

scoot/scheduler/setup/ports.go

Lines 24 to 30 in 9b02ac8

	// Use exec.Command because we don't worry about these getting orphaned,
	// and don't want to fill up our Cmds's list of running cmds
	cmd := exec.Command("nc", "-z", "localhost", strconv.Itoa(port))
	if err := cmd.Run(); err == nil {
	log.Infof("Port %v active", port)
	return nil
	}

# yum install -y nc

In the future, there should be better error message. Possibly a check for all binaries up front? Possibly avoid using external nc, since go has reasonable facilities for network programing in the std lib.

Sandboxing

Does Scoot have any support for sandboxing builds?

Missing Bazel 'capabilities' endpoint

Bazel implemention is missing Capabilities endpoint as far as I can tell. Please correct me if I'm wrong! :)

Expected behavior

Service implements this endpoint.

Actual behavior

When I attempt to run a bazel build, I get Failed to query remote execution capabilities: UNIMPLEMENTED: unknown service build.bazel.remote.execution.v2.Capabilities

setup-cloud-scoot fails to run and error is not clearly given

INFO[0000] Killing 1 cmds                                sourceLine="setup/cmds.go:75"
INFO[0005] Still waiting; killing all                    sourceLine="setup/cmds.go:102"
FATA[0005] fork/exec : no such file or directory         sourceLine="setup-cloud-scoot/main.go:53"
exit status 1

Which file does not exist?

[root@host-10-0-133-169 ~]# tar -C /usr/local -xzf  go1.14.linux-amd64.tar.gz 
e[root@host-10-0-133-169 ~]# export PATH+=:/usr/local/go/bin
[root@host-10-0-133-169 ~]# cd scoot/
[root@host-10-0-133-169 scoot]# go run ./binaries/setup-cloud-scoot/main.go --strategy local.local
go: downloading github.com/apache/thrift v0.0.0-20160616201512-0e9fed1e12ed
go: downloading github.com/sirupsen/logrus v1.4.2
go: downloading github.com/golang/protobuf v1.3.2
go: downloading golang.org/x/sys v0.0.0-20190422165155-953cdadca894
go: downloading google.golang.org/grpc v1.14.0
go: downloading google.golang.org/genproto v0.0.0-20180817151627-c66870c02cf8
go: downloading golang.org/x/net v0.0.0-20190628185345-da137c7871d7
go: downloading golang.org/x/time v0.0.0-20190308202827-9d24e82272b4
go: downloading github.com/davecgh/go-spew v1.1.1
go: downloading github.com/cenkalti/backoff v2.2.1+incompatible
go: downloading github.com/twitter/groupcache v0.0.0-20191021194929-8215cbf3be0f
go: downloading github.com/sethgrid/pester v0.0.0-20190127155807-68a33a018ad0
go: downloading github.com/nu7hatch/gouuid v0.0.0-20131221200532-179d4d0c4d8d
go: downloading golang.org/x/text v0.3.0
go: downloading github.com/rcrowley/go-metrics v0.0.0-20160613154715-cfa5a85e9f0a
INFO[0000] Starting up a Local ApiServer                 sourceLine="setup/api.go:49"
INFO[0000] RunCmd:  [-http_addr localhost:11100 -grpc_addr localhost:12100 -log_level info]  sourceLine="setup/cmds.go:122"
INFO[0000] Starting[ -http_addr localhost:11100 -grpc_addr localhost:12100 -log_level info]  sourceLine="setup/cmds.go:142"
INFO[0000] Killing 1 cmds                                sourceLine="setup/cmds.go:75"
INFO[0005] Still waiting; killing all                    sourceLine="setup/cmds.go:102"
FATA[0005] fork/exec : no such file or directory         sourceLine="setup-cloud-scoot/main.go:53"
exit status 1
[root@host-10-0-133-169 scoot]# go run ./binaries/setup-cloud-scoot/main.go --strategy local.local --log_level debug
INFO[0000] Starting up a Local ApiServer                 sourceLine="setup/api.go:49"
INFO[0000] RunCmd:  [-http_addr localhost:11100 -grpc_addr localhost:12100 -log_level debug]  sourceLine="setup/cmds.go:122"
INFO[0000] Starting[ -http_addr localhost:11100 -grpc_addr localhost:12100 -log_level debug]  sourceLine="setup/cmds.go:142"
INFO[0000] Killing 1 cmds                                sourceLine="setup/cmds.go:75"
INFO[0005] Still waiting; killing all                    sourceLine="setup/cmds.go:102"
FATA[0005] fork/exec : no such file or directory         sourceLine="setup-cloud-scoot/main.go:53"
exit status 1
[root@host-10-0-133-169 scoot]# go run ./binaries/setup-cloud-scoot/main.go --strategy local.local -log_level debug
INFO[0000] Starting up a Local ApiServer                 sourceLine="setup/api.go:49"
INFO[0000] RunCmd:  [-http_addr localhost:11100 -grpc_addr localhost:12100 -log_level debug]  sourceLine="setup/cmds.go:122"
INFO[0000] Starting[ -http_addr localhost:11100 -grpc_addr localhost:12100 -log_level debug]  sourceLine="setup/cmds.go:142"
INFO[0000] Killing 1 cmds                                sourceLine="setup/cmds.go:75"
INFO[0005] Still waiting; killing all                    sourceLine="setup/cmds.go:102"
FATA[0005] fork/exec : no such file or directory         sourceLine="setup-cloud-scoot/main.go:53"
exit status 1
[root@host-10-0-133-169 scoot]# go run ./binaries/setup-cloud-scoot/main.go --help
Usage of /tmp/go-build935509414/b001/exe/main:
  -api_strategy string
        api strategy to setup server w/bundlestore. (default "local")
  -apiservers int
        number of apiservers to use (default 3)
  -log_level string
        Log everything at this level and above (error|info|debug) (default "info")
  -repo_dir string
        backing repo to use as a basis for handling bundles
  -strategy string
        scheduler/worker strategy to setup (default "local.memory")
  -workers int
        number of workers to use (default 5)
exit status 2

Are the command line options given with double dashes, or single dash. Instructions in readme show double dashes, here in --help output I see single dash.

Hardcode support for empty/zero-length bazel directory

Using an empty directory is handy for testing when the input environment does not matter, but this can break when using a Store backend that does not handle zero-length blobs. Either add hardcoded support for this case or give an error on attempts to upload zero-length blobs to the CAS.

Bazel API should not set stdout/stderr digest for PreconditionFailed case

If input files are missing, I'm seeing these responses:

DEBUG] Got operation response: name: "d89e165c-6413-4fb4-4da1-4e95106f6328" metadata {type_url: "type.googleapis.com/google.devtools.remoteexecution.v1test.ExecuteOperationMetadata" value: "\010\004\022\000"} done: true response {type_url: "type.googleapis.com/google.devtools.remoteexecution.v1test.ExecuteResponse" value: "\n\0042\000B\000\032\222\001\010\t\032\215\001\n2type.googleapis.com/google.rpc.PreconditionFailure\022W\nU\n\007MISSING\022Jblobs/92329943dc749fab9e5dc006b2048bf78d28cfb0c9d939f9d5a56523e1d32aff/244"}
DEBUG] Got (nested) execute response: result {stdout_digest {} stderr_digest {}} status {code: 9 details {type_url: "type.googleapis.com/google.rpc.PreconditionFailure" value: "\nU\n\007MISSING\022Jblobs/92329943dc749fab9e5dc006b2048bf78d28cfb0c9d939f9d5a56523e1d32aff/244"}}

stdout_digest and stderr_digest appear to be being explicitly set to invalid values (with no digest or length), rather than not set.

fs_util errors on macOS

Something seems to be up with the macOS distribution of fs_util at 1.5.0.dev3+569f14c2. I cannot pass make tests locally on master:

go test -tags="integration property_test" github.com/twitter/scoot/snapshot/bazel ... --- FAIL: TestSaveEmptyDir (0.00s) fs_util_test.go:45: fork/exec /Users/dgassaway/workspace/bin/fs_util: exec format error ...

Also, running fs_util-1.5.0.dev3%2B569f14c2 --help yields no output (although an exit code of 0), while on a linux machine, the normal help dialog appears. Other commands on the mac binary return 0 but seem not to do anything.

Bazel gRPC API includes scoot header in stdout/stderr

We should not add these prefixes to the stdout/stderr we report through the gRPC API:

$ cat /var/folders/1j/jzw90nc51nn_358mmhnwc_fh0000gn/T/scoot-tmp-988923338/common-bundles/blob-cf722969a46641061fbf9809045a0e10b15d8a9ba573a4e62296c1d96a669749.blob
###########################################
###########################################


Date: 2018-02-12 22:13:38.899871 +0000 GMT
Out: http://localhost:10102/output//2-stdout?file=file://tw-mbp-dwagnerhall/var/folders/1j/jzw90nc51nn_358mmhnwc_fh0000gn/T/scoot-tmp-445635530/output/2-stdout109126150    Err: http://localhost:10102/output//2-stderr?file=file://tw-mbp-dwagnerhall/var/folders/1j/jzw90nc51nn_358mmhnwc_fh0000gn/T/scoot-tmp-445635530/output/2-stderr209918381    OutErr: http://localhost:10102/output//2-stdlog?file=file://tw-mbp-dwagnerhall/var/folders/1j/jzw90nc51nn_358mmhnwc_fh0000gn/T/scoot-tmp-445635530/output/2-stdlog637639464    Cmd:
runner.Command -- SnapshotID: bz-e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855-0 # Argv: ["/bin/bash" "-c" "exit 42"] # Timeout: 30m0s # JobID: 878bbb7e-8e3a-4eae-5923-d438072b7f67 # TaskID: Bazel_ExecuteRequest_40e8c5bcece27be634101baca92efc1d654921688a0c1528deed9abbf6fac94e_1518473618 # Tag: 878bbb7e-8e3a-4eae-5923-d438072b7f67  ExecuteRequest=action:<command_digest:<hash:"8a1eb3dc8694551ee161e6f616de205f7c2d5b52c8f5b93d2745d67c3b750e6a" size_bytes:24 > input_root_digest:<hash:"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" > platform:<> timeout:<> > 
<nil>

###########################################
###########################################



SCOOT_CMD_LOG

Bazel gRPC API does not report missing files

If the input directory is not found, the Operation response I get from the scheduler is:

name: "312a8f54-91a1-4ec9-598f-f435b2cdfd0a" metadata {type_url: "type.googleapis.com/google.devtools.remoteexecution.v1test.ExecuteOperationMetadata" value: "\010\004\022\000"} done: true response {type_url: "type.googleapis.com/google.devtools.remoteexecution.v1test.ExecuteResponse" value: "\n\000\032\002\010\001"}

which nests the ExecuteResponse: result {} status {code: 1}

Instead, it should populate a FailedPrecondition status as per the suggestion in the proto file: https://github.com/googleapis/googleapis/blob/eb2730ad8d88eb8e70ac647d1e91a3b41e4ca81c/google/devtools/remoteexecution/v1test/remote_execution.proto#L99

Otherwise clients can't know what to do to rectify the situation.

Ideally the command_digest missing would have the same effect.