Scoot is infrastructure to make developer tools smaller, simpler, and more distributed. The core concept of Scoot is the Snapshot, which represents an immutable filesystem state. Scoot allows for execution of commands against input Snapshots to create new output Snapshots.
The Scheduler receives and distrubutes jobs to workers, and maintains state. It is also responsible for serving the Cloud Scoot API for clients.
The Worker (workerserver) receives information about jobs and runs them, and is responsible for all Snapshot-related functionality.
The local daemon serves a Protobuf interface and can either send Scoot jobs to a remote scheduler or to embedded local workers.
- Cloud Scoot API
- Worker API
- Daemon API
Scoot work is broken down into independent jobs, which can consist of one or more tasks, which are executed sequentially.
Setup a scheduler and worker nodes locally:
go run ./binaries/setup-cloud-scoot/main.go --strategy local.local
Run a series of randomly generated tests against the local scheduler and workers:
go run ./binaries/scootapi/main.go run_smoke_test
Scoot has a few tests that exercise varying levels of common usages and workflows.
Invokes a scootapi client directly to run jobs against a local cluster and waits for the scheduled jobs to complete.
(./scootapi/client/smoke_test_cmd.go)
Invokes a scootapi client directly to run jobs against a local cluster, kills the cluster, attempts to spin up a new one, and waits for the originally scheduled jobs to complete.
(./binaries/recoverytest/main.go)
Invokes a scootapi and scoot-snapshot-db client via CLI to run a job against a local cluster and waits for the job to complete
(./tests/integration_test.go)
(open source scoot code is in workspace/github.com/twitter/scoot)
Generating thrift files (scootapi used as an example)
- To Generate files run from scoot's scootapi directory:
thrift --gen go:package_prefix=github.com/twitter/scoot/scootapi/gen-go/,package=scoot,thrift_import=github.com/apache/thrift/lib/go/thrift scoot.thrift
Generating go protobuf files (for local Scoot Daemon) cd to scoot's daemon/protocol
protoc -I . daemon.proto --go_out=plugins=grpc:.
Generating python client files (for client library accessing local Scoot Daemon) cd to scoot's daemon/protocol
python -m grpc.tools.protoc -I. --python_out=./python/scoot --grpc_python_out=./python/scoot daemon.proto
###Thrift (version >= 0.9.3) macOS:
brew install thrift
go get github.com/apache/thrift/lib/go/thrift
- If necessary, remove any conflicting or older versions of Protobuf:
brew uninstall protobuf
pip install grpcio grpcio-tools
(If grpcio* is already installed you should probably update it by including '--upgrade' to the above pip command)
pip install docopt==0.6.2
*cd to scoot directory (workspace/github.com/twitter/scoot) *run: go install ./binaries/... ** the binaries will be installed in workspace/bin
### Python client library
Can be found at workspace/github.com/twitter/scoot/daemon/protocol/python/scoot/client_lib.py