crows-nest
A supervisor tool to launch and monitor multiple SauceLabs Sauce Connect tunnel in high avaiability mode.
The tool tracks the availability of each tunnel. If any of the tunnels is unresponsive, the supervisor will attempt to terminate the tunnel gracefully and start a new one. Additionally, the tool provides the ability to perform "rolling restarts" all the tunnels periodically to avoid known issues with long-running tunnel processes.
By launching sauce tunnels in high availability mode with crows-nest, you won't need to:
- manually check the availability of each tunnel
- manually launch up new tunnels to replace the dead ones
- manually and periodically restart all tunnels
Usage
Configuration
./config.json
{
"username": "",
"accessKey": "",
"verbose": false,
"proxy": null,
"tunnelIdentifier": "",
"waitTunnelShutdown": true,
"noRemoveCollidingTunnels": true,
"restartCron": "*/2 * * * *"
}
./config.json file has the basic configurations that are required by Sauce Connect. To launch your own tunnels, username, accessKey and tunnelIdentifier are mandatory.
More configurations please refer to this page sauce-connect-launcher.
In high availability mode all tunnels share the same tunnelIdentifier. tunnelIdentifier can be any string. One suggested convention is to use this ID to describe the geographical location where your tunnel terminates. For example, east or west.
You can set username and accessKey using one of the following methods:
- Specifying the values in
./config.json - Setting the environment variable
SAUCE_USERNAMEandSAUCE_ACCESS_KEY
If the rolling restart feature is enabeld, restartCron must be a valid cron value.
Help
./bin/supervise --help
Basic usage
Start one sauce tunnel in high availability mode:
./bin/supervise --tunnels 1
With rolling restart feature:
SAUCE_USERNAME=xxx SAUCE_ACCESS_KEY=yyy ./bin/supervise --tunnels 1 --rollingRestart
Advanced usage
Read sauce tunnel configuration from ./myproject/config.json and launch 20 sauce tunnels in high availability mode
./bin/supervise --tunnels 20 --config ./myproject/config.json
Daemon
We use pm2 to run the supervisor process as a daemon
To start:
./node_modules/.bin/pm2 start ./bin/supervise --kill-timeout 600000 --silent -- --tunnels 10 --rollingRestart
To stop:
./node_modules/.bin/pm2 stop supervise
To view the log:
./node_modules/.bin/pm2 log --lines 100
Design
Architecture
+---------------------------------------------------------------------+ +-----------------+
| | | |
| Crows-nest +---------------------------+ | | Saucelabs |
| | +----------------| | | |
| | Tunnel 1 | Sauce Tunnel || | | Sauce Tunnel |
| +-----> | | Child Process || +--------> | Proxy Cloud |
| | | +----------------| | | |
| | +---------------------------+ | | |
| | | | |
| | | | |
| | | | |
| | +---------------------------+ | | |
| | | +----------------| | | |
| +-----------------+ | | Tunnel 2 | Sauce Tunnel || | | |
| | | +-----> | | Child Process || +--------> | |
| | Supervisor +----------+ | +----------------| | | |
| | | | +---------------------------+ | | |
| +-----------------+ | | | |
| | . | | |
| | . | | |
| | . | | |
| | . | | |
| | | | |
| | +---------------------------+ | | |
| | | +----------------| | | |
| | | Tunnel n | Sauce Tunnel || | | |
| +-----> | | Child Process || +--------> | |
| | +----------------| | | |
| +---------------------------| | | |
+---------------------------------------------------------------------+ +-----------------+
Components
There are two major components in Crows-nest, Supervisor and Tunnel
+------------+ +---------+
| Supervisor | | Tunnels |
+-----+------+ +----+----+
| |
+++ |
| | +++
| | Initialize() | |
| +------------------------------------> | | monitor()
| | | +------------+
+++ | | |
| | | <----------+
| +++
| |
| |
+++ |
| | scheduleRestart() +++
| +------------------------------------> | | stop()
| | | +------------+
| | | | |
| | | | |
| | | | <----------+
| | | |
| | | | start()
| | | +------------+
| | | | |
| | | | |
| | | | <----------+
+++ +++
| |
| |
+ +
Tunnel
Crows-nest Tunnel maintains the life cycle of a Saucelabs Sauce Connect Tunnel. It is a one to one mapping to Saucelabs Sauce Connect Tunnel instance. It does following things
- Start a Sauce Connect Tunnel as child process
- Monitor the status of current child process
- Terminate current Sauce Connect Tunnel nicely if the connection drops and start a new one.
- Restart current Sauce Connect Tunnel if scheduled by
rollingRestart - Stop current Sauce Connect Tunnel
Supervisor
Crows-nest Supervisor keeps track of all Crows-nest Tunnels. It does following things
- Initialize all Tunnels
- Start all Tunnels by sending
startsignals to each Tunnel - Restart all tunnels by sending
restartsignals to each Tunnel according to schedule - Stop all Tunnels by sending
stopsignals to each Tunnel
Randomness
To avoid network blast (in case all tunnels are scheduled at the same time), some randomness are introduced
- Each tunnel takes random delay [0, 5000] ms to start
- Each tunnel takes random delay [0, 5000] ms to stop
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent