garysieling / clusterodm

A NodeODM API compatible autoscalable load balancer and task tracker for easy horizontal scaling ♆

ClusterODM

A reverse proxy, load balancer and task tracker with optional cloud autoscaling capabilities for NodeODM API compatible nodes. In a nutshell, it's a program to link together multiple NodeODM API compatible nodes under a single network address. The program allows to distribute tasks across multiple nodes while taking in consideration factors such as maximum number of images, queue size and slots availability. It can also automatically spin up/down nodes based on demand using cloud computing providers (currently only DigitalOcean).

The program has been battle tested on the WebODM Lightning Network for quite some time and has proven reliable in processing thousands of datasets. However, if you find bugs, please report them.

Installation

The only requirement is a working installation of NodeJS.

git clone https://github.com/OpenDroneMap/ClusterODM
cd ClusterODM
npm install

There's also a docker image available at opendronemap/clusterodm.

Usage

First, start the program:

node index.js [parameters]

Or with docker:

docker run --rm -ti -p 3000:3000 -p 8080:8080 opendronemap/clusterodm [parameters]

Then connect to the CLI and connect new NodeODM instances:

telnet localhost 8080
> HELP
> NODE ADD nodeodm-host 3001
> NODE LIST

Finally, use a web browser to connect to http://localhost:3000. A normal NodeODM UI should appear. This means the application is working, as web requests are being properly forwarded to nodes.

You can also check the status of nodes via a web interface available at http://localhost:10000.

See node index.js --help for all parameter options.

Autoscale Setup

ClusterODM can spin up/down nodes based on demand. This allows users to reduce costs associated with always-on instances as well as being able to scale processing based on demand.

To setup autoscaling you must:

• Make sure docker-machine is installed.
• Setup a S3-compatible bucket for storing results.
• Create a configuration file.

Example configuration file:

{
    "provider": "digitalocean",
    
    "accessToken": "CHANGEME!",
    "s3":{
        "accessKey": "CHANGEME!",
        "secretKey": "CHANGEME!",
        "endpoint": "nyc3.digitaloceanspaces.com",
        "bucket": "bucketname"
    },

    "maxRuntime": -1,
    "maxUploadTime": -1,
    "region": "nyc3",
    "monitoring": true,
    "tags": ["clusterodm"],
    
    "image": "ubuntu-16-04-x64",
    "snapshot": false,

    "imageSizeMapping": [
        {"maxImages": 20, "slug": "s-1vcpu-1gb"},
        {"maxImages": 100, "slug": "s-4vcpu-8gb"}
    ],

    "addSwap": 1,
    "dockerImage": "opendronemap/nodeodm"
}

Field	Description
accessToken	DigitalOcean API Access Token
s3	S3 keys and bucket configuration
dropletsLimit	Maximum number of droplets that are allowed to run at the same time. Set to -1 for no limit.
createRetries	Number of attempts to create a droplet before giving up. Defaults to 1.
maxRuntime	Maximum number of seconds a droplet is allowed to run ever. Set to -1 for no limit.
maxUploadTime	Maximum number of seconds a droplet is allowed to receive file uploads. Set to -1 for no limit.
region	Region identifier where the droplets should be created.
monitoring	Set to true to enable monitoring on the droplet.
tags	List of tags to associate to the droplet.
image	Image identifier (from public images) or snapshot identifier (private) if snapshot is set to true (see below).
snapshot	When set to true, image refers to a snapshot in the user account instead of an image name. Useful to speed up boot time if you already have a droplet with the docker image preloaded.
sshKey	Optionally specify an existing DigitalOcean SSH fingerprint and private key path instead of generating new keys.
imageSizeMapping	Max images count to droplet size mapping. The autoscaler will pick a droplet size based on the number of images of the incoming task. Use this to control what size of droplet should correspond to which image count. The least powerful droplet able to process a certain number of images is always selected. Valid slug identifiers are available from Digital Ocean's API list all sizes
addSwap	Optionally add this much swap space to the droplet as a factor of total RAM (RAM * addSwap). A value of 1 sets a swapfile equal to the available RAM.
dockerImage	Docker image to launch

You can then launch ClusterODM with:

node index.js --asr configuration.json

You should see the following messages in the console:

info: ASR: DigitalOceanAsrProvider
info: Can write to S3
info: Found docker-machine executable

You should always have at least one static NodeODM node attached to ClusterODM, even if you plan to use the autoscaler for all processing. If you setup auto scaling, you can't have zero nodes and rely 100% on the autoscaler. You need to attach a NodeODM node to act as the "reference node" otherwise ClusterODM will not know how to handle certain requests (for the forwarding the UI, for validating options prior to spinning up an instance, etc.). For this purpose, you should add a "dummy" NodeODM node and lock it:

telnet localhost 8080
> NODE ADD localhost 3001
> NODE LOCK 1
> NODE LIST
1) localhost:3001 [online] [0/2] <version 1.5.1> [L]

This way all tasks will be automatically forwarded to the autoscaler.

A docker-compose file is available to automatically setup both ClusterODM and NodeODM on the same machine by issuing:

docker-compose up

Roadmap

We have plenty of goals. If you want to help, or need help getting started contributing, get in touch on the OpenDroneMap community forum.

License

ClusterODM is licensed under the AGPL. Contact us for other licensing options.