ubclaunchpad / inertia

✈️ Effortless, self-hosted continuous deployment for small teams and projects

An effortless, self-hosted continuous deployment platform.

Usage · Motivation & Design · Contributing · Wiki

Inertia is a user-friendly, cross-platform command line application and serverside agent that enables quick and easy setup and management of continuous, automated deployment of a variety of project types on any virtual private server. The project is used, built, and maintained with ❤️ by UBC Launch Pad, UBC's student-run software engineering club.

	Main Features
🚀	Simple to use - set up a deployment from your computer without ever having to manually SSH into your remote
🍰	Cloud-agnostic - use any Linux-based remote virtual private server provider you want
⚒	Versatile project support - deploy any Dockerfile or docker-compose project
🚄	Continuous deployment - Webhook integrations for GitHub, GitLab, and Bitbucket means your project can be automatically updated, rebuilt, and deployed as soon as you git push
🛂	In-depth controls - start up, shut down, and monitor your deployment with ease from the command line or using Inertia's REST API
🏷	Flexible configuration - branch deployment, environment variables, easy file transfer for configuration files, build settings, and more
📦	Built-in provisioning - easily provision and set up VPS instances for your project with supported providers such as Amazon Web Services using a single command
👥	Built for teams - provide shared access to an Inertia deployment by adding users
🔑	Secure - secured with access tokens and HTTPS across the board, as well as features like 2FA for user logins

📦 Usage

Check out our new Inertia Usage Guide to get started with using Inertia for your project! The guide will walk you through installing Inertia, setting up a project, deploying to a remote, managing your deployment, and advanced usage tips.

Why Use Inertia?

If you...

• want a simple utility to quickly build and deploy the latest iterations of your projects
• are new to the concept of "deployment" and related tooling
• are on a tight budget and need to switch between cloud providers as your free trials run out
• want some lightweight team features for managing your deployment

Inertia might be for you! For example, UBC Launch Pad teams have used Inertia to set up automated deployments for projects like Rocket 2 and Bumper, and nwPlus used Inertia to stage previews of the nwHacks 2019 website during development.

💡 Motivation and Design

UBC Launch Pad is a student-run software engineering club at the University of British Columbia that aims to provide students with a community where they can work together to build a all sorts of cool projects, ranging from mobile apps and web services to cryptocurrencies and machine learning applications.

Many of our projects rely on hosting providers for deployment. Unfortunately we frequently change hosting providers based on available funding and sponsorship, meaning our projects often need to be redeployed. On top of that, deployment itself can already be a frustrating task, especially for students with little to no experience setting up applications on remote hosts. Inertia is a project we started to address these problems, with the goal of developing an in-house deployment system that can make setting up continuously deployed applications simple and painless, regardless of the hosting provider.

The primary design goals of Inertia are to:

• minimize setup time for new projects
• maximise compatibility across different client and VPS platforms
• offer an easy-to-learn interface for managing deployed applications

How It Works

There is a detailed Medium post that goes over the project, its motivations, the design choices we made, and Inertia's implementation. The team has also made a few presentations about Inertia that go over its design in some more detail:

• UBC Launch Pad internal demo
• Vancouver DevOpsDays 2018 (video)

In summary, Inertia consists of two major components: a deployment daemon and a command line interface.

The deployment daemon runs persistently in the background on the server, receiving webhook events from GitHub whenever new commits are pushed. The CLI provides an interface to adjust settings and manage the deployment - this is done through HTTPS requests to the daemon, authenticated using JSON web tokens generated by the daemon. Remote configuration is stored locally in .inertia.toml.

Inertia is set up serverside by executing a script over SSH that installs Docker and starts an Inertia daemon image with access to the host Docker socket. This Docker-in-Docker configuration gives the daemon the ability to start up other containers alongside it, rather than within it, as required. Once the daemon is set up, we avoid using further SSH commands and execute Docker commands through Docker's Golang API. Instead of installing the docker-compose toolset, we use a docker-compose image to build and deploy user projects.

📚 Contributing

Any contribution (pull requests, feedback, bug reports, ideas, etc.) is welcome!

Please see our contribution guide for contribution guidelines as well as a detailed guide to help you get started with Inertia's codebase.

inertia's Issues

Access deployment logs

inertia [REMOTE] logs -c [MY_CONTAINER_NAME]

Allow users to view/stream logs from their inertia deployment

Rename 'remote bootstrap' to '[REMOTE_NAME] init'

also see #44

Create a Testing Container That Acts Like a Remote

For development, we're currently using an actual host on gcloud or somewhere. We should write a script that starts a container that we can use as if it were a remote, enable SSH access to it, create a PEM, output login credentials etc, etc. It will save development time and $$.

Much like @bobheadxi's testing strategy here: #30

Logrus

Swap out our logging with logrus - replace some log.Fatal's with log.WithError's.

Daemon Up Shell Script Breaks When Daemon Stopped

ALREADY_RUNNING=`sudo docker ps -q --filter "name=$DAEMON_NAME"`

Doesn't capture the case when the container has already been started but may have exited (see here https://github.com/ubclaunchpad/inertia/blob/master/cmd/bootstrap/daemon-up.sh#L11).

One fix is to use status="exited".

SSH-less initialization

We are considering the benefits of an SSH-less initialization (and maybe management afterward).
From @chadlagore:

Pros:

• Easy to setup many instances at once (inertia init -l <list_of_instances>)
• No accidental box customization aside from what the setup script does.
• Easy onboarding for junior devs.

Cons:

• Might have to do weird stuff like ssh root@mybox 'bash -s' < setup_script.sh.
• Difficult to test.

Deployment requirements

We are considering two possible dependencies for deployment.

GitHub:

Pull a repository from GitHub, builds some Docker images, and runs them via a Compose file.

Pros:

• We already always use GitHub
• Don't need to manually copy any files to server (daemon automatically git clones everything required)
• Easier to expand to non-Docker apps later (Could have users specify a Heroku-like Procfile)
• You might even be able to simulate the Docker Image Repository method below using this method and some setup script logic that gets run when you docker-compose up.

Cons:

• We need to create a webhook manually through GitHub, and input a SSH key manually through GitHub (same as Docker Image Repo though, at least for the webhook)

Docker Image Repository:

Pull a docker image from a specified repository URL, run them via a Compose file. Use their webhook system to receive deploy events.

Pros:

• Only file need to move to server is inertia.yml. It is identical to the docker-compose.yml, but it has image urls specified for the image flag and production settings instead of development settings.
• No git cloning, just install docker and run docker-compose up -d -f /env/inertia.yml.
• No git login on server.

Cons:

• Need to set up continuous integration (manually?) with Docker Cloud
• Need docker login somewhere.
• Need to configure automated build on dockerhub (this is almost a show stopper for me and it doesn't appear possible to make an autobuild repo programmatically - the other option is to use travis, but I don't think its a good idea to touch the travis script here).

Implement Daemon Server-side Functionality

Definitions

• Client-side: CLI calls made via inertia binary, run from inside your repo locally, some of which make HTTP requests to the server, some of which pipe commands over SSH to bootstrap the server.
• Server-side: Daemon activity (receiving HTTP requests and responding appropriately) and bootstrapping activity (running scripts piped over SSH) on the server.

Background

The daemon runs on the server after we call:

▶ inertia remote bootstrap [REMOTE]

Then we can make sure its running with:

▶ inertia remote status [REMOTE]
# Remote instance '[REMOTE]' accepting requests at http://35.227.171.49:8081

• The daemon is simply a web server running in a docker container with access to the docker socket via this trick.
• The daemon was started when the client CLI tool ran this script over SSH (one of the steps in the implementation of the bootstrap call).
• The daemon accepts generic hooks from GitHub on /. It currently logs the information it receives and nothing more.
• The server provided the client with a public deploy key (after the bootstrap call) and asked the user to associate the key with their GitHub repository (effectively giving the daemon readonly access to the repository).
• The daemon currently has two other endpoints.

1. /up [POST] Bring the project up. This could fail for a myriad of reasons.
   • Upon receiving a POST to this endpoint, the project could not yet be cloned (cloning functionality laid out in the POC). We have to try to clone it, this could fail due to the user not yet giving the daemon readonly access (failing to have added the deploy key), return an appropriate error.
   • The project may already be cloned, conventionally we may either disregard the POST or force re-deploy the project with a git pull, docker-compose up; I'm partial to the latter, but maybe a client-side flag is in order.
   • The user may not have added the URL for webhooks. I'm not sure if this something the daemon can learn by querying the GitHub API, but it would be helpful user feedback - perhaps this falls into the "nice to have" category.
2. /down [POST] Bring the project down. Fail in the situation where no other containers besides the daemon itself are running. Use the docker POC to learn this.

Finally, the daemon needs some sort of authorization. A token needs to be transferred to the client on bootstrap - something to think about how to do.

Goveralls

User Expectations

1. User installs inertia locally, user runs inertia init.
   • Alternatively, user ssh login, user install inertia, user run inertia init.
2. User opens port on VPS.
   • Can't avoid, need for server-daemon design choice (see #12).
3. User copy-paste public deploy key to GitHub.
   • Can avoid with oauth GitHub login or by logging into github thru inertia (see #14)
4. User copy-paste webhook URL to GitHub.
   • Maybe can avoid later with oauth GitHub login.

go-git v4.x

look into new features, implement and upgrade

Heroku Buildbacks with herokuish

ie. setup in .inertia.toml

Remote Add Walkthrough

We currently silently expect certain parameters on inertia remote add. When the user doesnt actually give them, we use defaults which are not good.

Change it to this,

$ inertia add remote instance-1
Enter location of pem file:
Enter IP:

Then try to extract the SSH user from the pem public file (this has the benefit of failing if we cant find the public key - which would be necessary to authenticate over SSH anyways).

Secure client-server communication

Right now the daemon sets up an HTTP (not HTTPS) listener. This makes the token authorization system vulnerable to replay attacks since the token is sent in the clear.

To fix: Use HTTPS exclusively. Maybe there's a way to generate a cert when creating a new remote, and informing the client-side to trust that cert?

Configuration

Configuration Options

The following are some configuration parameters Inertia may have to accept:

• Whitelisted webhook URL(s) (we should only accept POST requests to a specific set
  of user-specified URLs).
• Repositories to build from
• Branches to build from for a given repo
• Events to build from (push, PR, etc.) for a given repo
• Secrets for secure communication and code-transfer

Configuration Methods

yml Config File

Pros

• Easy to implement

Cons

• User has to manually edit the config file over SSH or copy a new version
  over to their VPS every time they want to change config. They may also have
  to restart the daemon to load the new config, but we could implement some
  auto-refresh mechanism.

Configuration over HTTP or some RPC

Pros

• If we're going to have a CLI that communicates with our daemon it makes
  sense to design our daemon to accept remote config from the beginning anyway.
• Theoretically zero downtime for configuration changes because there is no
  need to restard the daemon to load new config.

Cons

• Harder to implement.

Hybrid of previous 2 options

Pros

• Sick

Cons

• It's the most work

Warn if 'inertia init' in non-Docker project

Set docker-compose project name

defaults to the folder name (in this case project) but can be changed with a flag - since it seems like some projects rely on hardcoded container names, would be good to pull project name from remote url just for deploy() so that our docker-compose doesn't append project_ to the name of every container

Setup go project

Setup dep management

`inertia [REMOTE]` commands segfault if no response

Reproduce:

$> inertia init
$> inertia remote add somevps
# manually change the daemon port in config to some garbage
$> inertia somevps status
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x40 pc=0x14dccdc]

goroutine 1 [running]:
github.com/ubclaunchpad/inertia/cmd.glob..func6(0x19be900, 0x19e4188, 0x0, 0x0)
	/Users/robertlin/go/src/github.com/ubclaunchpad/inertia/cmd/deploy.go:119 +0xbc
github.com/ubclaunchpad/inertia/vendor/github.com/spf13/cobra.(*Command).execute(0x19be900, 0x19e4188, 0x0, 0x0, 0x19be900, 0x19e4188)
	/Users/robertlin/go/src/github.com/ubclaunchpad/inertia/vendor/github.com/spf13/cobra/command.go:702 +0x2c6
github.com/ubclaunchpad/inertia/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0x19bf180, 0x1, 0x16394c0, 0x4)
	/Users/robertlin/go/src/github.com/ubclaunchpad/inertia/vendor/github.com/spf13/cobra/command.go:783 +0x30e
github.com/ubclaunchpad/inertia/vendor/github.com/spf13/cobra.(*Command).Execute(0x19bf180, 0x0, 0x19bf5c0)
	/Users/robertlin/go/src/github.com/ubclaunchpad/inertia/vendor/github.com/spf13/cobra/command.go:736 +0x2b
github.com/ubclaunchpad/inertia/cmd.Execute()
	/Users/robertlin/go/src/github.com/ubclaunchpad/inertia/cmd/main.go:38 +0x38
main.main()
	/Users/robertlin/go/src/github.com/ubclaunchpad/inertia/main.go:6 +0x20

Get Repo Name to Daemon

This was supposed to be caught via webhook, but for the time being lets pass it up on any call to inertia deploy [REMOTE] up.

Proof-of-concept webhook receiver

Get a server up and running that receives webhook events from a GitHub repo using a secret. This requires creating a webhook in the GitHub repo. This package might be useful.

Scaffold CLI

Scaffold a subpackage containing a Cobra CLI

Scaffold inertia web

Simple scaffold to show how inertia daemon could deliver web pages

Bootstrap Implementation

When inertia init is called, bootstrap the server.

Authorization

Add some auth to the daemon

One promising approach is to use a JWT. We've already generated an RSA key server-side. We can use it to create a JWT during the server bootstrapping process, and securely pipe it back to the client over SSH for storage in it's config folder. All of this should happen without involving the user.

`inertia reset`

$ inertia reset
# delete ./inertia folder

command up for discussion - delete or clear or reset?

Inertia Web

Would be cool to offer a clean web interface for managing your Inertia project when the daemon is online:

$> inertia myremote init
...
Inertia daemon is now online. Visit http://myhost.com:8081/admin to manage your project!

At minimum, the user should be able to:

• enable more accounts from the CLI and allow logins via web interfaces
• be assured of the security of using inertia web
• manage the deployment state of their project (up/down)
• manage their repository (change branches, check what commit is deployed)
• view things like docker-compose logs and container logs

Stretch goals:

• Analytics? Visitor count, deploys per day, uptime/downtime
• Container backups?
• Plugins?

Show more details about remotes

e.g.

$ inertia remote
gcloud { IP: 0.0.0.0, user: robertlin, ... }
testserver { IP: 0.0.0.0, user: robertlin, ... }

formatting up for discussion

Support Multiple Remotes

v0.0.2 demo

• slideshow
• get Sleuth to work with properly on testvps instance for demonstration

Compile executables for release

travis releases uploading

• automatically tag and build all merge commits on master as beta builds
• build tagged commits as releases
• possibly push to dockerhub as well

Better error when the git ops fail due to handshake fail

A reasonable error, with bad language. Heres how to replicate:

$ inertia init
$ inertia remote add ...
$ inertia [REMOTE] init   # Forget to add deploy key to github.
$ inertia [REMOTE] up
(Status code 412) Problem with deployment setup: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain

We need to make it more informative! It should say

I could't clone the project, did you add the deploy key? <deploy_key>"

Add --rm to deamon run

In the odd case that I kill the daemon manually, but try to run it again, I get this error:

▶ ./inertia gcloud init
Bootstrapping remote
Installing docker
Starting daemon
docker: Error response from daemon: Conflict. The container name "/inertia-daemon" is already in use by container "7d10a51407403a008a31abf1f962d2c43074bf2738d4c11c9628f25c3ce30e0c". You have to remove (or rename) that container to be able to reuse that name.
See 'docker run --help'.

FATA[0001] Process exited with status 125

The reason is we need to use docker run --rm .... when we run the daemon so that it removes the container if it crashes.

Inertia versioning

Might be a good idea to version inertia so that the desired version of the tool can be used, such as:

inertia init -v 1.0

Because bootstrap pulling the latest version of inertia could cause problems with version discrepancies

Bonus for being able to specify a branch version of inertia for testing 😛

As noted in #95, also add to Travis builds

List of docker-compose applications that don't work when deployed by Inertia

Repos that might be worth looking into

• volumes do not seem to get mounted correctly - this one fails at building the server container (can't find start-server.sh), tested on an Ubuntu VPS

Dockerize Testing Environment

A lot of our initialization code does some small file system changes. This may be something we want to dockerize in order to test properly. Discuss?

container1 (fake local, file system changes) <---- ssh pipe ----> container2 (fake remote, accepts ssh commands)

Replacing remote breaks config.json

e.g.

inertia remote add gcloud 123456 -u brunocodesbad -i /Users/robertlin/.ssh/brunocodesbad

{
 "name":"gcloud",
 "remote":{
  "User":"brunocodesbad",
  "IP":"123456",
  "PEM":"/Users/robertlin/.ssh/brunocodesbad",
  "Port":"8081"
 }
}

Then:

inertia remote add local 0.0.0.0

{
 "name":"local",
 "remote":{
  "User":"root",
  "IP":"0.0.0.0",
  "PEM":"/Users/robertlin/.ssh/id_rsa",
  "Port":"8081"
 }
}rt":"8081"}}

Travis

run test

Proof-of-Concept Serverless Daemon

The deamon requires a port to be opened for it. Thats a bit annoying.

Another option is to run a process that just checks it's current version of the project against some remote version (git hash, image hash, whatever), and upon noticing a difference, it re-deploy's the project. This is a pull instead of a push from github.

Requirements: Either the deamon needs a github deploy key or an image container deploy key to use for checking for updates, cloning, pulling, etc.

Proof-of-Concept SSH GitHub connection

Figure out how to get a SSH connection to pull from a GitHub repo. This requires adding the SSH key to the repo through settings. This package might be useful.

Inertia SSH Command

Add a command inertia SSH [REMOTE] that establishes an interactive session with your remote (just a wrapper around ssh user@ip).

Proof of Concept Deploy Key

As a result of calling inertia init <vps_ip> <github_repo> (locally), at least the following steps should take place:

0. Inertia gets installed on the VPS.
1. Create a key pair locally.
2. Show the public key to the user (they copy-pasta to GitHub).
3. Pipe the pem file to the VPS ssh <vps_ip> 'bash -c' < inerta set-key <key> <repo>.
4. Show the webhook url to the user locally (they copy-pasta to GitHub.

OR, if we opt to call inertia init on the VPS kill step 3 and replace step 0 with ssh <vps_ip>:

Example:

$> inertia init <vps_ip> <github_repo>

😏 Your new inertia public deploy key (add it here https://github.com/<user>/<repo>/settings/keys/new):
ssh-rsa AAAAB3NzaC1yc2EAA...

Enter VPS SSH password: *****************

🎉 Your new inertia GitHub webhook URL (add it here https://github.com/<user>/<repo>/settings/hooks/new):
https://<vps_ip>:8888

$>

Note: (2) can actually be done with:

curl -kX POST -u <user> https://api.github.com/repos/<user>/<repository>/keys \
    -d '{
    "key": "<public-key>", 
    "read_only": true, 
    "title": "inertia"
}'

# Prompts for GitHub password :(

but then the user has to login to github via inertia, which seems bad - but might be OK? This can be done with OAuth too, but is OAuth in scope for MVP?

Remove Deploy Command

Replace with simply inertia [REMOTE] up. Do the same for init and down.

Dockerize Daemon

It would be nice to just have to run docker run ubclaunchpad/inertia-daemon over SSH and have the daemon come up.

Restructure project

Possible packages:

• cmd: top-level package for cobra and command line stuff (probably excluding daemon commands?)
• daemon: server-side functionality (daemon)
• client: client-side functionality (init, remote, deploy, bootstrap)
• common: bottom-level package for shared types, utils, bootstrap scripts, etc

        cmd
      /  |  \
 daemon  |  client
      \  |  /
       common

'inertia init' does nothing

Scaffold daemon

Create a subpackage for the daemon.

I think the best way to do this is to write the daemon as a server (without the knowledge that it'll be running as a daemon or the ability to bootstrap itself) and have the CLI just run the daemon either as a Docker container (using exec.Cmd) or by registering it as a system service using this maybe (?).

Open to suggestions about the best way to do this

Proof Of Concept SSH-Command Wrapper

Write a wrapper that pipes commands to a server, and returns the output.

Set up Vling server for testing

Merge `inertia remote status` with `inertia [REMOTE] status`

Right now there are two remote-status-related commands:

$> inertia remote add gcloud
$> inertia remote status gcloud
Remote instance 'gcloud' accepting requests at http://myhost.com:8081
$> inertia gcloud status
(Status code 200) 7b7be0b7097a26169e17037f4220fd0ce039bde1 refs/heads/master
There are currently no active containers.

I'm in favour of putting the inertia remote status functionality into inertia [REMOTE] status:

$> inertia gcloud status
Could not connect to daemon
Try running inertia [REMOTE] init
$> inertia gcloud init
$> inertia gcloud status
(Status code 200) Daemon at remote 'gcloud' online at http://myhost.com:8081
Project status:
    7b7be0b7097a26169e17037f4220fd0ce039bde1 refs/heads/master
    There are currently no active containers.