chrismckenzie / dropship Goto Github PK

View Code? Open in Web Editor NEW

64.0 4.0 12.0 18.27 MB

Super simple deployment tool

Home Page: https://chrismckenzie.io/posts/deploy-anything/

License: Apache License 2.0

Go 83.06% HCL 3.51% Shell 13.43%

deployment dropship hcl shipping shipit cd automation deploy

dropship's People

Contributors

Stargazers

Watchers

Forkers

cloudxtreme gitter-badger bhyvex the-control-group m4ttyw steenzout developgo joel-g shammishailaj sangdev90 standardgalactic ajunlonglive

dropship's Issues

Post-deployment testing + rollback

It would be awesome if we had the ability to rollback the deployment based on the pass or fail of a post-deployment script that can be anything, though in the case I'm thinking of, automated integration tests. Also provide hooks for notifications in case of failure, though to start the script could do this.

Persist Hash Value

Currently the file hash that is used to check if the remote and local are in sync is stored and memory and should be persisted.

I believe this should be configurable and have the ability to store this information in a distributed store like consul as well as on the local FS.

some ssh calls fail and don't print

sometimes ssh calls will fail to execute and never say anything. 😢

JSON Service config

you should be able to write service configs in JSON and not just hcl. maybe even yaml?

plugin system

It would be awesome to have a plugin system much like many CI systems that would allow users to write pre-defined steps.

Management and Monitoring Interface

This meant as a planning issue for the new v2 UI additions.

The management and monitoring interface is pretty self explanatory and provides a central ui for all dropship instances to be monitored and configured. This will allow users to easily catch errors on deployment for there service on an overall or per-machine basis. This interface will also supply some much needed coordination for services and will allows us to fire hooks for when an entire service (all machines) have completed their deployment.

I have started development on this first as it will change quite a few things in the architecture of the application. This will be accomplished by using an grpc service for streaming and registering service data back and forth between agent and manager.

Manager

The manager is the responsible for serving both the UI and a portion of the RPC service. The UI will be a simple web service that will display all services that are registered with it and allow users to see past deployments as well as the state of any current deployments. The Manager RPC service will be a gRPC service that will contain a few functions for managing (adding, removing) Service Members, as well as a set of streaming interfaces for receiving results from current deployments.

Need rolling deployment

This might be related to #18, but I'm not certain so I'm creating a new issue. An application accessing the API that is being updated receives 502s for a short period of time during deployments. Ideally, there would be rolling updates, and each instance could be removed from the load balancer during the update, and added back in as soon as it was completed.

Issues with script Hook

It would seem that one of my commands fails in my before "script" hook which is there to stop the process before installing an updated binary this causes the update to fail. Unfortunately none of the commands output is logged to the dropship logs, it would be nice to see this.

Thanks @Sdedelbrock for the report!

Support for piping commands

I have the following command in my dropship config file:

  before "script" {
    command = "ls | xargs echo"
  }

However, when I run it I get the following error:

2016/05/09 20:16:50 [INF]: Executing "script" hook with map[command:ls | xargs echo]
2016/05/09 20:16:50 [INFO]:
2016/05/09 20:16:50 [ERR]: Unable to execute "script" hook exit status 2

I'd like to be able to pipe commands in these scripts - is there a way to do that?

Wrong mime detection

The following dropship config errors out:

service "cadabra-api" {
  sequentialUpdates = true
  checkInterval = "10s"

  before "script" {
    command = "initctl stop cadabra-api"
  }

  artifact "rackspace" {
    bucket = "private-binaries"
    path = "cadabra/api/api.tar.gz"
    destination = "/sites"
  }

  after "script" {
    command = "initctl start cadabra-api"
  }
}

Dropship logs:

2016/02/28 02:11:42 [INF]: Acquiring lock for cadabra-api
2016/02/28 02:11:42 [INF]: Downloading update for cadabra-api...
2016/02/28 02:11:42 [INF]: Executing "script" hook with map[command:initctl stop cadabra-api]
2016/02/28 02:11:42 [INF]: Update for cadabra-api installed successfully. [hash: b39e9351ecd98df60752ef7cf99ca878] [files written: 1]
2016/02/28 02:11:42 [INF]: Executing "script" hook with map[command:initctl start cadabra-api]
2016/02/28 02:11:42 [ERR]: Unable to execute "script" hook fork/exec /sbin/initctl: not a directory

this ends up creating a binary file (not the one in the tarball weirdly enough) at /sites (supposed to be the install directory)

Hook for after all deployments

It would be awesome to be able to run a script after all deployments across machines have finished.

Timeout for distributed deploys

@dewey4iv brought up a good point today sometimes we may want to have a sleep timeout on our lock so that we for example can say 1 deploy can happen every 10 minutes or some thing like that currently it unlock the second it is finished, which makes it feel like all of the deploys are happening at once.

Configs for Consul

currently the system assumes the machine it is running on is a consul agent would be nice if we could direct it at a given consul server.

Github repo support

I would like to see the ability to download artifacts from github releases.

Mult-task definitions

I would like to be able to define custom tasks in my dropship.yml for example a task that migrates database schemas as well as a task that just deploys the app

Example:

tasks:
  "deploy:migrate":
      steps:
         - ...
  deploy:
      steps:
        - ...

Add Environment support

Github API can optionally send a deployment environment it would be nice to be able to define a different set of servers for each environment.

maybe something like this:

servers:
  development:
    provider: list
    options:
      list:
          -
            address: 127.0.0.1

@ericgreene || @krisbarrett || @mattwarren1234 do you have any input?

Build UI for deployment logs

We need a way to view the response from the remote servers so that it is easier to debug and monitor. I imagine something akin to _Wercker, or CI's alike.

Race condition onClose

after a long download is finished exiting the agent will panic do to a send on closed channel.

panic: send on closed channel

goroutine 8 [running]:
github.com/ChrisMcKenzie/dropship/work.(*Dispatcher).start(0xc8200f8750)
        /Users/chrism/gocode/src/github.com/ChrisMcKenzie/dropship/work/dispatcher.go:59 +0x293
created by github.com/ChrisMcKenzie/dropship/work.NewDispatcher
        /Users/chrism/gocode/src/github.com/ChrisMcKenzie/dropship/work/dispatcher.go:45 +0x178

goroutine 1 [semacquire]:
sync.runtime_Semacquire(0xc820104534)
        /usr/local/Cellar/go/1.5.1/libexec/src/runtime/sema.go:43 +0x26
sync.(*WaitGroup).Wait(0xc820104528)
        /usr/local/Cellar/go/1.5.1/libexec/src/sync/waitgroup.go:126 +0xb4
github.com/ChrisMcKenzie/dropship/work.(*Runner).Shutdown(0xc820104520)
        /Users/chrism/gocode/src/github.com/ChrisMcKenzie/dropship/work/runner.go:41 +0x3e
github.com/ChrisMcKenzie/dropship/commands.agent(0x7dab00, 0xc820019040, 0x0, 0x2)
        /Users/chrism/gocode/src/github.com/ChrisMcKenzie/dropship/commands/agent.go:46 +0x4c4
github.com/spf13/cobra.(*Command).execute(0x7dab00, 0xc820018f20, 0x2, 0x2, 0x0, 0x0)
        /Users/chrism/gocode/src/github.com/spf13/cobra/command.go:503 +0x869
github.com/spf13/cobra.(*Command).Execute(0x7dacc0, 0x0, 0x0)
        /Users/chrism/gocode/src/github.com/spf13/cobra/command.go:586 +0x46a
github.com/ChrisMcKenzie/dropship/commands.Execute()
        /Users/chrism/gocode/src/github.com/ChrisMcKenzie/dropship/commands/dropship.go:38 +0x28
main.main()
        /Users/chrism/Projects/dropship/main.go:6 +0x14

goroutine 17 [syscall, locked to thread]:
runtime.goexit()
        /usr/local/Cellar/go/1.5.1/libexec/src/runtime/asm_amd64.s:1696 +0x1

goroutine 5 [syscall]:
os/signal.loop()
        /usr/local/Cellar/go/1.5.1/libexec/src/os/signal/signal_unix.go:22 +0x18
created by os/signal.init.1
        /usr/local/Cellar/go/1.5.1/libexec/src/os/signal/signal_unix.go:28 +0x37

goroutine 6 [select]:
github.com/ncw/swift.(*Connection).doTimeoutRequest(0xc820138120, 0xc8201e0100, 0xc82035a620, 0xc8203da120, 0x0, 0x0)
        /Users/chrism/gocode/src/github.com/ncw/swift/swift.go:226 +0x21d
github.com/ncw/swift.(*Connection).Call(0xc820138120, 0xc8201e0000, 0x3d, 0xc8200fc861, 0x6, 0xc8200197c1, 0x19, 0x55b8c0, 0x4, 0x0, ...)
        /Users/chrism/gocode/src/github.com/ncw/swift/swift.go:468 +0x6ec
github.com/ncw/swift.(*Connection).storage(0xc820138120, 0xc8200fc861, 0x6, 0xc8200197c1, 0x19, 0x55b8c0, 0x4, 0x0, 0x0, 0xc820014e10, ...)
        /Users/chrism/gocode/src/github.com/ncw/swift/swift.go:527 +0x128
github.com/ncw/swift.(*Connection).Object(0xc820138120, 0xc8200fc861, 0x6, 0xc8200197c1, 0x19, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /Users/chrism/gocode/src/github.com/ncw/swift/swift.go:1679 +0xe2
github.com/ChrisMcKenzie/dropship/updater.(*RackspaceUpdater).IsOutdated(0xc820847cf0, 0xc8204ae340, 0x20, 0xc820847ee0, 0x1, 0x0, 0x0)
        /Users/chrism/gocode/src/github.com/ChrisMcKenzie/dropship/updater/rackspace.go:42 +0xb2
github.com/ChrisMcKenzie/dropship/work.(*Dispatcher).Work(0xc8200f8750)
        /Users/chrism/gocode/src/github.com/ChrisMcKenzie/dropship/work/dispatcher.go:74 +0x418
github.com/ChrisMcKenzie/dropship/work.NewRunner.func1(0xc820104520)
        /Users/chrism/gocode/src/github.com/ChrisMcKenzie/dropship/work/runner.go:26 +0x7e
created by github.com/ChrisMcKenzie/dropship/work.NewRunner
        /Users/chrism/gocode/src/github.com/ChrisMcKenzie/dropship/work/runner.go:29 +0xe4

goroutine 9 [select, locked to thread]:
runtime.gopark(0x63e798, 0xc82002ff28, 0x561f88, 0x6, 0x2cc18, 0x2)
        /usr/local/Cellar/go/1.5.1/libexec/src/runtime/proc.go:185 +0x163
runtime.selectgoImpl(0xc82002ff28, 0x0, 0x18)
        /usr/local/Cellar/go/1.5.1/libexec/src/runtime/select.go:392 +0xa64
runtime.selectgo(0xc82002ff28)
        /usr/local/Cellar/go/1.5.1/libexec/src/runtime/select.go:212 +0x12
runtime.ensureSigM.func1()
        /usr/local/Cellar/go/1.5.1/libexec/src/runtime/signal1_unix.go:227 +0x323
runtime.goexit()
        /usr/local/Cellar/go/1.5.1/libexec/src/runtime/asm_amd64.s:1696 +0x1

goroutine 25 [select]:
net/http.(*persistConn).writeLoop(0xc8204b40b0)
        /usr/local/Cellar/go/1.5.1/libexec/src/net/http/transport.go:1009 +0x40c
created by net/http.(*Transport).dialConn
        /usr/local/Cellar/go/1.5.1/libexec/src/net/http/transport.go:686 +0xc9d

goroutine 24 [IO wait]:
net.runtime_pollWait(0x2101098, 0x72, 0xc8200122e0)
        /usr/local/Cellar/go/1.5.1/libexec/src/runtime/netpoll.go:157 +0x60
net.(*pollDesc).Wait(0xc8204c2140, 0x72, 0x0, 0x0)
        /usr/local/Cellar/go/1.5.1/libexec/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc8204c2140, 0x0, 0x0)
        /usr/local/Cellar/go/1.5.1/libexec/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc8204c20e0, 0xc820536000, 0x8000, 0x8000, 0x0, 0xfa0050, 0xc8200122e0)
        /usr/local/Cellar/go/1.5.1/libexec/src/net/fd_unix.go:232 +0x23a
net.(*conn).Read(0xc82013c040, 0xc820536000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
        /usr/local/Cellar/go/1.5.1/libexec/src/net/net.go:172 +0xe4
crypto/tls.(*block).readFromUntil(0xc820487260, 0x21011a8, 0xc82013c040, 0x5, 0x0, 0x0)
        /usr/local/Cellar/go/1.5.1/libexec/src/crypto/tls/conn.go:455 +0xcc
crypto/tls.(*Conn).readRecord(0xc8204d6000, 0x63e817, 0x0, 0x0)
        /usr/local/Cellar/go/1.5.1/libexec/src/crypto/tls/conn.go:540 +0x2d1
crypto/tls.(*Conn).Read(0xc8204d6000, 0xc8204f8000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        /usr/local/Cellar/go/1.5.1/libexec/src/crypto/tls/conn.go:901 +0x167
net/http.noteEOFReader.Read(0x210df88, 0xc8204d6000, 0xc8204b4108, 0xc8204f8000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        /usr/local/Cellar/go/1.5.1/libexec/src/net/http/transport.go:1370 +0x67
net/http.(*noteEOFReader).Read(0xc8204f01a0, 0xc8204f8000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        <autogenerated>:126 +0xd0
bufio.(*Reader).fill(0xc82049cba0)
        /usr/local/Cellar/go/1.5.1/libexec/src/bufio/bufio.go:97 +0x1e9
bufio.(*Reader).Peek(0xc82049cba0, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
        /usr/local/Cellar/go/1.5.1/libexec/src/bufio/bufio.go:132 +0xcc
net/http.(*persistConn).readLoop(0xc8204b40b0)
        /usr/local/Cellar/go/1.5.1/libexec/src/net/http/transport.go:876 +0xf7
created by net/http.(*Transport).dialConn
        /usr/local/Cellar/go/1.5.1/libexec/src/net/http/transport.go:685 +0xc78

goroutine 4046 [select]:
net/http.(*persistConn).writeLoop(0xc8204b4370)
        /usr/local/Cellar/go/1.5.1/libexec/src/net/http/transport.go:1009 +0x40c
created by net/http.(*Transport).dialConn
        /usr/local/Cellar/go/1.5.1/libexec/src/net/http/transport.go:686 +0xc9d

goroutine 4045 [select]:
net/http.(*persistConn).readLoop(0xc8204b4370)
        /usr/local/Cellar/go/1.5.1/libexec/src/net/http/transport.go:976 +0xac7
created by net/http.(*Transport).dialConn
        /usr/local/Cellar/go/1.5.1/libexec/src/net/http/transport.go:685 +0xc78

goroutine 4027 [select]:
net/http.(*persistConn).writeLoop(0xc820413970)
        /usr/local/Cellar/go/1.5.1/libexec/src/net/http/transport.go:1009 +0x40c
created by net/http.(*Transport).dialConn
        /usr/local/Cellar/go/1.5.1/libexec/src/net/http/transport.go:686 +0xc9d

goroutine 4026 [IO wait]:
net.runtime_pollWait(0x2100f18, 0x72, 0xc8200122e0)
        /usr/local/Cellar/go/1.5.1/libexec/src/runtime/netpoll.go:157 +0x60
net.(*pollDesc).Wait(0xc82001a3e0, 0x72, 0x0, 0x0)
        /usr/local/Cellar/go/1.5.1/libexec/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc82001a3e0, 0x0, 0x0)
        /usr/local/Cellar/go/1.5.1/libexec/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc82001a380, 0xc8202be800, 0x400, 0x400, 0x0, 0xfa0050, 0xc8200122e0)
        /usr/local/Cellar/go/1.5.1/libexec/src/net/fd_unix.go:232 +0x23a
net.(*conn).Read(0xc820030038, 0xc8202be800, 0x400, 0x400, 0x0, 0x0, 0x0)
        /usr/local/Cellar/go/1.5.1/libexec/src/net/net.go:172 +0xe4
crypto/tls.(*block).readFromUntil(0xc8203e22d0, 0x21011a8, 0xc820030038, 0x5, 0x0, 0x0)
        /usr/local/Cellar/go/1.5.1/libexec/src/crypto/tls/conn.go:455 +0xcc
crypto/tls.(*Conn).readRecord(0xc8204d6580, 0x63e817, 0x0, 0x0)
        /usr/local/Cellar/go/1.5.1/libexec/src/crypto/tls/conn.go:540 +0x2d1
crypto/tls.(*Conn).Read(0xc8204d6580, 0xc820261000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        /usr/local/Cellar/go/1.5.1/libexec/src/crypto/tls/conn.go:901 +0x167
net/http.noteEOFReader.Read(0x210df88, 0xc8204d6580, 0xc8204139c8, 0xc820261000, 0x1000, 0x1000, 0xc820222840, 0x0, 0x0)
        /usr/local/Cellar/go/1.5.1/libexec/src/net/http/transport.go:1370 +0x67
net/http.(*noteEOFReader).Read(0xc8203beee0, 0xc820261000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        <autogenerated>:126 +0xd0
bufio.(*Reader).fill(0xc82047d7a0)
        /usr/local/Cellar/go/1.5.1/libexec/src/bufio/bufio.go:97 +0x1e9
bufio.(*Reader).Peek(0xc82047d7a0, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
        /usr/local/Cellar/go/1.5.1/libexec/src/bufio/bufio.go:132 +0xcc
net/http.(*persistConn).readLoop(0xc820413970)
        /usr/local/Cellar/go/1.5.1/libexec/src/net/http/transport.go:876 +0xf7
created by net/http.(*Transport).dialConn
        /usr/local/Cellar/go/1.5.1/libexec/src/net/http/transport.go:685 +0xc78

goroutine 4023 [select]:
net/http.(*persistConn).roundTrip(0xc820413970, 0xc82046a0a0, 0x0, 0x0, 0x0)
        /usr/local/Cellar/go/1.5.1/libexec/src/net/http/transport.go:1164 +0xb0d
net/http.(*Transport).RoundTrip(0xc8201bad80, 0xc82035a620, 0xb8, 0x0, 0x0)
        /usr/local/Cellar/go/1.5.1/libexec/src/net/http/transport.go:235 +0x530
net/http.send(0xc82035a620, 0xfa44b0, 0xc8201bad80, 0xc82047c660, 0x0, 0x0)
        /usr/local/Cellar/go/1.5.1/libexec/src/net/http/client.go:220 +0x52c
net/http.(*Client).send(0xc82043ad20, 0xc82035a620, 0x5e, 0x0, 0x0)
        /usr/local/Cellar/go/1.5.1/libexec/src/net/http/client.go:143 +0x15a
net/http.(*Client).doFollowingRedirects(0xc82043ad20, 0xc82035a620, 0x63e390, 0x0, 0x0, 0x0)
        /usr/local/Cellar/go/1.5.1/libexec/src/net/http/client.go:380 +0xbc0
net/http.(*Client).Do(0xc82043ad20, 0xc82035a620, 0x2246b1, 0x0, 0x0)
        /usr/local/Cellar/go/1.5.1/libexec/src/net/http/client.go:175 +0x9c
github.com/ncw/swift.(*Connection).doTimeoutRequest.func1(0xc820138120, 0xc82035a620, 0xc82047c480)
        /Users/chrism/gocode/src/github.com/ncw/swift/swift.go:222 +0x36
created by github.com/ncw/swift.(*Connection).doTimeoutRequest
        /Users/chrism/gocode/src/github.com/ncw/swift/swift.go:224 +0x8b

Implement Plain file downloading

Currently dropship only supports Tarball installation and it would be nice to be install for example just a single binary file. this should be super simple to implement.

Docker registry support

I think it would be super useful to be able to pull and update docker containers using dropship. it could possibly even reload containers that it detects are running that image.

Implement locker support for redis

Currently Consul is only supported for perform sequential updates. would be nice if we could accomplish this with redis as well.

Add tool to verify dropship.yml

It would be awesome to have a tool that can validate/run your dropship.yml from a cli on a developer machine. This would greatly reduce the commit->deploy->check workflow that most hook systems have.

Add validation command

It would be awesome to be able to validate service definitions.

Implement locker support for etcd

Currently Consul is only supported for perform sequential updates. would be nice if we could accomplish this with etcd as well.

onFailure, onSuccess hooks

A friend of mine asked for the ability to run hooks on failure or on success (currently they only run on success).

Release, Build, and Testing

Currently I am the only active developer on this project and so I have tended not to get crazy with automation initially so I build, test and release somewhat manually through my own machine which is not the best. I would like to use Werckers new workflows feature and setup the system to do all these things for me!

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.