orthros / docker-gen

Generate files from docker container meta-data

docker-gen

docker-gen is a file generator that renders templates using docker container meta-data.

It can be used to generate various kinds of files for:

• Centralized logging - fluentd, logstash or other centralized logging tools that tail the containers JSON log file or files within the container.
• Log Rotation - logrotate files to rotate container JSON log files
• Reverse Proxy Configs - nginx, haproxy, etc. reverse proxy configs to route requests from the host to containers
• Service Discovery - Scripts (python, bash, etc..) to register containers within etcd, hipache, etc..

===

Installation

There are three common ways to run docker-gen:

• on the host
• bundled in a container with another application
• separate standalone containers

Host Install

Linux binaries for release 0.3.5

• amd64
• i386

Download the version you need, untar, and install to your PATH.

$ wget https://github.com/jwilder/docker-gen/releases/download/0.3.5/docker-gen-linux-amd64-0.3.5.tar.gz
$ tar xvzf docker-gen-linux-amd64-0.3.5.tar.gz
$ ./docker-gen

Bundled Container Install

Docker-gen can be bundled inside of a container along-side and applications.

jwilder/nginx-proxy trusted build is an example of running docker-gen within a container along-side nginx. jwilder/docker-register is an example or running docker-gen within a container to do service registration with etcd.

Separate Container Install

It can also be run as two separate containers using the jwilder/docker-gen image virtually any other image.

This is how you could run the official nginx image and have dockgen-gen generate a reverse proxy config in the same way that nginx-proxy works. You may want to do this to prevent having the docker socket bound to an publicly exposed container service.

Start nginx with a shared volume:

$ docker run -d -p 80:80 --name nginx -v /tmp/nginx:/etc/nginx/conf.d -t nginx

Fetch the template and start the docker-gen container with the shared volume:

$ mkdir -p /tmp/templates && cd /tmp/templates
$ curl -o nginx.tmpl https://raw.githubusercontent.com/jwilder/docker-gen/master/templates/nginx.tmpl
$ docker run -d --name nginx-gen --volumes-from nginx \
   -v /var/run/docker.sock:/tmp/docker.sock \
   -v /tmp/templates:/etc/docker-gen/templates \
   -t jwilder/docker-gen:0.3.4 -notify-sighup nginx -watch --only-published /etc/docker-gen/templates/nginx.tmpl /etc/nginx/conf.d/default.conf

===

Usage

$ docker-gen
Usage: docker-gen [-config file] [-watch=false] [-notify="restart xyz"] [-notify-sighup="nginx-proxy"] [-interval=0] [-endpoint tcp|unix://..] <template> [<dest>]

Options:

  -config="": Use the specified config file instead of command-line options.  Multiple templates can be defined and
              they will be executed in the order that they appear in the config file.
  -endpoint="": docker api endpoint [tcp|unix://..]. This can also be set w/ a `DOCKER_HOST` environment.
  -interval=0:run notify command interval (s). Useful for service registration use cases.
  -notify="": run command after template is regenerated ["restart xyz"]. Useful for restarting nginx,
              reloading haproxy, etc..
  -notify-sighup="": send HUP signal to container.  Equivalent to `docker kill -s HUP container-ID`
  -only-exposed=false: only include containers with exposed ports
  -only-published=false: only include containers with published ports (implies -only-exposed)
  -version=false: show version
  -watch=false: run continuously and monitors docker container events.  When containers are started
                or stopped, the template is regenerated.

If no <dest> file is specified, the output is sent to stdout. Mainly useful for debugging.

===

Templating

The templates used by docker-gen are written using the Go text/template language. In addition to the built-in functions supplied by Go, docker-gen provides a number of additional functions to make it simpler (or possible) to generate your desired output.

Within those templates, the object emitted by docker-gen will have this structure.

Functions

• contains $map $key: Returns true if $map contains $key. Takes maps from string to string.
• exists $path: Returns true if $path refers to an existing file or directory. Takes a string.
• groupBy $containers $fieldPath: Groups an array of RuntimeContainer instances based on the values of a field path expression $fieldPath. A field path expression is a dot-delimited list of map keys or struct member names specifying the path from container to a nested value, which must be a string. Returns a map from the value of the field path expression to an array of containers having that value. Containers that do not have a value for the field path in question are omitted.
• groupByMulti $containers $fieldPath $sep: Like groupBy, but the string value specified by $fieldPath is first split by $sep into a list of strings. A container whose $fieldPath value contains a list of strings will show up in the map output under each of those strings.
• split $string $sep: Splits $string into a slice of substrings delimited by $sep. Alias for strings.Split
• replace $string $old $new $count: Replaces up to $count occurences of $old with $new in $string. Alias for strings.Replace
• dict $key $value ...: Creates a map from a list of pairs. Each $key value must be a string, but the $value can be any type (or nil). Useful for passing more than one value as a pipeline context to subtemplates.
• sha1 $string: Returns the hexadecimal representation of the SHA1 hash of $string.
• json $value: Returns the JSON representation of $value as a string.
• last $array: Returns the last value of an array.

===

Examples

• Automated Nginx Reverse Proxy for Docker
• Docker Log Management With Fluentd
• Docker Service Discovery Using Etcd and Haproxy

NGINX Reverse Proxy Config

jwilder/nginx-proxy trusted build.

Start nginx-proxy:

$ docker run -d -p 80:80 -v /var/run/docker.sock:/tmp/docker.sock -t jwilder/nginx-proxy

Then start containers with a VIRTUAL_HOST env variable:

$ docker run -e VIRTUAL_HOST=foo.bar.com -t ...

If you wanted to run docker-gen directly on the host, you could do it with:

$ docker-gen -only-published -watch -notify "/etc/init.d/nginx reload" templates/nginx.tmpl /etc/nginx/sites-enabled/default

Fluentd Log Management

This template generate a fluentd.conf file used by fluentd. It would then ships log files off the host.

$ docker-gen -watch -notify "restart fluentd" templates/fluentd.tmpl /etc/fluent/fluent.conf

Service Discovery in Etcd

This template is an example of generating a script that is then executed. This tempalte generates a python script that is then executed which register containers in Etcd using it's HTTP API.

$ docker-gen -notify "/bin/bash /tmp/etcd.sh" -interval 10 templates/etcd.tmpl /tmp/etcd.sh

Development

This project uses glock for managing 3rd party dependencies. You'll need to install glock into your workspace before hacking on docker-gen.

$ git clone <your fork>
$ glock sync github.com/jwilder/docker-gen
$ make

TODO

• Add event status for handling start and stop events differently
• Add a way to filter out containers in templates