Set certificate from volume about dockercloud-haproxy

[mrmachine]

@modius I have also created a variant of dockercloud/haproxy and a companion letsencrypt image that automate creation and renewal of SSL certificates from Let's Encrypt - https://github.com/ixc/letsencrypt-dockercloud-haproxy

The haproxy image uses inotify-tools to monitor the /etc/letsencrypt/live directory from the letsencrypt image. When changes are detected, it installs combined certificates (privkey.pem + fullchain.pem) into the HAproxy /certs directory and reloads HAproxy.

The letsencrypt image will try to create or renew certificates for domains specified in a DOMAINS environment variable on startup, and daily thereafter. DOMAINS defines both certificates (separated by ;) and domains (separated by ,). E.g. DOMAINS=foo.com,www.foo.com;bar.com,www.bar.com would create two certificates, each covering two names.

I have included a sample stack file using the dockercloud/hello-world image.

[nickbreen]

Yes I did, more or less. I setup a combined certificate installer for LE:

• https://github.com/nickbreen/letsencrypt-combined-installer

and a docker-container for LE and the plugin:

• https://github.com/nickbreen/docker-letsencrypt-combined

It's been working happily for new certs with haproxy:v1.2.1, but I haven't tried adding new certs with v1.3 yet.

I also haven't had an automatic renew actually happen yet with any cert.

Example (fragment) from my docker-cloud.yml file:

le:
  image: nickbreen/letsencrypt-combined:v1.1.0
  volumes:
    - /etc/letsencrypt
    - /var/lib/letsencrypt
  volumes_from:
    - haproxy
    - www
  environment:
    CRON_D_LE: |
      @monthly root le renew 2>&1 | logger --tag le-renew

haproxy:
  image: dockercloud/haproxy:1.3
  ports:
    - "80:80"
    - "443:443"
  volumes:
    - /certs
  environment:
    CERT_FOLDER: /certs/
  links:
    - www
  roles:
    - global

www:
  image: nickbreen/wp-php:apache-v1.0.5
  environment:
    EXTRA_SETTINGS: >
      acl https dst_port 443,
      reqdel X-Forwarded-Proto,
      reqadd X-Forwarded-Proto:\ http unless https,
      reqadd X-Forwarded-Proto:\ https if https,

Also notice the EXTRA_SETTINGS which corrects the X-Forwarded-Proto value so that it's https only when the actual request was https as my backend, otherwise I'd need to set VIRTUAL_HOST=*,https://* which I find slightly less obvious to it's intent, and I want the backend to support a large-ish number of domains of which some are https.

[modius]

Are you thinking of using something like quay.io/letsencrypt/letsencrypt to generate certificates in /etc/letsencrypt/live on the host node? (see http://letsencrypt.readthedocs.org/en/latest/using.html#running-with-docker)

On DockerCloud would deploying this container via every_node enable you to maintain a scaling nodecluster?

Would love to see something like this running but an example Stack file for getting this going would be much appreciated.

[passcod]

Given the haproxy probably runs on a single node on most small-to-medium scale deploys, there's even no need for every_node, just point it to whatever node is running the haproxy instance (with deploy tag haproxy to be flexible, for example).

[nickbreen]

All the certificates are stored in /certs/.

I'm experimenting with using this as a volume and, for now, copying the fullchain.pem file(s) from /etc/letsencrypt/live/... to /certs/.

I think the --fullchain-path argument to letsencrypt could shortcut that copy.

I also used a self-signed certificate to trick haproxy into configuring itself for SSL termination. E.g.

haproxy:
  image: dockercloud/haproxy:1.2.1
  volumes:
      - /certs
  ports:
      - "80:80"
      - "443:443"
      - "1936:1936"
  environment:
      TIMEOUT: "connect 1000, client 5000, server 60000"
      STATS_AUTH: stats:stats
      EXTRA_SSL_CERTS: SELF_SSL_CERT
      SELF_SSL_CERT: |
        -----BEGIN PRIVATE KEY-----
        blah... 
        -----END PRIVATE KEY-----
        -----BEGIN CERTIFICATE-----
        blah...
        -----END CERTIFICATE-----
  links:
    - www

www:
  image: php:5-apache
  volumes:
   - /var/www/html
  environment:
    VIRTUAL_HOST: "*,https://*"

le:
  image: quay.io/letsencrypt/letsencrypt
  command: >
     certonly
      --email [email protected]
      --domains foo.example.com
      --no-self-upgrade
      --non-interactive
      --agree-tos
      --webroot
      --webroot-path /var/www/html
      --text
      --fullchain-path /certs/
  volumes:
    - "/etc/letsencrypt"
    - "/var/lib/letsencrypt"
  volumes_from:
    - www
    - haproxy

This doesn't solve the problem of scheduling renewals, etc.

[modius]

I also used a self-signed certificate to trick haproxy into configuring itself for SSL termination.

@nickbreen I'm a bit of an SSL noob -- what does that mean exactly?

[nickbreen]

See https://github.com/docker/dockercloud-haproxy#pem-files; generating a key and certificate using that method is called "self signing" (you're literally signing the certificate yourself, rather than have a trusted 'CA' sign it).

The referenced page tells a small lie, you can use the | character in the YAML stack/compose file to avoid having to faff around with new lines.

In my case I did this:

# Gerenate the key and certificate
openssl req -x509 -newkey rsa:2048 -keyout key.pem -out ca.pem -days 1080 -nodes -subj '/CN=*.example.com/O=Example Limited/C=NZ'
# Copy 'n' paste the output below into the environment variable
cat key.pem ca.pem

[passcod]

@modius To add to the explanation, this is a necessary because this haproxy image only enables SSL termination if one (or more) of the SSL environment variables is set. Setting it to a self-signed certificate triggers this easily, without needing to get a real certificate, and it doesn't matter that it's self-signed because HAProxy will never actually use it (it will use the ones in /certs).

[nickbreen]

It copes it into /certs/! Oddly, contrary to the docs, it seems that it stores it as /certs/cert0.pem which should be the the DEFAULT_SSL_CERT only.

I think it's OK, if I read the docs right, it'll prefer the most specific certificate for the domain. I've not quite got this working yet though.

This may be a red-herring as I'm currently having difficulties with this (https://forums.docker.com/t/redeploy-a-service-that-share-volumes/7034) issue.

[nickbreen]

Installing certificates directly in /certs/ is working for me, however haproxy does not 'see' the new certificates until redeployment of the service/container. /reload.sh does not seem to let it pick up the new certificates.

[passcod]

Redeploys are cheap enough and easy enough to automate that it's not a bad tradeoff, methinks.

[tifayuki]

This PR #23 should all you to mount certs from a volume

[nickbreen]

FYI

Note: when reload.sh is invoked, it doesn't necessarily mean that HAProxy will be restarted. In fact, dockercloud/haproxy will try to get the current information of the the service and calculate a new configuration. HAProxy will only be restarted when the newly generated configuration differs from the current one.

From the README.

Given that changes in the contents of /certs/ does not change the configuration haproxy is not reloaded!

[nickbreen]

I'd quite like to be able to force /reload.sh to send USR1 to the (proper) haproxy process. My two use cases would be, sort of in order of severity:

1. reload haproxy if required (configuration has changed) via USR1
2. reload haproxy (unconditionally) via USR1
3. restart haproxy
4. redeploy container/service

Currently my only options are 1 and 4 and naught betwixt the two (at least as documented).

[nickbreen]

I've just been invoking it with exec. E.g.

docker-cloud container exec haproxy-1 /reload.sh

I dug through the python for dockercloud-haproxy and it has some magic in there to determine if the configuration has changed and chooses to issue USR1 to the actual haproxy process.

---

It's also occurred to me, and what I'm currently experimenting with, is a Docker Cloud API client that will update the EXTRA_SSL_CERTS environment variable and add new certificates that way... which would be considered a configuration change.

[passcod]

Yeah, sorry, I deleted my comment when I started reading the source.

[passcod]

I think a better solution to this PR would be to amend the dockercloud-haproxy magic to also check if /certs/ has changed, and set ssl_updated = True in that case, which would let a normal USR1 reload to do the right thing.

This:

I'd quite like to be able to force /reload.sh to send USR1 to the (proper) haproxy process.

looks to me like a separate feature. A useful one, probably, but separate from this.

[passcod]

a Docker Cloud API client that will update the EXTRA_SSL_CERTS environment variable and add new certificates that way... which would be considered a configuration change.

Yes, that's what I'm currently doing. I have a script that runs every month, renews certs if necessary, then uses the docker-cloud CLI tool to update the relevant environment variable and reload the service. It feels like a hack, though.

[passcod]

I'm also thinking that it could be possible to use inotify to watch /certs/ inside the container, and automatically reload if its contents change. More advanced, perhaps, but now that would be truly magic.

[nickbreen]

... force /reload.sh ...

... separate feature ...

Agreed.

[modius]

@nickbreen did you say you had a beautiful stack running quay.io/letsencrypt/letsencrypt and haproxy without complaint? ie. should i be racing out to try and implement something in DockerCloud

Or is it still all a bit "experimental"?

[tifayuki]

I am going to reset Haproxy.cls_cfg on USR1 event before actually running the reload, and this will result in force reload whenever /reload.sh is invoked.

[tifayuki]

released in dockercloud/haproxy:1.3

[modius]

@nickbreen @passcod did you guys get this going with the LE container in an unmodified Docker Cloud host node? Does anyone have a working Stack file that works with hello-world or similar?