Can't seem to run HAProxy about containerpilot

[tgross]

Hi @mbbender! The support team sent me your ticket from Zendesk so I've looked at the linked Bitbucket repo from there to try and figure out what's up.

I've pulled the container image mbbender/haproxy (which is the :latest tag) and it does not have Containerbuddy installed at /opt/containerbuddy. But in your docker-compose files, you're using the image mbbender/haproxy:cb, which does have Containerbuddy installed and starts up HAProxy as expected.

I'm going to copy the relevant section of your Zendesk ticket here too if you don't mind.

Not sure if you support this shim through this support interface or not, but if so...
I got the example to run on Triton just fine for the most part. In trying to adapt it to use HAProxy and PHP. For some reason my services don't seem to be reporting themselves to consul as the example app did. Is it possible you could take a peak at my Dockerfiles to see why?
The Command line for haproxy was moved into the local-docker-compose file (I did have it baked into the build and that didn't work either). The app image has it in the hello-world Dockerfile.
The services seem to be running fine but container buddy doesn't seem to have informed consul of their existence.

In order for Containerbuddy to tell Consul that HAProxy is healthy, it you need to have a working health directive. In this line you're making an HTTP GET on /health.txt but haven't configured that end point for HAProxy so it'll always fail health checks.

Thanks for checking out Containerbuddy and please let me know if these changes work for you.

[mbbender]

Yeah sorry that was just a typo from local testing (I haved pushed up the latest image to docker hub that had containerbuddy on it). I'm definitely using a container that has containerbuddy installed. I had it download and put it in the /bin directory and the tried including it in the copied /opt/containerbuddy folder per the examples and run it from there instead but no difference. I've also tried with absolute paths to everything. I think the challenge is just trying to get container buddy to launch and report HAProxy base image. No matter what I try I can't seem to get it to work.

[tgross]

@mbbender can you push that :latest image again? I just tried this myself:

$ docker pull mbbender/haproxy
Using default tag: latest
latest: Pulling from mbbender/haproxy
81ea8837dab7: Already exists
3b5671666ac3: Already exists
aac3775d15b2: Already exists
3ab442ed0fa5: Already exists
dc8b732678ec: Already exists
fab84cd96083: Already exists
66752d611115: Already exists
0d278595e9a0: Already exists
a8d62e9f0856: Already exists
Digest: sha256:587eee11c66c93e7ab0aceac5502892f53d960a5eceda297e9210a110763eaeb
Status: Image is up to date for mbbender/haproxy:latest

$ docker run --rm mbbender/haproxy /opt/containerbuddy/containerbuddy -config file:///opt/containerbuddy/haproxy.json /usr/local/sbin/haproxy -f /usr/local/etc/haproxy/haproxy.cfg
Error response from daemon: Cannot start container b16e3daf2e222912b26da96fe3d044ca3b217b8796efb34ff934b2cdba428767: [8] System error: exec: "/opt/containerbuddy/containerbuddy": stat /opt/containerbuddy/containerbuddy: no such file or directory

$ docker run -it mbbender/haproxy /bin/bash
root@dac5fb956948:/# ls -lah /opt/
total 8.0K
drwxr-xr-x  2 root root 4.0K Oct 22 17:09 .
drwxr-xr-x 38 root root 4.0K Nov  6 20:23 ..
root@dac5fb956948:/#

[tgross]

Oh, I see what the problem is. When I pulled your mbbender/haproxy:cb, the Containerbuddy service isn't in the image:

$ docker run -it mbbender/haproxy:cb /bin/bash
root@3f8ec86e6032:/# ls -lah /opt/
total 12K
drwxr-xr-x  3 root root 4.0K Nov  5 19:51 .
drwxr-xr-x 48 root root 4.0K Nov  6 20:29 ..
drwxr-xr-x  2 root root 4.0K Nov  5 19:51 containerbuddy
root@3f8ec86e6032:/# ls -lah /opt/containerbuddy/
total 16K
drwxr-xr-x 2 root root 4.0K Nov  5 19:51 .
drwxr-xr-x 3 root root 4.0K Nov  5 19:51 ..
-rw-r--r-- 1 root root  363 Nov  5 17:15 haproxy.json
-rw-r--r-- 1 root root  509 Nov  5 17:16 reload-haproxy.sh
root@3f8ec86e6032:/#

In this line you've installed it to /bin and not /opt/containerbuddy. So we need to do:

docker run -d mbbender/haproxy:cb /bin/containerbuddy -config file:///opt/containerbuddy/haproxy.json /usr/local/sbin/haproxy -f /usr/local/etc/haproxy/haproxy.cfg

[mbbender]

I pushed lastest. Here is the dockerfile. I just took the HAProxy docker file for 1.6 and removed the default CMD in testing. This version includes containerbuddy in the /opt/containerbuddy location so you can run it like

docker run -it mbbender/haproxy /bin/bash

Running

/opt/containerbuddy/containerbuddy -config file://opt/containerbuddy/haproxy.json haproxy -f /usr/local/etc/haproxy/haproxy.cfg 

which is what I understand to be the right format to launch haproxy with containerbuddy runs for a few seconds then gives the error

fork/exec : no such file or directory

Dockerfile:

FROM debian:wheezy

RUN apt-get update && apt-get install -y libssl1.0.0 libpcre3 --no-install-recommends && rm -rf /var/lib/apt/lists/*

ENV HAPROXY_MAJOR 1.6
ENV HAPROXY_VERSION 1.6.2
ENV HAPROXY_MD5 d0ebd3d123191a8136e2e5eb8aaff039

# see http://sources.debian.net/src/haproxy/1.5.8-1/debian/rules/ for some helpful navigation of the possible "make" arguments
RUN buildDeps='curl gcc libc6-dev libpcre3-dev libssl-dev make' \
    && set -x \
    && apt-get update && apt-get install -y $buildDeps --no-install-recommends && rm -rf /var/lib/apt/lists/* \
    && curl -SL "http://www.haproxy.org/download/${HAPROXY_MAJOR}/src/haproxy-${HAPROXY_VERSION}.tar.gz" -o haproxy.tar.gz \
    && echo "${HAPROXY_MD5}  haproxy.tar.gz" | md5sum -c \
    && mkdir -p /usr/src/haproxy \
    && tar -xzf haproxy.tar.gz -C /usr/src/haproxy --strip-components=1 \
    && rm haproxy.tar.gz \
    && make -C /usr/src/haproxy \
        TARGET=linux2628 \
        USE_PCRE=1 PCREDIR= \
        USE_OPENSSL=1 \
        USE_ZLIB=1 \
        all \
        install-bin \
    && mkdir -p /usr/local/etc/haproxy \
    && cp -R /usr/src/haproxy/examples/errorfiles /usr/local/etc/haproxy/errors \
    && rm -rf /usr/src/haproxy \
    && apt-get purge -y --auto-remove $buildDeps

ENV consul_template_version 0.11.1
ENV CONTAINERBUDDY file:///opt/containerbuddy/haproxy.json

# Add consul-template v0.11.1
RUN apt-get update && \
    apt-get install -y \
    curl \
    unzip && \
    rm -rf /var/lib/apt/lists/*

# RUN wget --no-check-certificate https://releases.hashicorp.com/consul-template/0.11.1/consul-template_0.11.1_linux_amd64.zip -O /tmp/consul-template.zip
# RUN cd /tmp && unzip consul-template.zip && mv consul-template /usr/local/bin

RUN curl -Lo /tmp/consul_template_${consul_template_version}_linux_amd64.zip https://releases.hashicorp.com/consul-template/${consul_template_version}/consul-template_${consul_template_version}_linux_amd64.zip && \
    unzip /tmp/consul_template_${consul_template_version}_linux_amd64.zip && \
    mv consul-template /bin


# add containerbuddy and all our configuration
ADD opt/containerbuddy /opt/containerbuddy/
ADD etc/haproxy/haproxy.cfg /usr/local/etc/haproxy/haproxy.cfg

EXPOSE 80
EXPOSE 1936

[tgross]

You have the line ADD opt/containerbuddy /opt/containerbuddy, but the opt/containerbuddy directory in your repo doesn't contain the Containerbuddy binary. Have you downloaded that from the tarball on Github separately? If you look at the makefile for the example app you'll see that we're building Containerbuddy as part of the build process. If your app doesn't include that (which I think you're right in that it should not), you'll want to make sure you're downloading the Containerbuddy release in your Dockerfile.

I would include that in a makefile like I did for this triton-cloudflare project or in the Dockerfile like I did for this triton-touchbase project.

That all being said, I just pulled down your :latest build and I get success:

$ docker pull mbbender/haproxy
Using default tag: latest
latest: Pulling from mbbender/haproxy
a9db1fa797b9: Pull complete
038119f4c62a: Pull complete
f3fe013e82e2: Pull complete
9260cfd2c7f8: Pull complete
71496f96a938: Pull complete
9cf257d218c7: Pull complete
fc1b5664487c: Pull complete
db3247b8c258: Pull complete
27d60332b1c3: Pull complete
416a24d22a4d: Pull complete
bd2984a8a6fc: Pull complete
81ea8837dab7: Already exists
3b5671666ac3: Already exists
aac3775d15b2: Already exists
3ab442ed0fa5: Already exists
Digest: sha256:23b45e139bf6989e24fef940b7355271dccd3506674dcb0dabe0648993c3c423
Status: Downloaded newer image for mbbender/haproxy:latest

$ docker run -it mbbender/haproxy ls -lah /opt/containerbuddy
total 7.3M
drwxr-xr-x 2 root root 4.0K Nov  6 04:33 .
drwxr-xr-x 3 root root 4.0K Nov  6 04:33 ..
-rwxr-xr-x 1 root root 7.3M Nov  6 03:12 containerbuddy
-rw-r--r-- 1 root root  235 Nov  6 04:00 haproxy.json
-rw-r--r-- 1 root root  511 Nov  5 22:04 reload-haproxy.sh

$ docker run -d mbbender/haproxy:latest /opt/containerbuddy/containerbuddy -config file://opt/containerbuddy/haproxy.json haproxy -f /usr/local/etc/haproxy/haproxy.cfg
17e0edc56cebddf9e4e815aaccf90bfd726f90c5c75876af1721a408a7e5ca7b

$ docker ps
CONTAINER ID        IMAGE                 COMMAND                  CREATED             STATUS              PORTS               NAMES
a8e85ed27bd5        mbbender/haproxy:latest   "/bin/containerbuddy "   37 minutes ago      Up 37 minutes       80/tcp, 1936/tcp    pensive_bhaskara

[mbbender]

I did have one container version download containerbuddy on build but I was just trying lots of different things even if it seemed like it shouldn't matter.

37 mins... what kind of magic machine do you have?! Mine still exits with the same error. Can you run it on triton in joyent? I get the same error when trying to run within joyent as I do locally. Would be interested to see if it works or fails for you in the same place.

[mbbender]

@tgross Have you had a chance to try this out within the joyent enivornment? I'm using us-east-1 if that matters.

[mbbender]

Nice talk today!

I think my issue might be that I removed the health parameter. Did you include a health check when trying my examples? Do you know if it is required? I was messing around with another container and containerbuddy just now and it wasn't reporting so I added a health check and it is showing up in consul now.

[mbbender]

Adding a health check doesn't solve the issue for me.

[tgross]

Nice talk today!

Thanks! Sorry I left you hanging here for a bit; I had another conference to attend in SF last week and then my travels back home took a lot out of me.

I'm having a little trouble figuring out exactly what you're trying to run. The health check in the Containerbuddy configuration you have in that image but your runs above show you using :latest. When I pull both this is what I'm seeing:

$ docker images | grep mbbender
mbbender/haproxy                latest                bd2984a8a6fc        10 days ago         127.6 MB
mbbender/haproxy                cb                    2fec01c4be24        10 days ago         129.9 MB

Your docker-compose files are running mbbender/haproxy:cb. That version does not have Containerbuddy installed:

docker run -it mbbender/haproxy:cb ls -lah /opt/containerbuddy
total 16K
drwxr-xr-x 2 root root 4.0K Nov  5 19:51 .
drwxr-xr-x 3 root root 4.0K Nov  5 19:51 ..
-rw-r--r-- 1 root root  363 Nov  5 17:15 haproxy.json
-rw-r--r-- 1 root root  509 Nov  5 17:16 reload-haproxy.sh

The Containerbuddy configuration file in that image matches what's in master in your repo and looks like this:

{
  "consul": "consul:8500",
  "services": [
    {
      "name": "haproxy",
      "port": 80,
      "publicIp": true,
      "health": "/usr/bin/curl --fail -s http://localhost/health.txt",
      "poll": 10,
      "ttl": 25
    }
  ],
  "backends": [
    {
      "name": "app",
      "poll": 7,
      "onChange": "/opt/containerbuddy/reload-haproxy.sh"
    }
  ]
}

The health check you have here is not supported by your HAProxy configuration, so health checks will never pass and the service will never appear as healthy to Consul.

In the application server, your CMD line is problematic. You're creating a subshell there, daemonizing PHP, and then running Nginx in the foreground. I'm not sure which application Containerbuddy is being blocked on there. But if you're running applications like this you need to be running a supervisor or you're bound for trouble when one of them exits.

[misterbisson]

@mbbender any feedback on @tgross' suggestions? It's been a while and I don't want to let this issue go stale.

[misterbisson]

@mbbender: it's been a while without a reply, so I'm going close this one. Please re-open if this is still problematic, or open a new issue with any new bugs. We will be creating an image of our own with HAproxy eventually, but I can't say specifically when that will be available.