Service redeploy results in 503 error about dockercloud-haproxy

[tifayuki]

As I tested, the 503 issue after the redeployment comes from weave network we are using. weave 1.5.2 (the version we are using) has a 20-30 seconds delay, resulting in 20-30 seconds 503 down time. The latest version 1.6 doesn't have such a problem. We are currently testing the new weave version, and will make a new release ASAP.

[tifayuki]

@midito
no ETA yet, we are busing on testing it. We will release it as soon as it is ready.

[Dids]

I thought it was just me, then found this. This really needs to become top priority for this repo, as redeploys can happen often, and they shouldn't break sites.

[Dids]

Is anyone else also seeing redeploys that don't come back after 10-15 seconds? Meaning you need to redeploy HAProxy itself?

[revett]

Running into this issue.

When I added another service and VIRTUAL_HOST on two different domains.

[modius]

This issue is really taking a toll on our setup. Is there an eta? Can we help test?

[tifayuki]

@modius I don't know the exact date. The plan to activate it was before the end of this month.

[samuelgiles]

Sorry to pester but is there any news on this issue?

[tifayuki]

@cliffom

The support for v1.11 with the weave plugin integration is available: https://forums.docker.com/t/docker-cloud-1-3-release-notes/19618

With this upgrade, the 503 issue should be mitigated

[samuelgiles]

Yeah I'm not sure its fixed this issue. For over a minute after a redeploy HAProxy was reporting 'Layer 4 Timeout' (the instance was up and serving, tested via curl) - Could it be a case of tweaking the check options?

[h0lyalg0rithm]

We are facing the same issue since we updated to 1.11.1-cs1

[fermayo]

It seems that the redeploy was too fast and when haproxy tried to reload the configuration nothing changed (because the new container had the same configuration and the same IP).
Did the 503 disappear after a while, or it didn't? Did you try to redeploy haproxy? Did that fix the issue?

[Starefossen]

The issue disappears by it self after a minute. I have redeployed HAProxy multiple times on all tags available without any change in this behavior.

It looks like I have mitigated this temporarily by having two instances of every service with sequential deploy.

[tifayuki]

@revett
Does the haproxy configuration not update when you added the service, or it updates but you get a 503 error?

[samuelgiles]

I've just had it happen upon a redeploy, HAProxy stats says 'Layer4 connection problem: Connection refused' which results in a 503. The container log doesn't indicate any problem and seems to read the same as every other time a service is redeployed. The issue does disappear after about a minute, I don't think anything gets logged in this timeframe.

[revett]

@tifayuki this causes the 503 error:

ha-proxy:
  image: 'dockercloud/haproxy:1.2.1'
  deployment_strategy: every_node
  links:
    - hq
    - platform
  ports:
    - '80:80'
  restart: always
  roles:
    - global
  tags:
    - staging
    - platform

hq:
  image: 'dockercloud/hello-world:latest'
  environment:
    - VIRTUAL_HOST=hq.staging.example.co
  expose:
    - '80'
  restart: always
  tags:
    - staging
    - platform
  target_num_containers: 1

platform:
  image: 'dockercloud/hello-world:latest'
  environment:
    - VIRTUAL_HOST=app.staging.example.co
  expose:
    - '80'
  restart: always
  tags:
    - staging
    - platform
  target_num_containers: 1

Any ideas?

---

This stack.yml will successfully route requests from both subdomains (hq. + app.) to the two different services as expected:

ha-proxy:
  image: 'dockercloud/haproxy:1.2.1'
  deployment_strategy: every_node
  links:
    - hq
    - platform
  ports:
    - '80:80'
  restart: always
  roles:
    - global
  tags:
    - staging
    - platform

hq:
  image: 'dockercloud/hello-world:latest'
  expose:
    - '80'
  restart: always
  tags:
    - staging
    - platform
  target_num_containers: 1

platform:
  image: 'dockercloud/hello-world:latest'
  expose:
    - '80'
  restart: always
  tags:
    - staging
    - platform
  target_num_containers: 1

How is VIRTUAL_HOST breaking it?

[rafagsiqueira]

Exactly same problem here.

EDIT: Mine was a problem with parsing of the stack.yml. When I created containers using docker cloud create services wizard, my VIRTUAL_HOST variable was inserted in the yml with single quotes around both sides of the attribution. When I removed these single quotes, it worked.

Hmm... It's not clear to me whether my issue fits here, or is separate, but here goes..

My docker-compose.yml looks like below. Under this configuration, I get 503 for everything.

  lb:
    image: dockercloud/haproxy
    links:
      - entities
      - sentiment
      - topics
    environment:
      - DOCKER_TLS_VERIFY
      - DOCKER_HOST
      - DOCKER_CERT_PATH
    volumes:
      - $DOCKER_CERT_PATH:$DOCKER_CERT_PATH
    ports:
      - 8080:80

  entities:
    extends:
      file: common.yml
      service: microservice
    build:
      context: services/entities/
    environment:
      - VIRTUAL_HOST="*/entities, */entities/*"
    command: "python main.py"

  sentiment:
    extends:
      file: common.yml
      service: microservice
    build:
      context: services/sentiment/
    environment:
      - VIRTUAL_HOST="*/sentiment, */sentiment/*"
    command: "python main.py"

  topics:
    extends:
      file: common.yml
      service: microservice
    build:
      context: services/topics/
    environment:
      - VIRTUAL_HOST="*/topics, */topics/*"
    command: "python main.py"

If I remove the VIRTUAL_HOST for each service, I can query the haproxy server at / (with no path) and it cycles between each service in a round-robin style.

Any ideas?

Hmm... I think I may have solved it. I think it was due to incorrect VIRTUAL_HOST format... Here is my *.yml with working service direction:

  lb:
    image: dockercloud/haproxy
    links:
      - entities
      - sentiment
      - topics
    environment:
      - DOCKER_TLS_VERIFY
      - DOCKER_HOST
      - DOCKER_CERT_PATH
    volumes:
      - $DOCKER_CERT_PATH:$DOCKER_CERT_PATH
    ports:
      - 8080:80

  entities:
    extends:
      file: common.yml
      service: microservice
    build:
      context: services/entities/
    environment:
      - 'VIRTUAL_HOST=*/entities, */entities/*'
    command: "python main.py"

  sentiment:
    extends:
      file: common.yml
      service: microservice
    build:
      context: services/sentiment/
    environment:
      - 'VIRTUAL_HOST=*/sentiment, */sentiment/*'
    command: "python main.py"

  topics:
    extends:
      file: common.yml
      service: microservice
    build:
      context: services/topics/
    environment:
      - 'VIRTUAL_HOST=*/topics, */topics/*'
    command: "python main.py"

EDIT: just adding the detail that I had to make sure my microservices actually had those endpoints (e.g. the topics service had a /topics endpoint ... )

[kklepper]

Hm. Putting single quotes around environment statements doesn't look right at all.

Didn't seem intuitive to me - but it works. Any idea why the original method does not work?

[kklepper]

Sorry, no.

[midito]

I'm having the same problem and it's so random. It happens some times at redeploy. Taking a look to the console, it appears that some VIRTUAL_HOST aren't loaded in the configuration, so when trying to access to that host you get a 503.
What I don't know is why aren't the VIRTUAL_HOST loaded some times.

[tifayuki]

@midito

Looks like it has something to do with the overlay network. When you redeploy a service, it takes some time for our network daemon to attach the container to the network, which will cause 503 error for a short period of time.

Currently, we are implementing and testing to use docker plugin for the setup of the overlay network, which means the container will be attached to the overlay network before it gets started. This will help mitigate the 503 issue.

[midito]

Thanks @tifayuki
Any ETA on this?

[samuelgiles]

@tifayuki Has the plugin been implemented yet?

[tifayuki]

@samuelgiles
Yes. The code is done, but not activated yet. It will be available soon.

[samuelgiles]

@tifayuki Thats great to hear, thanks for updating us 👍

[cosmostail]

Somewhat surprised the repo is in docker cloud jumpstart as the only choice of proxy section but unstable like that.

I switched to https://github.com/madwire/dockercloud-nginx-proxy which works perfectly.. even without much documentation to go on...

If you have multiple domains, just remember instead using "," separator, using "space".

[cliffom]

@tifayuki Any updates on this? I'm encountering the same issue now that I've replaced all of our ELB instances with HAproxy services on Docker Cloud.

[cliffom]

@tifayuki thanks for the update but these newly deployed instances which are experiencing the issues are running 1.11.1-cs1, build bfd1f99.

[Dids]

No issues here so far, fingers crossed.

[cliffom]

@samuelgiles My 503s don't last that long - I'm using 50s timeouts with a 1 rise, 2 fall check. Still, I would really like to resolve this issue and now get 503s every redeploy.

[cliffom]

@Dids Given you are not having issues, would you mind posting a gist of your config (with any sensitive info removed). Are you using VIRTUAL_HOST variables in your services?

[Dids]

@cliffom Spoke too soon, still happening, but seems to be entirely random, and only happens every now than (maybe 1/10 redeploys?).
Also yes, I'm using VIRTUAL_HOST.

[cliffom]

@Dids sorry to hear that. It happens on 100% of our deploys. @tifayuki what are the proper next steps for us to get some support on this?

[aaronhayes]

Came across this issue as well! My virtual_host variable was the problem I fixed it by adding an extra entry (without the protocol) into the VIRTUAL_HOST.

e.g. Changing
'VIRTUAL_HOST=https://sub.example.com'

to

'VIRTUAL_HOST=https://sub.example.com, sub.example.com'

hope this helps someone!

[mvdstam]

I can confirm this. After a redeployment (or, in my case, when doing blue/green deployments and changing the linked_to_service property afterwards), I get 503 errors for around 10-15 seconds on every deployment.
This is frankly quite disappointing (especially since this issue has been opened 6 months ago), as I was hoping to use docker cloud (in combination with dockercloud-haproxy) as a zero-downtime solution.

CC @cliffom @tifayuki

[AndrewLugg]

We are having this problem also, only since updating to 1.11.1-cs1

When is this fix getting released?

[fermayo]

We are still investigating but so far it seems that it's due to the new containers not being immediately available when a redeploy happens. If you use sequential deployment and have enough containers for the new containers to start accepting requests while the redeploy is happening, this should not be an issue.

[cliffom]

Thanks for the update, @tifayuki. I'm really looking forward to that update.

[primeobsession]

Second the big thank you to @tifayuki. We are also experiencing these issues using DockerCloud on 1.11.1-cs1. Please let us know if we can help with testing.

When the nodes are recognized as up from HAP, they seem to perform well. Sometimes, we see nodes take minutes or hours to make it back into a healthy state when redeployed. We have 8 node instances behind HAP 1.4.1 with about 40 requests a second.

[samuelgiles]

Do we know when 1.6 is going to be live? Let us know if we could help in testing as we're frequently utilising the 'automatic redeploy' feature in DockerCloud several times a day so we're bumping into this issue every day.

[tifayuki]

@samalba

The upgrade to weave 1.6 is available :)
https://forums.docker.com/t/docker-cloud-release-notes-09-27-2016/22655

[samuelgiles]

From redeploying a couple of applications today I can say this is working really well! From 'redeploy' to functioning website is 1-2 seconds, most of which is the application itself starting up.

Thanks @tifayuki & @fermayo!