Comments (45)
As I tested, the 503 issue after the redeployment comes from weave network we are using. weave 1.5.2 (the version we are using) has a 20-30 seconds delay, resulting in 20-30 seconds 503 down time. The latest version 1.6 doesn't have such a problem. We are currently testing the new weave version, and will make a new release ASAP.
from dockercloud-haproxy.
@midito
no ETA yet, we are busing on testing it. We will release it as soon as it is ready.
from dockercloud-haproxy.
I thought it was just me, then found this. This really needs to become top priority for this repo, as redeploys can happen often, and they shouldn't break sites.
from dockercloud-haproxy.
Is anyone else also seeing redeploys that don't come back after 10-15 seconds? Meaning you need to redeploy HAProxy itself?
from dockercloud-haproxy.
Running into this issue.
When I added another service and VIRTUAL_HOST
on two different domains.
from dockercloud-haproxy.
This issue is really taking a toll on our setup. Is there an eta? Can we help test?
from dockercloud-haproxy.
@modius I don't know the exact date. The plan to activate it was before the end of this month.
from dockercloud-haproxy.
Sorry to pester but is there any news on this issue?
from dockercloud-haproxy.
The support for v1.11 with the weave plugin integration is available: https://forums.docker.com/t/docker-cloud-1-3-release-notes/19618
With this upgrade, the 503 issue should be mitigated
from dockercloud-haproxy.
Yeah I'm not sure its fixed this issue. For over a minute after a redeploy HAProxy was reporting 'Layer 4 Timeout' (the instance was up and serving, tested via curl) - Could it be a case of tweaking the check options?
from dockercloud-haproxy.
We are facing the same issue since we updated to 1.11.1-cs1
from dockercloud-haproxy.
It seems that the redeploy was too fast and when haproxy tried to reload the configuration nothing changed (because the new container had the same configuration and the same IP).
Did the 503 disappear after a while, or it didn't? Did you try to redeploy haproxy? Did that fix the issue?
from dockercloud-haproxy.
The issue disappears by it self after a minute. I have redeployed HAProxy multiple times on all tags available without any change in this behavior.
It looks like I have mitigated this temporarily by having two instances of every service with sequential deploy.
from dockercloud-haproxy.
@revett
Does the haproxy configuration not update when you added the service, or it updates but you get a 503 error?
from dockercloud-haproxy.
I've just had it happen upon a redeploy, HAProxy stats says 'Layer4 connection problem: Connection refused' which results in a 503. The container log doesn't indicate any problem and seems to read the same as every other time a service is redeployed. The issue does disappear after about a minute, I don't think anything gets logged in this timeframe.
from dockercloud-haproxy.
@tifayuki this causes the 503 error:
ha-proxy:
image: 'dockercloud/haproxy:1.2.1'
deployment_strategy: every_node
links:
- hq
- platform
ports:
- '80:80'
restart: always
roles:
- global
tags:
- staging
- platform
hq:
image: 'dockercloud/hello-world:latest'
environment:
- VIRTUAL_HOST=hq.staging.example.co
expose:
- '80'
restart: always
tags:
- staging
- platform
target_num_containers: 1
platform:
image: 'dockercloud/hello-world:latest'
environment:
- VIRTUAL_HOST=app.staging.example.co
expose:
- '80'
restart: always
tags:
- staging
- platform
target_num_containers: 1
Any ideas?
This stack.yml
will successfully route requests from both subdomains (hq.
+ app.
) to the two different services as expected:
ha-proxy:
image: 'dockercloud/haproxy:1.2.1'
deployment_strategy: every_node
links:
- hq
- platform
ports:
- '80:80'
restart: always
roles:
- global
tags:
- staging
- platform
hq:
image: 'dockercloud/hello-world:latest'
expose:
- '80'
restart: always
tags:
- staging
- platform
target_num_containers: 1
platform:
image: 'dockercloud/hello-world:latest'
expose:
- '80'
restart: always
tags:
- staging
- platform
target_num_containers: 1
How is VIRTUAL_HOST
breaking it?
from dockercloud-haproxy.
Exactly same problem here.
EDIT: Mine was a problem with parsing of the stack.yml. When I created containers using docker cloud create services wizard, my VIRTUAL_HOST variable was inserted in the yml with single quotes around both sides of the attribution. When I removed these single quotes, it worked.
from dockercloud-haproxy.
Hmm... It's not clear to me whether my issue fits here, or is separate, but here goes..
My docker-compose.yml
looks like below. Under this configuration, I get 503
for everything.
lb:
image: dockercloud/haproxy
links:
- entities
- sentiment
- topics
environment:
- DOCKER_TLS_VERIFY
- DOCKER_HOST
- DOCKER_CERT_PATH
volumes:
- $DOCKER_CERT_PATH:$DOCKER_CERT_PATH
ports:
- 8080:80
entities:
extends:
file: common.yml
service: microservice
build:
context: services/entities/
environment:
- VIRTUAL_HOST="*/entities, */entities/*"
command: "python main.py"
sentiment:
extends:
file: common.yml
service: microservice
build:
context: services/sentiment/
environment:
- VIRTUAL_HOST="*/sentiment, */sentiment/*"
command: "python main.py"
topics:
extends:
file: common.yml
service: microservice
build:
context: services/topics/
environment:
- VIRTUAL_HOST="*/topics, */topics/*"
command: "python main.py"
If I remove the VIRTUAL_HOST
for each service, I can query the haproxy server at / (with no path) and it cycles between each service in a round-robin style.
Any ideas?
from dockercloud-haproxy.
Hmm... I think I may have solved it. I think it was due to incorrect VIRTUAL_HOST
format... Here is my *.yml with working service direction:
lb:
image: dockercloud/haproxy
links:
- entities
- sentiment
- topics
environment:
- DOCKER_TLS_VERIFY
- DOCKER_HOST
- DOCKER_CERT_PATH
volumes:
- $DOCKER_CERT_PATH:$DOCKER_CERT_PATH
ports:
- 8080:80
entities:
extends:
file: common.yml
service: microservice
build:
context: services/entities/
environment:
- 'VIRTUAL_HOST=*/entities, */entities/*'
command: "python main.py"
sentiment:
extends:
file: common.yml
service: microservice
build:
context: services/sentiment/
environment:
- 'VIRTUAL_HOST=*/sentiment, */sentiment/*'
command: "python main.py"
topics:
extends:
file: common.yml
service: microservice
build:
context: services/topics/
environment:
- 'VIRTUAL_HOST=*/topics, */topics/*'
command: "python main.py"
EDIT: just adding the detail that I had to make sure my microservices actually had those endpoints (e.g. the topics service had a /topics endpoint ... )
from dockercloud-haproxy.
Hm. Putting single quotes around environment statements doesn't look right at all.
from dockercloud-haproxy.
Didn't seem intuitive to me - but it works. Any idea why the original method does not work?
from dockercloud-haproxy.
Sorry, no.
from dockercloud-haproxy.
I'm having the same problem and it's so random. It happens some times at redeploy. Taking a look to the console, it appears that some VIRTUAL_HOST aren't loaded in the configuration, so when trying to access to that host you get a 503.
What I don't know is why aren't the VIRTUAL_HOST loaded some times.
from dockercloud-haproxy.
Looks like it has something to do with the overlay network. When you redeploy a service, it takes some time for our network daemon to attach the container to the network, which will cause 503 error for a short period of time.
Currently, we are implementing and testing to use docker plugin for the setup of the overlay network, which means the container will be attached to the overlay network before it gets started. This will help mitigate the 503 issue.
from dockercloud-haproxy.
Thanks @tifayuki
Any ETA on this?
from dockercloud-haproxy.
@tifayuki Has the plugin been implemented yet?
from dockercloud-haproxy.
@samuelgiles
Yes. The code is done, but not activated yet. It will be available soon.
from dockercloud-haproxy.
@tifayuki Thats great to hear, thanks for updating us π
from dockercloud-haproxy.
Somewhat surprised the repo is in docker cloud jumpstart as the only choice of proxy section but unstable like that.
I switched to https://github.com/madwire/dockercloud-nginx-proxy which works perfectly.. even without much documentation to go on...
If you have multiple domains, just remember instead using "," separator, using "space".
from dockercloud-haproxy.
@tifayuki Any updates on this? I'm encountering the same issue now that I've replaced all of our ELB instances with HAproxy services on Docker Cloud.
from dockercloud-haproxy.
@tifayuki thanks for the update but these newly deployed instances which are experiencing the issues are running 1.11.1-cs1, build bfd1f99
.
from dockercloud-haproxy.
No issues here so far, fingers crossed.
from dockercloud-haproxy.
@samuelgiles My 503s don't last that long - I'm using 50s timeouts with a 1 rise, 2 fall check. Still, I would really like to resolve this issue and now get 503s every redeploy.
from dockercloud-haproxy.
@Dids Given you are not having issues, would you mind posting a gist of your config (with any sensitive info removed). Are you using VIRTUAL_HOST
variables in your services?
from dockercloud-haproxy.
@cliffom Spoke too soon, still happening, but seems to be entirely random, and only happens every now than (maybe 1/10 redeploys?).
Also yes, I'm using VIRTUAL_HOST
.
from dockercloud-haproxy.
@Dids sorry to hear that. It happens on 100% of our deploys. @tifayuki what are the proper next steps for us to get some support on this?
from dockercloud-haproxy.
Came across this issue as well! My virtual_host variable was the problem I fixed it by adding an extra entry (without the protocol) into the VIRTUAL_HOST
.
e.g. Changing
'VIRTUAL_HOST=https://sub.example.com'
to
'VIRTUAL_HOST=https://sub.example.com, sub.example.com'
hope this helps someone!
from dockercloud-haproxy.
I can confirm this. After a redeployment (or, in my case, when doing blue/green deployments and changing the linked_to_service
property afterwards), I get 503 errors for around 10-15 seconds on every deployment.
This is frankly quite disappointing (especially since this issue has been opened 6 months ago), as I was hoping to use docker cloud (in combination with dockercloud-haproxy) as a zero-downtime solution.
from dockercloud-haproxy.
We are having this problem also, only since updating to 1.11.1-cs1
When is this fix getting released?
from dockercloud-haproxy.
We are still investigating but so far it seems that it's due to the new containers not being immediately available when a redeploy happens. If you use sequential deployment and have enough containers for the new containers to start accepting requests while the redeploy is happening, this should not be an issue.
from dockercloud-haproxy.
Thanks for the update, @tifayuki. I'm really looking forward to that update.
from dockercloud-haproxy.
Second the big thank you to @tifayuki. We are also experiencing these issues using DockerCloud on 1.11.1-cs1. Please let us know if we can help with testing.
When the nodes are recognized as up from HAP, they seem to perform well. Sometimes, we see nodes take minutes or hours to make it back into a healthy state when redeployed. We have 8 node instances behind HAP 1.4.1 with about 40 requests a second.
from dockercloud-haproxy.
Do we know when 1.6 is going to be live? Let us know if we could help in testing as we're frequently utilising the 'automatic redeploy' feature in DockerCloud several times a day so we're bumping into this issue every day.
from dockercloud-haproxy.
The upgrade to weave 1.6 is available :)
https://forums.docker.com/t/docker-cloud-release-notes-09-27-2016/22655
from dockercloud-haproxy.
From redeploying a couple of applications today I can say this is working really well! From 'redeploy' to functioning website is 1-2 seconds, most of which is the application itself starting up.
from dockercloud-haproxy.
Related Issues (20)
- Unnecessary redirect from https to https scheme in SSL termination mode HOT 2
- TCP load balancing doesn't work in Docker Swarm Mode
- Basic auth on a single service? HOT 2
- DNS resolution of loggers at startup
- Still experimental in Swarm Mode? HOT 8
- You should be able to indicate explicit Service/Port combos.
- Source balancing does not work on Docker for Mac
- Some question about multi-ports balance and ADDITIONAL_SERVICES
- Is this project dead? HOT 16
- How can i achieve the desired result with NGINX? Anybody else knows?
- Not working properly when using compose mode + service with fixed container names HOT 1
- Swarm service deploy mode = global is not supported
- Put stats on Virtual Host on a port 80 / 443
- Service Unavailable (503) Custom HTML
- Support option for this application
- We have a requirment of altering url for backend service at haproxy, how can we achieve
- tcpη1mθΆ ζΆζδΉθ§£ε³
- Questions about the mysterious project deprecation HOT 1
- imcompatible with docker compose 1.23 HOT 5
- Can not link service or container in compose mode HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. πππ
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google β€οΈ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dockercloud-haproxy.