Comments (13)
Actually that workaround isn't safe, as it could cause wg
to go negative; obviously we don't know if a call to FinishRoutine()
is from a 'post-close reactivated idle conn' that shouldn't decrement the WaitGroup. I think the more fundamental changes to properly track and wait on / clean up idle connections during shutdown are the only solution here; let me know if there's something I can do to help there. As it is, we're seeing at least one WaitGroup race from manners
reported on about 5% of our CI test runs.
from manners.
I think this is broader than just reverse proxies. Firefox currently ships with a network.http.keep-alive.timeout
of 115 seconds, so it takes quite a while for the server to stop (possibly unbounded if a client never drops the connection?)
I think that the correct way to deal with this would be to set a low ReadTimeout
(either always, or after the Shutdown signal, if that's possible).
However, at a quick glance neither http.Server
nor the net.Connection
is exposed when using this package.
Rather than updating the README, is there something we can expose in the API to deal with this issue?
from manners.
Oops, just read through https://code.google.com/p/go/issues/detail?id=4674. Looks like this will be fixable in Go 1.3
from manners.
Fixed in 3.0.0.
from manners.
hi @lionelbarrow
not fixed for me, nothing in your code is killing idle connection (http.StateIdle), nor calling SetKeepAlivesEnabled(false).
For now i'm manually adding header Connection:close, so i just have to wait for the ongoing keepalive, but it's not perfect
from manners.
It is definitely not fixed.
I see some Connection reset by peer
in nginx error logs with this package, Wireshark shows RST packets sent in response for requests. I've read about this error. After looking and syscalls before my service gracefully exits it seems that my service doesn't close any connections, it just exits cleanly resulting ing RST.
I switched to github.com/tylerb/graceful and it works as expected, only raising Connection refused
errors in logs. This way nginx is completely sure that request wasn't served and retries it on different upstream.
Hope this helps.
from manners.
I'll take another look, sorry for closing prematurely.
from manners.
Any updates?
from manners.
Digging into this today.
I see the problem -- if the shutdown command is received and then a request on a keepalive connection finishes, we close the connection correctly. But if the connection is idle already, we won't do anything to it; we just shut down.
It seems the correct solution is for the library to hold onto the connection when it first is created and then close them all once it enters the final phase of shutdown. I'll play around with that, but it might be a bit involved, so I'll update the README with a warning in the mean time. Does that sound reasonable?
I don't think the behavior @coopernurse was describing can happen after we adopted the Go 1.3 TCP state function model.
from manners.
I don't think the behavior @coopernurse was describing can happen after we adopted the Go 1.3 TCP state function model.
Well, what I see now with Go 1.4.2 is that server.ListenAndServe()
(where server := manners.NewServer()
) does not return after server.Shutdown <- true
in configuration described in the first comment.
from manners.
@lionelbarrow - I'm not sure if you've had a chance to play around with the refactoring to accommodate idle connections during shutdown yet, but I suspect your changes there may eliminate, or influence the solution to, a race condition I've encountered. In a similar fashion, if an idle keepalive connection becomes active after shutdown, a data race will be reported by the race detector, since that connection's state change will attempt a wg.Add(1)
after the wg.Wait()
has occurred.
As a temporary workaround (trying to eliminate occasional race detector noise from our CI builds), we've added an atomic read of the closing
flag to StartRoutine()
- but the need to do that is further evidence that our manners aren't quite proper for idle connections ;)
from manners.
have this been solved already? I met a lot of negative WaitGroup counter
in my production environment. I am taking investigation about this now.
can you @allgeek show me any example which will result in wg to go negative?
from manners.
today, In Go 1.9.4, Manners does not correctly shut down long-lived keepalive connections when issued a shutdown command yet! so how can i do?
from manners.
Related Issues (20)
- race condition in tests HOT 3
- race condition in listener.go HOT 1
- doesn't shutdown when used with clients using persistent connections HOT 2
- Multiple calls to Close will panic HOT 1
- 1.0.0 Release?
- Please merge non-backwards-compat HOT 2
- TestShutdown intermittently fails HOT 4
- manners doesn't set http.Request.TLS HOT 1
- A blocking version of Close() HOT 2
- [retracted] HOT 1
- Allow /debug endpoints be access while service shutting down HOT 1
- Serving + Closing in 2 separate routines is racy HOT 2
- Panic... HOT 2
- How about websockets? HOT 1
- Signal example should use syscall.SIGTERM instead of os.Kill
- Fix ListenAndServe(addr, nil)
- ListenAndServeTLS and letsencrypt
- Can you please allow GracefulListener to be constructed outside of manners package? HOT 2
- How the channel would get the shutdown message? HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from manners.