Comments (4)
@drolando: nice find 🥇
That explains why we're seeing spurious 503 spikes. graceful_mesos_shutdown
is not the root cause as I initially thought, it's just what exposes this bug.
from casper.
So far I haven't able to solve this problem, but here's a repro:
make dev
curl -o /dev/null -iv -H 'X-Smartstack-Destination: yelp-main.internalapi' -H 'X-Smartstack-Source: spectre.main' -H 'X-Ctx-Tarpit: yelp-main.internalapi' 'localhost:32927/status'
. This triggers a slow request to internalapi (timeout of 15s.)- (in a different session):
docker ps | grep abrousse | cut -d' ' -f1 | xargs docker stop --time 30
. This shuts down the Casper container by sending SIGTERM first, then SIGKILL after 30s.
Expected:
The request to /status
finishes then nginx/docker shuts down gracefully
Actual:
Connection is aborted and we get "curl: (52) Empty reply from server", docker shuts down at the same time
Some things that might be useful to note here
- process hierarchy:
dumb-init start.sh
|___start.sh
|___nginx (master)
|___nginx (worker)
- From dumb-init's docs: "In its default mode, dumb-init establishes a session rooted at the child, and sends signals to the entire process group. This is useful if you have a poorly-behaving child (such as a shell script) which won't normally signal its children before dying."
- From nginx's docs: SIGTERM causes a fast shutdown, and SIGQUIT causes a graceful shutdown (that's what we want)
- To check that SIGQUIT was the right thing I hoped onto the container, fired up a slow request to
/status
and ran/usr/local/openresty/nginx/sbin/nginx -s quit
. The slow request finished, then nginx exited and the container shut down.
from casper.
We can use dumb-init to change the SIGTERM in a SIGQUIT
dumb-init --rewrite 15:3
from casper.
We can use dumb-init to change the SIGTERM in a SIGQUIT
Nice. Just verified, it totally works (CMD ["dumb-init", "--rewrite", "15:3", "/code/start.sh"]
)
from casper.
Related Issues (17)
- Cassandra fails to start with "Cassandra 2.0 and later require Java 7u25 or later" HOT 2
- "make dev" fails
- Add instructions on how to run cassandra locally and make casper use it
- id extraction for normal endpoints results in all cache misses
- Support for surrogate keys
- Support for "Vary" header
- headers with underscores are being dropped HOT 2
- API endpoint to expose TTL left for a key
- Support for caching POST requests
- Refactor cache_entry handling in cacheability_info map
- Have Casper set the X-Smartstack-Source header HOT 3
- Periodically refresh Cassandra cluster topology
- /status?check_cassandra=true should return the current topology HOT 1
- Some integration tests are flaky in Travis CI HOT 1
- Configuration option to let a portion of hit traffic through Casper
- Automatically create Cassandra keyspace and schema if they're not present
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from casper.