Server crashes when receiving little bit of load about tileserver

[rmarianski]

What error do you see when it crashes? The first thing I would look at is to see if it's got enough memory. Otherwise, I'd suggest either trying to pre-generate what you expect to serve, or caching aggressively.

[Martinlck]

Hello, hope you guys had a great weekend.
This is the error we are getting:
Error on request:
Traceback (most recent call last):
File "/home/ubuntu/env/lib/python2.7/site-packages/werkzeug/serving.py", line 209, in run_wsgi
execute(self.server.app)
File "/home/ubuntu/env/lib/python2.7/site-packages/werkzeug/serving.py", line 200, in execute
write(data)
File "/home/ubuntu/env/lib/python2.7/site-packages/werkzeug/serving.py", line 181, in write
self.wfile.write(data)

I'm using a t2.large on ec2 to host all of the service right now (pgsql and all dependencies) going to work in a more robust environment with nginx as reverse proxy and caching results plus separate machines for the db and a fleet of 3 or 4 web servers. But my real question, do you have any specs recommended? and if I want it to put it under heavy loads any suggestion? Thanks for your help appreciate it.

[zerebubuth]

We split our work up into several different service layers. The most important are:

• Tileserver running on r3.xlarge instances. These only process "on-demand" tiles which are not part of the "Tiles of Interest" set, which (currently) lists all the tiles which have been requested and we have copies of on S3.
• Tilequeue running on c4.xlarge instances. These process updates to the "Tiles of Interest", and so keeps S3 up-to-date so that those tiles can be served directly to clients using a service called Tapalcatl, which bypasses Tileserver and so helps reduce the number of requests.
• PostgreSQL running on db.r3.2xlarge instances. These are in an RDS cluster, and so separate from the Tileserver and Tilequeue machines.

Looking at the monitoring for the Tileserver machines, it looks like they're hovering around 27GB heap usage. This seems like a lot, but it's hard to tell how much of that is actually in use without forcing more garbage collections to take place.

[rmarianski]

We run tileserver through gunicorn, and set worker_processes to 150 for the r3.xlarge instances. With a smaller instance size, the number of worker processes running should be tweaked accordingly.

[Martinlck]

Thank you very much for your response @rmarianski @zerebubuth I will replicate your settings and I will conduct some stress testing.
I'll close this issue and re-visit it later with some results from the tests.

[zerebubuth]

Good luck!

We have a gitter channel, if you'd find it useful to chat.