Comments (10)
Her you say:
- the shared memory (shm-size) setting of the docker container needs to be increased, because the default (64 MB) isn't big enough for postgres (128 MB).
- use docker-compose.
I cant seem to get this working?
in docker compose I do this:
version: '3'
services:
tileserver:
container_name: tileserver
image: overv/openstreetmap-tile-server
volumes:
- '/data/docker/tileserver/merged.osm.pbf:/data.osm.pbf'
- 'openstreetmap-data:/var/lib/postgresql/10/main'
environment:
shm_size: '256M'
command: import
volumes:
openstreetmap-data:
external: true
But inside the container df -hT /dev/shm still says:
root@2dc9518b5eb9:/home/renderer/src/openstreetmap-carto# df -hT /dev/shm
Filesystem Type Size Used Avail Use% Mounted on
shm tmpfs 64M 8.0K 64M 1% /dev/shm
Any suggestions?
from openstreetmap-tile-server.
Do you only have this issue with lakes and not oceans?
from openstreetmap-tile-server.
I'm not working with a full planet import (only europe & africa) and I didn't run into your particular problem, but even at that size I ran into other import problems working with this project (postgres db size: ~400 GiB, 48 core system, 128 GB RAM, daily differential updates).
Judging from the image posted, the database seems to lack content. The import might not have been completed or there were errors that weren't printed / or that you didn't noticed.
Sadly I can't send in pull requests (yet), for performance issues, bugs and improvements, without the approval of my employer, but I can give you the following advise:
- you want to change the osm2pgsl import to use the flatnodes parameter to save time and space (the flatnodes file needs to be saved and kept within a volume).
- osm2pgsl has a -C parameter to increase the allowed memory during the import. It is hardcoded to 2 GiB inside this project's run.sh (
I think, but am not sure right now, that there was a similar settings for the renderd.conf). - the postgres config should be tuned (I had to increase max_connections to make the import possible at all - probably because of a higher THREADS value).
- the shared memory (shm-size) setting of the docker container needs to be increased, because the default (64 MB) isn't big enough for postgres (128 MB).
- use docker-compose.
- be aware that docker on linux saves volumes by default into /var/lib/docker/volumes/. You might want to change the paths on the host system manually (via local volume driver).
- check that the storage device that contains the database volume hasn't run full (extrapolating, from europe & africa 20.8 GiB PBF to 400 GiB, planet.osm with 41 GiB PBF might be around 790 GiB imported into postgres?).
- try to delete the contents of the tiles volume, so that you get freshly rendered tiles and aren't served with tiles that might have been created while the import was still in progress.
Outdated but still useful (2012):
from openstreetmap-tile-server.
Thank you guys for your answers.
@Overv: the issue is only with lakes, oceans are rendered correctly. I did an import of Europe yesterday and everything seems to render fine and the server is very responsive.
@Istador
My machine has an i7-8700 with 6 physical/12 hyperthreading cores 64 GB of RAM and 1 TB of NVMe Storage. I'm running the import with the -e THREADS=12 parameter. I did not notice any error messages on import, and I don't think I ran out of space. I will try to follow your suggestions and adjust the memory usage on import, the shm-size value and the postgres config. Again, thanks for the pointers
I'll keep you posted
from openstreetmap-tile-server.
Docker version 18.09.2, build 6247962
docker-compose version 1.24.0-rc1, build 0f3d4dda
If using the build option:
build:
context: .
shm_size: '256M'
i get :
ERROR: The Compose file './tileserver-import.yml' is invalid because:
services.tileserver.build contains unsupported option: 'shm_size'
EDIT1:
updated docker-compose to 1.24, still the above error
(Apperently I need to update docker-compose for this to work?)
FIXED:
make sure the file version is '3.5' (I had still '3')
But still the default SHM size of 64 MB ...?
EDIT2:
I toltally messed up sorry ofcourse this is the right complete compose file:
version: '3.5'
services:
tileserver:
container_name: tileserver
image: overv/openstreetmap-tile-server
volumes:
- '/data/docker/tileserver/merged.osm.pbf:/data.osm.pbf'
- 'openstreetmap-data:/var/lib/postgresql/10/main'
shm_size: '256mb'
command: import
volumes:
openstreetmap-data:
external: true
Somehow I thought the option needed to be in 'environment' or 'build' or something ....
from openstreetmap-tile-server.
I have some rendering issues on full planet import:
The import completes without errors, but some details are missing from rendered tiles. The tiles from my server lack at least lakes, rivers and building footprints, please take a look at the screenshot attached.
If I import the data for a smaller area, I don't get any render issues. Any pointers are really appreciated.
I observed semilar problem of my full planet import. The attached sample shows a comparsion with osm map in zoom level 8 and 9. In zoom level 8 the lake is visible in zoom level 9 not.
from openstreetmap-tile-server.
another sample from city of berlin. In zoom level 13 some parts of the highway are missing. In zoom level 12 everything looks fine.
from openstreetmap-tile-server.
I had a similar problem and tracked it down to an issue with too many database connections.
Connection to database failed: FATAL: sorry, too many clients already
On default postgres allows 100 simultaneously connections. When running osm2pgsql on a server with multiple cores, it can hapen that more connections are open at the same time. to solve this problem just start postgres with
command: postgres -c 'max_connections=300'
in your docker-compose file
from openstreetmap-tile-server.
I had a similar problem and tracked it down to an issue with too many database connections.
Connection to database failed: FATAL: sorry, too many clients already
On default postgres allows 100 simultaneously connections. When running osm2pgsql on a server with multiple cores, it can hapen that more connections are open at the same time. to solve this problem just start postgres with
command: postgres -c 'max_connections=300'
in your docker-compose file
Increasing max_connections
was already mentioned here in this issue #5 (comment). If you're using this docker image as it is, it is using the postgresql.custom.conf.tmpl file, which increases it from 100 to 250. But that might even be too less for some setups (see #32 (comment) for details).
If you try to separate postgres
into an own docker-compose
service, you might also want to consider the other settings from postgresql.custom.conf.tmpl.
from openstreetmap-tile-server.
You are correct, I forgot that I heavily modified this project. However as you noted, 250 can still be too few connections. osm2pgsql opens multiple connections per process and people with big machines could set the processes too high. Maybe max_connections should scale with number of processes for osm2pgsql.
from openstreetmap-tile-server.
Related Issues (20)
- Import command fails for OSM data
- Data/Labels not shown up on maps HOT 1
- How do I force latin names for countries like Greece or China? HOT 6
- Postgis Plugin: ERROR: could not resize shared memory segment "/PostgreSQL.2682777122" to 50438144 bytes: No space left on device HOT 1
- To optimise the rendering HOT 1
- Import not completing HOT 7
- Search filed?
- Postgres in container not working HOT 2
- Tile fetching behind reverse proxy HOT 1
- Postgres says rendrer user already exsists
- Error downloading tile file : i/0 time out HOT 3
- til-server automatic update: how to do? HOT 2
- Running out of space when downloading the planet HOT 3
- postgresql error HOT 1
- Automatize tile rendering of big country HOT 2
- Can We Use an External PostgreSQL in Kubernetes
- using 'https://tile.openstreetmap.org/{z}/{x}/{y}.png' HOT 2
- Renderd logs not visible in k8s container HOT 4
- Cannot recognize postgresql service
- How to log Http requests info to the docker logs with errors
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from openstreetmap-tile-server.