joe-akeem / contour-tiles

A docker image to create vector tiles for contour lines and hillshade raster tiles of europe (mbtiles files).

Countour Lines and Hillshade files for Europe

A docker image to create vector tiles for contour lines and hillshade (relief) raster tiles of europe (mbtiles files).

The relief and is based on data from SRTM 90m DEM Digital Elevation Database and OpenSlopeMap.

Getting it

git clone https://github.com/joe-akeem/contour-tiles.git

Building the mbtiles files

cd contour-tiles
docker-compose -f docker-compose-compute.yml build contour-tiles && docker-compose -f docker-compose-compute.yml up

This will build the mbtiles files in the folder ./osm/mbtiles

Area

By default, the Europe area will be covered (defined in .env). The configuration for other areas has been prepared in the .env-* files. For example, to build the contour tiles for South America, pass the corresponding environment file to docker-compose as follows:

docker-compose --env-file .env-south-america -f docker-compose-compute.yml up

To completely customize the area, please use this tool SRTM Tile Grabber. You'll be able to identify the tiles that will be needed to build your data:

On this image, the name of the tile is srtm_38_03.zip. X is 38, Y is 03. For example, to compute Hawaii only, you can export the custom area as environment variables before running docker-compose:

export MIN_X=05 MAX_X=06 MIN_Y=08 MAX_Y=09

Please note that MIN_Z and MAX_Z are the limit in terms of altitude.

For a dry run to verify the configuration before launching the whole process run:

docker-compose -f docker-compose-compute.yml config

Then start the building of the tiles as before:

docker-compose -f docker-compose-compute.yml up

Computing large areas

With the view to be able to compute areas, many improvements have been done. First of all, the main bottleneck was the use of gdal_contour. It's a single core process...

To be able to overcome this problem, this script can now compute contour using parallelized job. You'll be able to define the number of parallel jobs using the CONTOUR_JOBS variables, in the docker-compose-compute.yml file. Beware of the fact that you should limit the number of jobs to the number of core that your host has. Moreover, each job will consume a lot of RAM, essentially dependent upon the size of the area. For the default area, i use 8 jobs with 16 GB free RAM.

Last problem, the long time taken by gdal_translate... Adding cache can speed up the process, but don't expect a huge difference. The variable is GDAL_CACHEMAX, in MB.

Inspecting the tiles

To inspect the mbtiles files a local tileserver can be started as follows:

docker-compose up

The web interface of the tile server is made available at localhost:8080. A basic style is also available and can be used to view the generated data e.g. as Vector Tiles.

You will notice that there are contour lines for 100 meter and for 20 meter equidistance. The 100 meter equidistance lines will start showing from zoom level 10 while the 20 meter equidistance lines are showing from zoom level 13 upwards.

The contour lines are tagged with a field named elev representing the elevation of the contour line in meters above sea level.

Performances

The container has been tried with success on an Azure VM, Standard_D8ds_v5 with a P30 SSD (1To, 5kIOPS). The whole process for Europa took about 30 minutes.

The container has been used with success on an Azure VM, Standard_HB60-45rs with a P30 SSD (1To, 5kIOPS). The whole process for Planet took about 3 days. Even if this machine had a lot of cores, the RAM was the limiting factor. I had to use these parameters in the compose file :

MIN_X: 01 
MAX_X: 72
MIN_Y: 01
MAX_Y: 24
MIN_Z: 0
MAX_Z: 9000
CONTOUR_JOBS: 8
GDAL_CACHEMAX: 160000

contour-tiles's Issues

Makefile hangs

Hi,

frst of all, thank you for your great job. I've ran your image and it hangs with this error :

gdal_merge.py -o /data/tif/contour-4326.tif \
                                 /data/tif/srtm_34_02.tif \
                                 /data/tif/srtm_35_01.tif \
                                 /data/tif/srtm_35_02.tif \
                                 /data/tif/srtm_35_03.tif \
                                 /data/tif/srtm_35_04.tif \
                                 /data/tif/srtm_35_05.tif \
                                 /data/tif/srtm_36_01.tif \
                                 /data/tif/srtm_36_02.tif \
                                 /data/tif/srtm_36_03.tif \
                                 /data/tif/srtm_36_04.tif \
                                 /data/tif/srtm_36_05.tif \
                                 /data/tif/srtm_37_01.tif \
                                 /data/tif/srtm_37_02.tif \
                                 /data/tif/srtm_37_03.tif \
                                 /data/tif/srtm_37_04.tif \
                                 /data/tif/srtm_37_05.tif \
                                 /data/tif/srtm_38_01.tif \
                                 /data/tif/srtm_38_02.tif \
                                 /data/tif/srtm_38_03.tif \
                                 /data/tif/srtm_38_04.tif \
                                 /data/tif/srtm_38_05.tif \
                                 /data/tif/srtm_39_01.tif \
                                 /data/tif/srtm_39_02.tif \
                                 /data/tif/srtm_39_03.tif \
                                 /data/tif/srtm_39_04.tif \
                                 /data/tif/srtm_39_05.tif \
                                 /data/tif/srtm_40_01.tif \
                                 /data/tif/srtm_40_02.tif \
                                 /data/tif/srtm_40_03.tif \
                                 /data/tif/srtm_40_04.tif \
                                 /data/tif/srtm_40_05.tif
0...10...20...30...40...50...60...70...80...90...100 - done.
free(): invalid pointer
make: *** [Makefile:133: /data/tif/contour-4326.tif] Aborted (core dumped)
make: *** Deleting file '/data/tif/contour-4326.tif'

Crawling the web, i found this issue : GDAL/ogr2ogr/ogrinfo produces an invalid pointer error each time I run it.

What do you think about that ?

Regards,

Fabien

contour.json is mising

Hi,

In the "tileserver-style.json" file, it says "line 11 url: mbtiles://contour.json"

After docker image builted, there is no contour.json file.

Thanks.

Empty vector map preview

Hello,

Thank you for this work, so helpful. I followed the README and successfully created the mbtiles for switzerland.
In the osm directory, I can see the mbtiles are created, and I tried to import them into QGis and they did appear, which proves their creation was a success.

After that, I'm running docker compose up to launch klokantech's tileserver-gl, and the http://localhost:8080 does show the basic web page with the styles and data chapters.

In the data chapters, I can see the raster maps of slope and hillshades.
But in the first chapter, styles, I can click on "Vector" to see the map but I only get a blank map.
In this blank map, I can (pretend to) zoom, as I can see the URL changing. I can also rotate like in any vector map, but nothing but white map is showing.

In the docker compose logs, I see nothing happening.

I also tried to replace klokantek's tileserver-gl docker image with maptiler's one, as adviced on dockerhub, but to no avail and with the same result.

I see no error in my browsers dev tools logs (firefox and chromium).

geom vs wkb_geometry

Hi,

About "contour20.sql" file, line 5

I created contour.mbtiles samples when the "geom" value was present.
The "geom" value has been removed and replaced with the "wkb_geometry" value.
Is this change important? Do I need to recreate the samples?

Thanks.

hillshade higher max zoom

Hillshade tiles are in the zoom range 6..10
Is it possible to generate at higher zoom levels?
I'd like to show hillshade at level 12, but it comes out quite blocky when scaled from zoom level 10.

Thanks so much for this repository - it's fantastic.

Other DEM inputs ?

Hello,

Great projetc, exactly what I'm looking for thanks !

Any possibiliyt to allow other inputs and make the config files more customizable ?

For example being able to specify DEMs with higher resolution (we have 5m now for most western countries and even LIDAR) and as a fallback use SRTM ?

This will certinly require to specify URL, file format and projection.

Challenge will be to be carefull on how to manage potential artefacts at tiles and country borders when multiple DEMS overlap and do not have the same resolution.

add a .dockerignore file

Rebuilding of the image failed with a "not enough disk space" error message.
Reason was that docker included the already downloaded DEM data into the build context of the docker container. Adding a .dockerignore file to the root directory of the project solved that issue.
Content of the .dockerignore file:

osm/*
img/*
postgres-data/*

Configuration howtos/examples

Hi, it would be great if we could get some examples on how to configure this container based on our needs. For example ...

• How to set the line intervals? Right now this seems to be hardcoded to 20.
• How to set the min/max zoom? Right now 20m lines just disappear at a fixed zoom level
• Is there any way to smooth/interpolate the generated lines? I.e. they are pretty jagged right now, and a trivial interpolation step might be helpful to make them look much more natural

Smoothing contour lines

In some areas contour lines are edgy, any possibility to have control onthe douglas-peucker value to make them smoother ?

Hit 2G/4G file limit

gdal_contour -a elev -i 20 /data/tif/contour-3785.tif /data/tif/contour-3785-20m.shp                                                                 
0...10...20...30...40...50...60..Warning 1: 2GB file size limit reached for /data/tif/contour-3785-20m.shp. Going on, but might cause compatibility issues with third party software                                                                                                                      
.70...80...90ERROR 1: Failed to write shape object. File size cannot reach 4294962812 + 427144.                                                      
ERROR 1: cannot write linestring                                                                                                                     
ERROR 1: Failed to write shape object. File size cannot reach 4294967100 + 328.                                                                      
ERROR 1: cannot write linestring                                                                                                                                                                                                                                                                          
ERROR 1: Failed to write shape object. File size cannot reach 4294967100 + 296.                                                                      
ERROR 1: cannot write linestring                                                                                                                                                                                                                                                                          
ERROR 1: Failed to write shape object. File size cannot reach 4294967100 + 8552.                                                                     

The parent FS is XFS so generally there is no limit on filesize, but I read on the net that gdal/db has a 2 or 4 GB limit.

(Isn't it possible to generate only a smaller bbox?)

this SE says it's fixed in new versions of gdal.

README.md suggestions

While dealing with the repo a few questions came in my mind which could be covered in the README.md file.

1. approx how long does it take to build the mbtiles file on a "normal" PC?
2. how much disk space do you need to build the mbtiles
3. how much disk space will the resulting mbtiles file need
4. maybe this is a stupid question: are the hill-shades included in the processing result?
5. If hill-shades are included, what is the max zoom-level?
6. is attribution needed when displaying the tiles? (Your project and SRTM 90m)

Note: there is a typo in the readme: "OpenSlopwMap" instead if "OpenSlopeMap"

The United States?

I would like to be able to use the same process for the United States. What all would need changed in order to do that? Has anyone already done this?

env for planet

Any possibiliyt to have an .env file for the planet ?

Parallelising the whole shebang?

Hi, would be pretty cool if this rendering process could somehow be parallelised. Right now it seems that only one CPU core is utilized, but could also be related to my setup (i.e. docker inside LXC).

missing distance and type attributes

Problem:
The generated contour.mbtiles is missing the distance and type attributes. It seems like the distance attribute is only added for the _contour<20/100>.geojson files. Could it be that previously the contour.geojson files were generated using ogr2ogr by merging the contour20 and contour100 ones propagating the attributes, whilst now since gdal_contour is used the attributes are not generated? Both the readme and the style files are still referencing these attributes.

Potential Solution for distance attribute:
The distance attribute might be unnecessary as the mapbox gl style spec is supporting the modulo operator, such that 100m interval contours can be styled different from the 20m ones using an expression like this:

  "filter": [
    "==",
    ["%", ["get", "elev"], 100],
    0
  ], 

Building the countour-tiles docker image failes

Here is my console output:

root@v22019043209786315:/docker/contour-tiles# docker-compose run contour-tiles
Starting contour-tiles_postgres-osm_1 ... done
Building contour-tiles
Step 1/18 : FROM osgeo/gdal:ubuntu-full-latest
 ---> 0f0f74b4d6e0
Step 2/18 : MAINTAINER joeakeem "[email protected]"
 ---> Using cache
 ---> 8c6e920a4157
Step 3/18 : ENV DEBIAN_FRONTEND=noninteractive
 ---> Using cache
 ---> cfcd58df50b9
Step 4/18 : RUN apt-get clean
 ---> Using cache
 ---> c538cb782895
Step 5/18 : RUN apt-get update -y && apt-get install -y software-properties-common
 ---> Using cache
 ---> ee020a0828f3
Step 6/18 : RUN add-apt-repository ppa:ubuntugis/ppa
 ---> Using cache
 ---> 48a0d57eb58a
Step 7/18 : RUN apt-get update -y && apt-get install -y         postgis         make         curl         unzip         git         cmake         g++         libboost-dev         libboost-system-dev         libboost-filesystem-dev         libexpat1-dev zlib1g-dev         libbz2-dev         libpq-dev         libproj-dev         lua5.2         liblua5.2-dev         build-essential         libsqlite3-dev         zlib1g-dev         wget         python-gdal         gdal-bin
 ---> Running in fe710e1b4cbc
Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease
Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease
Hit:3 http://ppa.launchpad.net/ubuntugis/ppa/ubuntu focal InRelease
Hit:4 http://archive.ubuntu.com/ubuntu focal-updates InRelease
Hit:5 http://archive.ubuntu.com/ubuntu focal-backports InRelease
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
Package python-gdal is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source
However the following packages replace it:
  gdal-bin

E: Package 'python-gdal' has no installation candidate
ERROR: Service 'contour-tiles' failed to build: The command '/bin/sh -c apt-get update -y && apt-get install -y         postgis         make         curl         unzip         git         cmake         g++         libboost-dev         libboost-system-dev         libboost-filesystem-dev         libexpat1-dev zlib1g-dev         libbz2-dev         libpq-dev         libproj-dev         lua5.2         liblua5.2-dev         build-essential         libsqlite3-dev         zlib1g-dev         wget         python-gdal         gdal-bin' returned a non-zero code: 100