hywan / qwantmaps

Central repository to link to qwant maps ressources

Qwant Maps

Qwant Maps is a map application that respects your privacy.

It is an opensource project, learn how you can contribute!

Architecture

Qwant Maps can be seen as 4 separated components:

• a tile server
• a search engine (geocoder)
• an API to detail points of interest and places
• a front end

Tile server

A tile server is a service whose job is to give all that is needed to display a fraction of a map.

Qwant Maps provides only vector tiles so the tile server does not serve images (as it is done with raster tiles) but raw data. It is the front end, Erdapfel, that takes the data and renders it into a user browsable map.

The tile server is a combination of 2 great opensource projects:

• kartotherian, the Wikimedia stack to have a highly available tile server. It is itself based on a number of mapbox components.
• OpenMapTiles, for their great and flexible tile schema.

OpenMapTiles makes it possible for Qwant Maps to have an easy to define/extend vector tile schema.

All the data (see below) are imported in a PostgreSQL database using mainly imposm. All the world's vector tiles are then generated using kartotherian's tilerator and stored in Cassandra. This enables Qwant Maps to have a fast and scalable way to serve tiles.

Geocoder

The geographical search engine (also called geocoder) used for Qwant Maps is Mimirsbrunn.

Mimirsbrunn is a geocoder based on Elasticsearch and rust components developed by Kisio Digital.

POI API

To get more details on Points Of Interest (POIs), Qwant Maps uses an additional API: Idunn, that combines the information in the geocoder with external APIs to format detailed POIs data to display in the front end.

Front end

Qwant Maps's front, Erdapfel, is a Javascript single page app that allows to browse the map, search for places, see your position on the map, etc. You can also save your favorite places, using Masq.

The front end uses Mapbox GL to render the map, using both the tiles from the tile server, and Qwant Maps custom map style.

Data

Tiles data

The main source of data for the tile server is the awsome OpenStreetMap (OSM) data ❤️ but other data like natural earth, better water polygon... are also used.

All the tile's data import process is defined in a python script.

This script imports all the data in the PostgreSQL database and runs lots of post-process SQL functions (those functions are defined as postgresql triggers, so they will also be able to run on the OSM updates.

When the data is loaded in PostgreSQL, we use tilerator to generate all the tiles from the zoom level 0 to 14 and store them in Cassandra. This way, when a vector tile is requested to the tiles API kartotherian, no computation needs to be done, only a simple query by id on the highly available Cassandra to get the raw protobuf tiles stored in it.

Daily OSM updates

The import of the world's data is quite a long process, so after the initial import, we have a cron job to read the OSM differential updates and apply the diff in the PostgreSQL database. The tool used to import in PostgreSQL (imposm3) can output the list of tiles impacted by the changes, so we can ask tilerator to generate them again.

Geocoder data

The data import process in Mimirsbrunn is defined in a python script.

First, the OSM data is given to Cosmogony which outputs a big JSON file with all the world's administrative regions.

This file is then imported in Mimir by cosmogony2mimir.

We then import some addresses from OpenAddresses using openaddresses2mimir.

The streets are imported afterwards from the OSM pbf file with osm2mimir.

Finally the POIs are extracted from the PostgreSQL database (thus we have a unified POI handling) using Fafnir.

POI API data

Idunn does not need its own data import process, but the API depends on:

• the data of the geocoder
• Wikipedia data, fetched either on the Wikipedia APIs or in a custom Elasticsearch database.

Global picture

NB: To provide the Qwant Maps service, Qwant hosts instances of the tile server, the geocoder and the Idunn API on its own servers.

Please note that Qwant instances are internal components and are not meant (yet) to be used directly other than as part of Qwant Maps. You can learn more in our Terms of services

How to run

Tiles

To run the tileserver, the easiest way is to use Docker.

The project kartotherian_docker provides a docker-compose with all that is needed.

This repository heavily depends on kartotherian_config that contains the real Qwant Maps business logic:

• imposm mappings/openmaptiles configurations (generated from Qwant's openmaptiles's fork with the openmaptiles-tools, cf the readme)
• data import pipeline scripts
• update scripts
• kartotherian/tilerator generic configurations

Geocoder

The repository docker_mimir contains a docker-compose to easily spawn the needed Docker containers and to import the needed data in them.

POI API

Idunn is quite straightforward to use as it is a simple python API (using apistar).

The readme details the way to run it.

Front end

Erdapfel is a simple node application but you can also use Docker to run it.

You will need to set a few environment variables to link it to the other Qwant Maps components. Check out the readme to know more about the config.