bdew70 / planetutils

Scripts and a Docker container to maintain your own OpenStreetMap planet, terrain tiles, & Valhalla Tilepacks

Interline PlanetUtils

• Features
• Installation
  • Using Docker container
  • Using Homebrew on Mac OS
  • Using Python package
• Command-line Usage
  • osm_planet_update
  • osm_planet_extract
  • osm_extract_download
  • osm_planet_get_timestamp
  • elevation_tile_download
  • elevation_tile_merge
  • valhalla_tilepack_list
  • valhalla_tilepack_download
• Specifying bounding boxes
  • Bounding box file: CSV format
  • Bounding box file: GeoJSON format
• Switching toolchains
• Support

Features

Python-based scripts and a Docker container to work with planet-scale geographic data. Using PlanetUtils, you can:

• maintain your own copy of the OpenStreetMap planet (by applying incremental updates)
• cut your copy of the OSM planet into named bounding boxes
• download OSM Extracts from Interline for popular cities and regions
• download Mapzen Terrain Tiles from AWS for the planet or your bounding boxes
• merge and resample Terrain Tiles
• download Valhalla Tilepacks from Interline for the planet (subscription required)

PlanetUtils is packaged for use as a:

• Docker container, for use on any operating system
• Python package, for use on any operating system
• Homebrew formula, for use on Mac OS

PlanetUtils is a "high level" library that makes use of Osmosis, OSM C tools, and Osmium among other great open-source components.

Installation

Using Docker container

Make sure you have Docker installed. Then:

docker pull interline/planetutils:release-v0.4.8

Any of the example commands below can be executed with docker run. It may be helpful to mount a local directory inside the container for persistence and to access output files.

• Example of using docker run with the data directory mounted as /data:

docker run --rm -v ${PWD}/data:/data -t interline/planetutils:release-v0.4.8 <command>

Using Homebrew on Mac OS

Make sure you have Homebrew installed. Then:

brew install interline-io/planetutils/planetutils

Using Python package

If you want to install and use the Python package directly, you'll need to provide:

• Python 2.x or 3.x
• Java and Osmosis
• OSM C tools
• Osmium Tool
• PyOsmium
• GDAL (both binaries and Python scripts)

Then clone this repo, run the tests, and install the Python package:

git clone https://github.com/interline-io/planetutils.git
python ./setup.py test
pip install .

Command-line Usage

PlanetUtils supplies the following command-line utilities:

osm_planet_update

Update a local OSM planet. For example:

osm_planet_update planet-recent.osm.pbf planet-with-updates.osm.pbf

If planet-recent.osm.pbf does not exist locally, the most recent planet file will be downloaded, before applying hourly updates to it. (Note: This download is nearly 40Gb.) By default, files are downloaded from planet.openstreetmap.org. Amazon Web Services also provides OSM planets through its Public Datasets program. To instead download the planet file from AWS:

1. Make sure you have your AWS credentials configured locally.
2. Append the --s3 flag.

Note that an entire OSM planet may be upwards of 40Gb in size! In other words, you should have ~80Gb free disk space before running this command.

For complete help on command-line arguments:

osm_planet_update -h

osm_planet_extract

Cut up an OSM planet file into one or more extracts, defined by bounding boxes. Each bounding box is assigned a name. (This is like a mini version of Mapzen Metro Extracts!)

To create a single extract:

osm_planet_extract --outpath=data/osm_extracts --bbox=-122.737,37.449,-122.011,37.955 --name=san-francisco planet-latest.osm.pbf

To specify more than one bounding box of tiles to download, list the bounding boxes in a CSV file or GeoJSON file. For example:

osm_planet_extract --outpath=data/osm_extracts --csv=data/bboxes.csv planet-latest.osm.pbf

For complete help on command-line arguments:

osm_planet_extract -h

osm_extract_download

Download regularly updated OSM extracts for popular cities and regions from OSM Extracts by Interline. Browse available extracts using the web interface or the GeoJSON file. Anyone can browse the available extracts or propose changes to the extract bounding boxes on GitHub. A subscription is required to download extracts, to cover hosting costs and keep the service sustainable. (See the OSM Extracts website for more information on how profits are donated to OpenStreetMap and other "open" efforts.)

To download the latest copy of an extract (if abcd is your Interline API token and abidjan_ivory-coast is the ID for your chosen extract region):

osm_extract_download --api-token=abcd abidjan_ivory-coast

You can also download extracts in GeoJSON format by using --data-format=geojson. Warning: these can be very large files, but may be useful for filtering and displaying on a web map.

For complete help on command-line arguments:

osm_extract_download -h

(Note: OSM Extracts is a hosted and managed version of the PlanetUtils library. Every day, the pipeline runs the osm_planet_update and osm_planet_extract commands.)

osm_planet_get_timestamp

A simple utility to print the timestamp of an OpenStreetMap PBF file.

osm_planet_get_timestamp planet-latest.osm.pbf

elevation_tile_download

Download elevation tiles from the Terrain Tiles in the AWS Public Datasets program. Download for the entire planet, only tiles within a single bounding box, or within multiple bounding boxes.

Elevation tiles are available in a variety of formats. This command supports the download of:

• GeoTIFF (default): extension .tif in Web Mercator projection, 512x512 tiles
• Skadi: extension .hgt in unprojected latlng, 1°x1° tiles

To download the entire planet in Skadi tiles (which will require about 1.6Tb of space!):

elevation_tile_download --format=skadi --outpath=data/elevation

To download GeoTIFF tiles to cover a single bounding box:

elevation_tile_download --outpath=data/elevation --bbox=-122.737,37.449,-122.011,37.955

To specify more than one bounding box of tiles to download, list the bounding boxes in a CSV file or GeoJSON file. For example:

elevation_tile_download --outpath=data/elevation --csv=data/bboxes.csv

For complete help on command-line arguments:

elevation_tile_download -h

elevation_tile_merge

After downloading elevation tiles using the elevation_tile_download command, use this command to merge together multiple tiles. You can optionally resample elevation values as part of the merge process.

This command only operates on GeoTIFF format elevation tiles.

Warnings: merging lots of tiles can be resource intensive!

To merge a directory of GeoTIFF files:

elevation_tile_merge geo_tiff_tiles/ single_tile.tif

For complete help on command-line arguments:

elevation_tile_merge -h

valhalla_tilepack_list

Use Valhalla Tilepacks from Interline to power your own instances of the Valhalla routing engine. Anyone can list available planet tilepacks. A subscription and an API key are required to download tilepacks.

To list all available planet tilepacks:

valhalla_tilepack_list

For complete help on command-line arguments:

valhalla_tilepack_list -h

valhalla_tilepack_download

Download Valhalla Tilepacks from Interline to power your own instances of the Valhalla routing engine. A subscription and an API key are required to download tilepacks.

Initial set-up:

1. Sign up for Valhalla Tilepacks from Interline.
2. Set your API token as an environment variable (INTERLINE_API_TOKEN) or use it as an argument to the command

To download the latest planet tilepack (if abcd is your Interline API token):

valhalla_tilepack_download --api-token=abcd

or set your API token as an environment variable, and download the latest planet tilepack:

export INTERLINE_API_TOKEN=abcd
valhalla_tilepack_download

For complete help on command-line arguments:

valhalla_tilepack_download -h

Specifying bounding boxes

When extracting multiple bounding boxes from an OSM planet, or when downloading multiple bounding boxes of elevation tiles, you can specify your bounding boxes in a single file, either CSV or GeoJSON format.

Bounding box file: CSV format

Do not include a header row. The format is as follows:

[name for extract],[left longitude],[bottom latitude],[right longitude],[top latitude]

For example:

san-francisco,-122.737,37.449,-122.011,37.955
dar-es-salaam,38.894,-7.120,39.661,-6.502

To determine a bounding box, try the tool at http://bboxfinder.com/

Bounding box file: GeoJSON format

Alternatively, you can specify the bounding boxes as features in a GeoJSON file, using the --geojson argument.

osm_planet_extract --geojson=examples/test.geojson examples/san-francisco-downtown.osm.pbf

To draw bounding box polygons in GeoJSON, try the tool at http://geojson.io/. Currently, the bounding box for each feature is used. Future releases may support polygon clipping.

Switching toolchains

PlanetUtils wraps up a number of libraries, including Osmosis, Osmium, and OSM C Tools. Some PlanetUtils commands allow you to switch which library is used to perform the operation:

PlanetUtils command	argument flag	default	options
osm_planet_update	--toolchain	osmosis	osmosis, osmium
osm_planet_extract	--toolchain	osmosis	osmosis, osmium, osmctools

If you are using osm_planet_extract with --toolchain=osmium, you can also use the --strategy= option to select simple, complete_ways (default) or smart.

If you are using osm_planet_update with --toolchain=osmium, you can also use the --size= option to limit the amount of updates downloaded from the OSM replication server. Osmium requires this data to be held in memory. The default is 1024 megabytes.

Support

To report a bug, please open an issue.

Interline Technologies also provides professional support and consulting services around this and other related tools. Contact us at [email protected] for more information.