migurski / extractotron

Placeholder for some ideas about OpenStreetMap extracts

Extractotron

⚠️ This archived project is no longer active ⚠️

In one of my SotM 2011 talks, I suggested that OSM should have regular extracts of metropolitan areas for major world cities and their surrounding areas, regardless of state or country. This is a repository for a bit of code to make that real. I’ve decided to use Amazon S3 to host files, and EC2 to do the extracting. extract.sh is a script that performs the actual work with Osmosis and terminates its host machine upon completion.

This collection is no longer maintained.

The fine folks at Mapzen have built a newer and more up-to-date version of Metro Extracts. Get data from mapzen.com/metro-extracts and suggest changes at mapzen/metroextractor-cities.

Adding New Cities

Extractotron is run a few times per month, to http://metro.teczno.com. If you have cities to add, you can add them directly to cities.txt and send me a pull request in Github.

cities.txt is a tab-delimited file, and has eight fields:

• group

  One of "Africa", "Asia", "Europe", "Middle East", "North America", "South America", or Oceania.

• geonameid

  It's nice to have a link back to the Geonames dataset.

• top, left, bottom, right

  Geographic bounding box of the metropolitan area. These should be larger rather than smaller to surrounding countryside wherever possible. For example of coverage area, see Moscow.

• slug

  Short, lowercase version of the name with dashes for spaces that's used for files.

• name

  Full name of the city.

Rolling Your Own

To use Extractotron yourself, use run-instance.py:

python run-instance.py <your AWS access key> <your AWS secret> <your bucket name>

By default, run-instance.py instantiates an m2.4xlarge instance with Alestic's Ubuntu 10.04 Lucid instance-store AMI ami-bffa6fd6. You can change the instance type or AMI, see python run-instance.py --help for details.

Currently, the cities list is very short. Help me expand it by modifying cities.txt and sending a pull request via Github.

Code Documentation

The Makefile generates various derived files from the cities.txt file. These scripts are utilities to do the extracts, generate utility scripts, JPG previews, and an HTML index based on the regions specified in index.html. Note the result of this Makefile is checked in at github so there's no immediate need to run make on a fresh clone of the repo.

• build-osmosis-script.py

  Generate the shell script osmosis.sh which uses Osmosis to extract subsets of OSM based on the bounding boxes in cities.txt

• build-osm2pgsql-script.py

  Generate the shell script osm2pgsql.sh which uses imposm to import generated OSM metro subsets into a PostGIS database.

• build-coastshapes-script.py

  Generate the shell script coastshapes.sh which uses ogr2ogr to generate coastline shapefiles.

• build-index.py

  Generate index.html, the current HTML interface to allow download of various extracted OSM data.

• compose-city-previews.py

  Generate the JPG images in previews, snapshot maps of the various metros being extracted.

The scripts generated above are then used by Extractotron to generate the actual extracts. The code for doing the extracts includes

• run-instance.py

  Utility script to create an EC2 instance to run the extract. See above for details.

• extract.sh

  Template for the shell script to do the extraction. run-instance.py substitutes pathnames for the scripts generated in the Makefile so that EC2 can then run the extraction. See also upload-files.sh and kill-self.sh.

extractotron's Issues

Time for another run? Last updated Dec 21, 2013

Is it running?

OpenStreetMap import will break if links aren't updated before May 7

http://planet.openstreetmap.org is available over https, and will start redirecting to https on 2018-05-07.

Your project seems to be using osmosis, which will not follow redirects by default.

Would it please be possible to update all files from http://planet.openstreetmap.org to httpS://planet.openstreetmap.org before then?

Ideally, would it be possible to make all requests to openstreetmap.org over https? All services respond over https and will start redirecting soon.

Add Chattanooga, TN

Add house numbers

I was not able to get house numbers on my map with Tilemill when using the Helsinki area data, so I'm assuming they are not included in the extracts. In some cities most of numbers have been added by the OSM community.

It would be awesome to be able to use them!

how to add a city

Could you please add some information to the readme how to add a new city and where get the information from. Small explanation of the different fields would be great.

Really nice project!

Regards
Philipp

Please add Beauvais

Hi! How are you!

Is it possible to add the town of Beauvais, in France?

The coordinates are:
Top : 49.4689
Left (west): 2.0153
Down: 49.3992
Right (east): 2.1749

Thank you so much!!!

Eileen

Philadelphia

Kudos for providing extracts in all the formats on fast download servers.

I found the following issues with the Philadelphia extracts:

1. If a way starts in the boundaries, it is included; however, nodes that are outside of the extracts are missing.

2. I see roads outside the city and the county in the extract. One of the ways was in Atlantic City, NJ.

Add Chattanooga

(Tennessee)

sort data for shapefile extracts

Mapnik's painters algorithm renders features in the order they come from the datasource. It would be useful to generate shapefiles in an order that makes sense for this. I believe adding order by z_order asc to the queries will handle this for both the Imposm and osm2pgsql shapefiles.