larroy / osmcompiler

A framework in Python to process OpenStreetMap data

Open Street Map to Mongodb compiler

This is a framework to process OSM data dumps in PBF format.

There's an included backend to store data in mongodb, to implement a new backend it's very simple, just implement osm.sink which does an action on the processed entities and osm.factory which creates the instances of OSM entitities. Then call the parser with your implementations of sink and factory.

Boot

Initialize the submodules:

git submodule init
git submodule update

Dependencies:

python-pymongo python-protobuf

Usage:

./osm_mongo_compiler.py spain.osm.pbf 
Loading: spain.osm.pbf
Counting blobs...  3829.
Bounding Box: (-9.779014,35.91539) (5.098525,44.14855)
[      0% 7/3829 blocks. 56 K nodes 2012-08-26 18:58      ]

MapReduce

Count the number of data blocks with:

./osm_mongo_compiler.py file --count yes

By running with -f and -n options the file can be devided in ranges and thus processing be done in a map kind of job across several processors or nodes.

./osm_mongo_compiler.py file -f 0 -n 1000
./osm_mongo_compiler.py file -f 1000 -n 1000
...
etc

Credits

Feedback welcome to [email protected] please put [osmcompiler] on the subject or your mails will be probably ignored. Based initially on the parsepbf tool by Chris Hill.

osmcompiler's Issues

I'm unable to connect to mongodb

I'm trying to use the compiler from the command line and it's not being able to save data to the mongo db.

# ./osm_mongo_compiler.py argentina-latest.osm.pbf 
Loading: argentina-latest.osm.pbf
Counting blobs (this might take a while for big dumps)...  952.
Bounding Box (lat, lon): (-55.35943,-73.64277) (-21.72221,-53.63647)
Traceback (most recent call last):/951 blocks.  --                             ]
  File "./osm_mongo_compiler.py", line 248, in <module>
    sys.exit(main())
  File "./osm_mongo_compiler.py", line 240, in main
    parser.parse(options.frm, options.num)
  File "/home/ariel/Projects/osmcompiler/osm/compiler.py", line 159, in parse
    self.processDense(pg.dense)
  File "/home/ariel/Projects/osmcompiler/osm/compiler.py", line 260, in processDense
    self.osm_sink.processNode(node)
  File "./osm_mongo_compiler.py", line 143, in processNode
    node.save()
  File "minimongo/minimongo/model.py", line 218, in save
    self.collection.save(self, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/pymongo/collection.py", line 269, in save
    manipulate, safe, check_keys=check_keys, **kwargs)
  File "/usr/lib/python2.7/dist-packages/pymongo/collection.py", line 487, in update
    check_keys, self.__uuid_subtype), safe)
  File "/usr/lib/python2.7/dist-packages/pymongo/mongo_client.py", line 954, in _send_message
    raise AutoReconnect("not master")
pymongo.errors.AutoReconnect: not master

But using the same configuration I am able to save a document into the DB through the python command line like this without any trouble.

# python
Python 2.7.5+ (default, Sep 19 2013, 13:48:49) 
[GCC 4.8.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import osm_mongo_compiler
>>> class Foo(osm_mongo_compiler.minimongo.Model):
...     class Meta:
...         # Here, we specify the database and collection names.
...         # A connection to your DB is automatically created.
...         collection = "rocks"
...         # Now, we programatically declare what indices we want.
...         # The arguments to the Index constructor are identical to
...         # the args to pymongo"s ensure_index function.
...         indices = (
...             osm_mongo_compiler.minimongo.Index("a"),
...         )
... 
>>> a = Foo()
>>> a.save()
{'_id': ObjectId('52feb7a2e5ebf21c3951f013')}

Is there something I should be doing that I'm not aware of?

Thanks!