mikeqfu / pydriosm

PyDriosm: an open-source tool for downloading, reading and PostgreSQL-based I/O of OpenStreetMap data

PyDriosm

PyDriosm is an open-source tool that provides an effortless way to download and access OpenStreetMap (OSM) data in popular file formats, such as shapefile and protobuf binary format (PBF), which are freely available from Geofabrik and BBBike. Additionally, the package offers a comprehensive solution for convenient I/O operations and efficient storage capabilities for parsed OSM data within PostgreSQL databases. This means that users can easily read from and write to PostgreSQL databases, enabling efficient data manipulation, querying, and other essential tasks. Whether you are a researcher, practitioner, or simply interested in working with OSM data, PyDriosm can be useful and helpful to streamline your workflow and enhance your experience.

Installation

To install the latest release of PyDriosm from PyPI via pip:

pip install --upgrade pydriosm

Please also refer to Installation for more information.

Quick start

For a concise guide with practical examples, please check out the quick-start tutorial. This tutorial showcases how to utilise PyDriosm for various tasks, such as downloading, parsing, and performing storage I/O operations on OSM data using a PostgreSQL database.

Documentation

The complete PyDriosm documentation: [HTML] [PDF]

It is hosted on ReadTheDocs and provides a wealth of detailed examples.

License

• PyDriosm is licensed under GNU General Public License v3.0 or later (GPLv3+).
• The free OpenStreetMap data, which is used for the development of PyDriosm, is licensed under the Open Data Commons Open Database License (ODbL) by the OpenStreetMap Foundation (OSMF).

Acknowledgement

The development of PyDriosm, including the example code that demonstrates how to use the package, heavily relies on freely available OpenStreetMap data. The author would like to express sincere gratitude to all the OpenStreetMap contributors for their invaluable contributions in making this data accessible to the community.

Cite as

Fu, Q. (2020). PyDriosm: an open-source tool for downloading, reading and PostgreSQL-based I/O of OpenStreetMap data. Zenodo. doi:10.5281/zenodo.4281194

@software{qian_fu_pydriosm_4281194,
  author    = {Qian Fu},
  title     = {{PyDriosm: an open-source tool for downloading, reading
                and PostgreSQL-based I/O of OpenStreetMap data}},
  year      = 2020,
  publisher = {Zenodo},
  doi       = {10.5281/zenodo.4718623},
  url       = {https://doi.org/10.5281/zenodo.4281194}
}

(Please also refer to the export options from Zenodo to reference the specific version of PyDriosm as appropriate.)

Contributors

Qian Fu 🌱 💻 🧪 📖

pydriosm's Issues

Library not working

Hi,
I tried to install the library and use it. Looks like its broken. I did check on linux as well as mac.
`>>> from pydriosm.downloader import GeofabrikDownloader

geofabrik_downloader = GeofabrikDownloader()
geofabrik_download_catalogue = geofabrik_downloader.get_download_catalogue()
geofabrik_download_catalogue
geofabrik_download_catalogue.columns.tolist()
Traceback (most recent call last):
File "", line 1, in
AttributeError: 'NoneType' object has no attribute 'columns'`

Incompatible with newer releases of pyhelpers

Several modules of the pyhelpers package has changed names in newer releases, which results in ModuleNotFoundErrors, e.g.

ModuleNotFoundError: No module named 'pyhelpers.dir'

Examples of renamed modules in pyhelpers are:
dir --> dirs
sql --> dbms
(There might be more. I have not investigated this thouroghly)

See pyhelpers releases page for more info.

This is a problem for new installs of pydriosm because setup.py requires pyhelpers>=1.2.15.

Make generator to parse data for performance improvement with large files

Since some *.osm.pbf files are quite large, do you think it would be possible to yield each row of data instead of appending it to a dict? So that parse_osm_pbf() would become a generator. This would be great for my use-case, if you want I can start a PR for this feature if you think its useful/feasible. @mikeqfu

Keep on failing installing because of the missing files

Hi there, I tried both ways the documentatino provided, but it seems doesn't work for me

pydriosm Import fails due to "ModuleNotFoundError: No module named 'pyhelpers.dir'"

As the title says, I tried to install pydriosm via "pip install --upgrade git+https://github.com/mikeqfu/pydriosm.git" without success as it returned an "ModuleNotFoundError: No module named 'pyhelpers.dir'".
Installation via "pip install --upgrade pydriosm" works but I can't import afterwards since it also gives me an "ModuleNotFoundError: No module named 'pyhelpers.dir'".
I uninstalled and reinstalled pyhelpers and pydriosm but it didn't fix the problem.
I'm working with Windows 11 and Python 3.10. Any idea how to fix this?

Compatible with Historic OSM data file (.osh.pbf)

Using a tool to work with historic osm data would be great.

Process large PBF files by chunks

hi, as of today is it possible to process large pbf files by chunks with pydriosm?

Originally posted by @Evgeny-ubrr in #2 (comment)

Command "python setup.py egg_info" failed with error code 1

I am unable to install your package.

I ran python3 -m pip install --upgrade pydriosm and got the following:

ERROR: Complete output from command python setup.py egg_info:
    ERROR: running egg_info
    creating pip-egg-info/psycopg2.egg-info
    writing pip-egg-info/psycopg2.egg-info/PKG-INFO
    writing dependency_links to pip-egg-info/psycopg2.egg-info/dependency_links.txt
    writing top-level names to pip-egg-info/psycopg2.egg-info/top_level.txt
    writing manifest file 'pip-egg-info/psycopg2.egg-info/SOURCES.txt'

    Error: pg_config executable not found.

    pg_config is required to build psycopg2 from source.  Please add the directory
    containing pg_config to the $PATH or specify the full executable path with the
    option:

        python setup.py build_ext --pg-config /path/to/pg_config build ...

    or with the pg_config option in 'setup.cfg'.

    If you prefer to avoid building psycopg2 from source, please install the PyPI
    'psycopg2-binary' package instead.

    For further information please check the 'doc/src/install.rst' file (also at
    <http://initd.org/psycopg/docs/install.html>).

    ----------------------------------------
ERROR: Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-install-8v3gj_z2/psycopg2/

Any help would be greatly appreciated, this seems like an excellent library!

Edit: I am running this on a Linux machine (Ubuntu 18.04.3 LTS)

Error parsing datetime string in `BBBikeDownloader.get_subregion_catalogue()`

Related class/method:

• BBBikeDownloader.get_subregion_catalogue()

Example error message "... Failed. Unknown datetime string format, unable to parse: last update: Sat Apr 27 20:13:20 2024 UTC, at position 0." resulting from running the following code:

>>> from pydriosm.downloader import BBBikeDownloader
>>> bbd = BBBikeDownloader()
>>> subrgn_name = 'birmingham'
>>> bham_dwnld_cat = bbd.get_subregion_catalogue(subrgn_name, confirmation_required=False, verbose=True)
Compiling the data of a download catalogue for "Birmingham" ... Failed. Unknown datetime string format, unable to parse: last update: Sat Apr 27 20:13:20 2024 UTC, at position 0.

Find Nodes Associated With a Way

Hello again,

Since the parsing is layer-specific, I was wondering if its possible to associated each WayId to a list of NodeIds? This information does not seem to be available but maybe it's possible, any thoughts?

Something like this:

{
    'osm_id': some_way_id, 
    'connected_nodes': [list of node id's associated with this way], 
    ...
}