Comments (8)
If you have a network that large won't you be unable to load it into memory anyway, even if it does make it to disk?
from pandana.
Well, the final network shouldn't be that big.
Before the OSM loader was added, @fscottfoti built me a network that covers the state of MD (an h5 file about 50mb). I want to create a similar network from scratch using the OSM loader, but when I try, eg:
from pandana.loaders import osm
osm.network_from_bbox(37.8856, -79.4872, 39.7905, -74.9852, network_type='walk', two_way=True)
lcn = network.low_connectivity_nodes(10000, 10, imp_name='distance')
network.save_hdf5('input/osmnetwork.h5', rm_nodes=lcn)
python eats up all of the machine's RAM, then quits with a memory error message. The same thing happens on an amazon server with 80gb of ram. There's no way the network is actually that big. Is it possible that something weird is happening during the overpass query?
maybe something like this? http://stackoverflow.com/questions/16694907/how-to-download-large-file-in-python-with-requests-py
from pandana.
The overpass request response is JSON so there's really no way to stream it. You have to have the entire document in order to parse it.
When I get a chance to work on this the first step will be to figure out exactly what's using all the memory, whether it's the query data or some subsequent step.
from pandana.
BTW, I tried this out. First I tried using a network from the Bay Area - basically all the 9 counties out basically to Sacramento. It ran fine in only about 5GB of memory and about 24 minutes. So that was great.
So I tried @knaaptime's query and did get a memory error on a 32GB machine. The place it errored for me was very strange though - in the from_records
Pandas call?
Traceback (most recent call last):
File "go.py", line 3, in <module>
network = osm.network_from_bbox(37.8856, -79.4872, 39.7905, -74.9852)
File "/home/ubuntu/pandana/pandana/loaders/osm.py", line 312, in network_from_bbox
lat_min, lng_min, lat_max, lng_max, network_type)
File "/home/ubuntu/pandana/pandana/loaders/osm.py", line 202, in ways_in_bbox
lat_min, lng_min, lat_max, lng_max, network_type=network_type)))
File "/home/ubuntu/pandana/pandana/loaders/osm.py", line 178, in parse_network_osm_query
pd.DataFrame.from_records(nodes, index='id'),
File "/home/ubuntu/anaconda/lib/python2.7/site-packages/pandas/core/frame.py", line 888, in from_records
columns)
File "/home/ubuntu/anaconda/lib/python2.7/site-packages/pandas/core/frame.py", line 4808, in _arrays_to_mgr
return create_block_manager_from_arrays(arrays, arr_names, axes)
File "/home/ubuntu/anaconda/lib/python2.7/site-packages/pandas/core/internals.py", line 3555, in
create_block_manager_from_arrays
blocks = form_blocks(arrays, names, axes)
File "/home/ubuntu/anaconda/lib/python2.7/site-packages/pandas/core/internals.py", line 3645, in
form_blocks
object_items, np.object_)
File "/home/ubuntu/anaconda/lib/python2.7/site-packages/pandas/core/internals.py", line 3677, in _simple_blockify
values, placement = _stack_arrays(tuples, dtype)
File "/home/ubuntu/anaconda/lib/python2.7/site-packages/pandas/core/internals.py", line 3741, in _stack_arrays
stacked = np.empty(shape, dtype=dtype)
MemoryError
from pandana.
Not the newish https://github.com/UDST/osmnet may be relevant here.
from pandana.
Thanks @Eh2406 that is correct the new OSMnet package fixes the issue with large bounding box queries: https://github.com/UDST/osmnet. The PR to replace the functions inside Pandana is still pending but will be merged soon so in the meantime anyone can use OSMnet to extract the nodes and edges from OSM and then place it inside a pandana network object. Will keep this issue open until the PR is merged and ready.
from pandana.
Thanks for the heads up. I noticed osmnet when i saw urbanaccess was released, and figured it would solve this issue.
I saw the PR get merged on friday so I ran the query again using osmnet. It finished in about an hour on my current-gen macbook pro
Downloaded OSM network data within bounding box from Overpass API in 40 request(s) and 907.23 seconds
657946 duplicate records removed. Took 160.33 seconds
Returning OSM data with 8,415,342 nodes and 774,600 ways...
Edge node pairs completed. Took 2,394.01 seconds
Returning processed graph with 986,809 nodes and 1,316,755 edges...
Completed OSM data download and Pandana node and edge table creation in 3,599.82 seconds
All looks good. Thanks for your great work
from pandana.
Fixed with #63
from pandana.
Related Issues (20)
- [Feature Request] Aggreggation queries to subset of nodes
- Binary incompatibility error with older versions of NumPy HOT 1
- Shortest path length exists but shortest path is empty HOT 5
- shortest path distance between non connected nodes
- Allow specifying mapping distance for POIs
- Issue: the POI Accessibility computation is taking forever
- OSM loader - POIs that are ways in OSM HOT 1
- Pandana network initiation slow for large network HOT 4
- how to increase the performance of the aggregate when impedance is changed to time?
- net.shortest_paths doesn't work with multiprocessing
- tables sub-dependency versions conflict with python 3.10 compatibility when installing with pip HOT 1
- incorrect shortest paths solutions when impendance values are particularly small or large HOT 1
- Plotting with Pandana does not render title HOT 1
- osm.pdna_network_from_bbox error HOT 6
- Pulling network object from OSM bugged HOT 2
- New release ?
- Google Colab install pandana error HOT 1
- query the OSM API for the street network within the specified bounding box error HOT 1
- OSM Network Loader Error HOT 1
- Retrieving contracted nodes and edges of shortest paths HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pandana.