Dead-end residential nodes not kept by osm.network_from_bbox about pandana

[gboeing]

Sam - I've got my own Overpass API queries working with metro-scale and even state-wide data downloads. Combination of recursive timeout increases and explicit memory allocation. Glad to share code if it'd be useful.

[fscottfoti]

This is interesting - did you figure out anything further with this? Do the nodes definitely get returned from a direct query of the Overpass API? As I recall we're not processing this much inside network_from_bbox, but it's always possible there's an issue.

[stefancoe]

Actually, it looks like this is being done purposefully on all dead end segments. It seems that Nodes that are not intersecting 2 or more ways are removed. See line 251 here:
https://github.com/UDST/pandana/blob/master/pandana/loaders/osm.py

[jiffyclub]

I do seem to remember wanting only intersection nodes but it's been long enough that I don't remember exactly why. Probably worth revisiting to see what happens when you have all nodes and see if anything breaks.

[stefancoe]

I'll try to do a test by keeping all the nodes. BTW, we are using this library as part of our activity based model to generate parcel accessibility variables and so far it is an improvement over our old method. Thanks!

-Stefan

[fscottfoti]

Cool - are you at PSRC? I'm glad it's helping!

[stefancoe]

Sorry for the late response. I am at PSRC and we are using Pandana to build accessibility metrics for our ABM, which you can see here: https://github.com/psrc/soundcast/blob/dev/scripts/accessibility/accessibility.py

I am finally getting around to test including dead end nodes by commenting out line 257 in OSM.py. However, I am having issues getting an OSM network for the bounding box of our region. When I run this code:

from pandana.loaders import osm
network = osm.network_from_bbox(46.73, -123.03, 48.3, -120.95, network_type='drive')

I get a long error message and at the end it says: RuntimeError: OSM query results contain no data.

I can get it to work by reducing the size of the bbox, so it seems like a size limit issue, which is odd because it worked several months ago using the exact same coordinates. I have tried this on several different machines with lots of RAM and on a Mac as well. I saw the other issue about support for large areas, but the error messages described in that thread explicit say something about being out of memory, which is not what I am getting. Plus, it happens very quickly. Also, I was able to download an OSM data set using Overpass directly from https://www.openstreetmap.org/export#map=9/47.5209/-121.9675 without any issues.

Here is the full text of the error message:
Traceback (most recent call last):
File "", line 1, in
File "C:\Anaconda\lib\site-packages\pandana\loaders\osm.py", line 312, in network_from_bbox
lat_min, lng_min, lat_max, lng_max, network_type)
File "C:\Anaconda\lib\site-packages\pandana\loaders\osm.py", line 202, in ways_in_bbox
lat_min, lng_min, lat_max, lng_max, network_type=network_type)))
File "C:\Anaconda\lib\site-packages\pandana\loaders\osm.py", line 163, in parse_network_osm_query
raise RuntimeError('OSM query results contain no data.')
RuntimeError: OSM query results contain no data.

[sablanchard]

Hi Stefan,
Thanks for the update, we are aware of issues with large area bbox queries from within the pandana osm loader and have a few fixes in mind in the queue to address this but haven't gotten to them yet. Mainly, I think it would help to specify a timeout interval for overpass api requests when it fails and to recursively increase the timeout time as each request fails. I think the issue may stem from throttling from the overpass api side but will take some testing to determine if this is really true.

[stefancoe]

Overdue update-I ended up creating an OSM network in GIS and then creating text files for the edges and nodes, which are used to build the pandana network. This was actually preferable because it allowed us to customize the network to include the way types of interest to us.

[sablanchard]

Hi Stefan,
Thanks for the update. We know there are a number of improvements that can be made to the built in Pandana OSM loader. Many of these improvements I have implemented in the UrbanAccess OSM downloader tool so until the changes get pushed to Pandana you can use UrbanAccess to download the data once it is released. UrbanAccess will also include drive and walk networks using a more refined criteria to pull those data, fixes the large area OSM query issue, and saves all the OSM attributes for nodes/edges so you can subset the data before running a accessibility query.

[sablanchard]

@stefancoe we have recently released a new OSM network extraction tool called OSMnet: https://github.com/UDST/osmnet that is designed to extract OSM street network data for Pandana. Pandana has been updated to now use OSMnet for its street network loading functionality. OSMnet: extracts drive and walk networks using a more refined criteria to pull those data, fixes the large area OSM query issue, and saves all the OSM attributes for nodes/edges so you can subset the data before running a accessibility query. Any new issues to loading OSM street network data should be submitted to the OSMnet repo.

Regarding your comment about re-testing the original issue reported here about dead end nodes: #50 (comment) would you be able to try testing again and give us more information on your results?

[stefancoe]

No problem- I went through the same testing process and it is still an issue. It looks like line 688 in load.py is only getting edges that have nodes that participate in intersections:

intersections = intersection_nodes(waynodes)

Do you want me to try to modify the code and try it?

-Stefan

[gboeing]

I had this issue in the past with pandana. It essentially calculates the degrees of all nodes in the graph, then discards all nodes of degree 1 and all isolated nodes.

This method also results in retaining all the nodes in the middle of a single street segment that allow it to curve around a bend as a series of straight-line edges. This is fine for accessibility calculation along the network, but such points are not intersections in the street network (it's basically an expansion graph). The only real drawback is that if you want intersection accessibility, you are also calculating accessibility for 2x or 3x as many nodes as you need to be. But pandana is really fast so that might not matter at all.

[sablanchard]

@stefancoe and @gboeing thanks for the update, this may be something that can be exposed as a parameter that a user can modify thus preserving the current method while adding extra flexibility in how the network is generated.