Comments (12)
@fscottfoti I felt I should report back after the help I got here and from @sablanchard. Thanks again.
Long story short, I gave up on using NetworkX, and found a way to construct a pandana graph from scratch just using geopandas. The notebook documenting this will be shared here in the future, but I felt should at least share the graph construction methods for future notice. If you think parts of this might be nice to include in the library, let me know.
def get_nodes_and_edges(shp_file,rounding=5):
"""Use geopandas to read line shapefile and compile all paths and nodes in a line file based on a rounding tolerance.
shp_file:path to polyline file with end to end connectivity
rounding: tolerance parameter for coordinate precision"""
edges = gpd.read_file(shp_file)
edges["from_x"]=edges["geometry"].apply(lambda x:round(x.coords[0][0],rounding))
edges["from_y"]=edges["geometry"].apply(lambda x:round(x.coords[0][1],rounding))
edges["to_x"]=edges["geometry"].apply(lambda x:round(x.coords[-1][0],rounding))
edges["to_y"]=edges["geometry"].apply(lambda x:round(x.coords[-1][1],rounding))
nodes_from = edges[["from_x","from_y"]].rename(index=str,columns={"from_x":"x","from_y":"y"})
nodes_to = edges[["to_x","to_y"]].rename(index=str,columns={"to_x":"x","to_y":"y"})
nodes = pd.concat([nodes_from,nodes_to],axis=0)
nodes["xy"] = list(zip(nodes["x"], nodes["y"]))
nodes = pd.DataFrame(nodes["xy"].unique(),columns=["xy"])
nodes["x"] = nodes["xy"].apply(lambda x: x[0])
nodes["y"] = nodes["xy"].apply(lambda x: x[1])
nodes = nodes[["x","y"]].copy()
return nodes , edges
def generate_pandana_store_from_shp(hdf5_path,shp_file,weights=["weight"],oneway=None,overwrite_existing=True,rounding=6):
"""Generate a pandana ready HDF5 store using geopandas (gdal required) and pandas. Python 3.5.
hdf5_path(str): output path of HDF5 store holding two dataframes ["nodes","edges"]
shp_file(str): input file that geopandas reads to make a graph based on end to end connectivity
weights(list): weights columns transfered to the store edges. Name is maintained.
oneway(str): series where oneway streets (edges) are denoted with a 1, 0 denotes twoway. None, assumes
twoway edge.
overwrite_existing(bool): if true, the existing store is overwritten.
rounding(int): the number of digits to round line coordinates to get unique nodes (precision)
returns hdf5_path(str)"""
if os.path.exists(hdf5_path):
if overwrite_existing:
print("Overwriting existing store...")
os.remove(hdf5_path)
else:
print("Existing store at path: {0}".format(hdf5_path))
return hdf5_path
all_edges_twoway = True
oneway_field_list = []
if oneway is not None:
all_edges_twoway = False
oneway_field_list.append(oneway)
print("Reading shapefile with geopandas: {0}...".format(shp_file))
nodes, edges =get_nodes_and_edges(shp_file,rounding)
h5store = pd.HDFStore(hdf5_path)
print("Establishing node store...")
df_nodes = nodes
h5store['nodes'] = df_nodes
df_nodes['nodeid'] = df_nodes.index.values
edge_cnt = len(edges)
print("Establishing edge store for {0} edges...".format(edge_cnt))
df_edges= edges[['from_x','from_y','to_x','to_y'] + weights + oneway_field_list].copy()
print("Transferring nodeids to edges...")
df_edges=pd.merge(df_edges, df_nodes, how='left', left_on=['from_x','from_y'], right_on=['x','y'])
df_edges=pd.merge(df_edges, df_nodes, how='left', left_on=['to_x','to_y'], right_on=['x','y'], suffixes=('_from', '_to'))
#nodeids are duplicated on from the joined nodes, joined first to from, suffix to on next set
df_edges.rename(columns= {'nodeid_from': 'from', 'nodeid_to': 'to'}, inplace=True)
df_edges=df_edges[['from','to'] + weights + oneway_field_list]
if all_edges_twoway:
pass
else:
print("Setting up twoway edges...")
twoway_edges = df_edges[df_edges[oneway]==0].copy()
twoway_to = twoway_edges["to"].copy()
twoway_edges["to"] = twoway_edges["from"]
twoway_edges["from"] = twoway_to
df_edges = pd.concat([df_edges,twoway_edges])
h5store['edges']=df_edges
h5store.close()
print("Graph store construction complete...")
return hdf5_path
Oh, and I hate when people post snippits without declaring a license. I use MIT (Copyright (c) 2017 David Wasserman) for snippits. You can reference here. Please use as you like if you find this on the forums.
from pandana.
One of the suggestions in @Holisticnature's first comment was to change the default in the Network()
constructor from twoway=True
to twoway=False
, which I think is probably a good idea.
I'd always assumed that networks were loaded as directed graphs, because the edge parameters are named edge_from
and edge_to
, and only recently realized this is not the case.
And I know it sucks to break API's, but should we also rename the parameter from twoway
to directed
? To me it's easier to tell at a glance what it's referring to.
twoway=False
-> directed=True
(new default)
twoway=True
-> directed=False
Here's the relevant code:
https://github.com/UDST/pandana/blob/master/pandana/network.py#L19-L56
from pandana.
Hi @fillipefeitosa,
I recently updated this code with a few minor changes that resolves some bugs. To keep the discussion related to Pandana, I have moved this code to this gist.
https://gist.github.com/d-wasserman/2d50671b37ee46b088e155293399a90c
The code change I made did not relate to weights, but the "weights" field should be a column or column of "impedance" or "distance" you want to use with the network.
Citation information is in the gist. Thank you!
from pandana.
If I recall correctly all edges are one way by default and are directed from node a to node b. You can add whichever new edges you want which are directed from b to a. I'm not sure an API of booleans indicating which edges to flip adds much to the API. In other words, I think Pandana supports what you need, you just need to write code outside of Pandana to read from a shapefile and add additional edges as required. At the least you could start there and we could review the code to see if it makes sense to integrate. Thoughts?
from pandana.
Hi @fscottfoti , In networkx, the edges are one-way by default. In pandana, your docs say twoway=True is the default. Is this correct?
The way I would approach this outside of pandana on the networkx side would be to flip and append any edge to the networkx graph where one-way!=1/True. I was hoping this could be done on the pandana side as networkx is pure python, and most center line networks would require their segments be flipped to accomodoate them (not really clean though).
The more realistic option is just to modify the tables in pandas based on the one-way field (filter edges, flip, append back to dataframe). I am going to try to replicate this idea in pandas and report back here.
Another approach is I could select out the pandana edges that are oneway, and have their two-way option flagged as false. Then I would merge them with the other pandana edges. However, I am not seeing a way to merge pandana graphs. Is that correct?
Generally, I think there is a good reason to add a convenience function for this purpose. I think it would follow the convention of other network analysis tools, and also respect typical typical municipal center line structures (divided one-way on access controlled roads, two-way on others).
from pandana.
Yup, two way by default.
It just seems to me like a handful of lines of Python/Pandas. You have an edges dataframe of A and B nodes and a mask of which edges are 2-way edges. You filter on that mask, reverse the edges and concat to the original dataframe.
I'm fine with a convenience function, but it would just take the edges dataframe and mask and return the new edges datagrame - it wouldn't touch any of the rest of the code really.
from pandana.
That is what I was thinking too in pandas. Do the edge flip then append myself and set twoway=False. I think this could be pretty isolated convenience function for pandana too. I will give it a shot and report back. Hopefully it is simple add so pandana can take a series alongside a simple bool. Maybe we just keep it outside of the network class (which looks like it would be more complicated to include).
from pandana.
Works for me.
from pandana.
@smmaurer I also think this makes sense, but also I have usually seen the convention be that you have to declare one-way segments (directed edges) rather twoway segments. This makes sense in urban networks where most segments are undirected (twoway), and only higher order/downtown streets are directed. I think this convention results from the fact it is easier to flag one-way streets in a database rather than flag everything.
I am ambivalent between to differentiating between directed/oneway, but I agree the change makes sense. I am not sure I agree on the default being true. While this is convention for graph analysis, I am not sure it makes sense for urban network analysis. Most segments in a street network are twoway. You can get reasonable results with the assumption that directed is false, not vice versa. I think generally defaults should reflect some type of direction to the user for most analysis needs. Especially with the fact some of this is for pedestrian network analysis (UrbanAccess etc), the assumption of one-way streets is just really an important assumption for bikes and automobile travel basically. That said, NetworkX makes this assumption, so I understand the desire to follow convention.
from pandana.
@Holisticnature Thanks, this is helpful. I don't have a good sense of what the conventions are. Definitely true that most urban street segments are undirected, and that 'one-way' and 'two-way' make sense as labels if we're talking about streets. Removing my earlier post about penciling this in for the next release -- let's see if other people have comments.
from pandana.
Agreed. To be honest, I am not sure anyone is really qualified to speak to "convention", but I will qualify it by saying 'based on my understanding of the convention based on working with a few municipal centerline networks/OSM data (where oneway is something that requires a tag etc)'. That is where I draw my basis for a broad generalization.
from pandana.
def generate_pandana_store_from_shp(hdf5_path,shp_file,weights=["weight"],oneway=None,overwrite_existing=True,rounding=6):
"""Generate a pandana ready HDF5 store using geopandas (gdal required) and pandas. Python 3.5.
hdf5_path(str): output path of HDF5 store holding two dataframes ["nodes","edges"]
shp_file(str): input file that geopandas reads to make a graph based on end to end connectivity
weights(list): weights columns transfered to the store edges. Name is maintained.
oneway(str): series where oneway streets (edges) are denoted with a 1, 0 denotes twoway. None, assumes
twoway edge.
overwrite_existing(bool): if true, the existing store is overwritten.
rounding(int): the number of digits to round line coordinates to get unique nodes (precision)
returns hdf5_path(str)"""
This code helped a lot. It was not clear to me how network should be created from shapefiles, or even by scratch.
@d-wasserman When I try to do use the second method (store to hdf5) i get this error message:
"['weight'] not in index"
Am I supposed to create a new column with the weights value? Any help would be appreciated.
from pandana.
Related Issues (20)
- [Question] How to get different Key|Value combinations in osm.node_query
- [Feature Request] Aggreggation queries to subset of nodes
- Binary incompatibility error with older versions of NumPy HOT 1
- Shortest path length exists but shortest path is empty HOT 5
- shortest path distance between non connected nodes
- Allow specifying mapping distance for POIs
- Issue: the POI Accessibility computation is taking forever
- OSM loader - POIs that are ways in OSM HOT 1
- Pandana network initiation slow for large network HOT 4
- how to increase the performance of the aggregate when impedance is changed to time?
- net.shortest_paths doesn't work with multiprocessing
- tables sub-dependency versions conflict with python 3.10 compatibility when installing with pip HOT 1
- incorrect shortest paths solutions when impendance values are particularly small or large HOT 1
- Plotting with Pandana does not render title HOT 1
- osm.pdna_network_from_bbox error HOT 6
- Pulling network object from OSM bugged HOT 2
- New release ?
- Google Colab install pandana error HOT 1
- query the OSM API for the street network within the specified bounding box error HOT 1
- OSM Network Loader Error HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pandana.