Oneway by Series about pandana

[d-wasserman]

@fscottfoti I felt I should report back after the help I got here and from @sablanchard. Thanks again.

Long story short, I gave up on using NetworkX, and found a way to construct a pandana graph from scratch just using geopandas. The notebook documenting this will be shared here in the future, but I felt should at least share the graph construction methods for future notice. If you think parts of this might be nice to include in the library, let me know.

def get_nodes_and_edges(shp_file,rounding=5):
    """Use geopandas to read line shapefile and compile all paths and nodes in a line file based on a rounding tolerance.
    shp_file:path to polyline file with end to end connectivity
    rounding: tolerance parameter for coordinate precision"""
    edges = gpd.read_file(shp_file)
    edges["from_x"]=edges["geometry"].apply(lambda x:round(x.coords[0][0],rounding))
    edges["from_y"]=edges["geometry"].apply(lambda x:round(x.coords[0][1],rounding))
    edges["to_x"]=edges["geometry"].apply(lambda x:round(x.coords[-1][0],rounding))
    edges["to_y"]=edges["geometry"].apply(lambda x:round(x.coords[-1][1],rounding))
    nodes_from = edges[["from_x","from_y"]].rename(index=str,columns={"from_x":"x","from_y":"y"})
    nodes_to = edges[["to_x","to_y"]].rename(index=str,columns={"to_x":"x","to_y":"y"})
    nodes = pd.concat([nodes_from,nodes_to],axis=0)
    nodes["xy"] = list(zip(nodes["x"], nodes["y"]))
    nodes = pd.DataFrame(nodes["xy"].unique(),columns=["xy"])
    nodes["x"] = nodes["xy"].apply(lambda x: x[0])
    nodes["y"] = nodes["xy"].apply(lambda x: x[1])
    nodes = nodes[["x","y"]].copy()
    return nodes , edges

def generate_pandana_store_from_shp(hdf5_path,shp_file,weights=["weight"],oneway=None,overwrite_existing=True,rounding=6):
    """Generate a pandana ready HDF5 store using geopandas (gdal required) and pandas. Python 3.5. 
    hdf5_path(str): output path of HDF5 store holding two dataframes ["nodes","edges"]
    shp_file(str): input file that geopandas reads to make a graph based on end to end connectivity
    weights(list): weights columns transfered to the store edges. Name is maintained. 
    oneway(str): series where oneway streets (edges) are denoted with a 1, 0 denotes twoway. None, assumes
    twoway edge. 
    overwrite_existing(bool): if true, the existing store is overwritten.
    rounding(int): the number of digits to round line coordinates to get unique nodes (precision)
    returns hdf5_path(str)"""
    if os.path.exists(hdf5_path):
        if overwrite_existing:
            print("Overwriting existing store...")
            os.remove(hdf5_path)
        else:
            print("Existing store at path: {0}".format(hdf5_path))
            return hdf5_path
    all_edges_twoway = True
    oneway_field_list = []
    if oneway is not None:
        all_edges_twoway = False
        oneway_field_list.append(oneway)
    print("Reading shapefile with geopandas: {0}...".format(shp_file))
    nodes, edges =get_nodes_and_edges(shp_file,rounding)
    h5store = pd.HDFStore(hdf5_path)
    print("Establishing node store...")
    df_nodes = nodes
    h5store['nodes'] = df_nodes
    df_nodes['nodeid'] = df_nodes.index.values
    edge_cnt = len(edges)
    print("Establishing edge store for {0} edges...".format(edge_cnt))
    df_edges= edges[['from_x','from_y','to_x','to_y'] + weights + oneway_field_list].copy()
    print("Transferring nodeids to edges...")
    df_edges=pd.merge(df_edges, df_nodes, how='left', left_on=['from_x','from_y'], right_on=['x','y'])
    df_edges=pd.merge(df_edges, df_nodes, how='left', left_on=['to_x','to_y'], right_on=['x','y'], suffixes=('_from', '_to'))
    #nodeids are duplicated on from the joined nodes, joined first to from, suffix to on next set
    df_edges.rename(columns= {'nodeid_from': 'from', 'nodeid_to': 'to'}, inplace=True)
    df_edges=df_edges[['from','to'] + weights + oneway_field_list]
    if all_edges_twoway:
        pass
    else:
        print("Setting up twoway edges...")
        twoway_edges = df_edges[df_edges[oneway]==0].copy()
        twoway_to = twoway_edges["to"].copy()
        twoway_edges["to"] = twoway_edges["from"]
        twoway_edges["from"] = twoway_to
        df_edges = pd.concat([df_edges,twoway_edges])
    h5store['edges']=df_edges
    h5store.close()
    print("Graph store construction complete...")
    return hdf5_path

Oh, and I hate when people post snippits without declaring a license. I use MIT (Copyright (c) 2017 David Wasserman) for snippits. You can reference here. Please use as you like if you find this on the forums.

[smmaurer]

One of the suggestions in @Holisticnature's first comment was to change the default in the Network() constructor from twoway=True to twoway=False, which I think is probably a good idea.

I'd always assumed that networks were loaded as directed graphs, because the edge parameters are named edge_from and edge_to, and only recently realized this is not the case.

And I know it sucks to break API's, but should we also rename the parameter from twoway to directed? To me it's easier to tell at a glance what it's referring to.

twoway=False -> directed=True (new default)
twoway=True -> directed=False

Here's the relevant code:
https://github.com/UDST/pandana/blob/master/pandana/network.py#L19-L56

[d-wasserman]

Hi @fillipefeitosa,

I recently updated this code with a few minor changes that resolves some bugs. To keep the discussion related to Pandana, I have moved this code to this gist.
https://gist.github.com/d-wasserman/2d50671b37ee46b088e155293399a90c

The code change I made did not relate to weights, but the "weights" field should be a column or column of "impedance" or "distance" you want to use with the network.

Citation information is in the gist. Thank you!

[fscottfoti]

If I recall correctly all edges are one way by default and are directed from node a to node b. You can add whichever new edges you want which are directed from b to a. I'm not sure an API of booleans indicating which edges to flip adds much to the API. In other words, I think Pandana supports what you need, you just need to write code outside of Pandana to read from a shapefile and add additional edges as required. At the least you could start there and we could review the code to see if it makes sense to integrate. Thoughts?

[d-wasserman]

Hi @fscottfoti , In networkx, the edges are one-way by default. In pandana, your docs say twoway=True is the default. Is this correct?
The way I would approach this outside of pandana on the networkx side would be to flip and append any edge to the networkx graph where one-way!=1/True. I was hoping this could be done on the pandana side as networkx is pure python, and most center line networks would require their segments be flipped to accomodoate them (not really clean though).
The more realistic option is just to modify the tables in pandas based on the one-way field (filter edges, flip, append back to dataframe). I am going to try to replicate this idea in pandas and report back here.

Another approach is I could select out the pandana edges that are oneway, and have their two-way option flagged as false. Then I would merge them with the other pandana edges. However, I am not seeing a way to merge pandana graphs. Is that correct?

Generally, I think there is a good reason to add a convenience function for this purpose. I think it would follow the convention of other network analysis tools, and also respect typical typical municipal center line structures (divided one-way on access controlled roads, two-way on others).

[fscottfoti]

Yup, two way by default.

It just seems to me like a handful of lines of Python/Pandas. You have an edges dataframe of A and B nodes and a mask of which edges are 2-way edges. You filter on that mask, reverse the edges and concat to the original dataframe.

I'm fine with a convenience function, but it would just take the edges dataframe and mask and return the new edges datagrame - it wouldn't touch any of the rest of the code really.

[d-wasserman]

That is what I was thinking too in pandas. Do the edge flip then append myself and set twoway=False. I think this could be pretty isolated convenience function for pandana too. I will give it a shot and report back. Hopefully it is simple add so pandana can take a series alongside a simple bool. Maybe we just keep it outside of the network class (which looks like it would be more complicated to include).

[fscottfoti]

Works for me.

[d-wasserman]

@smmaurer I also think this makes sense, but also I have usually seen the convention be that you have to declare one-way segments (directed edges) rather twoway segments. This makes sense in urban networks where most segments are undirected (twoway), and only higher order/downtown streets are directed. I think this convention results from the fact it is easier to flag one-way streets in a database rather than flag everything.
I am ambivalent between to differentiating between directed/oneway, but I agree the change makes sense. I am not sure I agree on the default being true. While this is convention for graph analysis, I am not sure it makes sense for urban network analysis. Most segments in a street network are twoway. You can get reasonable results with the assumption that directed is false, not vice versa. I think generally defaults should reflect some type of direction to the user for most analysis needs. Especially with the fact some of this is for pedestrian network analysis (UrbanAccess etc), the assumption of one-way streets is just really an important assumption for bikes and automobile travel basically. That said, NetworkX makes this assumption, so I understand the desire to follow convention.

[smmaurer]

@Holisticnature Thanks, this is helpful. I don't have a good sense of what the conventions are. Definitely true that most urban street segments are undirected, and that 'one-way' and 'two-way' make sense as labels if we're talking about streets. Removing my earlier post about penciling this in for the next release -- let's see if other people have comments.

[d-wasserman]

Agreed. To be honest, I am not sure anyone is really qualified to speak to "convention", but I will qualify it by saying 'based on my understanding of the convention based on working with a few municipal centerline networks/OSM data (where oneway is something that requires a tag etc)'. That is where I draw my basis for a broad generalization.

[fillipefeitosa]

def generate_pandana_store_from_shp(hdf5_path,shp_file,weights=["weight"],oneway=None,overwrite_existing=True,rounding=6):
"""Generate a pandana ready HDF5 store using geopandas (gdal required) and pandas. Python 3.5.
hdf5_path(str): output path of HDF5 store holding two dataframes ["nodes","edges"]
shp_file(str): input file that geopandas reads to make a graph based on end to end connectivity
weights(list): weights columns transfered to the store edges. Name is maintained.
oneway(str): series where oneway streets (edges) are denoted with a 1, 0 denotes twoway. None, assumes
twoway edge.
overwrite_existing(bool): if true, the existing store is overwritten.
rounding(int): the number of digits to round line coordinates to get unique nodes (precision)
returns hdf5_path(str)"""

This code helped a lot. It was not clear to me how network should be created from shapefiles, or even by scratch.

@d-wasserman When I try to do use the second method (store to hdf5) i get this error message:

"['weight'] not in index"

Am I supposed to create a new column with the weights value? Any help would be appreciated.