Use case: load a large shapefile / geographical filter / handle with geopandas about dask-geopandas

[martinfleis]

Hi, I think that dask-geopandas may help you with it.

load the entire shapefile

File IO still needs to be implemented. There's proof of a concept in #11. You can either try to write a custom code to do that in your case or try to implement fiona-based IO in dask-geopandas (following @jorisvandenbossche's example). LAtter is of course preferable :).

project to a plane area
get the centroids for each polygon
reprojet to a geodic projection

both to_crs and centroid are implemented and should work as you know them from geopandas.

keep polygons with the nearest centroids (maybe 12-16 of them)

This may be tricky. How do you do that for smaller data? Ideally, this would require spatial index on top of dask.GeoDataFrame which is not yet implemented (a bit of a discussion is in #8). Also, you will need spatially partitioned data to make this operation efficient enough (again #8).

Just out of curiosity - since you are working with a grid, wouldn't be actually easier to convert it to a raster and use something like xarray-spatial?

[gansanay]

Thank you, I will look into these issues.

Right now I am filtering nearest centroids using
sklearn.neighbors.BallTree with geopy.distance.geodesic as a metric, which is horribly long considering I am just looking for points closer than a certain radius to my target point.

Concerning your last question... you probably know much more than I do and it's a great learning opportunity for me!

[Zeroto521]

I'm also finding some ways to improve geopandas io performance.

Geopandas may cost too much time when reading or writing the million-level data.