BUG: set_index results in invalid dask GeoDataFrame (partitions are DataFrames) about dask-geopandas

[jorisvandenbossche]

@DahnJ indeed, good catch. I opened an issue on the dask/partd side about this: dask/partd#52

Alternative for now is to specify set_index(.., shuffle="tasks"), so it doesn't use the default "disk".

[DahnJ]

Interestingly, this is not the case for all the partitions:

>>> [type(partition.compute()) for partition in ddf2.partitions]

[pandas.core.frame.DataFrame,
 geopandas.geodataframe.GeoDataFrame,
 pandas.core.frame.DataFrame,
 pandas.core.frame.DataFrame]

The change seems to happen in rearrange_by_column

[jorisvandenbossche]

Ah, interesting. Then it might also be a bug in GeoPandas (if some operation on a GeoDataFrame results in a pandas DataFrame where it could have preserved the GeoDataFrame type)

[DahnJ]

I believe the problem is in the fact that Dask uses partd.pandas to (de)serialize the dataframe:

import geopandas as gpd
from partd.pandas import serialize, deserialize
df = gpd.read_file(gpd.datasets.get_path('naturalearth_lowres'))
type(deserialize(serialize(df)))  # pandas.core.frame.DataFrame

The reason we sometimes get GeoDataFrame is that dask.shuffle.collect returns the meta object for empty partitions and the meta is a GeoDataFrame.