ArrowIndexError: Negative buffer slice length about h3ronpy

[nmandery]

I will look into this.

Are there any NULLs or empty geometries in the dataframe you are using?

[nmandery]

So far I have been unable to reproduce this error. Could you provide a test-dataset which triggers this error? Maybe a small subset of your dataframe is sufficient.

What version of pyarrow are you using?

[nmandery]

Thanks, I now can reproduce it.

Fixing this will take some time I guess. The error originates in this check calling this. Not sure how this could be triggered without a bug in pyarrow.

[nmandery]

Hmm. Maybe it is memory related. I am using https://arrow.apache.org/docs/python/generated/pyarrow.Table.html#pyarrow.Table.select which creates a new table. This should have little effect on RAM usage due to arrow only copying pointers to data. The take afterwards lets then memory explode. When I am processing your dataset in batches of max. 10000 rows it works. When computing all of it at once, the error occurs and my 32Gb machine does not have much RAM left. If that is the reason, the error message is quite bad ;) At least you can work around the problem using batches. I will look into how to make the process less memory intensive.

[diehl]

@nmandery No worries Nico. I appreciate your prompt investigation of the issue and your suggestions about how to proceed. Thank you!

[diehl]

Thank you @nmandery - I'm not seeing any NaNs or empty geometries in the dataset.

[diehl]

Here's a link to the exact dataset I'm using: https://web.tresorit.com/l/J3zSS#XkNiAArLbMRKOWoCaL1XpA

The PyArrow version installed in my virtual environment is 14.0.1.

[diehl]

That's great news @nmandery - no worries on the timing. I can explore other pathways in the meantime. Just glad I could contribute something here that will help make the library better.

[diehl]

I wouldn't be surprised. I'm breaking things left and right over here. ;-)

I've been experimenting a bit with h3pandas as well and I'm able to get to the stage where I've got a GeoDataframe with the cell geometries. When I do a GeoPandas dissolve operation to group cells into larger geometries, memory usage seems to grow without bound and crash my machine. All this to say that it seems challenging to integrate multiple attribute layers at high resolution and national scale.

[nmandery]

I know the feeling of running low on resources when dealing with large numbers of H3 cells ;-)

I suppose for your case you may want to try to work in batches. In the past I often used batches defined by low-resolution H3-cells (r=4 may work here).

I am afraid there is little I can do here to improve things. To give the user at least a bit of a hint that memory exhaustion may be the issue I added a warning in #41