Iceberg partitioning based on transformed DataFrame columns not supported? about aws-sdk-pandas

[kukushking]

Hi @petebachant unfortunately only partition columns are supported at the moment (no functions). You can transform the data in your data frame prior to the insert.

[alvaro-ponce]

Hi @kukushking that's not what documentation says:

https://aws-sdk-pandas.readthedocs.io/en/stable/stubs/awswrangler.athena.to_iceberg.html#awswrangler.athena.to_iceberg

I was using append mode a couple of days ago successfully on version 3.4, now it fails on version 3.6, I tried to check the wrangler code but I do not totally understand why I did not get error in 3.4 since the code involved in this error seems not to be updated since 3.4 version

[alvaro-ponce]

Ok, it was 3.3.0, I have checked and I can append multiple times with function in partition cols in 3.3.0 version, not 3.4 as I said before.

I believe the problem appeared in version 3.4.1 with the addition of _determine_differences function in _write_iceberg.py script.

[LeonLuttenberger]

Hey, thanks for bringing this up. You are correct, this feature was broken in the PR you tagged. I'm looking into whether we can fix this feature. If not, I will update the documentation.

[alvaro-ponce]

Thanks for the quick response and work!

You are awesome!