Question regarding wr.athena.to_iceberg about aws-sdk-pandas

[jaidisido]

The temporary table is created/deleted from the same database. As the temp table has a random name, you must have Lake Formation permissions to create, describe and delete any table in that database

[rossiellog]

@jaidisido Thanks! I finally got what the problem was thanks to your answer. Now I am facing a different problem, but looking at wrangler's source code I think I know what's happening.
Basically, it seems that the temporary table is missing a column. This column is in fact the index of the pandas dataframe I am trying to merge into the Iceberg table.
Check this.
Unlike the final merge operation that takes in consideration the index parameter given as input to the to_iceberg method, the linked invocation of to_parquet does not pass the value of the parameter. Since the default is False, the temporary table will not be aligned with the final one in this particular case. This leads to the error I am having.
Now, I guess I can reset the dataframe index in order to retrieve the index as a column, but I think it would be nice to fix this. Let me know if my assumption is correct, or if I missed some other line that covers this case!

[jaidisido]

The to_iceberg API is already significantly overloaded with parameters. We would prefer not to add yet another parameter especially since as you mentioned you can reset the index before calling the method

[rossiellog]

I understand, but that's not the point I was trying to make. You do not need to add a new parameter, you just have to pass the index parameter from outer method to inner method. If you'd like, I can fork and add a pull request so that you can check what I mean and evaluate if it makes sense.