`catalog.create_parquet_table` silently ignores `table_type='ICEBERG'` about aws-sdk-pandas

[jaidisido]

In the athena.to_iceberg call, the Iceberg table is created with an Athena DDL query not a Glue call. This rePost article points to the same solution.

[plbremer]

In the athena.to_iceberg call, the Iceberg table is created with an Athena DDL query not a Glue call. This rePost article points to the same solution.

Does the following makes sense then:

• One cannot use awswrangler.catalog.* to create iceberg tables for athena
• Instead, if one wants to use awswrangler to create iceberg tables without downloading the contents into a local dataframe, one can only hope to use awswrangler.athena.read_sql_query with a query that might have been templated based on properly parsed parquet metadata from something like awswrangler.s3.read_parquet_metadata

I assume that the syntactically proper CREATE TABLE query would also deposity the appropriate iceberg metadata in the s3 containing the parquet file...?

[jaidisido]

Below is a simplified version of the code I linked in my previous comment using the start_query_execution API:

   sql = (
        f"CREATE TABLE IF NOT EXISTS `{table}` ({cols_str}) "
        f"LOCATION '{path}' "
        f"TBLPROPERTIES ('table_type'='ICEBERG', 'format'='parquet')"
    )

    wr.athena.start_query_execution(sql=sql, database=database, wait=True)

It simply follows what is described in the Iceberg Athena docs. You might have to play a bit with the parameters to the API but that is the gist of it

[plbremer]

Below is a simplified version of the code I linked in my previous comment using the start_query_execution API:

   sql = (
        f"CREATE TABLE IF NOT EXISTS `{table}` ({cols_str}) "
        f"LOCATION '{path}' "
        f"TBLPROPERTIES ('table_type'='ICEBERG', 'format'='parquet')"
    )

    wr.athena.start_query_execution(sql=sql, database=database, wait=True)

It simply follows what is described in the Iceberg Athena docs. You might have to play a bit with the parameters to the API but that is the gist of it

Ok great, I agree with this, thanks