Reading an iceberg table in a vpc fails with connection time out on endpoint URL: "https://sts.amazonaws.com/" about aws-sdk-pandas

[kukushking]

Hi @AdrianoNicolucci , does it succeed with a vanilla boto3 call and not with aws sdk for pandas? is athena endpoint also set?

[AdrianoNicolucci]

Hi @AdrianoNicolucci , does it succeed with a vanilla boto3 call and not with aws sdk for pandas? is athena endpoint also set?

@kukushking After testing it with a table that is just a csv table instead, I'm still getting the same issue of the lambda function timing out since it can't connect to athena. So perhaps it has nothing to do with the iceberg table specifically. However, When I attempt to query directly with boto3, I can see the query reach the athena service which tells me there is an issue somewhere with this method.

[AdrianoNicolucci]

After reviewing the logs and doing some further debugging, it looks like the read_sql_query method is trying to call the sts service to get the account_id, using the global instead of the regional endpoint. I'm confused why it's trying to use the global sts endpoint instead of the regional one.

error message

 File "/opt/python/awswrangler/sts.py", line 33, in get_account_id
    return _utils.client(service_name="sts", session=boto3_session).get_caller_identity()["Account"]

Here are the global endpoint configurations I have tried to set to force it to use the region endpoints but with no luck:

os.environ["AWS_STS_REGIONAL_ENDPOINTS"] = "regional"
wr.config.glue_endpoint_url = "https://glue.us-east-2.amazonaws.com"
wr.config.athena_endpoint_url = "https://athena.us-east-2.api.aws.com"
wr.config.sts_endpoint_url = "https://sts.us-east-2.amazonaws.com"