combination of managed layer AWSSDKPandas-Python312 and AWSLambdaPowertoolsPythonV2 result in ModuleNotFoundError: No module named '_cffi_backend' about aws-sdk-pandas

[heitorlessa]

Powertools for AWS Lambda on-call checking in - we're investigating. Thank you so much for the preliminary investigation.

[jaidisido]

After testing with version 2.34.2 of powertools using layers:

• arn:aws:lambda:eu-west-1:017000801446:layer:AWSLambdaPowertoolsPythonV2:65
• arn:aws:lambda:eu-west-1:336392948345:layer:AWSSDKPandas-Python312:4

on Python 3.12, I can confirm that the issue is fixed, thanks @heitorlessa. Closing the ticket

[jaidisido]

So here's what is going on here. awswrangler packages pymysql as a dependency in its lambda layers. In turn, pymysql has an optional dependency on the cryptography package. Because it's optional, we don't package it in the zip layer and therefore pymysql does not try to import it. This explains why you have no issue when using the awswrangler layer on its own.

On the other hand, recent versions of aws_lambda_powertools do package cryptography as a dependency in their layer. As a result, when both layers are used, pymysql detects that cryptography is available from powertools and it tries to import it.

Now on the exception No module named '_cffi_backend'. There seems to be an issue with the way cryptography was packaged in the powertools layer. I am able to reproduce the exception with this simple code:

import json

from aws_lambda_powertools.utilities.data_masking.provider.kms.aws_encryption_sdk import (
    AWSEncryptionSDKProvider,
)

def lambda_handler(event, context):
    # TODO implement
    return {
        'statusCode': 200,
        'body': json.dumps('Hello from Lambda!')
    }

which also raises No module named _cffi_backend and as you can see does not involve awswrangler. From the little research I have done, it looks like powertools uses the same base image to package their layers but cffi is sensitive to the base python version on which it's installed:
https://stackoverflow.com/questions/76443834/aws-lambda-python-3-10-no-module-named-cffi-backend
We have the same issue with numpy and this is why we have one layer per python version when powertools does not.

In short, this is a powertools specific issue that they need to solve. I can open an issue with them to inform them.

[Eerk1975]

an extra test I did: the combination of arn:aws:lambda:eu-west-1:336392948345:layer:AWSSDKPandas-Python311:6 and arn:aws:lambda:eu-west-1:017000801446:layer:AWSLambdaPowertoolsPythonV2:11 in Python 3.11 did work

[heitorlessa]

we're making a patch release now to remove aws-encryption-sdk from the Powertools Layer. I'll update as soon as it's out -- it'll be version 65.

[heitorlessa]

an extra note for whoever bumps into this -- we will be investigating the operational effort in adding a new layer per Python version in the near future.