Bug: High init duration when deployed through SAM. about aws-sam-cli

[lucashuy]

Hi, I was not able to reproduce your slow cold start with a project deployed via SAM CLI:

SAM CLI

1. sam init a Python 3.11 hello world template
2. Modified the template to add the following resource:

  HelloWorldLayer:
    Type: AWS::Serverless::LayerVersion
    Properties:
      ContentUri: ./layer
      CompatibleRuntimes:
        - python3.11
      CompatibleArchitectures:
        - x86_64
    Metadata:
      BuildMethod: python3.11

The only thing that is inside of ./layer is a requirements.txt file with the same dependencies noted in the original issue.

3. Modified the hello world function to add the following new properties:

EphemeralStorage:
  Size: 512
Layers:
  - !Ref HelloWorldLayer

4. Update the handler to import a package that is defined from the layer (just import, not using any methods)
5. Ran sam build -u
6. Ran sam deploy
7. Ran sam remote invoke

$ sam remote invoke --stack-name sam-app-6715-2
Invoking Lambda Function HelloWorldFunction                                                                                                                         
START RequestId: ee5bfe82-1b21-4c20-af40-fc9eaea477c3 Version: $LATEST
END RequestId: ee5bfe82-1b21-4c20-af40-fc9eaea477c3
REPORT RequestId: ee5bfe82-1b21-4c20-af40-fc9eaea477c3  Duration: 1.43 ms       Billed Duration: 2 ms   Memory Size: 128 MB     Max Memory Used: 76 MB  Init Duration: 679.37 ms
{"statusCode": 200, "body": "{\"message\": \"hello world\"}"}%

AWS CLI

When creating a Lambda manually via the AWS CLI:

1. Zipped the function and layer built directory from previous sam build
2. Create a Lambda layer:

aws lambda publish-layer-version --layer-name manual-layer3 --zip-file fileb://hellolayer.zip --compatible-runtimes python3.11 --compatible-architectures x86_64

3. Create a Lambda function:

aws lambda create-function --function-name manual3 --role <some role that is able to `InvokeFunction`> --layers <layer ARN from previous command> --zip-file fileb://hellofunction.zip --handler app.lambda_handler --runtime python3.11 --ephemeral-storage Size=512

4. Invoke the Lambda function

$ aws lambda invoke --function-name manual3 /dev/stdout       
{"statusCode": 200, "body": "{\"message\": \"hello world\"}"}%

Inspecting the Cloudwatch logs for the function, we can see that the init time is roughly the same as one the one deployed with SAM CLI:

REPORT RequestId: 44daa145-cf6f-4ccb-a86c-379107756e76	Duration: 1.54 ms	Billed Duration: 2 ms	Memory Size: 128 MB	Max Memory Used: 76 MB	Init Duration: 665.80 ms

Is there a reproducible project that you can share somewhere where we can reproduce this? Something interesting to point out, is that in your function runtime logs it looks like the memory is being maxed out. This may have something to do with the slow init times?

[AmruthjithSunil]

Found the issue. It was because of same module importing twice in a python file which was in the layer.

import phonenumbers
from phonenumbers import geocoder, carrier, timezone

removed second line. Then init duration during warm up reduced from 5s to 0.5s. Also Max Memory Used gone down from 250MB to 70MB.

[hnnasit]

Hi @AmruthjithSunil, glad you found the issue. Does that address your concerns when running sam commands now?

[mildaniel]

Closing as the issue seems to be resolved.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.