aws-samples / aws-lambda-layer-awscli

AWS Lambda Layer for AWS CLI

lambda-layer-awscli

AWS CDK(Cloud Development Kit) comes with lambda-layer-awscli which allows you to build your private AWS Lambda layer with AWS CLI executable. This repository demonstrates how to create your own AWS Lambda layer with AWS CLI in AWS CDK.

Basic Usage

import { App, CfnOutput, Construct, Stack, StackProps } from '@aws-cdk/core';
import * as layer from '@aws-cdk/lambda-layer-awscli';

export class MyStack extends Stack {
  constructor(scope: Construct, id: string, props: StackProps = {}) {
    super(scope, id, props);

    const awscliLayer = new layer.AwsCliLayer(this, 'AwsCliLayer');
    new CfnOutput(this, 'LayerVersionArn', { value: awscliLayer.layerVersionArn })

  }
}

const devEnv = {
  account: process.env.CDK_DEFAULT_ACCOUNT,
  region: process.env.CDK_DEFAULT_REGION,
};

const app = new App();

new MyStack(app, 'awscli-layer-stack', { env: devEnv });

app.synth();

After deployment, the AWS Lambda layer version ARN will be returned and you can use this ARN in your Lambda functions in the same AWS region.

Outputs:
awscli-layer-stack.LayerVersionArn = arn:aws:lambda:us-east-1:123456789012:layer:AwsCliLayerF44AAF94:34

Customize your layer

The AwsCliLayer from AWS CDK upstream does not allow you to pass custom Dockerfile(see the built-in Dockerfile). To customize the layer, we simply create our own AwsCliLayer construct class in our CDK application with our custom Dockerfile.

cd src/custom-layer
# edit and customize the Dockerfile under the `custom-layer` directory
# login ECR public
aws ecr-public get-login-password --region us-east-1 | docker login --username AWS --password-stdin public.ecr.aws
# generate the layer.zip from Dockerfile
bash build.sh

Now prepare your custom AwsCliLayer construct class and run cdk deploy to generate your own layer.

import { App, CfnOutput, Construct, Stack, StackProps } from '@aws-cdk/core';
import * as layer from '@aws-cdk/lambda-layer-awscli';
import * as customlayer from './custom-layer/custom-layer';

export class CustomLayerStack extends Stack {
  constructor(scope: Construct, id: string, props: StackProps = {}) {
    super(scope, id, props);

    const awscliLayer = new customlayer.AwsCliLayer(this, 'CustomAwsCliLayer');
    new CfnOutput(this, 'LayerVersionArn', { value: awscliLayer.layerVersionArn });
  }
}

// for development, use account/region from cdk cli
const devEnv = {
  account: process.env.CDK_DEFAULT_ACCOUNT,
  region: process.env.CDK_DEFAULT_REGION,
};

const app = new App();

new CustomLayerStack(app, 'custom-awscli-layer-stack', { env: devEnv });

app.synth();

License Summary

This sample code is made available under the MIT-0 license. See the LICENSE file.

aws-lambda-layer-awscli's Issues

Error: fork/exec /var/task/bootstrap: permission denied

Hello,
thanks for this great work, this Repo. saved my day.

I just work through aws-version example and deploy CDK stack from AWS Cloud9, everything works.

then I tried to do the cdk synth and deploy from GitHub Actions, deploy process was done successfully but I got Lambda Runtime Error while I press the test button in the AWS Console.

{
"errorMessage": Error: fork/exec /var/task/bootstrap: permission denied",
"errorType": "Runtime.InvalidEntrypoint"
}

The following step I tried but no help.

1. chmod +x for main.sh bootstrap then push to GitHub
2. run chmod +x in GitHub Actions then run cdk synth and deploy

run ls -al in GitHub Actions and confirm the permission should be fine, but still got runtime error in Lambda.

ls -al ./src/stacks/lambda/handler/awscliFunc.d/
total 16
drwxr-xr-x 2 runner docker 4096 Apr 6 03:21 .
drwxr-xr-x 3 runner docker 4096 Apr 6 03:21 ..
-rwxrwxr-x 1 runner docker 1046 Apr 6 03:21 bootstrap
-rwxrwxr-x 1 runner docker 66 Apr 6 03:21 main.sh

Any suggestion will be helpful.

Issue with libpython2.7.so.1.0 when running on nodejs v10 runtime

Hello,

I am getting an error with the following code:

const shell = require('child_process').execSync

module.exports.handler = () => {
    const version = shell('/opt/awscli/aws --version')
    console.log(version)
}

The error is as following: Command failed: /opt/awscli/aws --version\n/opt/awscli/bin/python: error while loading shared libraries: libpython2.7.so.1.0: cannot open shared object file: No such file or directory\n

OBS: I got the layer by clicking this button:

What might have been wrong with my deployment?! Thank a lot for your work :)

Unable to get the awscli call to run, it always errors out

I am running the following command:

/opt/awscli/aws --version

expecting following output
aws-cli/1.16.213 Python/2.7.15+ Linux/4.15.0-1043-aws botocore/1.12.203

but getting this error instead

usage: aws [options] <command> <subcommand> [<subcommand> ...] [parameters]
To see help text, you can run:

  aws help
  aws <command> help
  aws <command> <subcommand> help
aws: error: too few arguments

AwsCliLayer has no compatible runtimes defined

When I create this layer per the README, the generated layer does not have any compatible runtimes, which means I cannot add it to any of my Lambda functions.

Looks like the runtimes need to be specified here: https://github.com/aws-samples/aws-lambda-layer-awscli/blob/main/src/custom-layer/custom-layer.ts#L12-L18

I'm attempting to work around this locally like the below, but it's not working yet. Will update if I get a good workaround.

import { App, CfnOutput, Construct, Stack, StackProps } from '@aws-cdk/core';
import * as lambdaLayerAwscli from '@aws-cdk/lambda-layer-awscli';

// https://www.typescriptlang.org/docs/handbook/2/mapped-types.html#mapping-modifiers
// we want to strip the readonly property from lambdaLayerAwscli.AwsCliLayer.compatibleRuntimes,
// and this is a clean way to do it.
type CreateMutable<Type> = {
  -readonly [Property in keyof Type]: Type[Property];
};

type AwsCliLayerWithRuntimes = CreateMutable<lambdaLayerAwscli.AwsCliLayer>

export class MyStack extends Stack {
  constructor(scope: Construct, id: string, props: StackProps = {}) {
    super(scope, id, props);

    const awscliLayer: AwsCliLayerWithRuntimes = new lambdaLayerAwscli.AwsCliLayer(this, 'AwsCliLayer');
    awscliLayer.compatibleRuntimes = [
      lambda.Runtime.PROVIDED_AL2,
    ]
    new CfnOutput(this, 'LayerVersionArn', { value: awscliLayer.layerVersionArn })
  }
}

Not compatible with runtime `PROVIDED_AL2`

I manually uploaded layers.zip from this package to a new Lambda layer with --compatible-runtimes provided.al2.

When I invoke /opt/awscli/aws --version on my Lambda, I get this error:

/opt/awscli/bin/python: error while loading shared libraries: libpython2.7.so.1.0: cannot open shared object file: No such file or directory

It works fine if I re-upload the layer as --compatible-runtimes provided and switch my function to use "Custom runtime".

Slow execution of basic aws --version command

I'm checking to see if this is a just me thing or if others are experiencing similar performance.

Specs:

• Custom lamba layer (sudo make layer-build-python27)
• Node.js 12
• 256mb

With a basic handler function to simply show the aws version, it is taking 6 seconds to execute.

const util = require('util');
const exec = util.promisify(require('child_process').exec);

exports.handler = async (event) => {

    ({ stdout, stderr } = await exec('/opt/awscli/aws --version'));

    console.log(`stdout: ${stdout}`);
    console.error(`stderr: ${stderr}`);

    // TODO implement
    const response = {
        statusCode: 200,
        body: JSON.stringify('Hello from Lambda!'),
    };
    return response;
};

Does it support awscli v2?

upgrade proxy-agent to address the security vulnerability

Depends on

• aws/aws-cdk#16257

Does this layer support Python3.8 runtime?

Getting '/opt/awscli/bin/python: error while loading shared libraries: libpython2.7.so.1.0: cannot open shared object file: No such file or directory' during execution of aws cli command
Looks like I have to use lambda- python 3.8 docker image to build the layer, am i missing something?
@pahud

Does this work with Ruby runtimes?

Forgive my ignorance. I’m trying to figure how to add aws s3 cli as a layer to my ruby function. Im not exactly sure how to package and deploy these things so I wanted to make sure - does this work with Ruby 2.5 or Ruby 2.7 runtime?

Problems with Dockerfile

Hi Pahud.
I ran into an issue using this layer that looks to me like you left a few semicolons in the COPY statements for aws, jq and make that shouldn't be there.
Additionally I have extended the PATH variable in the end to the new directory.

corrected:

FROM amazonlinux:latest as builder

WORKDIR /root

RUN yum update -y && yum install -y unzip make wget

ADD https://s3.amazonaws.com/aws-cli/awscli-bundle.zip /root
    
RUN unzip awscli-bundle.zip && \
    cd awscli-bundle;
    
#RUN ./awscli-bundle/install -i /usr/local/aws -b /usr/local/bin/aws
RUN ./awscli-bundle/install -i /opt/awscli -b /opt/awscli/aws
  
# install jq
RUN wget https://github.com/stedolan/jq/releases/download/jq-1.6/jq-linux64 \
&& mv jq-linux64 /opt/awscli/jq \
&& chmod +x /opt/awscli/jq
  
#
# prepare the runtime at /opt/awscli
#
  
FROM lambci/lambda:provided as runtime

USER root

RUN yum install -y zip 

COPY --from=builder /opt/awscli/lib/python2.7/site-packages/ /opt/awscli/ 
COPY --from=builder /opt/awscli/bin/ /opt/awscli/bin/ 
COPY --from=builder /opt/awscli/bin/aws /opt/awscli/aws 
COPY --from=builder /opt/awscli/jq /opt/awscli/jq
COPY --from=builder /usr/bin/make /opt/awscli/make 

# remove unnecessary files to reduce the size
RUN rm -rf /opt/awscli/pip* /opt/awscli/setuptools* /opt/awscli/awscli/examples

#&& mv /opt/awscli/site-packages/* /opt/awscli/

# get the version number
RUN grep "__version__" /opt/awscli/awscli/__init__.py | egrep -o "1.[0-9.]+" > /AWSCLI_VERSION

# zip up
RUN cd /opt; zip -r ../layer.zip *; \
echo "/layer.zip is ready"; \
ls -alh /layer.zip

#extend PATH with new layer
ENV PATH="/opt/awscli:${PATH}"

Thanks for some really useful work otherwise.

publish to SAR with the same SemanticVersion as awscli

refactor this repo with @aws-cdk/lambda-layer-awscli

Background

This repository was initially created when AWS Lambda layer was announced in 2018, during which time AWS CDK was not available and we had to build everything from scratch to create our own lambda layer with AWS CLI.

As AWS CDK now comes with @aws-cdk/lambda-layer-awscli , I am not planning to maintain this layer, instead, I am planning to refactor this repo to align with AWS CDK @aws-cdk/lambda-layer-awscli construct library instead.

What will change

1. Instead of sharing how to build the layer from scratch, we are planning to provide a sample CDK script which leverages the upstream @aws-cdk/lambda-layer-awscli to deploy your own awscli lambda layer in your own account. And you will be able to customize it in the Dockerfile provided. This means you can determine which will be bundled into this layer by customizing the Dockerfile.

2. The SAR for aws-lambda-layer-awscli will not update anymore and will be removed from the README. You are encouraged to build and deploy your own layer with AWS CDK.

If you are interested in the difference between this layer and @aws-cdk/lambda-layer-awscli , check this issue comment for more details:
aws/aws-cdk#13979 (comment)

Any feedbacks are appreciated.

permission denied

I am getting a permission denied error

Public layer?

Not sure what’s involved, but I’m wondering if it’d be possible to publish a public version of this layer that could be used directly?

Type error

Hi pahud.

This cp Makefile.lambda ./lambda-bundle/Makefile; in below section of Makefile command should be cp Makefile ./lambda-bundle/Makefile;

func-zip:
	rm -rf ./lambda-bundle; mkdir ./lambda-bundle
	chmod +x main.sh
	cp main.sh bootstrap ./lambda-bundle;
	cp Makefile.lambda ./lambda-bundle/Makefile;

update Makefile and use the SAM CLI for deployment

Parameterize `RetentionPolicy`

First, thank you for making this handy Layer available! 💯

Problem ❓

The template has hard-coded RetentionPolicy: Retain

Proposed Solution: 👍

Add a RetentionPolicy parameter, we can configure when using from SAR.

ie.

LambdaLayerAwsCli:
  Type: AWS::Serverless::Application
  Properties:
    Parameters:
      RetentionPolicy: Delete
    Location:
      ApplicationId: arn:aws:serverlessrepo:us-east-1:903779448426:applications/lambda-layer-awscli
      SemanticVersion: 1.18.142

Not able to run aws cli commands in lambda function

Not able to run aws cli commands after adding this lambda layer, Here is my code

import subprocess
import logging

logger = logging.getLogger()
logger.setLevel(logging.INFO)

def run_command(command):
    command_list = command.split(' ')
    #print(command_list)
    try:
        logger.info("Running shell command: \"{}\"".format(command))
        result = subprocess.run(command_list, stderr=subprocess.STDOUT, stdout=subprocess.PIPE);
        #print(result)
        logger.info("Command output:\n---\n{}\n---".format(result.stdout.decode('UTF-8')))
    except Exception as e:
        logger.error("Exception: {}".format(e))
        return False

    return result

def lambda_handler(event, context):
    
    run_command('/opt/awscli/aws --version')
    run_command('/opt/awscli/aws s3 ls')

It is failing with these errors

Function Logs
START RequestId: 10dd8da3-3325-4195-a522-ed79d0120430 Version: $LATEST
[INFO]	2021-10-15T22:25:29.943Z	10dd8da3-3325-4195-a522-ed79d0120430	Running shell command: "/opt/awscli/aws --version"
[INFO]	2021-10-15T22:25:29.967Z	10dd8da3-3325-4195-a522-ed79d0120430	Command output:
---
/opt/awscli/bin/python: error while loading shared libraries: libpython2.7.so.1.0: cannot open shared object file: No such file or directory

---
[INFO]	2021-10-15T22:25:29.985Z	10dd8da3-3325-4195-a522-ed79d0120430	Running shell command: "/opt/awscli/aws s3 ls"
[INFO]	2021-10-15T22:25:30.5Z	10dd8da3-3325-4195-a522-ed79d0120430	Command output:
---
/opt/awscli/bin/python: error while loading shared libraries: libpython2.7.so.1.0: cannot open shared object file: No such file or directory

---
END RequestId: 10dd8da3-3325-4195-a522-ed79d0120430
REPORT RequestId: 10dd8da3-3325-4195-a522-ed79d0120430	Duration: 83.23 ms	Billed Duration: 84 ms	Memory Size: 128 MB	Max Memory Used: 38 MB	Init Duration: 126.17 ms

Request ID
10dd8da3-3325-4195-a522-ed79d0120430