claranet / terraform-aws-lambda

Terraform module for AWS Lambda functions

DEPRECATION WARNING

This module is lacking active maintainance and is being deprecated. Users are encouraged to migrate to the actively maintained https://github.com/terraform-aws-modules/terraform-aws-lambda community module. This repository will be marked as archived but will stay online for the foreseeable future.

terraform-aws-lambda

This Terraform module creates and uploads an AWS Lambda function and hides the ugly parts from you.

Features

• Only appears in the Terraform plan when there are legitimate changes.
• Creates a standard IAM role and policy for CloudWatch Logs.
  • You can add additional policies if required.
• Zips up a source file or directory.
• Installs dependencies from requirements.txt for Python functions.
  • It only does this when necessary, not every time.

Requirements

• Python 2.7 or higher
• Linux/Unix/Windows

Terraform version compatibility

Module version	Terraform version
1.x.x	0.12.x
0.x.x	0.11.x

Usage

module "lambda" {
  source = "github.com/claranet/terraform-aws-lambda"

  function_name = "deployment-deploy-status"
  description   = "Deployment deploy status task"
  handler       = "main.lambda_handler"
  runtime       = "python3.6"
  timeout       = 300

  // Specify a file or directory for the source code.
  source_path = "${path.module}/lambda.py"

  // Add additional trusted entities for assuming roles (trust relationships).
  trusted_entities = ["events.amazonaws.com", "s3.amazonaws.com"]

  // Attach a policy.
  policy = {
    json = data.aws_iam_policy_document.lambda.json
  }

  // Add a dead letter queue.
  dead_letter_config = {
    target_arn = aws_sqs_queue.dlq.arn
  }

  // Add environment variables.
  environment = {
    variables = {
      SLACK_URL = var.slack_url
    }
  }

  // Deploy into a VPC.
  vpc_config = {
    subnet_ids         = [aws_subnet.test.id]
    security_group_ids = [aws_security_group.test.id]
  }
}

Inputs

Inputs for this module are the same as the aws_lambda_function resource with the following additional arguments:

Name	Description	Type	Default	Required
source_path	The absolute path to a local file or directory containing your Lambda source code	string		yes
build_command	The command to run to create the Lambda package zip file	string	"python build.py '$filename' '$runtime' '$source'"	no
build_paths	The files or directories used by the build command, to trigger new Lambda package builds whenever build scripts change	list(string)	["build.py"]	no
cloudwatch_logs	Set this to false to disable logging your Lambda output to CloudWatch Logs	bool	true	no
lambda_at_edge	Set this to true if using Lambda@Edge, to enable publishing, limit the timeout, and allow edgelambda.amazonaws.com to invoke the function	bool	false	no
policy	An additional policy to attach to the Lambda function role	object({json=string})		no
trusted_entities	Additional trusted entities for the Lambda function. The lambda.amazonaws.com (and edgelambda.amazonaws.com if lambda_at_edge is true) is always set	list(string)		no
enabled	Enabling and disaling of resources	bool	true	no

The following arguments from the aws_lambda_function resource are not supported:

• filename (use source_path instead)
• role (one is automatically created)
• s3_bucket
• s3_key
• s3_object_version
• source_code_hash (changes are handled automatically)

Outputs

Name	Description
function_arn	The ARN of the Lambda function
function_invoke_arn	The Invoke ARN of the Lambda function
function_name	The name of the Lambda function
function_qualified_arn	The qualified ARN of the Lambda function
role_arn	The ARN of the IAM role created for the Lambda function
role_name	The name of the IAM role created for the Lambda function

terraform-aws-lambda's Issues

Support pipenv

It would be nice if pipenv was supported.

https://pipenv.readthedocs.io/en/latest/

It doesn't have a requirements.txt file, instead it has Pipfile and Pipfile.lock files

pipenv run pip install -r <(pipenv lock -r) --target .

Source: pypa/pipenv#746 (comment)

Would be enough to install all of its dependencies

Simplify logic where ternary operator can return a list directly

The ternary operator can return any type now. Rather than machinations with slice/list/etc, just return the desired list value directly.

https://www.terraform.io/docs/configuration/expressions.html#conditional-expressions

Originally posted by @lorengordon in #49

Windows support?

Thanks for this awesome module! I tend to flip between platforms a lot, developing on Windows for Linux-based systems. If I can get this working on Windows, would you be open to a pull request? Looks like some folks have been recommending/doing things that definitely wouldn't work, like using python2/python3 executables, relying on the shebang, etc... So, if you want to keep that, then I won't bother and will just keep using the Windows subsystem for Linux.

Leading quote in `source_path` causing failures on Windows

Discovered an issue in the new custom build script implementation that causes failures on Windows... I believe there are some differences in how shells or sys.argv are interpreting arguments with quotes around them. Here is the error:

null_resource.archive (local-exec): Traceback (most recent call last):
null_resource.archive (local-exec):   File "build.py", line 129, in <module>
null_resource.archive (local-exec):     with cd(source_dir):
null_resource.archive (local-exec):   File "C:\Python36\lib\contextlib.py", line 81, in __enter__
null_resource.archive (local-exec):     return next(self.gen)
null_resource.archive (local-exec):   File "build.py", line 23, in cd
null_resource.archive (local-exec):     os.chdir(path)
null_resource.archive (local-exec): OSError: [WinError 123] The filename, directory name, or volume label syntax is incorrect: ''C:\\...

You can see the leading ' in the directory name, there at the end. Note that there is no closing quote. I'm obviously removing the full path, but there is indeed no closing quote by the time the value gets to os.chdir(path).

I threw in some debug statements to see where there are differences between linux and windows, and I can see that hash.py is outputting the json the same way on both (thankfully). The difference appears to be where build.py is reading the arguments from sys.argv.

diff --git a/build.py b/build.py
index c32737e..38de3ad 100644
--- a/build.py
+++ b/build.py
@@ -108,6 +108,7 @@ filename = sys.argv[1]
 runtime = sys.argv[2]
 source_path = sys.argv[3]

+print('source_path = {0}'.format(source_path))
 absolute_filename = os.path.abspath(filename)

 # Create a temporary directory for building the archive,

On linux, that will output the following, with no quotes at all:

null_resource.archive (local-exec): source_path = /home/...

On windows, it outputs the following, complete with opening and closing quotes:

null_resource.archive (local-exec): source_path = 'C:\...'

Some further debugging reveals that the closing quote is stripped by source_dir = os.path.dirname(source_path).

The behavior of os.path.dirname is the same on windows and linux, so the failure is simply due to how the value of sys.argv[3] is not wrapped in quotes on linux but is wrapped in quotes on windows. The quotes come from the default value in variables.tf.

Still noodling possible fixes...

No release was created for the latest commit in master branch

Hi! Latest commit to master branch is this: f9ff6ee

However, latest release tag is this: https://github.com/claranet/terraform-aws-lambda/commits/v1.1.0 and it's pointing to this commit: ace2bc9 which is one commit behind f9ff6ee (master).

Could you create a new release tag for the latest commit in master branch? Thanks!

CloudWatch Log Group not automagically created

There's a small problem I'm having when using the module: The CloudWatch Log Group is only created when the Lambda Function tries to send its first logs. This poses a challenge when you want to use that Log Group somewhere else which relies on its existence (in my use case, Kinesis Data Stream).

I'll gladly put in a PR to create the Log Group if enable_cloudwatch_logs is true, but was just wondering whether this was something that has come up before?

Code example:

module "lambda" {
  source = "github.com/claranet/terraform-aws-lambda"

  function_name = "foo"
  description   = "foo"
  handler       = "foo.lambda_handler"
  runtime       = "python3.7"
  timeout       = 300

  source_path = "${path.module}/files/foo.py"
}

resource "aws_kinesis_stream" "bar" {
  name        = "bar"
  shard_count = "1"
}

resource "aws_cloudwatch_log_subscription_filter" "foo_to_bar" {
  name            = "foo-to-bar"
  log_group_name  = "/aws/lambda/foo"
  filter_pattern  = ""
  destination_arn = "${aws_kinesis_stream.bar.arn}"
}

Missing file error

If you delete the lambda function from the AWS console, and try to apply using this module again, it will try to create the lambda function using a filename that may or may not exist on disk. It might not exist because it was generated on someone else's laptop, or it was created on disk a long time ago and then cleaned up.

A workaround is to edit the lambda source (e.g. add a comment), apply, undo, apply.

Doesn't seem to work with version 12

Have the following code
environment {
variables {
BUCKET_NAME = "${aws_s3_bucket.s3_bucket.id}"
}
}

getting the following error

22: environment {

Blocks of type "environment" are not expected here. Did you mean to define
argument "environment"? If so, use the equals sign to assign it a value.

Support golang

Hi, Is there a version of this that supports go workflow?

Cheers!

Installs pip dependencies always using 'pip'

Instead of installing requirements.txt using pip2 or pip3 the module currently always installs using 'pip' this results in undefined behaviour as pip can either point to pip3 or pip2.

This can result in the wrong version of the required modules being installed for the Lambda function.

Download dependencies using --platform flag in pip

Currently this project downloads dependencies using pip install, but if you rely on any packages which require a native extension you'll have problems unless you are building from a machine with the same platform (in pip's parlance).

AWS provider 2.0.0 disables the function

https://github.com/terraform-providers/terraform-provider-aws/blob/master/CHANGELOG.md#200-february-27-2019

resource/aws_lambda_function: Setting reserved_concurrent_executions to 0 will now disable Lambda Function invocations, causing downtime for the Lambda Function. Previously reserved_concurrent_executions accepted 0 and below for unreserved concurrency, which means it was not previously possible to disable invocations. The argument now differentiates between a new value for unreserved concurrency (-1) and disabling Lambda invocations (0). If previously configuring this value to 0 for unreserved concurrency, update the configured value to -1 or the resource will disable Lambda Function invocations on update. If previously unconfigured, the argument does not require any changes. See the Lambda User Guide for more information about concurrency.

1.3.0 generates perpetual diffs when switching platform

We run terraform commands on both MacOS (dev) and Linux (CI). Since version 1.3.0 when a stack calling this module is applied in either environment it diffs in the other one and vice versa. Had to pin 1.2.0 to fix it.

I would guess this behavior is a result of 6632b89 @pdecat.

Run python scripts with python2

The terraform currently using the ambiguous 'python' command to call hash.py. This can be either python3 or python2 depending on what the default is on the user's Linux distribution.

hash.py is known to work fine with python2 so I suggest we just change python to python2 here:

terraform-aws-lambda/archive.tf

Line 4 in 7b8d42e

program = ["python", "${path.module}/hash.py"]

Cannot run on terraform entreprise

Just leaving this out there - This module cannot run on terraform entreprise because the zip command is not present. Not sure if there is a workaround for this..

* module.taskPlanner_cron.module.lambda.data.external.built: data.external.built: failed to execute "/terraform/scheduling-service/.terraform/modules/c205777ab1dadf9cdd9611349aa2639d/built.py": Traceback (most recent call last):
  File "/terraform/scheduling-service/.terraform/modules/c205777ab1dadf9cdd9611349aa2639d/build.py", line 146, in <module>
    create_zip_file(temp_dir, absolute_filename)
  File "/terraform/scheduling-service/.terraform/modules/c205777ab1dadf9cdd9611349aa2639d/build.py", line 100, in create_zip_file
    run('zip', '-r', target_file, '.')
  File "/terraform/scheduling-service/.terraform/modules/c205777ab1dadf9cdd9611349aa2639d/build.py", line 70, in run
    subprocess.check_call(args, **kwargs)
  File "/usr/lib/python3.5/subprocess.py", line 576, in check_call
    retcode = call(*popenargs, **kwargs)
  File "/usr/lib/python3.5/subprocess.py", line 557, in call
    with Popen(*popenargs, **kwargs) as p:
  File "/usr/lib/python3.5/subprocess.py", line 947, in __init__
    restore_signals, start_new_session)
  File "/usr/lib/python3.5/subprocess.py", line 1551, in _execute_child
    raise child_exception_type(errno_num, err_msg)
FileNotFoundError: [Errno 2] No such file or directory: 'zip'
Traceback (most recent call last):
  File "/terraform/scheduling-service/.terraform/modules/c205777ab1dadf9cdd9611349aa2639d/built.py", line 31, in <module>
    subprocess.check_output(build_command, shell=True)
  File "/usr/lib/python3.5/subprocess.py", line 626, in check_output
    **kwargs).stdout
  File "/usr/lib/python3.5/subprocess.py", line 708, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '/terraform/scheduling-service/.terraform/modules/c205777ab1dadf9cdd9611349aa2639d/build.py=[redacted]' returned non-zero exit status 1

module_relpath seems to be broken under version 0.12.0

I am fitting the module to hcl2.0 namely terraform version 0.12.0, but has difficulty to understand the path variable in archive.tf

locals {
  module_relpath = "${substr(path.module, length(path.cwd) + 1, -1)}"
}

what does it really mean ? shouldn't it be ?

module_relpath = "${path.module}"

thanks

CloudWatch Alarm

Make it create an alarm for invoke errors and notify any SNS topics that have been passed in via variables.

Add an "exclude" variable to exclude certain directories

I think it would be a good idea to allow for directory exclusion.
Just as an example, sometimes you have a local copy of a node_modules/ directory, but you have no intention of uploading it (or some of its modules).

Question - How do you provide and use python requirements.txt

The documentation indicates that a requirements.txt can be used to automatically build with the necessary python modules. Where is requirements.txt placed? How is it referenced? Documentation could be expanded a bit for this portion.

Thanks

timeout and memory_limit lambda not working

Hello!
setting timeout and memory_limit as lambda parameters terraform does not apply this params

Consider a variable that makes the module/resources optional

Since terraform doesn't support using count on modules (open issue), it is rather difficult to optionally include a module in a terraform config. The pattern I've seen to work around this has been to expose a new variable, maybe something like create_module or create_lambda, and use that to interpolate the count in every resource within the module. Yeah, it's a bit ugly.

Here's an example of what I mean:

• https://github.com/terraform-aws-modules/terraform-aws-vpc#conditional-creation

In particular, I find I need to do this rather often when writing configs that need to work both in the commercial regions and in the GovCloud regions. Since GovCloud is missing a lot of services, I need a way to make certain resources optional when it's the target region.

I'd be willing to work this if you like.

Generating node_modules for node runtimes

Any thoughts on running something like npm install --production to create the node_modules directory prior to zipping up a directory for node runtimes? Generally, the directory is added to a .gitignore, so the dependencies are not present unless installed first.

Better documentation for how to deploy non python lambda functions

I started using this module for a Java11 lambda function and I was having a hard time making the lambda function execute as I kept getting an error message saying that my class was not found.

I later realize that the module ZIPs the jar I provided and this causes it to break.

I spent some time looking at the code and I realize that a simple workaround would be to use the build_command property like so:

  build_command = "cp '$source' '$filename' "

Improving the documentation can help with the deployment of functions that do not require to be ZIP, like Java, or are already ZIP.

pip error when running make plan on debian

When running make plan on Debian the pip command that is ran to install the python modules:

pip install -r requirements.txt -t .

Seems to trigger the problem documented here: https://stackoverflow.com/questions/4495120/combine-user-with-prefix-error-with-setup-py-install

It appears that this can be partially fixed by setting up ~/.pydistutils.cfg to disable the prefix option however for this to work the pip install command needs to be ran with --user. This is a problem for the module as it does not specify this option.

Should we just specify --user as part of the command? If so only on Debian or on all distros?

Thanks,

Alan Jenkins