amazon-archives / cloudformation-validation-pipeline

WARNING- This package is no longer supported and will be replaced in the near future. An automated CI/CD Pipeline solution to help accelerate AWS CloudFormation template development

Usage

A demo and deployment guide describing this solution are available on the AWS Answers website.

To start writing your own tests and pipelines to test your templates it's likely your going to want to customize which tests are run and even write your own tests.

To ensure that any binary dependencies you may have are built for the correct platform it is recommended to create a new ec2 instance using the amzn-ami-hvm-2016.03.3.x86_64-gp2 AMI and run the build process from there.

The following s3 permissions are required for the IAM user/role configured in the aws cli:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:CreateBucket",
                "s3:PutObject",
                "s3:PutObjectAcl",
                "s3:ListBucket",
                "s3:GetObject"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}

Ensure that pip and setuptools are up to date:

sudo -H pip install --upgrade setuptools pip

Clone the validation pipeline repo:

git clone https://github.com/awslabs/aws-cloudformation-validation-pipeline.git

Install:

cd aws-cloudformation-validation-pipeline
sudo -H python setup.py install

Now you're ready to start building tests and pipelines, to get going create a project skeleton:

cd ~/
cfn-validation-pipeline-skeleton
cd validation_pipeline

In the validation_pipeline folder you will find all the required cloudformation templates, lambda function code, html docs and unit tests

Building/deploying dependencies, Lambda function zips and CloudFormation templates to S3 is done using the deployment utility:

cfn-validation-pipeline-deploy my-cfn-pipeline-bucket

Note: this command has several configurable options, to view help for the deployment tools run the command with the --help argument

Now you can launch stacks from the cloudformation console using the s3 location for the desired cloudformation template in the output of the previous command.

Creating your own tests

This can be done by creating a new folder in the 'lambda_functions' folder. The folder should contain:

• python file(s) - this is installed into the lambda function and executed when the pipeline runs.
• requirements.txt [optional] - a list of all python packages that are required. Can include pip modules or custom modules stored in the 'lambda_functions/lib/' folder

Using the CFNPipeline class in your function greatly simplifies writing tests and is recommended. The included tests can be used as examples on usage for both pre-create and post-create tests. The validate_template and ami_check are fairly simple examples of pre and post create tests respectively. HTML API docs for usage of CFNPipeline are available in the 'docs' folder.

Adding tests to the template and pipeline

Custom tests can be added to the 'cloudformation/central-microservices.template' file and added to the pipeline by modifying 'cloudformation/main-pipeline.template' file.

Once you've authored your new tests and pipeline it can be deployed to s3 using the cfn-validation-pipeline-deploy command.

cloudformation-validation-pipeline's Issues

Lambda function getting timed out while filtering cloudwatch logs in Generate Report stage

When I invoke the pipeline to validate the cfn template, the template gets validated at validate_template and Lint_template stages post which a report was getting generated and send.
After multiple execution of the pipeline, the generate_template lambda function is failing with error:
Action execution failed
The AWS Lambda function Generate_Report-XXXXX failed to return a result. Check the function to verify that it has permission to call the PutJobSuccessResult action and that it made a call to PutJobSuccessResult.

There are no errors logged in the cloudwatch logs. It only says that: Task timed out after 30.03 seconds

I tried increasing the timeout for the lambda function but still the same error

Experiencing failure in Codepipeline due to "gem install cfn-nag"

Lambda Lint_Template is failing with the following error --
"message": "1 lint failures [[u'TemplateArtifact', 'aws-vpc.template', u'linting of template aws-vpc.template failedError while executing command: gem install cfn-nag. Reason: exit status 1 - COMMAND_EXECUTION_ERROR']]",

Integrate cfn-python-lint

Would be nice if instead of cfn-nag, we had the option to use cfn-python-lint: https://github.com/aws-cloudformation/cfn-python-lint.

licensing question

hi @hyandell , @shsenior ,
like this solution and amount of work you've done to write all that functionality.

I'd like to integrate some of these libraries/files (with partial modifications) into solutions I create for my customers.

Since this repository is covered under Amazon Software License, a non open-source licence, I have clarify first:

• am I allowed to use it within my solutions?
• am I allowed to modify it?
• what else I should worry before using it?

Thank you in advance.