awslabs / aws-config-engine-for-compliance-as-code Goto Github PK

YAML ERROR LINE 38 - 39

Background

Description

The CloudFormation Templates should use Fn::Sub in place of Fn::Join with Fn::Ref and Fn::GetAttr as it makes for easier reading.

I believe making this change will make reading the CloudFormation Templates easier.

Documentation

https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/intrinsic-function-reference-sub.html

Desired Change

Use Fn::Sub as it is cleaner

The Application Account CFN is failing its update when a rule is added or modified in the compliance code pipline. The below error is shown.

The AWS Lambda function arn:aws:lambda:xxxxxxx:xxxxxxxxxxxxxxxx:function:RDK-Rule-Function-COMPLIANCERULESETLATESTINSTALLED cannot be invoked.

It appears that this section of the CFN is where the issue is. The only modifications to this CFN has been to add the Compliance account #. The Application account CFN shows this as the Lambda name in the compliance account but the Compliance account CFN does not create a Lambda function with this name.

LambdaFunctionName:
Description: Lambda name in the Compliance Account
Default: RDK-Rule-Function-COMPLIANCERULESETLATESTINSTALLED
Type: String

Situation:

• The pipeline runs every hour
• The datalake is deployed

Challenges:

• After 30 days or so, the pipeline fails due to hitting the change set limit on the datalake CFn.

Reason:

• The datalake gets a change set created everytime the pipeline runs. If no change, the number of change set keep growing.

Workaround:

• delete the change sets manually on the datalake CFn
• Re-run the pipeline

When deploying the Application CFN to the main and other regions in the Compliance account, the config rules are not deploying. I am selecting "False" in the DeployAWSConfig parameters section.

Hi,

While configuring this in a single account and multi account environment, I noticed that the Compliance-Engine-Pipeline pipeline is successful even though Compliance-Rule-Template-Deploy process is not successful if it can't assume a role.

> [Container] 2020/02/01 16:16:53 Running command python ./deploy_rule_templates.py $AWS_DEFAULT_REGION $OUTPUT_BUCKET_NO_REGION $ENGINE_RULE_NAME $OTHER_ACTIVE_REGIONS
> Failed to assume role into remote account. An error occurred (AccessDenied) when calling the AssumeRole operation: User: arn:aws:sts::XXXXXXXXXXXX:assumed-role/ComplianceEngine-CodeBuildRole/AWSCodeBuild-713a7773-7e20-4212-bfa4-0206631304cb is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::XXXXXXXXXXXX:role/service-role/AWSConfigAndComplianceAuditRole-DO-NOT-DELETE
> 
> [Container] 2020/02/01 16:16:55 Phase complete: POST_BUILD State: SUCCEEDED

Not sure if this is specific to my situation, ideally it should fail the build

Another environment variable for AWS default region needs to be added to compliance-account-initial-setup.yaml since - python ./deploy_rule_templates.py $AWS_DEFAULT_REGION $OUTPUT_BUCKET_NO_REGION $ENGINE_RULE_NAME $OTHER_ACTIVE_REGIONS in buildspec_deploytemplates.yaml takes 4 arguments.

  CodeBuildRuleDeployProject:
    Condition: IsMainRegion
    Type: AWS::CodeBuild::Project
    Properties:
      ServiceRole: !Join [ ":", ["arn:aws:iam:", !Ref 'AWS::AccountId', "role/ComplianceEngine-CodeBuildRole"]]
      Artifacts:
        Type: CODEPIPELINE
      Name: Compliance-Rule-Template-Deploy
      Description: Deploy the Config Rules CloudFormation to be deployed
      Environment:
        Type: LINUX_CONTAINER
        ComputeType: BUILD_GENERAL1_SMALL
        Image: aws/codebuild/python:3.6.5
        EnvironmentVariables:
         - Name: AWS_DEFAULT_REGION
           Value: !Ref 'AWS::Region'
         - Name: OUTPUT_BUCKET_NO_REGION
           Value: !Join [ "-", [ !Ref CodebuildDestinationS3BucketConfig, !Ref 'AWS::AccountId']]
         - Name: OTHER_ACTIVE_REGIONS
           Value: !If [ OtherActiveRegions, !Ref OtherActiveRegions, 'none']
         - Name: ENGINE_RULE_NAME
           Value: !Ref EngineComplianceRule

when deploying the Compliance account setup pipeline, im getting an error

[Container] 2019/10/18 19:43:33 Running command if [ "$OTHER_ACTIVE_REGIONS" != "none" ]; then chmod a+x ./rulesets-build/multi-region/deploy_lambda.sh; ./rulesets-build/multi-region/deploy_lambda.sh $OTHER_ACTIVE_REGIONS $ENGINE_RULE_NAME $AWS_DEFAULT_REGION; fi
Deploy in us-east-2
Traceback (most recent call last):
File "/usr/local/bin/rdk", line 11, in
load_entry_point('rdk==0.6.1', 'console_scripts', 'rdk')()
File "/usr/local/lib/python3.6/site-packages/pkg_resources/init.py", line 480, in load_entry_point
return get_distribution(dist).load_entry_point(group, name)
File "/usr/local/lib/python3.6/site-packages/pkg_resources/init.py", line 2693, in load_entry_point
return ep.load()
File "/usr/local/lib/python3.6/site-packages/pkg_resources/init.py", line 2324, in load
return self.resolve()
File "/usr/local/lib/python3.6/site-packages/pkg_resources/init.py", line 2330, in resolve
module = import(self.module_name, fromlist=['name'], level=0)
File "/usr/local/lib/python3.6/site-packages/rdk/cli.py", line 13, in
from rdk import rdk
File "/usr/local/lib/python3.6/site-packages/rdk/rdk.py", line 686
<<<<<<< HEAD
^
IndentationError: expected an indented block

An error occurred (ResourceNotFoundException) when calling the UpdateFunctionConfiguration operation: Function not found: arn:aws:lambda:us-east-2:788247333161:function:RDK-Rule-Function-COMPLIANCERULESETLATESTINSTALLED

In the following file you have the following mistake;

This should be Ruleset_Status_Reporter which would match the Zip file contained in this repository, otherwise CloudFormation will fail when it tries to create. Additionally it might be good to keep the .py file available.

I am not seeing any saved Athena Queries in the "Master" or "Compliance" account. Do they get populated or deployed after the compliance engine runs the first time with rule populated in there?

I added a new manager rule to the pipeline and it is failing during the buildruletemplate phase. Below is the error message. I am not sure what I am missing here. I also have a copy of the json parameters file below.

[Container] 2019/07/09 16:39:21 Phase context status code: COMMAND_EXECUTION_ERROR Message: Error while executing command: rdk deploy -f --all > ../result.txt. Reason: exit status 1

{
"Version": "1.0",
"Parameters": {
"RuleName": "REQUIRED_TAGS",
"SourceRuntime": null,
"CodeKey": null,
"InputParameters": "{}",
"OptionalParameters": "{}",
"SourcePeriodic": "TwentyFour_Hours",
"SourceIdentifier": "REQUIRED_TAGS",
"RuleSets": "baseline"
},
"Tags": "[DefaultRuleSet]"
}

This key value needs to be ConfigRuleArn and not RuleARN to align with parsing in etl_evaluations.py
https://github.com/awslabs/aws-config-engine-for-compliance-as-code/blob/master/rulesets-build/compliance-whitelist.json#L3

The instructions for adding a new region are not complete. The actual steps are as follows:

• Update compliance-account-initial-setup.yaml to add the new region and run in the compliance account, in the new region. This will create the s3 codebuild-output bucket for the new region.
• Create an S3 bucket named config-rule-code-bucket-947791289618-%region%. This should be added to the CFN with an if doesn't exist. Alternatively, you could run rdk init in the new region
• Update the stack in the main region to include the parameter with the new region being added
• Invoke the pipeline, the proper lambda's will now be created.
• Run the application-account-initial-setup.yaml in the child account(s) in the new region

It should also be noted, if creating multiple regions at the beginning, all of the above needs to happen as well. Additionally, the defaults are for the "other regions" to only receive "otherregionsbaseline" tagged rules. Recommend that we change this to the same "baseline" tag as default.

I will fork the repo this weekend and create pull requests with new readme and the bucket creation that is missin

Part 1:
get_configuration(resource_type, resource_id, configuration_capture_time) fails when resource_type == "AWS::KMS::Key" or resource_type == "AWS::::Account".

The reason for this is GetResourceConfigHistory does not support these two resource types.

Part 2:
get_configuration(resource_type, resource_id, configuration_capture_time) also fails when resource_type == "AWS::ApiGateway::RestApi" since resource_id recorded by AWS Config service for ApiGateway::RestApi is of "arn:aws:apigateway:"+compliance_account_region+"::/restapis/"+resource_id format compared to all other resource types vs resource ids where you can pass resource_id as is.

any existing rule or additional rule will break the export as csv.

I setup the compliance and the application account yamls in the same region but I did add additional regional support in the compliance account. Both CFT's ran successfully. The code pipeline failed to deploy config rule successfully. I have the out error below.

[Container] 2019/06/21 00:25:57 Running command python ./deploy_rule_templates.py $AWS_DEFAULT_REGION $OUTPUT_BUCKET_NO_REGION $ENGINE_RULE_NAME $OTHER_ACTIVE_REGIONS
Skipping default.json
Skipping rulesets_list.txt
Traceback (most recent call last):
File "./deploy_rule_templates.py", line 31, in
default_template = json.loads(default_template_obj.get()['Body'].read().decode('utf-8'))
File "/usr/local/lib/python3.6/site-packages/boto3/resources/factory.py", line 520, in do_action
response = action(self, *args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/boto3/resources/action.py", line 83, in call
response = getattr(parent.meta.client, operation_name)(**params)
File "/usr/local/lib/python3.6/site-packages/botocore/client.py", line 320, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/usr/local/lib/python3.6/site-packages/botocore/client.py", line 623, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the GetObject operation: The specified bucket does not exist

[Container] 2019/06/21 00:25:59 Command did not exit successfully python ./deploy_rule_templates.py $AWS_DEFAULT_REGION $OUTPUT_BUCKET_NO_REGION $ENGINE_RULE_NAME $OTHER_ACTIVE_REGIONS exit status 1
[Container] 2019/06/21 00:25:59 Phase complete: BUILD State: FAILED
[Container] 2019/06/21 00:25:59 Phase context status code: COMMAND_EXECUTION_ERROR Message: Error while executing command: python ./deploy_rule_templates.py $AWS_DEFAULT_REGION $OUTPUT_BUCKET_NO_REGION $ENGINE_RULE_NAME $OTHER_ACTIVE_REGIONS. Reason: exit status 1

Many rules have a 'scope' parameter that does not seem to be supported. For example, the rule "Required_Tags" has a scope defining the 'ComplianceResourceTypes' that I would like to limit based on local policy/requirements.

ENV:
Python 3
installed rdk with
python3 -m pip install rdk

sample of config rule
{ "Version": "1.0", "Parameters": { "RuleName": "access_keys_rotated", "SourceRuntime": null, "CodeKey": null, "InputParameters": "{}", "OptionalParameters": "{\"maxAccessKeyAge\": \"30\"}", "SourcePeriodic": "TwentyFour_Hours", "SourceIdentifier": "ACCESS_KEYS_SOMETHING", "RuleSets": [ "tagOne" ] }, "Tags": [] }

CLI used to generate template:
rdk create-rule-template --rulesets tagOne --output-file test.json --rules-only

Output:
Generating CloudFormation template!
Traceback (most recent call last):
File "/usr/local/bin/rdk", line 11, in
sys.exit(main())
File "/usr/local/lib/python3.6/site-packages/rdk/cli.py", line 21, in main
return_val = my_rdk.process_command()
File "/usr/local/lib/python3.6/site-packages/rdk/rdk.py", line 341, in process_command
exit_code = method_to_call()
File "/usr/local/lib/python3.6/site-packages/rdk/rdk.py", line 1806, in create_rule_template
tags_str += "Key={},Value={} ".format(tag['Key'], tag['Value'])]]

Any one have solution to this issue, however if I have one config rule it works but not with multiple one using Rulesets tagging.

unable to deploy application-account-initial-setup.yaml in application account while creating stack using this template it gives status roll back complete @[email protected]

Template validation error: Parameter SourceLambdaBucket failed to satisfy constraint: Enter the name not the ARN, must be in the same region than the stack

is the error when one tries to deploy the CFN template.

Overall, the methodology is invasive (createchangeset and deletechangeset). Reconsidering the approach in read only.
Workaround, do not parameterize the GLOBAL varialbe in SecurityEpic7 lambda.

Currently Have Deployed this in US-east-1 region fine to multiple accounts. I have Deployed Compliance engine and application to us-east-2. I run the Code build which runs correct. The issue is no rules are ever created in the second region. When looking at cloudformation The RDK-Config-Rule-Functions go into a rollback_failed status. Which I have no clue why.

15:28:14 UTC-0500 | ROLLBACK_IN_PROGRESS | AWS::CloudFormation::Stack | RDK-Config-Rule-Functions | The following resource(s) failed to create: [IAMGROUPNOPOLICYFULLSTARLambdaFunction, IAMUSERNOPOLICYFULLSTARLambdaFunction, INTERNETGATEWAYAUTHORIZEDONLYLambdaFunction, IAMROLENOPOLICYFULLSTARLambdaFunction, COMPLIANCERULESETLATESTINSTALLEDLambdaFunction, EBSENCRYPTEDVOLUMESV2LambdaFunction, ROOTNOACCESSKEYLambdaFunction]. . Rollback requested by user.
  | 15:28:14 UTC-0500 | CREATE_FAILED | AWS::Lambda::Function | IAMUSERNOPOLICYFULLSTARLambdaFunction | RDK-Rule-Function-IAMUSERNOPOLICYFULLSTAR already exists
  | 15:28:14 UTC-0500 | CREATE_FAILED | AWS::Lambda::Function | ROOTNOACCESSKEYLambdaFunction | RDK-Rule-Function-ROOTNOACCESSKEY already exists
  | 15:28:14 UTC-0500 | CREATE_FAILED | AWS::Lambda::Function | COMPLIANCERULESETLATESTINSTALLEDLambdaFunction | RDK-Rule-Function-COMPLIANCERULESETLATESTINSTALLED already exists
  | 15:28:14 UTC-0500 | CREATE_FAILED | AWS::Lambda::Function | IAMROLENOPOLICYFULLSTARLambdaFunction | RDK-Rule-Function-IAMROLENOPOLICYFULLSTAR already exists
  | 15:28:14 UTC-0500 | CREATE_FAILED | AWS::Lambda::Function | EBSENCRYPTEDVOLUMESV2LambdaFunction | RDK-Rule-Function-EBSENCRYPTEDVOLUMESV2 already exists
  | 15:28:14 UTC-0500 | CREATE_FAILED | AWS::Lambda::Function | IAMGROUPNOPOLICYFULLSTARLambdaFunction | RDK-Rule-Function-IAMGROUPNOPOLICYFULLSTAR already exists
  | 15:28:14 UTC-0500 | CREATE_FAILED | AWS::Lambda::Function | INTERNETGATEWAYAUTHORIZEDONLYLambdaFunction | RDK-Rule-Function-INTERNETGATEWAYAUTHORIZEDONLY already exists
  | 15:28:13 UTC-0500 | CREATE_IN_PROGRESS | AWS::Lambda::Function | IAMUSERNOPOLICYFULLSTARLambdaFunction |  
  | 15:28:13 UTC-0500 | CREATE_IN_PROGRESS | AWS::Lambda::Function | ROOTNOACCESSKEYLambdaFunction |  
  | 15:28:13 UTC-0500 | CREATE_IN_PROGRESS | AWS::Lambda::Function | IAMROLENOPOLICYFULLSTARLambdaFunction |  
  | 15:28:13 UTC-0500 | CREATE_IN_PROGRESS | AWS::Lambda::Function | COMPLIANCERULESETLATESTINSTALLEDLambdaFunction |  
  | 15:28:13 UTC-0500 | CREATE_IN_PROGRESS | AWS::Lambda::Function | EBSENCRYPTEDVOLUMESV2LambdaFunction |  
  | 15:28:13 UTC-0500 | CREATE_IN_PROGRESS | AWS::Lambda::Function | IAMGROUPNOPOLICYFULLSTARLambdaFunction |  
  | 15:28:13 UTC-0500 | CREATE_IN_PROGRESS | AWS::Lambda::Function | INTERNETGATEWAYAUTHORIZEDONLYLambdaFunction

When running the application account CFN, and an existing delivery channel (aws config set to false) a number of things don't run. I've been working through some of those things (role creation) but one thing that does not get run is the Compliance-Engine-Benchmark stack.

Would like a way to get everything run in the application account when the config/delivery channel already exists.

--pk

Is there a roadmap item to add support for AWS Organizations? I'm battling through manually enabling and configuring AWS Config under an AWS Organizations setup and was hoping I could refer to the codebase to help confirm my IAM, etc. settings