stelligent / cfn-model Goto Github PK
View Code? Open in Web Editor NEWAn object model for CloudFormation templates
License: MIT License
An object model for CloudFormation templates
License: MIT License
The Globals section is not parsed by cfn-model and there are some instances where the Raw model is being used to access Globals. Update cfn-model to parse the Globals section and then re-write the Raw model references to use.
Reference: stelligent/cfn_nag#141
As it stands, JSON is a subset of YAML so YAML parser is used at the basic level but.... when actually trying to parse JSON, some degenerate YAML will let junk through so when we know we've got JSON - use a JSON parser instead
Given illegal JSON
When parsing and applying JSON values
Then a JSON::ParserError is raised
Given JSON that isn't a dictionary
When parsing and applying JSON values
Then a JSON::ParserError is raised
Given JSON that doesn't have a Parameters key
When parsing and applying JSON values
Then a JSON::ParserError is raised
Given JSON that includes extra key-value pairs in Parameters vs the template
When parsing and applying JSON values
Then the extra key-value pair is quietly ignored
13:54:14 + scl enable rh-ruby23 'gem install cfn-nag'
13:54:15 Successfully installed cfn-nag-0.3.53
13:54:15 Parsing documentation for cfn-nag-0.3.53
13:54:15 Done installing documentation for cfn-nag after 0 seconds
13:54:15 1 gem installed
13:54:15 + find CloudFormation -name '*.template'
13:54:15 + xargs -t -n 1 cfn_nag
13:54:15 cfn_nag CloudFormation/Serverless.template
13:54:15 /var/lib/jenkins/.gem/ruby/gems/cfn-model-0.1.24/lib/cfn-model/transforms/serverless.rb:29:in `object_key_from_uri': undefined method `join' for nil:NilClass (NoMethodError)
13:54:15 Did you mean? JSON
13:54:15 from /var/lib/jenkins/.gem/ruby/gems/cfn-model-0.1.24/lib/cfn-model/transforms/serverless.rb:39:in `replace_serverless_function'
13:54:15 from /var/lib/jenkins/.gem/ruby/gems/cfn-model-0.1.24/lib/cfn-model/transforms/serverless.rb:11:in `block in perform_transform'
13:54:15 from /var/lib/jenkins/.gem/ruby/gems/cfn-model-0.1.24/lib/cfn-model/transforms/serverless.rb:9:in `each'
13:54:15 from /var/lib/jenkins/.gem/ruby/gems/cfn-model-0.1.24/lib/cfn-model/transforms/serverless.rb:9:in `perform_transform'
13:54:15 from /var/lib/jenkins/.gem/ruby/gems/cfn-model-0.1.24/lib/cfn-model/parser/transform_registry.rb:20:in `perform_transforms'
13:54:15 from /var/lib/jenkins/.gem/ruby/gems/cfn-model-0.1.24/lib/cfn-model/parser/cfn_parser.rb:41:in `parse'
13:54:15 from /var/lib/jenkins/.gem/ruby/gems/cfn-nag-0.3.53/lib/cfn-nag/cfn_nag.rb:82:in `audit'
13:54:15 from /var/lib/jenkins/.gem/ruby/gems/cfn-nag-0.3.53/bin/cfn_nag:70:in `<top (required)>'
13:54:15 from /var/lib/jenkins/bin/cfn_nag:23:in `load'
13:54:15 from /var/lib/jenkins/bin/cfn_nag:23:in `<main>'
13:54:15 Build step 'Execute shell' marked build as failure
template file:
AWSTemplateFormatVersion: '2010-09-09'
Transform: 'AWS::Serverless-2016-10-31'
Description: 'Create Serverless application.'
Parameters:
Environment:
Description: Name of the AWS deployment environment
Type: String
AllowedValues:
- rd
- di
- qa
- ct
- pr
ProductIdentifier:
Description: Tag for the name of the product
Type: String
Resources:
ServerlessFunc:
Type: 'AWS::Serverless::Function'
Properties:
FunctionName: !Sub '${Environment}-${ProductIdentifier}'
Handler: handler.hello
Runtime: python3.6
CodeUri: ../app
Description: >-
Sample application1
MemorySize: 128
Timeout: 3
Role: !Sub 'arn:aws:iam::${AWS::AccountId}:role/${Environment}-${ProductIdentifier}-Role'
Environment:
Variables:
AWS_ENV: !Ref Environment
Hi!
I am using cfn_nag lastest version 0.3.84, we have a usecase with wildcard on policy
with an sns policy we would authorize any account in our aws organization to publish on it, on our policy we have an condition aws:principalOrgId.
https://aws.amazon.com/fr/blogs/security/control-access-to-aws-resources-by-using-the-aws-organization-of-iam-principals/
We have devellop a small and durty function to catch this condition, my question is
make sense to implement this on model/statements, for add a condition if orgid is specify and value equal with our orgid ?
thank's for your work on cfn_nag and your support
CloudFormation has added a Transform concept for templates. There are currently two flavors: Include transforms, and Serverless transforms. Docs at https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/transform-aws-serverless.html
If a template parsed by CfnModel relies on resources created by SAM like the .Alias
resource for functions, CfnModel will raise an error due to having no knowledge of the resource that would (inside CloudFormation) be created by the Transform. See stelligent/cfn_nag#115 for a concrete example
There are currently per-resource handlers in https://github.com/stelligent/cfn-model/tree/master/lib/cfn-model/model and one could envision a way in which SAM-supplied resources like AWS::Serverless::Function
modify the raw structure representing the template (to add resources that would be created).
However, a model that would fit the render model used by CloudFormation closer would be to create lib/cfn-model/model/transforms/serverless.rb
and add a call in CfnParser around https://github.com/stelligent/cfn-model/blob/master/lib/cfn-model/parser/cfn_parser.rb#L130 , to pass it the template yaml and to let the serverless.rb
transform handler mutate the model's view of the template.
The current kwalify rules are manually updated based upon the resource types specifications.
Two things to look into:
http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-resource-specification.html
Perhaps also trigger on updates to this URL:
http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/ReleaseHistory.html
Hi!
I am working on stelligent/cfn_nag#49 which I think this feature may be useful for us.
Did some research and looks like inside cfn_model.rb
we only have resources
available which the resources are loaded by the parser, and if we would like to access anything else(eg. metadata and parameters) we need to use raw_model
to get it.
Since a big perspective of cfn_nag
is security and credentials are commonly loaded into the cfn templates via parameters so i think it would be beneficial to have this feature implemented ๐
e.g. {"Fn::Join"=>["", ["logs.", {"Ref"=>"AWS::Region"}, ".amazonaws.com"]]}
Using a fresh Git Bash and Ruby 2.4.1p111 I can't install cfn-nag because the installation of cfn-model fails with the following message:
ERROR: While executing gem ... (Errno::EINVAL) Invalid argument @ rb_sysopen - C:/Ruby24/lib/ruby/gems/2.4.0/gems/cfn-model-0.0.6/lib/cfn-model/schema/AWS::CloudFront::Distribution.yml
The schema folder is empty and when I try to copy the files there, it just deletes them during the next installation.
Currently, the cfn_model vscode remote development configuration in the dev container uses the same port (9001) as cfn_nag. These should not conflict and should use a unique port so they can be run simultaneously if desired.
If a Sub references a parameter value.... we can probably compute this just fine and use the interpolated value for rules.
If the Sub references the output of a constructed resource... no way to know that value so just leave it be like always
A changelog would allow users to view the differences between tagged releases, which may be helpful in selecting a version or understanding the impact that upgrading a version will have.
We need a full re-evaluation of the kwalify validator in light of this. For now, this is breaking for a particular user (on the load balancer)
In the case of security group - the SecurityGroup object was missing tags which led to a failure. Easy enough to rectify, but more generally speaking if attributes are added to resources over time by AWS.... we don't want objects to be too strict about mapping those new properties. So.... collapse DynamicModelElement into ModelElement. Still allow unseen model elements to be created whole cloth but for defined ModelElement... use the defined attributes and then be tolerant of missing properties - acting like the DynamicModelElement does.
While writing a parser for EC2::Instance resources it felt awkward to reassign securityGroupIds to be the parsed SecurityGroup objects themselves. After thinking about it a bit.... overwriting that mapping deprives rules of the ability to do their own analysis on them - otherwise they have to go down to the raw model. So.... thinking mapped fields should be preserved and synthetic fields should just be added with appropriate names to distinguish them
Per the AWS documentation
"IpProtocol
The IP protocol name (tcp, udp, icmp) or number (see Protocol Numbers). (VPC only) Use -1 to specify all protocols. If you specify -1, or a protocol number other than tcp, udp, icmp, or 58 (ICMPv6), traffic on all ports is allowed, regardless of any ports you specify. For tcp, udp, and icmp, you must specify a port range. For protocol 58 (ICMPv6), you can optionally specify a port range; if you don't, traffic for all types and codes is allowed."
When specifying the IpProtocol as -1 in a template, the parse will throw an exception stating to/from is required when it actually is not.
Expected 0.2.x to generate 0.2.37
Make 0.3.x start at 0.3.0
Parameters:
VpcId:
Type: "AWS::EC2::VPC::Id"
ExtraIngress:
Type: String
Default: "true"
Conditions:
ExtraIngress: !Equals [ !Ref ExtraIngress, true ]
Resources:
sg2:
Type: "AWS::EC2::SecurityGroup"
Properties:
GroupDescription: "some_group_desc"
SecurityGroupIngress:
- CidrIp: "10.1.2.3/32"
FromPort: 34
ToPort: 36
IpProtocol: tcp
- Fn::If:
- ExtraIngress
- CidrIp: "10.1.2.4/32"
FromPort: 44
ToPort: 46
IpProtocol: tcp
- !Ref AWS::NoValue
SecurityGroupEgress:
- CidrIp: "1.2.3.4/32"
FromPort: 55
ToPort: 56
IpProtocol: tcp
VpcId:
Ref: VpcId
with
NameError: @fn::If=' is not allowed as an instance variable name ./lib/cfn-model/model/model_element.rb:82:in
instance_variable_get'
./lib/cfn-model/model/model_element.rb:82:in method_missing' ./lib/cfn-model/parser/security_group_parser.rb:31:in
block (2 levels) in objectify_ingress'
./lib/cfn-model/parser/security_group_parser.rb:29:in each' ./lib/cfn-model/parser/security_group_parser.rb:29:in
block in objectify_ingress'
./lib/cfn-model/parser/security_group_parser.rb:27:in map' ./lib/cfn-model/parser/security_group_parser.rb:27:in
objectify_ingress'
./lib/cfn-model/parser/security_group_parser.rb:13:in parse' ./lib/cfn-model/parser/cfn_parser.rb:56:in
block in post_process_resource_model_elements'
./lib/cfn-model/parser/cfn_parser.rb:50:in each' ./lib/cfn-model/parser/cfn_parser.rb:50:in
post_process_resource_model_elements'
./lib/cfn-model/parser/cfn_parser.rb:42:in parse' ./spec/parser/cfn_parser_security_group_spec.rb:205:in
block (4 levels) in <top (required)>'
./spec/parser/cfn_parser_security_group_spec.rb:204:in each' ./spec/parser/cfn_parser_security_group_spec.rb:204:in
block (3 levels) in <top (required)>'
-e:1:in load' -e:1:in
not nec a sequence of anys
Parameters:
Base:
Type: "String"
Default: "10"
CustomResourceId:
Type: "String"
Default: "SomeCustomResourceId"
Resources:
TestCustomResource:
Type: "Custom::TestCustomResourceWithLambda"
Properties:
ServiceToken: !GetAtt "TestLambdaFunction.Arn"
Base: !Ref Base
Id: !Ref CustomResourceId
actions-csv: some_value
Apparently legal (despite the Function specification?) to have neither CodeUri nor InlineCode and look for the handler in (js) file in same directory as template.
cfn-model should transform this function into a Lambda function with no code... basically just don't throw an exception
These fields are lists, but can be a ref to a list of literals in the Parameters block. CommaDelimitedList or SecurityGroupId.
afaik there isn't a way to specify a List of Refs, so these Lists are collections of literals that we can't do much with (so just ignore them instead of blowing up on them)
The potential use of Fn::If can muck with even the most basic restriction to have a sequence. The Fn::If is a Hash, but could have a legit array "output".
This could play in with #10 but in the meantime, probably makes sense to reduce the schema validators to only include required fields and set them to any. For each field we make this change, we need to check there isn't a core rule that will be affected in an undesirable way (i.e. something it needs to check itself that previously it depended upon being correct).
Custom resources with a type other than AWS
or Custom
cause a cfn_nag exception.
Update /lib/cfn-model/parser/cfn_parser.rb to allow other types of custom resources.
Example template
Resources:
VPC:
Type: Versent::Network::VPC
[...]
Transform:
Name: "123456789012::VPC"
Error
$ cfn_nag --debug ./aws/*.yaml
/usr/local/bundle/gems/cfn-model-0.4.0/lib/cfn-model/parser/cfn_parser.rb:218:in `generate_resource_class_from_type': Unknown namespace in resource type: Versent (RuntimeError)
from /usr/local/bundle/gems/cfn-model-0.4.0/lib/cfn-model/parser/cfn_parser.rb:189:in `rescue in class_from_type_name'
Reference:
Track line numbers of resources that are violations
Hi, I try to parse template that use CloudFormation Intrinsic Functions to generate resource names and it brings to parsing error:
`pre_validate_model': Basic CloudFormation syntax error:[#<Kwalify::ValidationError: [/Resources/ECSTaskRolePolicies/Properties/PolicyName] not a string.>, #<Kwalify::ValidationError: [/Resources/ECSServiceRolePolicy/Properties/PolicyName] not a string.>] (ParserError)
Here is resource sample that fail parser:
"ECSTaskRolePolicies": {
"Type": "AWS::IAM::Policy",
"Properties": {
"PolicyName" : { "Fn::Sub" : "${EnvPrefix}-my-policy-${EnvSuffix}" },
"Roles" : [ { "Ref": "ECSTaskRole" } ],
"PolicyDocument" : {
"Version" : "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents"
],
"Resource": "arn:aws:logs:*:*:*"
}
]
}
}
}
Chased down a couple places where Fn::If or And, etc. cause the parser to freak out and I hope there won't be anymore exceptions but.... the deeper issue is to attempt to evaluate some of these conditions. Otherwise there will potentially be specifications that aren't analyzed for security issues (like an ingress being ignored because it's in a conditional)
Investigate a "mode" for parsing that attempts to invoke a transform Lambda.... and then turn the model into what comes back. A few things to consider:
This could enable some cross-template static analysis possibly...
The models for wildcards, such as in the Statement Model, are too broad. This ends up impacting cfn-nag users by identifying warnings on resources that are considerably scoped down yet still have an asterisk - such as "arn:${AWS::Partition}:s3:::${S3Bucket}/*"
on an s3:GetObject
policy, that could be intentional.
I would like to see this check scoped down if possible - I know this is a considerable challenge since ARN conventions differ from service to service.
Check for an Array in ResourceTypeValidator and throw the exception that says there is no Resources block
the mixing in of references and whatnot complicate matters but.... can perhaps compute some of the static stuff
Umbrella ticket for ongoing linting efforts of codebase
"Type": "Custom::lowercasename" will yield "NameError: wrong constant name lowercasename" because the name is mapped to a Class name which is a constant which must be upper case.
The parser crashes when using AWS::EC2::SecurityGroup with tags.
According to AWS tags are supported.
I get the following message:
/usr/local/bundle/gems/cfn-model-0.0.6/lib/cfn-model/parser/cfn_parser.rb:88:in block in assign_fields_based_upon_properties': undefined method tags=' for #<AWS::EC2::SecurityGroup:0x0055a5a4f90688> (NoMethodError)
from /usr/local/bundle/gems/cfn-model-0.0.6/lib/cfn-model/parser/cfn_parser.rb:87:in each'
from /usr/local/bundle/gems/cfn-model-0.0.6/lib/cfn-model/parser/cfn_parser.rb:87:in assign_fields_based_upon_properties'
from /usr/local/bundle/gems/cfn-model-0.0.6/lib/cfn-model/parser/cfn_parser.rb:63:in block in transform_hash_into_model_elements'
from /usr/local/bundle/gems/cfn-model-0.0.6/lib/cfn-model/parser/cfn_parser.rb:56:in each'
from /usr/local/bundle/gems/cfn-model-0.0.6/lib/cfn-model/parser/cfn_parser.rb:56:in transform_hash_into_model_elements'
from /usr/local/bundle/gems/cfn-model-0.0.6/lib/cfn-model/parser/cfn_parser.rb:31:in `parse'
from /usr/local/bundle/gems/cfn-nag-0.1.3/lib/cfn-nag/cfn_nag.rb:61:in audit'
from /usr/local/bundle/gems/cfn-nag-0.1.3/bin/cfn_nag:28:in <top (required)>'
from /usr/local/bundle/bin/cfn_nag:23:in load'
from /usr/local/bundle/bin/cfn_nag:23:in <main>
From cfn-nag issue:
With the V2 LoadBalancer with type network you can specify either Subnets or SubnetMappings under the Properties key, but currently the model requires Subnets.
CloudFormation snippet:
NetworkLoadBalancer:
Type: AWS::ElasticLoadBalancingV2::LoadBalancer
Properties:
Type: network
Scheme: internet-facing
SecurityGroups:
- !Ref LoadBalancerSecurityGroup
SubnetMappings:
- AllocationId: 'eipalloc-xxxxxx'
SubnetId: !Ref Subnet1ID
- AllocationId: 'eipalloc-xxxxx
SubnetId: !Ref Subnet2ID
Result:
{
"failure_count": 1,
"violations": [
{
"id": "FATAL",
"type": "FAIL",
"message": "Basic CloudFormation syntax error:[#<Kwalify::ValidationError: [/Resources/NetworkLoadBalancer/Properties] key 'Subnets:' is required.>]",
"logical_resource_ids": null
}
]
}
Previously, model supported substituting JSON in the form:
{
"Parameters": {
<parameter_name> : <parameter_value> ....
}
}
Continues to support that format, but now also supports:
[
{ "ParameterKey": "xxx", "ParameterValue": "yyy" }
]
For people who prefer interactive commit message writing, #IssueNumber is super painful. We should accept "^Issue #" in addition.
The ultimate goal here is that when security groups are linked to resources internal to a template that we can figure out if its to an ENI, EC2 instance or ELB. From an analysis point of view, sg ingress on any EC2 instance or ENI should be locked down big time, but for an ELB there is more give.
LambdaPrincipal (used in LambdaPermission) can only be a string.... and an integer on the down-low for an AWS account ID.
The Principal (used in IAM policies) can be a Hash or a String.... so break these guys apart.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.