Coder Social home page Coder Social logo

go-cloudformation's Introduction

Build Status

This package provides a schema and related functions that allow you to parse and serialize CloudFormation templates in golang. The package places an emphasis on type-safety so that the templates it produces are (slightly) more likely to be correct, and maybe you can avoid endless cycles of UPDATE_ROLLBACK_IN_PROGRESS.

Parsing example:

t := Template{}
json.NewDecoder(os.Stdin).Decode(&t)
fmt.Printf("DNS name: %s\n", t.Parameters["DnsName"].Default) 

Producing Example:

t := NewTemplate()
t.Parameters["DnsName"] = &Parameter{
  Type: "string",
  Default: "example.com",
  Description: "the top level DNS name for the service"
}
t.AddResource("DataBucket", &S3Bucket{
  BucketName: Join("-", String("data"), Ref("DnsName"))
})
json.NewEncoder(os.Stdout).Encoder(t)

See the examples directory for a more complete example of producing a cloudformation template from code.

Producing the Schema

As far as I can tell, AWS do not produce a structured document that describes the CloudFormation schema. The names and types for the various resources and objects are derived from scraping their HTML documentation (see scraper/). It is mostly, but not entirely, complete. I've noticed several inconsistencies in the documentation which suggests that it is constructed by hand. If you run into problems, please submit a bug (or better yet, a pull request).

Object Types

Top level objects in CloudFormation are called resources. They have names like AWS::S3::Bucket and appear as values in the "Resources" mapping. We remove the punctuation and redundant words from the name to derive a golang structure name like S3Bucket.

There are other non-resource structs that are refered to by resources or other non-resource structs. These objects have names with spaces in them, like "Amazon S3 Versioning Configuration". To derive a golang type name the non-letter characters and redundant words are removed to get S3VersioningConfiguration.

Type System

CloudFormation uses three scalar types: string, int and bool. When they appear as properties we represent them as *StringExpr, *IntegerExpr, and *BoolExpr respectively.

type StringExpr struct {
  Func    StringFunc
  Literal string
}

// StringFunc is an interface provided by objects that represent 
// CloudFormation functions that can return a string value.
type StringFunc interface {
  Func
  String() *StringExpr
}

These types reflect that fact that a scalar type could be a literal value ("us-east-1") or a JSON dictionary representing a "function call" ({"Ref": "AWS::Region"}).

Another vagary of the CloudFormation language is that in cases where a list of objects is expected, a single object can provided. For example, AutoScalingLaunchConfiguration has a property BlockDeviceMappings which is a list of AutoScalingBlockDeviceMapping. Valid CloudFormation documents can specify a single AutoScalingBlockDeviceMapping rather than a list. To model this, we use a custom type AutoScalingBlockDeviceMappingList which is just a []AutoScalingBlockDeviceMapping with extra functions attached so that a single items an be unserialized. JSON produced by this package will always be in the list-of-objects form, rather than the single object form.

Known Issues

The cloudformation.String("foo") is cumbersome for scalar literals. On balance, I think it is the best way to handle the vagaries of the CloudFormation syntax, but that doesn't make it less kludgy. A similar approach is taken by aws-sdk-go (and is similarly cumbersome).

There are some types that are not parsed fully and appear as interface{}.

I worked through public template files I could find, making sure the library could accurately serialize and unserialize them. In this process I discovered some of the idiosyncracies described. This package works for our purposes, but I wouldn't be surprised if there are more idiosyncracies hidden in parts of CloudFormation we are not using.

Feedback, bug reports and pull requests are gratefully accepted.

go-cloudformation's People

Contributors

crewjam avatar leavengood avatar mweagle avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

go-cloudformation's Issues

Add support for Parameter Groups

Newbie in Go. So far this package was very useful in the project I'm working. I'm using this to generate CFN template. I noticed one feature is missing that is support for Parameter Groups using AWS::CloudFormation::Interface which is used by the AWS console to display relevant parameters in groups.

This requires Template struct to include Metadata field.

I see the schema.go includes various type (CloudFormationInterface, ParameterGroupsList etc etc) as resource types and wasn't sure how to use them to implement the Metadata field in Template struct.

I have come up with an alternate approach (patch attached) which works but probably not a good approach (again I'm newbie and in very early stage of my learning GO). Appreciate some suggestion and guidance.
metadata.txt

IAM PolicyDocument would be a useful type to have

https://github.com/crewjam/etcd-aws/blob/master/aws/iam.go

type PolicyDocument struct {
    Version   string `json:",omitempty"`
    Statement []Policy
}

type Policy struct {
    Sid            string              `json:",omitempty"`
    Effect         string              `json:",omitempty"`
    Principal      *Principal          `json:",omitempty"`
    Action         *cfn.StringListExpr `json:",omitempty"`
    Resource       *cfn.StringListExpr `json:",omitempty"`
    ConditionBlock interface{}         `json:",omitempty"`
}

type Principal struct {
    Service *cfn.StringListExpr `json:",omitempty"`
}

Need a little help

Hi

I have the following while spinning up a cloudformation stack


func makeTemplate() *cf.Template {
    t := cf.NewTemplate()
   t.Description = "example production infrastructure"
    t.Parameters["Name"] = &cf.Parameter{
        Description: "This is a test",
        Type:        "String",
        Default:     "live long and prosper",
    }

    t.Outputs["myoutput1"] = &cf.Output{
        Description: "test output 1",
        Value:     "warp",
    }

    t.AddResource("ec2_vol", cf.EC2Volume{
        AutoEnableIO: cf.Bool(false),
        AvailabilityZone: cf.String("us-east-1b"),
        Encrypted: cf.Bool(false),
        Size: cf.String("1"),
        Tags: &cf.ResourceTag{
                 Key: cf.String("Name"),
                 Value: cf.String("my-vol"),
            },
        })


    etc etc 

The Tags are giving an error:
cannot use cloudformation.ResourceTag literal (type *cloudformation.ResourceTag) as type []cloudformation.ResourceTag in field value

Any advice here, please ?

Security Group Id for specifying ingress/egress rules creates incorrect JSON

When creating an ingress/egress rule using a security group ID, this creates incorrect JSON. For example, the following:

EC2SecurityGroupRule{
	SourceSecurityGroupIdXXSecurityGroupIngressXOnlyX: Ref(s.elbSecurityGroupLogicalName()).String(),
	IpProtocol:                                        String(tcpProtocol),
	FromPort:                                          Integer(httpsPort),
	ToPort:                                            Integer(httpsPort),
},

Generates the JSON:

{
    "FromPort": 443,
    "IpProtocol": "tcp",
    "SourceSecurityGroupId (SecurityGroupIngress only)": {
        "Ref": "LBSecurityGroupGamma"
    },
    "ToPort": 443
}

The expected output is:

{
    "FromPort": 443,
    "IpProtocol": "tcp",
    "SourceSecurityGroupId": {
        "Ref": "LBSecurityGroupGamma"
    },
    "ToPort": 443
}

This is probably due to how the scraper is scraping the AWS website, causing it to include the (SecurityGroupIngress only) as seen here:
http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-ec2-security-group-rule.html#cfn-ec2-security-group-rule-sourcesecuritygroupid

Regenerate schema

CloudFormation has released some new properties recently. Is it possible to regenerate the schema?

Also, thinking out loud, thoughts on running the schema generation code as a Lambda function with a cron-style trigger? Maybe post the generated code to public S3? Might be an interesting case study for serverless.

various resource properties should be list types but are not

I discovered that S3NotificationConfiguration has properties like LambdaConfigurations that are clearly from context supposded to be []SimpleStorageServiceNotificationConfigurationLambdaConfigurations but are actually SimpleStorageServiceNotificationConfigurationLambdaConfigurations.

The heuristic we use to determine when something is a list failed for this item. I generated a list of other possible similar errors which appears below with the following code:

for _, resource := range tr.Resources {
    for _, property := range resource.Properties {
        if n := property.GoName(); !strings.HasSuffix(n, "s") {
            continue
        }
        if t := property.GoType(&tr); strings.Contains(t, "List") {
            continue
        }
        if t := property.GoType(&tr); strings.Contains(t, "[]") {
            continue
        }
        if t := property.GoType(&tr); strings.Contains(t, "interface{}") {
            continue
        }
        fmt.Fprintf(os.Stderr, "* in "+
            "[%s](https://godoc.org/github.com/crewjam/go-cloudformation#%s) "+
            "property `%s` has type `%s` but should maybe be a list type. [ref](%s)\n",
            resource.GoName(), resource.GoName(),
            property.GoName(),
            property.GoType(&tr),
            rootURL+resource.Href)
    }
}

This list:

add information to the schema if field is required

That would very nice if you could have information if a property is required. This would a way road for linters based on your library.

If I am not mistaken, *Expr types can be extended with a bool field.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.