awslabs / aws-service-catalog-factory

This is a framework where you define a Service Catalog portfolio, products and versions using YAML. For versions of your products you specify where the source code for them can be found and the framework publishes the portfolio, products and versions in every* AWS Region after validating, linting and testing them.

aws-service-catalog-factory

What is it?

This is a python3 framework that makes it easier to build multi region AWS Service Catalog portfolios.

With this framework you define a portfolio in YAML. For each product version in your portfolio you specify which git repository it is in and the framework will build out AWS CodePipelines for each product version.

These CodePipelines can run CFN_NAG and Cloudformation_rspec on your templates enabling you to check your templates are good quality that they are functionally correct.

Getting started

You can read the installation how to or you can read through the every day use guides.

You can read the documentation to understand the inner workings.

Going further

The framework is one of a pair. The other is aws-service-catalog-puppet. With Service Catalog Puppet you can provision products into multiple regions of multiple accounts using YAML and you can share portfolios across multiple regions of multiple accounts.

License

This library is licensed under the Apache 2.0 License.

aws-service-catalog-factory's Issues

Can all buckets be made server side encrypted

PuppetRole trust policy

Issue

When I enroll a new account as a spoke in an existing organization, it provisions a role in the spoke account that allows two sets of principals to assume it:

• All principals in the master account
• All principals in the spoke account

Description

The PuppetRole's permissions policy allows Administrator access (via managed policy) along with other permissions to accept portfolio shares.

Two issues here:

• Anyone with sts:AssumeRole in a spoke account can assume this role. While allowing sts:AssumeRole for all roles is a red flag in itself, it is unfortunately used as a pattern for 'jump accounts' where a security auditor might log into one account using SAML and then switch roles to different accounts with ReadOnly roles. This role (since it is a named IAM role) could allow them to become Admin.
• The PuppetRole has managed Admin as a policy but also other policies attached to it. Wouldn't the Admin managed policy give them all the permissions that they need (since it allows all actions for all resources)?

Potential solutions

• Allow the PuppetRole to be assumed by a known set of principals (either service principals, or standard role) from the master account
• Allow the PuppetRole to be assumed only by known principals in the spoke account. This is going to be tricky, since we may not know who these principals should be until they are provisioned. If the principal in the spoke account that can assume this is known (i.e. FooRole), someone else could create a low privileged role with that name and then switch before service-catalog-factory has had a chance to create the role that can assume PuppetRole

(docs) use AWS_PROFILE instead of AWS_DEFAULT_PROFILE

according to the CLI docs, https://docs.aws.amazon.com/cli/latest/topic/config-vars.html the AWS_PROFILE variable is the env var used for profile selection, AWS_DEFAULT_PROFILE works but is legacy. for consistency it would be better to advise to use AWS_PROFILE in the docs

Renaming a portfolio does not create new products

Remove sc-factory-artifacts-* bucket lifecycle policy

The sc-factory-artifacts-<account_id>-<region> buckets have a lifecycle policy to delete files after 90 days.

This causes issues as Service Catalog uses this bucket to store the product files (like CF templates). Expiration means product versions stop working 90 days after creation.

The policy needs to be removed.

add bootstrap to sdk

github.com support is broken

Docs are wrong and the code is broken

Remove product from portfolio cli command fails

Remove product from portfolio does not work when products are defined outside of portfolios

Removal of product from the portfolio

There should be a tool/command to remove all traces of a product from your account

Sharing a portfolio

It should be possible to share a portfolio from the manifest file.

OU sharing would be ideal.

Stockholm region not included

Currently the eu-north-1 region is not included in the ALL_REGIONS constant, so service catalog portfolios/products are not created in that region.

Duplicate products created

Since updating to the latest release I'm noticing that duplicate products are being created when we define new products. One of them is empty, with the default - version, and the other one has the first version we define, but none of the subsequent ones.

ALL_REGIONS is hard coded and cannot be changed

It would be nice to choose which regions can be enabled for the factory so we do not deploy to regions not in use.

tags for the factory resources

Is it possible to also create tags for the factory itself so the s3 buckets can be tagged as well as the other factory resources?

this was forked from #91

Ability to define a product once for multiple protfolios

I'd like a product to be available in multiple portfolios.

At the moment I need to duplicate the entire product definition in each portfolio, which also duplicates pipelines and other supporting resources, as well as needing to change product definitions multiple times (once for each portfolio its defined in).

It would be good if we could define a product in a portfolio that refers to another product definition.

This could just be a way of pointing the product config in one portfolio to the definition in another, or to split out the definition of portfolios & products in the manifest.

E.g.

portfolios:
  - DisplayName: portfolio-one
    Description: portfolio-one
  - DisplayName: portfolio-two
    Description: portfolio-two

products:
      - Name: my-product
        Owner: Test
        Description: This is my-product
        Distributor: Test
        SupportDescription: N/A
        SupportEmail: [email protected]
        SupportUrl: https://donotuse.com
        Tags:
          - Key: product-type
            Value: test
        Portfolios:
          - portfolio-one
          - portfolio-two
        Versions:
          - Name: develop
            Description: Development version. Do not use for Production!
            Source:
              Provider: CodeCommit
              Configuration:
                RepositoryName: my-product
                BranchName: develop

Full docs for the portfolio schema yaml

Can we please have full documentation for the portfolio yaml

Cross account terraform products break: the S3 bucket is wrong

The template: https://github.com/awslabs/aws-service-catalog-factory/blob/master/servicecatalog_factory/templates/terraform.template.yaml.j2#L199
uses AWS::AccountId instead of the literal account id. When the pipeline runs in a spoke account the AWS::AccountId evaluates to the wrong value.

Better handling of cloudformation stacks in status: ROLLBACK_COMPLETE

When a product is added and errors on first provisioning the stack is in state ROLLBACK_COMPLETE meaning the stack must be removed manually. This is painful.

Simple example does not showcase github examples

Could you please add a github example to the seed command.

Docs : missing begginer step ?

Hey,

I'm just playing around so far, and it seems to me that we're missing something either in the doc or in the boostrap cfn.

Arrived at the cloning the repo part of the doc, I saw that no repo existed. I had to manually trigger the pipeline to get the codecommit repo created.

Are we missing something to trigger the first build or a documentation step ? I can help with the latter :>

Automatic creation of codecommit repos for product version pipelines

It should be possible to enable the auto creation of codecommit repos when building a manifest

Terraform support

Branched from a conversation as a result of issue #31

Users would like support for terraform based products.

Please subscribe to this issue or +1 if you would like to see this happen.

Troposphere support

Branched from issue #31

Users would like support for troposphere based products.

Please subscribe to this issue or +1 if you would like to see this happen.

Product Pipelines not triggered from commits

Might be related to #47

Since the update to turn off pipeline polling, our product pipelines aren't executing on commit anymore.

I suspect the CloudWatch Event Rules haven't been created (as this is done for you when configuring through the console, I suspect they need to be created separately through the API).

Autogenerated stack name is invalid

When an IAM role with an underscore is used for an association, it breaks the autogenerated stack name, e.g.:

botocore.exceptions.ClientError: An error occurred (ValidationError) when calling the CreateStack operation: Template format error: Resource name Role_roles_admin is non alphanumeric.

  File "/usr/local/lib/python3.7/site-packages/servicecatalog_factory/cli.py", line 245, in generate_and_run 
    Capabilities=['CAPABILITY_IAM'], 

Factory is no longer creating associations for portfolios

When you create a portfolio with associations they are no longer being created.

Having two factory accounts in the same organization clobbers each other

Bootstrapping an organization clobbers the previous bootstrapped.

Ensure portfolios and products match code

It would be nice if the pipeline work would be able to not only create new portfolios and products, but also recognise when portfolios or products are removed from the code and tidy up any related resources, i.e. remove products (including product templates), portfolios and stacks/code* elements when removed from the definition. This could perhaps be achieved by comparing git diffs or with the current state stored in an s3 bucket.
This would then make the code the source of truth for all created products/portfolios without requiring a manual clean up.

Deactivate a product version from the portfolio yaml

It should be possible to deactivate a version of a product from the portfolio yaml

requests library is missing from setup.py requirements

Extra tagging

Would it be possible to pick up the tags from the products defined in the portfolios and tag the Cloudwatch and CodeBuild resources which get created? Is it possible to also create tags for the factory itself so the s3 buckets can be tagged as well as the other factory resources?

Bootstrap to a non released version

To enable better testing of preleased versions I would like to be able to bootstrap an account and say use a github url for a branch of the source code instead of using a published pypi version.

Removal of product versions from the portfolio

There should be a tool/command to remove all traces of a product version from your account

Add upload config to sdk

Launching a product into the puppet account fails

• Listing imported shares is not working as the portfolio is not shared
• IAM principal association is missing

document all CLI commands

the following commands require documenting:

• bootstrap-branch
• generate
• quick-start

0.2.0 onwards is sensitive to API throttling limits

Generated CWE rule name is too long

ERROR MainThread CREATE_FAILED: 1 validation error detected: Value 'omitted' at 'name' failed to satisfy constraint: Member must have length less than or equal to 64 (Service: AmazonCloudWatchEvents; Status Code: 400; Error Code: ValidationException

Can all buckets be made not public

Support for AWS CDK based products

Branched from issue #31

Users would like support for AWS CDK based products.

Please subscribe to this issue or +1 if you would like to see this happen.

version name is missing from the pipeline name

seed command is failing to create a portfolio directory within the target dir

Running the following gives an error:

servicecatalog-factory seed simple Foo

Schema does not contain the github configuration

When you use github source for a product or its versions the validate command does not pass configuration as valid

Support for running tests during pipeline

it'd be great to be able to run tests during the pipeline (AWSpec, serverless unit tests etc.) at various stages. perhaps pre/post hooks could be used to run a list of commands the user supplies at each phase? Obvious things to consider are security and altering/customising the build environment

Read Custom Buildspec from Product Repository

I'd prefer it if we could tell SCF to look at buildspec files in the Product Repo instead of defining them in-line in the SCF Manifest.

Maybe have a toggle in the manifest along the lines of load_buildspec_from_repo: true to toggle this.

This would be for the Tests step, and it should still output the correct artefacts for the Package step to pick up.

docutils install clash

Ability to use the Folder in the Repository for Product source

Possibility to have a folder in the repository to be the source of the product instead of creating separate repository for each product. This way we can have less number of repositories for the products.

account-bootstrap-shared-product should read aws-service-catalog-puppet version from SSM and not from its template

Reading from SSM at run time will make it easier to maintain and easier to keep up to date.

Change from polling to event based change detection for pipelines

We've just hit the below error in our account

ERROR MainThread CREATE_FAILED: Maximum number of pipelines using periodic source checks in account (60) exceeded. 
Switch to event-based change detection such as Amazon CloudWatch Events or webhooks to save this pipeline. 
Learn more: https://docs.aws.amazon.com/console/codepipeline/polling-to-events-based 
(Service: AWSCodePipeline; Status Code: 400; Error Code: LimitExceededException; Request ID: xxx)  

Can the Product Pipelines be switched to using CloudWatch Events for change detection?

jinja2 support

It would be great to have support for jinja2 syntax within the product.template.yaml file.