stelligent / mu Goto Github PK

Create a subcommand mu pipeline logs [-f] to view the logs for the recent build and -f to follow the logs.

• Allow specifying hostedzone in mu.yml to have a resource record added to that zone for the ALB
• Default the hostname in the hostedzone to the env name
• Allow override hostname via name in the mu.yml
• Allow providing certificate in mu.yml to be the ARN of the cert to reference from the ELB. will create ELB listeners for 443 if defined.

As a developer, I want to be able to view the pipeline status for a service to understand the current configuration and utilization of a pipeline.

AC:

• When running mu svc show, mu will list the pipeline stage status for the service along with the following: stage, action, revision, status, last update

As a mu user, I want to be able to define my service and environments for mu in a YAML file so that I can have a simple definition of my application infrastructure in a file that I can commit with my app to source code.

AC:

• mu.yml would be loaded from current directory by default
• path to mu.yml can be specified via -c flag
• mu should return error if cannot find mu.yml file
• mu should return error if cannot parse yaml in file
• mu should allow the following config options:

---
### Region to utilize
region: us-west-2

### Define a list of environments
environments:

  # The unique name of the environment  (required)
  - name: dev

    ### Attributes for the ECS container instances
    cluster:
      imageId: ami-xxxxxx           # The AMI to use for the ECS container instances (default: latest ECS optimized AMI)
      instanceTenancy: default      # Whether to use default or dedicated tenancy (default: default)
      desiredCapacity: 1            # Desired number of ECS container instances (default 1)
      maxSize: 2                    # Max size to scale the ECS ASG to (default: 2)
      keyName: my-keypair           # name of EC2 keypair to associate with ECS container instances (default: none)
      sshAllow: 0.0.0.0/0           # CIDR block to allow SSH access from (default: 0.0.0.0/0)
      scaleOutThreshold: 80         # Threshold for % memory utilization to scale out ECS container instances (default: 80)
      scaleInThreshold: 30          # Threshold for % memory utilization to scale in ECS container instances (default: 30)

    ### attributes for the VPC to target.  If not defined, a VPC will be created. (default: none)
    vpcTarget:
        vpcId: vpc-xxxxx            # The id of the VPC to launch ECS container instances into
        publicSubnetIds:            # The list of subnets to use for ECS container instances
          - sg-xxxxx
          - sg-xxxxy
          - sg-xxxxz

### Define the service for this repo
service:
  name: my-service                   # The unique name of the service (default: the name of the directory that mu.yml was in)
  desiredCount: 4                    # The desired number of tasks to run for the service (default: 2)
  dockerfile: ./Dockerfile           # The relative path to the Dockerfile to build images (default: ./Dockerfile)
  imageRepository: tutum/hello-world # The repository to push images to and deploy services from.  Leave unset to have mu manage an ECR repository (default: none)
  port: 80                           # The port to expose from the container (default: 8080)
  healthEndpoint: /health            # The endpoint inside the container to determine if the task is healthy (default: /health)
  cpu: 20                            # The number of CPU units to allocate to each task (default: 10)
  memory: 400                        # The amount of memory in MiB to allocate to each task (default: 300)

  # The paths to match on in the ALB and route to this service.  Leave blank to not create an ALB target group for this service (default: none)
  pathPatterns:
    - /bananas
    - /apples

As a developer, i want to be able to create my environment with mu, so that i can run my application.

AC:

• Should create new environment stack by running mu env up <env_name>
• Should update (with message of no change) if rerun mu env up <env_name>
• Should require the environment with matching name exists in mu.yml
• Will create VPC stack if no vpcTarget is specified
• If vpcTarget is specified, then the ECS cluster and ALB will be created in the referenced VPC
• Cluster stack will define an ECS cluster, ASG/LC for ECS cluster instances, with autoscale based on memory, and ALB
• VPC stack will define IGW, Subnets, NACL, SG

What are all the commands we expect this tool to offer?

Need to have access in codebuild to the commit id from codepipeline to tag the image properly. Currently this isn't exposed...not sure how to address? One option is to use CODEBUILD_* env variables to find the current codepipeline execution and then run aws cli to query details of the pipeline execution and get revisionId from there.

As a developer, I want to be able to undeploy a service from an environment so that I can remove services that aren't needed anymore

AC:

• When running mu service undeploy <env_name> the service will be removed from the provided environment
• ECR repo will remain unchanged

As a developer, I want to be able to terminate an environment with mu so that I can discard unused resources.

AC:

• Running mu env term <env_name> will teardown the cluster stack
• Will also teardown the VPC stack, unless vpcTarget was specified in mu.yml

Language - TBD. Need to determine the language to use for developing the tool. Options in order of preference include Go, Node.js, Python.

Document how to contribute to the project

As a developer, i want to be able to show the details of a service so that I can know which version is deployed in which environments.

AC:

• Running mu svc show <svc_name> will display the list of which environments the service is currently running in

Aggregate container logs to CloudWatch logs. Create log group for each service and log stream for each container

As a developer, I want to be able to terminate a pipeline with mu so that I can discard unused resources.

AC:

• Running mu pipeline term will teardown the pipeline

As a microservice developer, i want to be able to create a pipeline for my service to i can continuously deliver my service into production.

AC:

• Run mu pipeline up with a mu file to have a pipeline created
• Will prompt for a GitHub token if none was provided via -t <token
• Will create a CodePipeline with 4 stages: source, build, acceptance, production
• Will create a manual blocker action in Production stage
• Will create a CodeBuild job to compile the source
• Will create a CodeBuild job to create the docker image
• Will create CodeBuild jobs to deploy to acpt and prod

Is there a clear example of the pain people suffer without Mu? Is there a way to describe the cost/pain/suffering/errors one suffers when one has to work in a world without Mu? If we have that it would go a long to way to motivate and encourage support.

Run CodeBuild to execute tests in test stage of pipeline

As a developer, i want to be able to create with mu, so that i can run my application.

AC:

• Should upsert ECR repo, build docker image and push to ECR with mu service push
• Will allow override of repo to push to by setting imageRepository in mu.yml. If set, ECR repo won't be created
• Will allow override of path to Dockerfile
• Will default image name to the directory that mu.yml is in, can be overridden in mu.yml
• Will default image tag to the short commit id for current git repo, but can be overriden with -t flag
• Should deploy service to ECS environment with mu service deploy <env_name>
• Will allow specifying container port and path patterns for configuration of the ALB target group
• Allow specifying the cpu and memory reservations
• Allow specifying the desired # containers
• Allow specifying the health endpoint for ALB

• Create makefile
• Create Travis or circleci config
• Create unit tests

• ECS subnets are private
• ELB subnets are private for "internal" and public for "internet-facing"
• Allow option for ELB to be "internal" (default: false)
• ECS container instances use private subnets
• Create NAT for managed VPC
• Allow optional "proxy" for ECS container instances

Review the IAM policies and lock down. For sure the service roles for CodeBuild and CodePipeline are too loose at the moment.

Why do we need plugins?

What are example plugins?

What capabilities will plugins be able to provide?

As a mu developer, i want to be able to pin my dependencies to specific version so that I can have deterministic and repeatable builds.

AC:

• should be able to define dependencies with version numbers, ranges
• ci of mu should use dependency definition

Consider: https://github.com/Masterminds/glide

• Use Fn::Split
• Consider reference Subnet/VPC direct rather than via Import

Would we need to prescribe the standard pipeline or could we support their multi-variant stage pipelines?

How opinionated do we anticipate this to be?

Would they want or would we want to support different types of build, orchestration, and other tools?

Do we need to know anything about their source provider?

As a developer, I want to be able to list the environments in my account/region so that I can know which environments are currently defined.

AC:

• Running mu env ls will display the following for each environment: environment name, CFN stack name, status, last update, and the version of mu used to create/update stack

When running mu svc build you'll see logs repeating from the docker daemon

As a developer, I want to be able to list the pipelines in my account/region so that I can know which pipelines are currently defined.

AC:

• Running mu pipeline ls will display the following for each environment: service name, CFN stack name, status, last update, and the version of mu used to create/update stack

As a developer, I want to be able to view the list of services running in an environment to understand the current configuration and utilization of an environment.

AC:

• When running mu env show <env_name>, mu will list the services deployed to the cluster along with the following: docker image repo, status, last update, mu version

As a developer, I want to be able to show the details of an environment so that I can understand it's current configuration and utilization.

AC:

• Running mu env show <env_name> will display the stack names for the Cluster and VPC stacks
• Will show base URL for the ALB
• Will list the container instances with the following information: instance id, ami, instance type, AZ, status, # tasks, mem and cpu avail

Preview CFN changeset for the change