aws-samples / amazon-ecs-and-aws-step-functions-design-patterns-starter-kit

Amazon ECS and AWS Step Functions Design Patterns Starter kit

This starter kit demonstrates how to run Amazon Elastic Container Service (ECS) tasks using AWS Step Functions. We will implement the following design patterns:

1. Running ECS tasks using AWS Lambda
2. Running ECS tasks using Step Functions native integration

We use AWS Cloud Development Kit (CDK) to deploy application resources.

---

Contents

• Prerequisites
• Overview
  • ECS Task Business Logic
  • Amazon DynamoDB Tables
  • Workflow Specification
  • AWS CDK Stacks
• Patterns
  • Pattern 1: Running ECS tasks using AWS Lambda
  • Pattern 2: Running ECS tasks using Step Functions native integration
• Build
• Deploy
• Test
  • Pattern 1: Testing ECS tasks using AWS Lambda
  • Pattern 2: Testing ECS tasks using Step Functions native integration
• Cleanup
• Contributors

---

Prerequisites

1. Docker software is installed on your MacBook / Laptop
2. Docker daemon is running
3. You have AWS account credentials

---

Overview

ECS Task Business Logic

We run a simple business logic within an ECS task. It creates a copy of the input file in S3 bucket. We will run multiple instances of the task simultaneously.

---

Amazon DynamoDB Tables

Each pattern requires 2 DynamoDB tables. They are workflow_summary and workflow_details. workflow_summary is used to audit the status of overall workflow execution status. workflow_details is used to audit the status of individual ECS tasks. The schema of the DynamoDB tables is described in the below table.

Table	Schema	Capacity
workflow_summary_pattern_x	Partition key = workflow_name (String), Sort key = workflow_run_id (Number)	Provisioned read capacity units = 5, Provisioned write capacity units = 5
workflow_details_pattern_y	Partition key = workflow_run_id (Number), Sort key = ecs_task_id (String)	Provisioned read capacity units = 5, Provisioned write capacity units = 5

Note: here, x and y represent either 1 or 2.

---

Workflow Specification

We create 2 Step Functions State machines to demonstrate the design patterns. State machine is executed with a JSON specs as an input. The specs have two parts - 1) values for ECS cluster, DynamoDB tables, subnets, security groups, S3 bucket etc. 2) list of ECS tasks to run. Table below describes the specs.

JSON Attribute	Description
region	AWS region used
s3BucketName	Amazon S3 bucket used demonstrate ECS Task Business Logic
subnetIdLiteral	List of Subnet Ids separated by a separator
separator	The separator used in subnetIdLiteral
workflowName	Name of the workflow name for e.g. amazon_ecs_starter_kit-pattern-1
securityGroupId	The security group id used to run ECS tasks
ddbTableNameWFSummary	Name of the DynamoDB table for workflow summary
hashKeyWFSummary	The hash key of workflow summary table
rangeKeyWFSummary	The sort key of workflow summary table
ddbTableNameWFDetails	Name of the DynamoDB table for workflow details
hashKeyWFDetails	The hash key of workflow details table
rangeKeyWFDetails	The sort key of workflow details table
clusterName	Name of the ECS cluster
containerName	Name of the container
taskDefinition	Name of the ECS task definition name
taskList	It has specs for one more ECS tasks. These specs drive the business logic of a task. Each task has three attributes - 1) taskName (Name of the ECS task) 2) s3BucketName (S3 bucket name) 3) objectKey (Object key)

---

AWS CDK Stacks

CdkApp runs the following stacks

Stack Name	Purpose
ECSTaskSubmissionFromLambdaPattern	This stack provisions resources needed to demonstrate Pattern 1
ECSTaskSubmissionFromStepFunctionsPattern	This stack provisions resources needed to demonstrate Pattern 2

---

Patterns

Running ECS tasks using AWS Lambda

As show in the below figure, this pattern (Pattern 1) uses AWS Lambda function to run ECS tasks. We call the Lambda function as ECS Task Launcher. It parses workflow specs, submits ECS tasks to ECS Cluster and invokes second AWS Lambda function called ECS Task Monitor.

ECS Task Monitor tracks the completion status of running ECS tasks. Each time it runs, it checks the number of completed tasks versus the total number of tasks submitted and updates the DynamoDB table workflow_summary.

The task executed on ECS cluster is called ECS Task. It takes the following actions - 1) reads input parameters 2) inserts a record in DynamoDB table for auditing 3) copies the input file to a target folder 4) marks the status of its job to Complete in the the DynamoDB table workflow_detail.

---

Running ECS tasks using Step Functions native integration

As shown in the below figure, this pattern (Pattern 2) uses AWS Step Functions' native integration with Amazon ECS. Unlike the usage of a Lambda function in Pattern 1, we use Parallel state to run ECS tasks. The number of tasks run depends on the size of "taskList":[] in workflow_specs_pattern_2.json. The role of ECS Task Monitor and the way ECS Task executes are similar to Pattern 1.

---

Build

1. Clone this repository to your Mac/Laptop

2. Open your IDE for e.g. Eclipse or Spring Tools or Intellij IDEA

3. Import the project as a Maven project by pointing to <Path_to_cloned_repo>/Amazon-ecs-java-starter-kit/pom.xml | This imports 4 module projects.

4. Select parent project Amazon-ecs-java-starter-kit and build it using the below instructions

   1. Using standalone Maven, go to project home directory and run command mvn -X clean install
   2. From Eclipse or STS, run command -X clean install. Navigation: Project right click --> Run As --> Maven Build (Option 4)

5. Expected output 1: In your IDE, you will see the following output

   [INFO] Reactor Summary for amazon-ecs-java-starter-kit 1.0:
   [INFO] 
   [INFO] amazon-ecs-java-starter-kit ........................ [SUCCESS [  0.717 s]
   [INFO] amazon-ecs-java-starter-kit-cdk .................... [SUCCESS [ 14.230 s]
   [INFO] amazon-ecs-java-starter-kit-tasklauncher ........... [SUCCESS [  8.418 s]
   [INFO] amazon-ecs-java-starter-kit-task ................... [SUCCESS [ 21.857 s]
   [INFO] amazon-ecs-java-starter-kit-taskmonitor ............ [SUCCESS [  4.587 s]
   [INFO] ------------------------------------------------------------------------
   [INFO] BUILD SUCCESS
   [INFO] ------------------------------------------------------------------------
   [INFO] Total time:  49.979 s
   [INFO] Finished at: 2020-12-21T13:03:30-06:00

6. Expected output 2: Build process generates the following jar file in their respective directories

   Module artifact name	Approximate Size
   amazon-ecs-java-starter-kit-cdk-1.0.jar	32 KB
   amazon-ecs-java-starter-kit-tasklauncher-1.0.jar	21 MB
   amazon-ecs-java-starter-kit-task-1.0.jar	19 MB
   amazon-ecs-java-starter-kit-taskmonitor-1.0.jar	21 MB

---

Deploy

1. In the terminal, go to path /<Path_to_your_cloned_rep>/amazon-ecs-and-aws-step-functions-design-patterns-starter-kit/amazon-ecs-java-starter-kit-cdk. Now, you are in the CDK module of this project.

2. Replace 1234567890 with your AWS Account Id wherever applicable in the following steps.

3. Set these to your account and region

   export AWS_ACCOUNT_ID=1234567890
   export AWS_REGION=us-east-2

4. Bootstrap CDK

   cdk bootstrap aws://${AWS_ACCOUNT_ID}/$AWS_REGION

5. Output 1: In the command line, you will get the following output

   (node:63268) ExperimentalWarning: The fs.promises API is experimental
   ⏳  Bootstrapping environment aws://AWS_ACCOUNT_ID/us-west-2...
   ✅  Environment aws://AWS_ACCOUNT_ID/us-west-2 bootstrapped (no changes).

6. Output 2: In the AWS console under CloudFormation, you will see a Stack created as follows

   

7. Output 3: In the AWS console under S3, you will see a bucket created with name cdktoolkit-stagingbucket-*

8. Deploy both stacks

   cdk deploy --require-approval never --all --outputs-file outputs.json

9. Expected output 1: Stack for amazon-ecs-java-starter-pattern-1 created with the following resources:

   Resource Type	Resource Details
   VPC	1 VPC to launch resources needed by the starter kit
   Subnet	2 public subnets and 2 private subnets
   Route Table	1 route table per public, and private subnet
   Security Group	1 security group per ECR, ECS, and ECS Agent endpoints
   Security Group	1 security group per ECS Task Launcher and ECS Task Monitor
   VPC Endpoint	1 VPC endpoint per Amazon DynamoDB, Amazon S3, Amazon ECS, Amazon ECS Agent, and Amazon ECR API.
   ECS Cluster	1 ECS cluster to run ECS tasks
   ECR Repository	1 ECR repository to store Docker image for ECS Task binary
   ECS Task Definition	1 ECS task definition for ECS Task
   Amazon DynamoDB	2 tables. Refer to Amazon DynamoDB Tables for more details
   Step Functions state machine	1 State machine for orchestration
   AWS Lambda	Lambda Function to submit ECS tasks
   AWS Lambda	Lambda Function to monitor the progress of ECS tasks
   Amazon IAM Role	1 IAM role per Step Functions State machine, ECS Task Launcher, ECS Task Monitor. 2 IAM roles for ECS Task Definition - 1) ECS Task Role 2) ECS Task Execution Role

10. Expected output 2: Stack for amazon-ecs-java-starter-pattern-2 created with the following resources:

    Resource Type	Resource Details
    VPC	1 VPC to launch resources needed by the starter kit
    Subnet	2 public subnets and 2 private subnets
    Route Table	1 route table per public, and private subnet
    Security Group	1 security group per ECR, ECS, and ECS Agent endpoints
    Security Group	1 security group per ECS Task Launcher and ECS Task Monitor
    VPC Endpoint	1 VPC endpoint per Amazon DynamoDB, Amazon S3, Amazon ECS, Amazon ECS Agent, and Amazon ECR API.
    ECS Cluster	1 ECS cluster to run ECS tasks
    ECR Repository	1 ECR repository to store Docker image for ECS Task binary
    ECS Task Definition	1 ECS task definition for ECS Task
    Amazon DynamoDB	2 tables. Refer to Amazon DynamoDB Tables for more details
    Step Functions state machine	1 State machine for orchestration
    AWS Lambda	Lambda Function to monitor the progress of ECS tasks
    Amazon IAM Role	1 IAM role per Step Functions State machine, and ECS Task Launcher. 2 IAM roles for ECS Task Definition - 1) ECS Task Role 2) ECS Task Execution Role

11. Expected output 3: A file outputs.json is created with a list of AWS resource values provisioned by the CDK.

---

Test

Testing ECS tasks using AWS Lambda

1. Open file workflow_specs_pattern_1.json in your IDE and update JSON attributes based on the values for amazon-ecs-java-starter-pattern-1 of outputs.json.

2. Open your command prompt / Mac terminal

3. Go to path /<Path_to_your_cloned_rep>/amazon-ecs-and-aws-step-functions-design-patterns-starter-kit/amazon-ecs-java-starter-kit-tasklauncher/

4. Copy jar file to S3 bucket. Use the following command

   aws s3 cp ../amazon-ecs-java-starter-kit-tasklauncher/target/amazon-ecs-java-starter-kit-tasklauncher-1.0.jar s3://${AWS_ACCOUNT_ID}-amazon-ecs-java-starter-kit-pattern-1-bucket/amazon_ecs_java_starter_kit_jar/

5. Go to path /<Path_to_your_cloned_rep>/amazon-ecs-and-aws-step-functions-design-patterns-starter-kit/amazon-ecs-java-starter-kit-cdk

6. Start Step Functions execution. Use the following command

   aws stepfunctions start-execution --state-machine-arn "arn:aws:states:${AWS_REGION}:${AWS_ACCOUNT_ID}:stateMachine:amazon-ecs-java-starter-kit-pattern-1" --input "$(cat workflow_specs_pattern_1.json)"

7. Expected output 1: You will get a response as follows

   {
    "executionArn": "arn:aws:states:us-east-2:1234567890:execution:amazon-ecs-java-starter-kit-pattern-1:4ea1f256-a0bb-4692-b63a-6b80edc02cb7",
    "startDate": "2020-12-21T09:54:29.385000-06:00"
   }

8. Expected output 2: In Step Functions console, state machine amazon-ecs-java-starter-kit-pattern-1 changes to Running state

9. Expected output 3: After a minute or two, your State machine execution status will be successful as follows.

   

10. Expected output 4: In S3 bucket, you will see 10 extra jar files which are copied by ECS task instances.

    

11. Expected output 5: In DynamoDB tables, you will find new items as follows:

    Table	Number of items	Sample column values
    workflow_summary_pattern_1	1	number_of_tasks = 10, completed_tasks = 10
    workflow_details_pattern_1	10	status = Completed

Testing ECS tasks using Step Functions native integration

1. Open file workflow_specs_pattern_2.json in your IDE and update JSON attributes based on the values for amazon-ecs-java-starter-pattern-2 of outputs.json.

2. Open your command prompt or Mac terminal

3. Go to path /<Path_to_your_cloned_rep>/amazon-ecs-and-aws-step-functions-design-patterns-starter-kit/amazon-ecs-java-starter-kit-tasklauncher/

4. Copy jar file to S3 bucket. Use the following command

   aws s3 cp ../amazon-ecs-java-starter-kit-tasklauncher/target/amazon-ecs-java-starter-kit-tasklauncher-1.0.jar s3://${AWS_ACCOUNT_ID}-amazon-ecs-java-starter-kit-pattern-2-bucket/amazon_ecs_java_starter_kit_jar/

5. Go to path /<Path_to_your_cloned_rep>/amazon-ecs-and-aws-step-functions-design-patterns-starter-kit/amazon-ecs-java-starter-kit-cdk

6. Start Step Functions execution. Use the following command

   aws stepfunctions start-execution --state-machine-arn "arn:aws:states:${AWS_REGION}:${AWS_ACCOUNT_ID}:stateMachine:amazon-ecs-java-starter-kit-pattern-1" --input "$(cat workflow_specs_pattern_2.json)"
   

7. Expected outputs are similar to Pattern 1

---

Cleanup

1. Go to /<Path_to_your_cloned_rep>/Amazon-ecs-java-starter-kit/amazon-ecs-java-starter-kit-cdk

2. Delete DynamoDB tables

   ./delete_ddb_items.sh workflow_details_pattern_1 workflow_summary_pattern_1

   ./delete_ddb_items.sh workflow_details_pattern_2 workflow_summary_pattern_2

3. Empty S3 buckets

   aws s3 ls s3://${AWS_ACCOUNT_ID}-amazon-ecs-java-starter-kit-pattern-1-bucket/amazon_ecs_java_starter_kit_jar/ | grep _ | awk '{print $NF}' | while read OBJ; do aws s3 rm s3://${AWS_ACCOUNT_ID-}amazon-ecs-java-starter-kit-pattern-2-bucket/amazon_ecs_java_starter_kit_jar/$OBJ;done

   aws s3 ls s3://${AWS_ACCOUNT_ID-}-amazon-ecs-java-starter-kit-pattern-2-bucket/amazon_ecs_java_starter_kit_jar/ | grep _ | awk '{print $NF}' | while read OBJ; do aws s3 rm s3://${AWS_ACCOUNT_ID-}amazon-ecs-java-starter-kit-pattern-2-bucket/amazon_ecs_java_starter_kit_jar/$OBJ;done

4. Delete S3 buckets

   aws s3 rm s3://${AWS_ACCOUNT_ID-}amazon-ecs-java-starter-kit-pattern-1-bucket/amazon_ecs_java_starter_kit_jar/amazon-ecs-java-starter-kit-tasklauncher-1.0.jar

   aws s3 rm s3://${AWS_ACCOUNT_ID-}amazon-ecs-java-starter-kit-pattern-2-bucket/amazon_ecs_java_starter_kit_jar/amazon-ecs-java-starter-kit-tasklauncher-1.0.jar

5. Delete ECR Repositories

   aws ecr delete-repository --force --repository-name amazon-ecs-java-starter-kit-pattern-1

   aws ecr delete-repository --force --repository-name amazon-ecs-java-starter-kit-pattern-2

6. Cleanup stacks

   cdk destroy --force --all

---

Contributors

1. Sarma Palli, Senior DevOps Cloud Architect, Amazon Web Services
2. Ravi Itha, Senior Big Data Consultant, Amazon Web Services

---

License Summary

This sample code is made available under the MIT-0 license. See the LICENSE file.