sylviewn / cfn-aws-ecs-elk

Deploying Elastic Stack with AWS ECS Docker

ELK-stack-on-AWS-ECS

Deploying Elastic Stack on AWS Elastic Container Service

I will outline the procedure I followed to deploy Elastic Stack on AWS ECS.

Note: The following arch has been deprecated as Docker released Here is the architecture I followed to build this

In the diagram above, we have the following containers:

• Three ElasticSearch containers to ensure high availability. Odd number to minimize split brain scenarios.
• Two Kibana containers for UI
• Two logstash containers to handle traffic

Split Brain Scenario is a situation where a network failure caused the network to split. The split means that the parts can no longer communicate with each other. They need to decide what to do next on their own based on the latest membership data of the cluster. This is being handled by using 3 ECS clusters.

Working with immutable infrastructure to replace defective componenets of this architecture.

After adding DockerFile and elasticsearch.yml configuration files, run the aws-ecr-bake-and-push.sh file to push a new docker image into the private docker repo on AWS ECR.

./elk-stack-aws-ecr-bake-and-push.sh elasticsearch

After we have the customized docker image with elasticsearch on ECR, we need to create a cluster in ECS with 3 EC2 instances.

Few points that need to be addressed:

• We might not need all 3 instances all the time. For this reason, we will be using Autoscaling to start with 1 instance and scale to 3 whenever needed.
• There is no way to disable public IP assignments from the ECS wizard. Instead, we need to manually ensure our private cluster stays private, safely tucked inside our VPC.
• ElasticSearch will fail due to a low mmap count; we need to increase that.
• We need to allocate more space to our ElasticSearch nodes than the standard 8GB granted by Amazon.
• We need to customize the docker daemon to take advantage of the extra space.

Cluster	dev-elk-cluster
Instance type	t2.micro
Desired number of instances	3
Key pair	elk-stack
ECS AMI ID	ami-066ce9bb9f4cbb03d
VPC	vpc-071e76a617d759d04
Subnet 1	subnet-02f18e573300175b6
Subnet 1 route table association	rtbassoc-06119201929019675
Subnet 2	subnet-0b8babda5dab2c0b7
Subnet 2 route table association	rtbassoc-0881ec1492ec2c853
VPC Availability Zones	us-east-1a, us-east-1b, us-east-1c, us-east-1d, us-east-1e, us-east-1f
Security group	sg-0cfae2130059a5738
Internet gateway	igw-0a96cb67118059687
Route table	rtb-0de91746702e12029
Amazon EC2 route	EC2Co-Publi-VF872SCT2020
Virtual private gateway attachment	EC2Co-Attac-17Z48YC2S8CVW
Launch configuration	EC2ContainerService-dev-elk-cluster-EcsInstanceLc-154J2FEYWM6UX
Auto Scaling group	EC2ContainerService-dev-elk-cluster-EcsInstanceAsg-1OEDR2NDUIUHE

---

Adding the following content to User Data section of the EC2 instances in CloudFormation Changeset.

Content-Type: multipart/mixed; boundary="//"
MIME-Version: 1.0

--//
Content-Type: text/cloud-boothook; charset="us-ascii"

# Set Docker daemon options
cloud-init-per once docker_options echo 'OPTIONS="${OPTIONS} --storage-opt dm.basesize=250G"' >> /etc/sysconfig/docker

--//
Content-Type: text/x-shellscript; charset="us-ascii"

#!/bin/bash
echo ECS_CLUSTER=dev-elk-cluster >> /etc/ecs/ecs.config;
echo ECS_BACKEND_HOST=dev-elk-cluster >> /etc/ecs/ecs.config;
echo ECS_ENGINE_TASK_CLEANUP_WAIT_DURATION=15m >> /etc/ecs/ecs.config;
echo ECS_IMAGE_CLEANUP_INTERVAL=10m >> /etc/ecs/ecs.config;
sysctl -w vm.max_map_count=262144;
mkdir -p /usr/share/elasticsearch/data/;
chown -R 1000.1000 /usr/share/elasticsearch/data/;

--//--

Storage

Adding EBS stores for storage and using PIOPS as advised on elastic's documentation. Or we can go with using Instance Store that has the advantage of hard disk physical attached to the host and also benefit of avoiding to pay extra for EBS.

Note: EBS Storage option in ECS Cluster configuration is not the storage for containers to store their data. This volume is strictly for docker images.

Amazon explains this:

The volume is configured as a Logical Volume Management (LVM) device and it is accessed directly by Docker via the devicemapper backend. Because the volume is not mounted, you cannot use standard storage information commands (such as df -h) to determine the available storage.

AutoScaling

We will be using Autoscaling to scale up or down the EC2 instances as required. Autoscaling Launch config can be used as reference.

We can use CloudFormation to deploy ECS containers with ElasticSearch and Kibana.

To create custom ElasticSearch Docker images, I am using AWS ECR push script using the CloudFormation stack with AWS ECR params and this CloudFormation template.

I am using this bash script to push custom elasticsearch images to AWS ECR when needed.

Once the custom images are updated in ECR, we can work on deploying ECS instances. This can be done using CloudFormation as well.

Note: CloudFormation will require privileges to create IAM roles.

To create EC2 instances with ECS and using the images from ECR repo, I am using AWS ECS bash script and the CloudFormation template that will use these parameters.

The need for Logstash has been deprecated since AWS ECS can utilize awslogs Log driver.