devopscorner / iac-terraform-emr

AWS Summit 2022 ASEAN --- COM203 Using IaC with Terraform to provision Big Data Platform on Amazon EMR

IaC Terraform EMR

Provisioning Amazon EMR cluster using Terraform as Infrastructure-as-Code (IaC) tools

---

Available Tags

Alpine

Image name	Size
devopscorner/terraform-emr:alpine
devopscorner/terraform-emr:alpine-latest
devopscorner/terraform-emr:alpine-3.16

Ubuntu

Image name	Size
devopscorner/terraform-emr:ubuntu
devopscorner/terraform-emr:ubuntu-latest
devopscorner/terraform-emr:ubuntu-22.04

AWS Linux (CodeBuild)

Image name	Size
devopscorner/terraform-emr:latest
devopscorner/terraform-emr:codebuild
devopscorner/terraform-emr:codebuild-latest
devopscorner/terraform-emr:codebuild-2.0
devopscorner/terraform-emr:codebuild-3.0
devopscorner/terraform-emr:codebuild-4.0

---

Prerequirements

• Docker (docker)
• Docker Compose (docker-compose)
• AWS Cli version 2 (aws)
• Terraform Cli (terraform)
• Terraform Environment (tfenv)

Documentation

• Index Documentation, go to this link
• Reproduce DEMO, detail here
• EMR DEMO Script, listed in test folder

Using Cloud9 IDE

• Provisioning Cloud9 in EC2 onspot Instances

  aws cloud9 create-environment-ec2 --name example-env --description "environment" --instance-type t3.small --subnet-id subnet-id --automatic-stop-time-minutes 60 --owner-arn arn:aws:iam::123:user/User
  
  -- or --
  
  Go to `terraform/environment/providers/aws/infra/resources/cloud9` folder
  
  terraform init
  terraform workspace select staging
  terraform plan
  terraform apply
  

Cleanup

• Destroy Environment Lab

  terraform destroy
  

Tested Environment

Versioning

• Docker version

  docker version
  
  Client:
  Cloud integration: v1.0.22
  Version:           20.10.12
  API version:       1.41
  Go version:        go1.16.12
  Git commit:        e91ed57
  Built:             Mon Dec 13 11:46:56 2021
  OS/Arch:           darwin/amd64
  Context:           default
  Experimental:      true
  

• Docker-Compose version

  docker-compose -v
  ---
  Docker Compose version v2.2.3
  

• AWS Cli

  aws -v
  ---
  Note: AWS CLI version 2, the latest major version of the AWS CLI, is now stable and recommended for general use. For more information, see the AWS CLI version 2 installation instructions at: <https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html>
  

• Terraform Cli

  terraform version
  ---
  Terraform v1.1.6
  on darwin_amd64
  - provider registry.terraform.io/hashicorp/aws v3.74.3
  - provider registry.terraform.io/hashicorp/local v2.1.0
  - provider registry.terraform.io/hashicorp/null v3.1.0
  - provider registry.terraform.io/hashicorp/random v3.1.0
  - provider registry.terraform.io/hashicorp/time v0.7.2
  

• Terraform Environment Cli

  tfenv -v
  ---
  tfenv 2.2.2
  

Security Check

Make sure that you didn't push sensitive information in this repository

• AWS Credentials (AWS_ACCESS_KEY, AWS_SECRET_KEY)
• AWS Account ID
• AWS Resources ARN
• Username & Password
• Private (id_rsa) & Public Key (id_rsa.pub)
• DNS Zone ID
• APP & API Key

Copyright

• Author: Dwi Fahni Denni (@zeroc0d3)
• Vendor: DevOps Corner Indonesia (devopscorner.id)
• License: Apache v2