monegim / terraform-eks-efs

Overall View

In this Chapter, the directory structures and how each module has been implemented is described.

1. VPC

To provision a VPC, private and public subnets, natgateway and respective security groups, AWS VPC module has been used. According to Hashicorp-Provision an EKS Cluster (AWS), the following tf files in the eks directory have been implemented:

vpc.tf: create a VPC with a public and private subnet, a natgateway and related options.

• sgs.tf : provisions the security groups used by the EKS cluster.

2. EKS

• eks-cluster.tf: provisions all the resources (AutoScaling Groups, etc...) required to set up an EKS cluster using the AWS EKS Module. 3 node groups (with the private subnet) - system pod node, cpu-application node and gpu application node with desired capacity=1 has been created. Taint is also could be assigned to the nodes to allow the pods to schedule onto nodes with matching cpu and gpu taints.

      taints = [{
          key    = "node-type"
          value  = "gpu"
          effect = "NO_SCHEDULE"
          }
      ]

• kubernetes.tf: the Kubernetes provider is included in this file so the EKS module can complete successfully.

3. Ingress Controller

To add ingress controller and get a public Network load balancer, Helm Module and Kubernetes-ingress-nginx have been used. According to the AWS and Hashicorp documentation, the following tf files in the nginx directory have been implemented:

• helm_release.tf: official ingress controller from kubernetes repository has been installed using helm module. Please note that this chart will create a Kubernetes service of type:loadbalancer with the Network Load Balancer(NLB) annotations, and this load balancer sits in front of the ingress controller.
• kubernetes.tf : using terraform_remote_state data block, information of eks cluster has been extracted from eks module.

4. EFS

To create an encrypted EFS and attach it to the EKS cluster with its storage class as default storage class, according to the AWS, the following tf files in the storage directory have been implemented:

• iam.tf: Create an IAM policy and assign it to an IAM role. The policy will allow the Amazon EFS driver to interact with the file system. Then, it runs an local-exec provisioner to create AWS Identity and Access Management (IAM) OpenID Connect (OIDC) provider for the cluster. It is assumed that eksctl has been installed on the machine running this script. After that, it creates the IAM role and Kubernetes service account. It also attaches the policy to the role, annotates the Kubernetes service account with the IAM role ARN, and adds the Kubernetes service account name to the trust policy for the IAM role.
• efs_csi_helm.tf: installs the Amazon EFS CSI driver using Helm.
• security.tf: creates a security group with an inbound rule that allows inbound NFS traffic for the Amazon EFS mount points.
• efs.tf: creates an encrypted Amazon EFS file system for the Amazon EKS cluster and creates mount targets.
• storage.tf: deploys StorageClass manifest for Amazon EFS and makes it default and the other storageclass non-default.

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: efs-sc
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: efs.csi.aws.com
parameters:
  provisioningMode: efs-ap
  fileSystemId: "${fileSystemId}"
  directoryPerms: "700"
  gidRangeStart: "1000" # optional
  gidRangeEnd: "2000" # optional
  basePath: "/dynamic_provisioning" # optional

Deploy a sample application

To test the storageclass is working correctly a sample pod with a pvc attached to it could be deployed that dynamically creates a persistent volume:

• deploy_sample.pvc : it applies a pod definition with a pvc attached.

How to run

Follow these steps to run these scripts, please. It is assumed that, kubectl, eksctl and aws cli have been installed on the machine running these scripts. Avoiding installing kubectl and aws is possible using module like kubectl, however, installing eksctl is necessary.

1. Install vpc and eks cluster :
   1. cd eks
   2. terraform init
   3. terraform apply and press y or terraform apply --auto-approve
2. Install ingress controller :
   1. cd nginx
   2. terraform init
   3. terraform apply and press y or terraform apply --auto-approve
3. Install efs storage for the cluster :
   1. cd storage
   2. terraform init
   3. terraform apply and press y or terraform apply --auto-approve

Notes

1. local backend is used for these scripts for the sake of simplicity. A s3 backend with the dynamoDB is better solution for the production use cases.