In this Chapter, the directory structures and how each module has been implemented is described.
To provision a VPC
, private
and public subnets
, natgateway
and respective security groups
, AWS VPC module has been used. According to Hashicorp-Provision an EKS Cluster (AWS), the following tf
files in the eks
directory have been implemented:
vpc.tf
: create a VPC with a public and private subnet, a natgateway and related options.
sgs.tf
: provisions the security groups used by the EKS cluster.
eks-cluster.tf
: provisions all the resources (AutoScaling Groups, etc...) required to set up an EKS cluster using the AWS EKS Module. 3 node groups (with the private subnet) - system pod node, cpu-application node and gpu application node with desired capacity=1 has been created. Taint is also could be assigned to the nodes to allow the pods to schedule onto nodes with matching cpu and gpu taints.
taints = [{
key = "node-type"
value = "gpu"
effect = "NO_SCHEDULE"
}
]
kubernetes.tf
: the Kubernetes provider is included in this file so the EKS module can complete successfully.
To add ingress controller and get a public Network load balancer, Helm Module and Kubernetes-ingress-nginx have been used. According to the AWS and Hashicorp documentation, the following tf
files in the nginx
directory have been implemented:
helm_release.tf
: official ingress controller from kubernetes repository has been installed using helm module. Please note that this chart will create a Kubernetes service of type:loadbalancer with the Network Load Balancer(NLB) annotations, and this load balancer sits in front of the ingress controller.kubernetes.tf
: usingterraform_remote_state
data block, information of eks cluster has been extracted fromeks
module.
To create an encrypted EFS and attach it to the EKS cluster with its storage class as default storage class, according to the AWS, the following tf
files in the storage
directory have been implemented:
iam.tf
: Create an IAM policy and assign it to an IAM role. The policy will allow the Amazon EFS driver to interact with the file system. Then, it runs anlocal-exec
provisioner to create AWS Identity and Access Management (IAM) OpenID Connect (OIDC) provider for the cluster. It is assumed thateksctl
has been installed on the machine running this script. After that, it creates the IAM role and Kubernetes service account. It also attaches the policy to the role, annotates the Kubernetes service account with the IAM role ARN, and adds the Kubernetes service account name to the trust policy for the IAM role.efs_csi_helm.tf
: installs the Amazon EFS CSI driver using Helm.security.tf
: creates a security group with an inbound rule that allows inbound NFS traffic for the Amazon EFS mount points.efs.tf
: creates an encrypted Amazon EFS file system for the Amazon EKS cluster and creates mount targets.storage.tf
: deploysStorageClass
manifest for Amazon EFS and makes it default and the other storageclass non-default.
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: efs-sc
annotations:
storageclass.kubernetes.io/is-default-class: "true"
provisioner: efs.csi.aws.com
parameters:
provisioningMode: efs-ap
fileSystemId: "${fileSystemId}"
directoryPerms: "700"
gidRangeStart: "1000" # optional
gidRangeEnd: "2000" # optional
basePath: "/dynamic_provisioning" # optional
To test the storageclass is working correctly a sample pod with a pvc attached to it could be deployed that dynamically creates a persistent volume:
deploy_sample.pvc
: it applies a pod definition with a pvc attached.
Follow these steps to run these scripts, please. It is assumed that, kubectl
, eksctl
and aws cli
have been installed on the machine running these scripts. Avoiding installing kubectl
and aws
is possible using module like kubectl
, however, installing eksctl
is necessary.
- Install vpc and eks cluster :
cd eks
terraform init
terraform apply
and pressy
orterraform apply --auto-approve
- Install ingress controller :
cd nginx
terraform init
terraform apply
and pressy
orterraform apply --auto-approve
- Install efs storage for the cluster :
cd storage
terraform init
terraform apply
and pressy
orterraform apply --auto-approve
- local backend is used for these scripts for the sake of simplicity. A s3 backend with the dynamoDB is better solution for the production use cases.