β΅ Build Amazon EKS using Terraform, GitHub Actions and GitOps
... managed by Flux/ArgoCD and serviced with Renovate ...
- GitHub repository: https://github.com/ruzickap/k8s-tf-eks-gitops
- Web Pages: https://ruzickap.github.io/k8s-tf-eks-gitops
πΒ Overview
β¨ Β Cluster setup
π¨ Β Cluster components
- cilium: For cluster networking.
- ingress-nginx: Provides ingress cluster services.
- SOPS: Encrypts secrets which is safe to store - even to a public repository.
- external-dns: Creates DNS entries in Cloud Provider's DNS service.
- cert-manager: Configured to create TLS certs for all ingress services automatically using LetsEncrypt.
πΒ Repository structure
flux tree kustomization flux-system --compact
Output:
Kustomization/flux-system/flux-system
βββ Kustomization/flux-system/cluster-apps
β βββ HelmRelease/metrics-server/metrics-server
β βββ HelmRelease/polaris/polaris
β βββ Kustomization/flux-system/cert-manager
β β βββ HelmRelease/cert-manager/cert-manager
β βββ Kustomization/flux-system/cert-manager-certificate
β βββ Kustomization/flux-system/cert-manager-clusterissuer
β βββ Kustomization/flux-system/cert-manager-crds
β βββ Kustomization/flux-system/cluster-autoscaler
β β βββ HelmRelease/cluster-autoscaler/cluster-autoscaler
β βββ Kustomization/flux-system/crossplane
β β βββ HelmRelease/crossplane-system/crossplane
β βββ Kustomization/flux-system/crossplane-provider
β βββ Kustomization/flux-system/crossplane-providerconfig
β βββ Kustomization/flux-system/dex
β β βββ HelmRelease/dex/dex
β βββ Kustomization/flux-system/external-dns
β β βββ HelmRelease/external-dns/external-dns
β βββ Kustomization/flux-system/ingress-nginx
β β βββ HelmRelease/ingress-nginx/ingress-nginx
β βββ Kustomization/flux-system/kube-prometheus-stack
β β βββ HelmRelease/kube-prometheus-stack/kube-prometheus-stack
β βββ Kustomization/flux-system/kubernetes-dashboard
β β βββ HelmRelease/kubernetes-dashboard/kubernetes-dashboard
β βββ Kustomization/flux-system/oauth2-proxy
β β βββ HelmRelease/oauth2-proxy/oauth2-proxy
β βββ Kustomization/flux-system/podinfo
β βββ HelmRelease/podinfo/podinfo
βββ Kustomization/flux-system/cluster-apps-secrets
βββ Kustomization/flux-system/sources
β βββ HelmRepository/flux-system/autoscaler
β βββ HelmRepository/flux-system/bitnami
β βββ HelmRepository/flux-system/crossplane
β βββ HelmRepository/flux-system/dex
β βββ HelmRepository/flux-system/fairwinds-stable
β βββ HelmRepository/flux-system/ingress-nginx
β βββ HelmRepository/flux-system/jetstack
β βββ HelmRepository/flux-system/kubernetes-dashboard
β βββ HelmRepository/flux-system/metrics-server
β βββ HelmRepository/flux-system/oauth2-proxy
β βββ HelmRepository/flux-system/podinfo
β βββ HelmRepository/flux-system/prometheus-community
βββ GitRepository/flux-system/flux-system
π€ Β Automate all the things
- GitHub Actions for checking code formatting
- Renovate Renovate GitHub action keeps my application charts and container images up-to-date
πΈοΈ Β Secrets
There are several secrets:
cluster-apps-vars-terraform-secret
- used for providing Terraform variables to Flux/Kustomizations: eks.tfcluster-apps-secrets
- secrets specific to cluster: cluster-apps-secrets.yamlcluster-apps-group-secrets
- secrets specific to cluster group: cluster-apps-secrets.yaml
π€·ββοΈ Β Notes
- Describe the directory structure
- Check emails form policy-reporter
- Put all
HelmRepository
objects toflux-system
instead of "namespaces" to be able to share them - Check snapshots (cnpg/velero) + KMS keys (if they are being deleted)
kubernetes-dashboard
- auto login not working
π€ Β Thanks
A lot of inspiration for my cluster came from the people that have shared their clusters over at awesome-home-kubernetes and many other "GitHub" repositories...
Requirements for K8s cluster management
Not complete...
- Run change on only 3% of the clusters, then another 3%, ...
- Changes applied to group of clusters
- Easily add / remove application from specific cluster / cluster group
- One source (file) for "variables/secrets"
- Move cluster from one cluster to another... ?
- Different TF code for different clusters / cluster groups