mathieu-benoit / myakscluster

How to setup a secure Kubernetes cluster on Azure

myakscluster

Here are blog articles related to this repository:

• setup a performant aks cluster
• private aks and private acr, safer you are
• my own custom and private azure pipelines agent as a docker container
• flexible kured deployment with its helm chart
• kubernetes network policies, how to secure the communications between your pods

To properly setup and secure your AKS cluster, there is a couple of features and components to enable in order to properly setup your Security Posture, here is the list:

Basic - without cost:

• Managed Identity #62
• Azure Lock #21
• kured #13
• Disable K8S Dashboard #24
• Network Policy with Calico #9
• AAD #10
• Azure Policy for AKS #11
• Azure KeyVault for Azure pipelines #3

Advanced - with cost:

• Uptime SLA #66
• Private AKS cluster + Private ACR + Jumpbox with Azure Private Endpoint #57
  • Alternatively: Use Authorized IP addresses with AKS and Azure Service Endpoint with ACR.
• Limit Egress Traffic with Azure Firewall #16

Complementary Azure services to leverage:

• Azure Advisor for AKS
• Azure Security Center for ACR
• Azure Security Center for AKS

Provisioning

Prerequisites

• Install Azure CLI
• Install Terraform
  • Optional if you would like to leverage Terraform instead of Azure CLI.
• Install Azure ConnectedK8s and KubernetesConfiguration CLI extensions
  • Optional if you would like to use Azure Arc enabled Kubernetes.

Configuration pre-provisioning

az login
#az account list -o table
#az account set -s <subscriptionId>

export LOCATION='canadacentral'
export NODES_COUNT=3
export NODE_SIZE='Standard_DS2_v2'
export ZONES=false

./run-pre-checks.sh

randomSuffix=$(shuf -i 1000-9999 -n 1)
export AKS=FIXME$randomSuffix
export K8S_VERSION=$(az aks get-versions \
  -l $LOCATION \
  --query "orchestrators[?isPreview==null].orchestratorVersion | [-1]" \
  -o tsv)

Provisioning Option 1: Azure CLI

cd cli
./create-aks-cluster.sh

FYI, current issues/workarounds with Azure CLI:

• az aks create|nodepool add --labels - labels ignored if prefix contains '.'
  • Need to manually apply the kubernetes.azure.com/mode=user k8s label on each user node.

Provisioning Option 2: Terraform

cd tf
sudo terraform init
terraform plan \
  -var aks_name=$AKS \
  -var k8s_version=$K8S_VERSION \
  -var location=$LOCATION \
  -var aks_node_count=$NODES_COUNT \
  -var aks_node_size=$NODE_SIZE
terraform apply \
  -auto-approve \
  -var aks_name=$AKS \
  -var k8s_version=$K8S_VERSION \
  -var location=$LOCATION \
  -var aks_node_count=$NODES_COUNT \
  -var aks_node_size=$NODE_SIZE

Configuration post-provisioning

You need to connect to the Jumpbox VM via the Bastion host and run the commands below:

az login \
  --service-principal \
  -u FIXME \
  -p FIXME \
  --tenant-id FIXME
az aks get-credentials \
  -g $RG \
  -n $RG
./configure-aks-cluster.sh

Pricing estimation

• AKS
  • $0.10 per cluster per hour.
• Virtual Machines related to AKS worker nodes
  • Estimation: For 2 Node pools X 3 Standard_DS2_v2 Ubuntu worker nodes, ~ $xx USD/month
  • To decrease the cost: you may want to leverage Azure Reserved VMs Instances to reduce the cost of your AKS worker nodes
• Managed Disks (related to the AKS's Nodes OS Disk)
  • Estimation: ~ $xx USD/month
• Log Analytics Workspace (related to the Azure Monitor for Containers add-on)
  • Estimation: For xx GB/month, ~ $xx USD/month
  • To decrease the cost: you may want to leverage logs filtering, retention by data type or New capacity-based pricing option.
• Standard Load Balancer
  • Estimation: TODO
  • To decrease the cost: you may want to leverage Basic Load Balancer instead which is free but you won't be able to use Availability Zones nor VNET Peering features
• Azure Firewall
  • Estimation: TODO
• Public IP
  • Related to the SLB + FW
  • Estimation: TODO
• Bandwidth (related to Availability Zones)
  • Estimation: TODO
  • Note: you may want to leverage PodAffinity to avoid cross-zones communications with Pods too chatty
• Private Endpoint
  • Estimation: For 2 Private Endpoints (AKS + ACR)
• Bastion - TODO

Other considerations:

• Regulatory compliance
• Security hardening in AKS virtual machine hosts
• Regularly update to the latest version of Kubernetes
• Audit logging in AKS
• Azure AD PIM
• Azure DDOS Protection
• Azure Front Door - TODO

Resources

• Camp Cloud Native, Best Practices in Securing Your Kubernetes Applications
• Attack matrix for Kubernetes
• Detect large-scale cryptocurrency mining attack against Kubernetes clusters
• Trusted Cloud: security, privacy, compliance, resiliency, and IP
• Microsoft Cloud Adoption Framework for Azure
• Azure webinar series - Help Deliver Applications Securely with DevSecOps
• Enterprise security in the era of containers and Kubernetes
• Azure Kubernetes Services: Container Security for a Cloud Native World
• 11 Ways (Not) to Get Hacked
• Tutorial: Bullet-Proof Kubernetes: Learn by Hacking - Luke Bond & Ana-Maria Calin
• Tutorial: Building Security into Kubernetes Deployment Pipelines - Michael Hough & Sam Irvine
• How Spotify Accidentally Deleted All its Kube Clusters with No User Impact
• Introduction to Kubernetes Security with GitOps
• Open Sourcing the Kubernetes Security Audit
• Kubernetes Security book
• https://aka.ms/aks/secure
• Exploiting and protecting containers with a few lines of scripting

myakscluster's Issues

Use --network-policy in az aks create to leverage Calico

• https://azure.microsoft.com/en-us/updates/user-defined-network-policy-in-azure-kubernetes-service-aks-is-now-available/
• https://docs.microsoft.com/en-us/azure/aks/use-network-policies
• https://vincentlauzon.com/2019/02/27/network-policies-with-calico-in-aks/
• https://azure.microsoft.com/en-us/blog/integrating-azure-cni-and-calico-a-technical-deep-dive/
• https://www.youtube.com/watch?v=131_TIa_ftI
• https://github.com/ahmetb/kubernetes-network-policy-recipes
• https://www.youtube.com/watch?v=3gGpMmYeEO8
• https://www.magalix.com/blog/kubernetes-network-policies-101
• https://medium.com/@reuvenharrison/an-introduction-to-kubernetes-network-policies-for-security-people-ba92dd4c809d

Use the --workspace-resource-id in az aks create

https://docs.microsoft.com/en-us/azure/azure-monitor/platform/template-workspace-configuration

https://docs.microsoft.com/en-us/cli/azure/resource?view=azure-cli-latest#az-resource-create

https://docs.microsoft.com/en-us/azure/azure-monitor/platform/manage-access

[ACR] VNET and Firewall rules integration

https://aka.ms/acr/vnet

Standard Load Balancer

• https://azure.microsoft.com/en-us/updates/standard-load-balancers-in-azure-kubernetes-service-aks/
• https://docs.microsoft.com/en-us/azure/aks/load-balancer-standard
• https://docs.microsoft.com/en-us/azure/load-balancer/load-balancer-overview#skus
• https://azure.microsoft.com/en-us/pricing/details/load-balancer/
• https://docs.microsoft.com/en-us/azure/architecture/guide/technology-choices/load-balancing-overview

Some gotchas:

• The load-balancer-sku property can only be used when your cluster is created. You cannot change the load balancer SKU after an AKS cluster has been created. Also, you can only use one type of load balancer SKU in a single cluster.
• If you want to use your own public IPs, use the load-balancer-outbound-ips, or load-balancer-outbound-ip-prefixes parameters. Both of these parameters can also be used when updating the cluster.
• VNET Peering is now possible with SLB, it was not with Basic Load Balancer
• SLB is required for Availability Zones #33

(Beta) Azure KeyVault Flex Volume

https://vincentlauzon.com/2019/02/21/flex-volume-in-aks/

It looks like FlexVolume won't be GA and its replacement will be CSI:

• https://github.com/deislabs/secrets-store-csi-driver
• https://kubernetes.io/blog/2019/01/15/container-storage-interface-ga/

(Beta) Helm 3

• https://cloudblogs.microsoft.com/opensource/2019/05/21/helm-3-release/
• https://github.com/squillace/helm-multinode/tree/master/HelloHelm3
• Security with Helm 3: https://helmsummit2019.sched.com/event/S8tr

TODO:

• remove tiller and associated serviceaccount setup
• remove helm init
• use helm registry login
• use helm chart push and helm chart pull
• use helm chart tag
• update index.yaml into index.json
• change helm delete by helm uninstall
• if k8s namespace doesn't exist, helm3 will raise an error, k8s namespace should be created ahead
• Migrate from helm2? https://helm.sh/blog/migrate-from-helm-v2-to-helm-v3/

Few more resources:

• https://helm.sh/blog/helm-3-preview-pt1/
• https://helm.sh/blog/helm-3-preview-pt2/
• https://helm.sh/blog/helm-3-preview-pt3/
• https://helm.sh/blog/helm-3-preview-pt4/
• https://helm.sh/blog/helm-3-preview-pt5/
• https://helm.sh/blog/helm-3-preview-pt6/
• https://helm.sh/blog/helm-3-preview-pt7/
• https://bridgetkromhout.com/speaking/2019/kubecon-eu/
• https://github.com/jldeen/helm3-demo
• https://v3.helm.sh/docs/faq/
• https://www.youtube.com/watch?v=BZFAMiVm0rM
• https://sweetcode.io/a-first-look-at-the-helm-3-plan/
• https://jdolitsky.github.io/helm2-helm3-migration/files/helm_3_sneak_preview.pdf
• https://bridgetkromhout.com/speaking/2019/ossummit-helm/
• Helm3 and Windows: https://github.com/squillace/helm-multinode

Availability Zones

• https://docs.microsoft.com/en-us/azure/aks/availability-zones

• https://channel9.msdn.com/Shows/Azure-Friday/Preview-of-Availability-Zones-for-your-Kubernetes-cluster-in-Azure

• You can only enable availability zones when the cluster is created

• Availability zone settings can't be updated after the cluster is created. You also can't update an existing, non-availability zone cluster to use availability zones

• You can't disable availability zones for an AKS cluster once it has been created

• The node size (VM SKU) selected must be available across all availability zones

• Clusters with availability zones enabled require use of Azure Standard Load Balancers for distribution across zones

• Volumes that use Azure managed disks are currently not zonal resources. Pods rescheduled in a different zone from their original zone can't reattach their previous disk(s). It's recommended to run stateless workloads that don't require persistent storage that may come across zonal issues.

• Supported regions: https://docs.microsoft.com/en-us/azure/availability-zones/az-overview#services-support-by-region

• Cross AZ traffic is charged if you have very chatty services you may want to allocate them in one Zone

  • Affinity with Pods could be used to achieve this, see example here.

[ACR] Patch base image

• https://docs.microsoft.com/en-us/azure/container-registry/container-registry-tasks-overview#automate-os-and-framework-patching

Add tests

(Alpha) Etcd with Azure CosmosDB

Security advantages:

• https://docs.microsoft.com/en-us/azure/cosmos-db/etcd-api-introduction#security--enterprise-readiness

How to setup it with AKS-Engine (not AKS yet):

• https://docs.microsoft.com/en-us/azure/cosmos-db/bootstrap-kubernetes-cluster

[IaC] Leverage ARM Templates

[ACR] Azure Monitor with ACR

• https://thorsten-hans.com/azure-container-registry-unleashed-integrate-acr-and-azure-monitor
• https://docs.microsoft.com/en-us/azure/container-registry/container-registry-diagnostics-audit-logs

[kured] Use the Helm chart

Use the associated Helm chart here https://github.com/helm/charts/tree/master/stable/kured instead of having my own custom YAML file.

Leverage Azure Security Center

https://azure.microsoft.com/en-us/blog/detecting-threats-targeting-containers-with-azure-security-center/
https://azure.microsoft.com/en-us/blog/protect-linux-containers-running-in-iaas-with-azure-security-center/

Managed Identity

Leveraging Managed Identity with AKS instead of Service Principal is more secure for 2 main reasons:

• There is not anymore the credentials file stored on any AKS nodes (/host/etc/kubernetes/azure.json)
• MI will wrap and managed the credentials (auto-rotate, when expired, etc.) for you

(Preview) Pod Security Policy

Need to be positioned in comparison of OPA+AzurePolicy first, what's the orientation from K8S here?

• https://docs.microsoft.com/en-us/azure/aks/use-pod-security-policies
• Non-root user containers https://kubernetes.io/blog/2018/07/18/11-ways-not-to-get-hacked/#8-run-containers-as-a-non-root-user
• https://medium.com/@mccode/understanding-how-uid-and-gid-work-in-docker-containers-c37a01d01cf
• https://medium.com/faun/kubernetes-pod-security-101-15fe8cda829e
• https://blog.aquasec.com/kubernetes-security-pod-escape-log-mounts
• https://github.com/kevingbb/demo-aks-psp
• http://zigmax.net/aks-pod-security-policy-psp/

Azure Application Gateway Ingress Controller

AGIC is now GA:

• https://docs.microsoft.com/en-us/azure/application-gateway/ingress-controller-overview
• https://github.com/Azure/application-gateway-kubernetes-ingress
• https://myignite.techcommunity.microsoft.com/sessions/82945
• https://jessicadeen.com/how-to-configure-azure-application-gateway-v2-on-an-existing-aks-cluster/

Limitations with my current implementation:

• doesn't support kubenet yet
• leverage AAD Pod Identity which is still in Preview

Private Cluster

• https://docs.microsoft.com/en-us/azure/aks/private-clusters
• https://blog.nillsf.com/index.php/2019/12/18/azure-kubernetes-private-clusters/
• https://medium.com/@denniszielke/fully-private-aks-clusters-without-any-public-ips-finally-7f5688411184

Uptime SLA

• https://techcommunity.microsoft.com/t5/azure-kubernetes-service/aks-introduces-uptime-sla/ba-p/1350832
• https://azure.microsoft.com/en-us/support/legal/sla/kubernetes-service/v1_1/

Windows nodes pool

• https://docs.microsoft.com/en-us/azure/aks/windows-container-cli
• https://docs.microsoft.com/en-us/azure/aks/windows-node-limitations
• Only Azure CNI supported

Service Principal

[ACR] Authentication restriction

https://docs.microsoft.com/en-us/azure/container-registry/container-registry-authentication

[ACR] Content Trust

https://docs.microsoft.com/en-us/azure/container-registry/container-registry-content-trust

Disable K8S Dashboard

Since Azure CLI version 2.0.66, we could disable the K8S dashboard: https://github.com/MicrosoftDocs/azure-docs-cli/blob/master/docs-ref-conceptual/release-notes-azure-cli.md#june-4-2019 which is a good practice on a security perspective.

Use --vnet-subnet-id in az aks create

https://docs.microsoft.com/en-us/azure/aks/configure-kubenet

Document how to leverage `az pipelines create`

https://docs.microsoft.com/en-us/cli/azure/ext/azure-devops/pipelines?view=azure-cli-latest#ext-azure-devops-az-pipelines-create

AAD integration

https://www.nccgroup.trust/us/about-us/newsroom-and-events/blog/2019/august/tools-and-methods-for-auditing-kubernetes-rbac-policies/

https://www.cncf.io/blog/2019/09/16/5-kubernetes-rbac-mistakes-you-must-avoid/

For users:

• https://docs.microsoft.com/en-us/azure/aks/azure-ad-integration-cli
• https://docs.microsoft.com/en-us/azure/aks/azure-ad-rbac
• https://medium.com/faun/playing-with-aks-aad-568bf7bc5ae8
• https://blog.jcorioland.io/archives/2018/11/20/azure-aks-kubernetes-rbac-azure-active-directory-terraform.html
• https://www.danielstechblog.io/azure-kubernetes-service-and-azure-active-directory-integration/

For SPs:

• https://feedback.azure.com/forums/914020-azure-kubernetes-service-aks/suggestions/35146387-support-non-interactive-login-for-aad-integrated-c
• Azure/AKS#600
• https://medium.com/@jakekitchener/kubernetes-serviceaccounts-for-use-in-automated-systems-515297974982
• https://devopscube.com/kubernetes-api-access-service-account/

Add a cleanup script

Azure KeyVault for Azure pipelines

[KeyVault] VNET and Firewall rules integration

https://docs.microsoft.com/en-us/azure/key-vault/key-vault-network-security

Hunt for security weaknesses with kube-hunter

https://github.com/aquasecurity/kube-hunter

[Pipelines] Private Agent in VNET

• https://github.com/microsoft/azure-pipelines-ephemeral-agents
• https://github.com/gambtho/aks-azuredevops-agent
• https://www.henrybeen.nl/fully-isolated-private-agents-for-azure-pipelines-in-azure-container-instances/
• https://github.com/microsoft/azure-pipelines-agent/blob/master/docs/design/byos.md
• https://juliocasal.com/2020/01/14/deploying-azure-pipelines-agents-as-containers-to-kubernetes/

Create a dashboard with Azure Security Center

Setup Azure DevOps pipeline

Add NSGs

• With Bastion/Jumpbox Subnet: https://docs.microsoft.com/en-us/azure/bastion/bastion-nsg
• With ACR's Subnet
  • Can't have NSG with Private Endpoint on ACR's subnet: https://docs.microsoft.com/en-us/azure/private-link/disable-private-endpoint-network-policy

CIS Kubernetes Benchmark with kube-bench

• Waiting for https://github.com/Azure/aks-engine/projects/7
• https://github.com/aquasecurity/kube-bench
• Azure/AKS#367
• https://github.com/sabbour/kube-bench-aks/blob/master/cfg/1.11/config.yaml
• https://blog.aquasec.com/kubernetes-security-cis-benchmarks

kubectl run --rm -i -t kube-bench-node --image=sabbour/kube-bench-aks:latest --restart=Never --overrides="{ \"apiVersion\": \"v1\", \"spec\": { \"hostPID\": true } }" -- node --version 1.11

(Beta) Pod Identity

https://vincentlauzon.com/2019/02/19/azure-ad-pod-identity-in-aks/

https://blog.jcorioland.io/archives/2018/09/05/azure-aks-active-directory-managed-identities.html

(Preview) Use Azure Policy

• https://docs.microsoft.com/en-us/azure/governance/policy/concepts/rego-for-aks
• https://azure.microsoft.com/en-us/updates/azure-policy-for-aks-is-now-in-public-preview/
• https://github.com/Azure/azure-policy/tree/master/built-in-references/KubernetesService
• https://blog.nillsf.com/index.php/2019/11/02/using-azure-policy-to-deny-public-ips-on-specific-vnets
• Release Gate? https://medium.com/microsoftazure/devsecops-policy-as-code-with-azure-pipelines-86cc1e27f03c

Use IP whitelisting for the Kubernetes API

• https://azure.microsoft.com/en-us/updates/authenticated-ip-for-azure-kubernetes-service-aks-is-now-in-preview/
• https://docs.microsoft.com/en-us/azure/aks/api-server-authorized-ip-ranges
• Azure DevOps agent need to be whitelisted too: https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/hosted?view=azure-devops#agent-ip-ranges

Prerequisities:

• Only with Standard Load Balancer
• Implementation with kubenet CNI otherwise with Azure CNI route table will have to be created manually like explained in the doc
• Min Azure CLI version: 2.0.76

Add ssh key for az aks create

Azure Lock

https://docs.microsoft.com/fr-ca/cli/azure/group/lock?view=azure-cli-latest#az-group-lock-create

az group lock create --lock-type CanNotDelete -n CanNotDelete -g $(rg)

Limit Egress Traffic

• https://azure.microsoft.com/en-us/updates/egress-lockdown-in-azure-kubernetes-service-aks-is-now-generally-available/
• https://docs.microsoft.com/en-us/azure/aks/limit-egress-traffic
• https://medium.com/@denniszielke/setting-up-azure-firewall-for-analysing-outgoing-traffic-in-aks-55759d188039

Setup kured for automatic OS patches

https://docs.microsoft.com/en-us/azure/aks/node-updates-kured

kubectl apply -f https://github.com/weaveworks/kured/releases/download/1.2.0/kured-1.2.0-dockerhub.yaml

[IaC] Leverage Terraform

• https://github.com/squasta/AzureKubernetesService-Terraform
• https://www.danielstechblog.io/terraform-deploy-an-azure-kubernetes-service-cluster-with-multiple-node-pools/

Use --service-principal in az aks create

https://docs.microsoft.com/en-us/azure/aks/kubernetes-service-principal

Important remark: every year you have to update your AKS service principal credentials: https://docs.microsoft.com/en-us/azure/aks/update-credentials

[kured] Notifications when OS upgrade process triggered/done

Multiple Node Pools

• https://azure.microsoft.com/en-us/updates/support-for-multiple-node-pools-in-azure-kubernetes-service-is-now-available/
• https://docs.microsoft.com/en-us/azure/aks/operator-best-practices-advanced-scheduler#provide-dedicated-nodes-using-taints-and-tolerations

Learnings and Gotchas:

• First Node Pool at cluster creation is Linux
• We can't delete the first Node Pool
• HTTP application routing add-on can't be used
• Linux node pools the length must be between 1-12 characters, for Windows node pools the length must be between 1-6 characters
• Maximum of eight node pools
• Maximum of 400 nodes across those 8 node pools
• All node pools must reside in the same subnet (Azure/azure-cli#10984)
• Must use virtual machine scale sets for the nodes
• When upgrading a cluster the new nodes will get the taints
• When you scale a node pool in AKS, taints and tolerations do not carry over by design (Azure/AKS#1061)

Taints and tolerations are used to logically isolate resources with a hard cut-off - if the pod doesn't tolerate a node's taint, it isn't scheduled on the node. An alternate approach is to use node selectors. You label nodes, such as to indicate locally attached SSD storage or a large amount of memory, and then define in the pod specification a node selector. Kubernetes then schedules those pods on a matching node. Unlike tolerations, pods without a matching node selector can be scheduled on labeled nodes. This behavior allows unused resources on the nodes to consume, but gives priority to pods that define the matching node selector.

Create a dashboard with Azure Monitor

https://blog.coffeeapplied.com/using-azure-monitor-logs-with-azure-kubernetes-service-aks-c0b0625295d1

[Monitor] Optimize data collection

https://medium.com/microsoftazure/azure-monitor-for-containers-optimizing-data-collection-settings-for-cost-ce6f848aca32