k8s.1e100's Introduction

Deploy a K8S cluster on Google Kubernetes Engine

Uses Google Cloud Cloud Foundation Fabric

Config

Copy cluster.example.yaml to cluster.yaml, disable features by setting to empty ({} or []) or false.

Prepare

Clone Clound Foundation Fabric Terraform modules:

git clone --depth 1 --branch v21.0.0 https://github.com/GoogleCloudPlatform/cloud-foundation-fabric.git tf/fabric

Update modules:

git fetch --tags
git checkout tags/v21.0.0

Deployment

cd tf/
terraform apply

First time only, migrate Terraform state to a remote bucket.

cat <<EOF > backend.tf
terraform {
  backend "gcs" {
    bucket = "$( terraform output -json | jq -r '."state_bucket".value' )"
    prefix = "terraform/state/bootstrap"
  }
}
EOF

tf init

Export helper variables locally:

REPO="$( git rev-parse --show-toplevel )"
TF_SUFFIX="$( cd ${REPO}/tf; terraform output -json | jq -r '.suffix.value' )"
CLUSTER_NAME="$( yq '.name' ${REPO}/cluster.yaml )-${TF_SUFFIX}"
PROJECT_ID="$( yq '.project' ${REPO}/cluster.yaml )"
REGION="$( yq '.region' ${REPO}/cluster.yaml )"
ZONE="$( yq '.zone' ${REPO}/cluster.yaml )"
BASTION="$( cd ${REPO}/tf; terraform output -json | jq -r '.iap_bastion_hostname.value' )"

Bastion

Bastion nodes are required for kubectl to access a private cluster from outside the VPC network.

For example, if you use kubectl on your desktop, you will need to enable the bastion in the configuration and run the following commands on your local machine:

https://cloud.google.com/kubernetes-engine/docs/tutorials/private-cluster-bastion

gcloud container clusters get-credentials $CLUSTER_NAME --region=$REGION --project=$PROJECT_ID
gcloud compute ssh $BASTION --tunnel-through-iap --project=$PROJECT_ID --zone=$ZONE -- -4 -L8888:localhost:8888 -N -q -f
kubectl config set-cluster $( kubectl config current-context ) --proxy-url http://localhost:8888

To start a preempted bastion node run: gcloud compute instances start bastion-vm --project $PROJECT_ID

BASTION is output by Terraform under "iap_bastion_hostname".

Interacting with Kubernetes

Google Cloud - Install kubectl and configure cluster access

gcloud container clusters get-credentials $CLUSTER_NAME --region=$REGION --project=$PROJECT_ID
kubectl get namespaces

Versions

Component / Tool	Version / Tag
Terraform	>= 1.3
Fabric modules	v21.0.0

k8s.1e100's People

Watchers

k8s.1e100's Issues

Config Connector

Config Connector enables us to manage Google Cloud resources via Kubernetes, awesome

Limitations:

GKE Addon does not support Autopilot clusters

Considerations:

Managed service Config Controller part of Anthos solution and comes with flat Anthos fee ($0.10 per hour)
Manually installing Operator apparently updates more frequently than Addon

Docs:

https://cloud.google.com/config-connector/docs/concepts/installation-types

Use remote modules instead of local fabric?

Does this simplify deployment?
Could use variable to set git tag?

Handle secrets with Mozilla SOPS

https://github.com/mozilla/sops#encrypting-using-gcp-kms

Bump FluxCD API to v1

https://fluxcd.io/blog/2023/05/april-2023-update/#flux-v200-release-candidate

FluxCD v2.0.0-rc.1 brings new API versions:

GitRepository v1
Kustomization v1
Receiver v1

Medium: Pod container is allowed to run as root

"A container has been explicitly allowed to run as the root user if the runAsUser or the USER directive in the image specifies the root user. The lack of preventive security controls when running as the root user increases the risk of container escape."

Restricted fields