fl64 / localk8s

Local k8s testing infra

k8s lab with gitops via ArgoCD

k8s cluster (1 master + X nodes )

Current config:

• k8s ver 1.21.0
• master - 2Gb Ram
• worker - 3Gb Ram
• network: cilium

Info

This lab is designed for local testing different k8s features and deployment with ArgoCD. All you need is: Vargant, VirtualBox and some time to deploy it.

The K8s cluster in its basic configuration consists of two virtual machines: master and node, if you want more nodes you can change default_node_count var, or create a lab with the K8S_NODE_COUNT=X vagrant up command.

Once the cluster is installed and configured, Argo comes in and installs all the applications from the argo/app directory for the current repository.

Requirements:

• virtualbox (tested on 6.1.12)
• vagrant (tested on 2.2.16)
• ansible (tested on 4.2.0)

ansible-galaxy collection install community.general
ansible-galaxy collection install ansible.posix
ansible-galaxy collection install community.kubernetes

Recommendations

• envrc
• k9s

Current setup

• k8s
• ArgoCD + ArgoCD applicationset
• Metallb
• Nginx ingress
• Prometheus
• Grafana
• nfs-client-provisioner (storageClass: nfs-client)
• vactor-agent

How to

# up VMs and run ansible playbook
vagrant up

# run playbooks for specified ansible tags
K8S_TAGS=common vagrant up

# ssh to VMs
vagrant ssh master
vagrant ssh node100
etc...

# how to get to UI with `k port-forward`
## how to get into the Argo UI
k port-forward service/argocd-server -n argocd 8080:80
browse to http://localhost:8080

## how to get into the Prometheus UI
k port-forward service/prometheus-server -n prometheus 9090:80
browse to http://localhost:9090

## how to get into the Prometheus UI
k port-forward service/grafana -n grafana 3000:80
browse to http://localhost:3000

# Another way with ingress and /etc/hosts
## Add local records to /etc/hosts
sudo bash -c "cp /etc/hosts /etc/hosts.backup && export HOSTS_PATCH=\"$(kubectl get svc -n ingress-nginx ingress-nginx-controller -o jsonpath=\"{.status.loadBalancer.ingress[0].ip}\") grafana.k8s.local argo.k8s.local prom.k8s.local\"; grep -qF \"${HOSTS_PATCH}\" -- /etc/hosts || echo \"${HOSTS_PATCH}\" >> /etc/hosts"

## Browse
https://grafana.k8s.local
https://argo.k8s.local
https://prom.k8s.local

## default login and pass everywhere: admin/password

# clean up
vagrant down

Some useful stuff

Bash completions:

cat <<EOF >> ~/.bashrc
source <(kubectl completion bash)
source <(kubeadm completion bash)
alias k=kubectl
complete -F __start_kubectl k
export do="--dry-run=client -oyaml"
EOF

source ~/.bashrc

Vim settings:

cat <<EOF >> ~/.vimrc
set number
set et
set sw=2 ts=2 sts=2
EOF

Tshoot traffic issues with hubble

k port-forward service/hubble-relay -n kube-system 4245:80 &
hubble observe --verdict DROPPED -f

krew plugin manager

• install krew: https://krew.sigs.k8s.io/docs/user-guide/setup/install/
• plugins list: https://krew.sigs.k8s.io/plugins/ plugins

kubectl krew install access-matrix
kubectl krew install view-utilization
kubectl krew install view-webhook
kubectl krew install example

Links:

• cilium: https://docs.cilium.io/en/stable/gettingstarted/k8s-install-helm/#installation-using-helm
• Awesome Kubernetes Resources: https://github.com/tomhuang12/awesome-k8s-resources
• kubernetes-homelab: https://github.com/lisenet/kubernetes-homelab

ToDo:

• move to containerd
• refactor roles
• use helm template instead of helm in ansible
• generate k8s certs with cfssl
• fix cert fqdns and IPs
• multiply masters?