nickazuredevops / kochhaus-home

Experimental homelab configuration for running things on kubernetes

kochhaus-home - Home Cloud via Flux v2 | GitOps Toolkit

GitOps state for my cluster using flux v2

K3S in a 7-node cluster running Arch Linux.

• Four Lenovo M900 Tinys
• Three VMs on my older homelab servers.

There's a full server list below.

Off-cluster support

• registry - I have a separate host running Kubernetes and an instance of the Harbor container registry, configured as a pull-through cache.
• named - primary home DNS running on a pair of (redundant) Raspberry Pi 3s.
• blocky - lightweight ad-blocking DNS resolver - this has replaced an older Pi-Hole. Thinking of checking out NextDNS.

Cluster components

• Flux 2 - GitOps manager that configures the cluster entirely from this GitHub repository.
• SOPS - Encrypts secrets which is safe to store - even to a public repository.

Networking

• calico - container networking with IPv6 support and policy enforcement.
• cert-manager - Configured to create TLS certs for all ingress services automatically using LetsEncrypt.
• external-dns - monitors service and ingress resources, and automatically generates DNS updates for them. This lets me maintain DNS mappings and LetsEncrypt certificates without a cloudflare account or domain.
• metallb - Kubernetes Load Balancer that runs on Kubernetes.
• nginx ingress - Ingress controller. I used to use Traefik, but it it much more challenging to configure correctly.

Storage

• democratic-csi - creates Persistent Volumes on a ZFS server as separate datasets, and exports them via NFS or iSCSI to the Kubernetes cluter.
• nfs-subdir-external-provisioner - creates Persistent Volumes on a pre-existing NFS mount.
• rook-ceph - on-cluster (hyperconverged) storage - eventually this will all be on SSDs attached to the cluster nodes for low power usage.

infrastructure

• descheduler - analyzes the cluster looking for overloaded or under-utilized nodes, as well as pods violating affinity rules, and evicts them so that they will be rescheduled "correctly".
• kube-fledged - caches critical images locally on each node for reliability during an Internet outage.
• kured - The Kubernetes Reboot Daemon.
• prometheus - metrics, monitoring, and alerting.
• reloader - reloads pods when a configMap and/or Secret changes - something that Flux 2 does not manage itself.
• system-upgrade-controller - Automatically upgrade the K3S kubernetes instance.
• cloudnative-pg - build and manage a postgresql cluster with HA and backups from a custom resource.
• ext-postgres-operator - create databases and users in an existing postgres cluster.
• authentik - integrated authentication and user management.
• volsync - data backup and restore. In a GitOps environment I don't need to backup the Kubernetes resources the way Velero and K10 do, and those tools are hard to manage. VolSync backs up my data.
• And more!

Home Infrastructure

• hajimari: a pretty start page with Kubernetes autodiscovery.
• openweathermap-exporter: a Prometheus exporter for Openweather.

Applications

• outline - full featured documentation platform.
• tautulli - Plex usage monitoring application.
• onedrive - syncs my OneDrive folder from Microsoft, as a local backup.
• syncthing - simple, peer-to-peer file synch app replacing Dropbox or NextCloud.
• actions-runner - Run GitHub Actions at home!
• tekton - simple CI/CD tooling.
• nextcloud - Finally - moving applications from Linode to my homelab.

Yes, this is a lot of infrastructure and heavy lifting - the point is to experiment with Kubernetes and GitOps in a safe space.

I have two longer-term goals:

1. migrate many of the apps that I currently run on Linode to my HomeLab.
2. Build a small Raspberry Pi cluster at home to run a lot of infrastructure, with the intent of being able to run off a small UPS during power outages.

---

Repository structure

The Git repository contains the following directories under cluster and are ordered below by how Flux will apply them.

📁 cluster      # k8s cluster defined as code
├─📁 flux       # flux, gitops operator, loaded before everything
├─📁 crds       # custom resources, loaded before 📁 core and 📁 apps
├─📁 charts     # helm repos, loaded before 📁 core and 📁 apps
├─📁 config     # cluster config, loaded before 📁 core and 📁 apps
├─📁 core       # crucial apps, namespaced dir tree, loaded before 📁 apps
└─📁 apps       # regular apps, namespaced dir tree, loaded last

---

Automation

• Flux 2 - GitOps automation for Kubernetes.
• Rancher System Upgrade Controller to apply updates to k3s.
• Renovate with the help of the k8s-at-home/renovate-helm-releases Github action keeps my application charts and container images up-to-date.
• Github Actions automatically runs renovate.
• Many, many kubernetes operators

---

💻 Nodes

Node	Hostname	CPU	RAM	Storage	Function	Operating System
Lenovo M900q tiny	k3st	4 Intel i5-6500T	8GB	128GB SSD	control-plane	Arch Linux
libvirtd VM	k3sj	4 AMD Ryzen 5 1600T	4GB	128GB HDD	control-plane	Arch Linux
libvirtd VM	k3sm	2 AMD Athlon 3000G	4GB	128GB HDD	control-plane	Arch Linux
Lenovo M910q tiny	k3s0	4 Intel i5-6500T	16GB	512GB NVMe	worker, ceph storage	Arch Linux
libvirtd VM	k3s1	6 AMD Ryzen 5 1600T	16GB	256GB HDD	worker, ceph storage	Arch Linux
Lenovo M900q tiny	k3s2	4 Intel i5-6500T	16GB	512GB SSD	worker, ceph storage	Arch Linux
Lenovo M910q tiny	k3s3	4 Intel i5-6500T	16GB	512GB NVMe	worker, ceph storage	Arch Linux

---

Community

This cluster in inspired by the work of others shared at awesome-home-kubernetes.