smartxworks / knest

Kubernetes-in-Kubernetes Made Simple

knest: Kubernetes-in-Kubernetes Made Simple

Installation

Prerequisites

• Your host Kubernetes cluster should meet Virtink's requirements
• Your local environment should have kubectl and clusterctl installed

Install knest

Binaries for Linux, Windows and Mac are available in the release page.

Getting Started

Create a Nested Kubernetes Cluster

You can create a nested Kubernetes cluster simply by typing:

knest create quickstart

knest would automatically install any missing components (Cluster API providers and Virtink) on the host cluster, create certain number of Virtink VMs, and form them into a new Kubernetes cluster. When the control plane of the new cluster is initialized, a corresponding kubeconfig file would be saved in the canonical kubeconfig directory ($HOME/.kube/) for you to further access and control the created cluster.

⚠️ Please be awared that the pod subnet and the service subnet of your nested cluster should not overlap with host cluster's pod subnet, service subnet or physical subnet. Use --pod-network-cidr and --service-cidr flags to configure nested cluster's pod subnet and service subnet respectively when necessary.

Create a Persistent Nested Kubernetes Cluster

A persistent nested Kubernetes cluster is a nested cluster that each of its nodes will have a persistent rootfs and a static IP address. To create a persistent nested Kubernetes cluster, the following prerequisites should be met:

• Your host cluster should have defined a default StorageClass, knest will use it to create PVC for each nested cluster node.
• Your host cluster's CNI plugin should support static IP assignment feature for pod, knest will use it to assign static IP for each nested cluster node. Currently knest has built-in static IP assignment support for Calico and Kube-OVN.

Below is an example to create a persistent nested cluster:

knest create quickstart-persistent --persistent --machine-addresses=172.22.127.100-172.22.127.110 --host-cluster-cni=calico

For other CNI plugins, you can download the default cluster template, modify it accordingly, and specify it using the --cluster-template flag.

Scale the Nested Kubernetes Cluster

You can scale your nested cluster easily as follows:

knest scale quickstart --control-plane-machine-count=3 --worker-machine-count=2

Delete the Nested Kubernetes Cluster

You can delete your nested cluster as follows:

knest delete quickstart

Please be noted that this operation would delete all VMs and data of the nested cluster.

Demo Recording

Known Issues

• Sometimes you may encounter an error with a message like ... rate limit for github api has been reached. Please wait one hour or get a personal API token and assign it to the GITHUB_TOKEN environment variable, this is a known issue with clusterctl. To work around this, create a personal access token on your GitHub settings page and assign it to the GITHUB_TOKEN environment variable.

• If no CNI plugin is installed in the nested cluster, worker nodes would get re-created about every 5 minutes. This is currently an expected behaviour due to our MachineHealthCheck settings. Once a valid CNI plugin is installed and running, this problem would disappear.

• Currently Calico and Cilium are the only two recommended CNI plugins for nested clusters, due to limited kernel modules was included in the image. Support for more CNI plugins is on the way. And overlay network is required for nested cluster CNI, the supports for CNI and encapsulation mode are as follows:

  CNI	Encapsulation Mode	Encryption
  Calico	IPIP	false
  Calico	VXLAN(4789/UDP)	false
  Cilium	VXLAN(8472/UDP)	false
  Cilium	Geneve(6081/UDP)	false

• The underlying network and firewalls must allow encapsulated packets. For example, if host cluster uses Calico as CNI plugin, the IPIP and VXLAN(4789/UDP) encapsulated packets from nested cluster will be DROP by default. See Configuring Felix to allow IPIP and VXLAN(4789/UDP) packets from workloads.

License

This project is distributed under the Apache License, Version 2.0.

knest's Issues

What is the username and password of the virtual machine quickstart-cp-gtcgg ?

I try to log in to the virtual machine through the console to troubleshoot some problems, but I don't know the username and password.

Or when I use "knest create quickstart", how to specify the username and password of the virtual machine?

quickstart-cp-gtcgg login: / #
/ #
/ #
/ # microcom -s 115200 /dev/pts/0

Failed to start Apply the …ngs specified in cloud-config.

Getting some errors during the "Waiting for control plane to be initialized..." phase:

kubectl logs vm-quickstart-cp-dc74x-qrvvf

[ 12.822875] cloud-init[1039]: Cloud-init v. 22.3.4-0ubuntu1~22.04.1 running 'modules:config' at Thu, 01 Jun 2023 02:40:30 +0000. Up 12.67 seconds.
[ 12.900775] cloud-init[1039]: 2023-06-01 02:40:30,914 - util.py[WARNING]: Running module locale (<module 'cloudinit.config.cc_locale' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_locale.py'>) failed
[ 12.939189] cloud-init[1039]: 2023-06-01 02:40:30,952 - cc_set_passwords.py[WARNING]: Ignoring config 'ssh_pwauth: None'. SSH service 'ssh' is not installed.
[FAILED] Failed to start Apply the …ngs specified in cloud-config.
See 'systemctl status cloud-config.service' for details.
Starting Execute cloud user/final scripts...

[ 57.034344] cloud-init[1055]: CGROUPS_BLKIO: missing
[ 57.035390] cloud-init[1055]: [WARNING SystemVerification]: missing optional cgroups: blkio
[ 57.037697] cloud-init[1055]: [WARNING SystemVerification]: failed to parse kernel config: unable to load kernel module: "configs", output: "", err: exec: "modprobe": executable file not found in $PATH

[ 101.474076] cloud-init[1055]: [kubelet-check] Initial timeout of 40s passed.
[ 322.483336] cloud-init[1055]: Unfortunately, an error has occurred:
[ 322.497123] cloud-init[1055]: timed out waiting for the condition
[ 322.501165] cloud-init[1055]: This error is likely caused by:
[ 322.501887] cloud-init[1055]: - The kubelet is not running
[ 322.509514] cloud-init[1055]: - The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)

Please give me some advice how to solve this problem, TKS