tinkerbell / k8s-sandbox

The easiest way to setup the Tinkerbell Stack on Kubernetes

k8s-sandbox

This repository is Experimental meaning that it's based on untested ideas or techniques and not yet established or finalized or involves a radically new and innovative style! This means that support is best effort (at best!) and we strongly encourage you to NOT use this in production.

Tinkerbell is made of different components: osie, boots, tink-server, tink-worker and so on. Currently they are under heavy development and we are working around the release process for all the components.

Here is a quick way to get the Tinkerbell stack up and running on Kubernetes.

Currently it supports:

1. Vagrant with libvirt and VirtualBox
2. Terraform on Equinix Metal

Getting Started

Follow documentation Local Setup with Vagrant or Packet Setup with Terraform and replace:

• docker-compose up -d → kubectl apply -f /vagrant/deploy/kubernetes/
• docker-compose ps → kubectl get pods
• docker-compose logs -f tink-server boots nginx → kubectl logs -f -l 'app in (tink-server, boots, nginx)'
• docker exec -i deploy_tink-cli_1 tink ... → kubectl exec -i $(kubectl get pod -l app=tink-cli -o name) -- tink ...

Deploying on a standalone Kubernetes cluster is not yet supported.

Limitations

Tinkerbell is unlikely to run on an existing Kubernetes cluster without additional configurations that require privileged, node access to Kubernetes. Also, multi-node clusters are not supported at the moment.

Docker and Shell Scripts

The installation process is ported from the Sandbox, which uses Docker Compose, and is still heavily dependent on shell scripts running locally and Docker CLI.

Host Path

The NGINX data directory requires to be filled with about 4GB of data (mostly OSIE). It is actually initialized from the setup.sh script before Tinkerbell is installed. The data is installed in a local directory and Kubernetes access it through a hostPath.

Host Network and Service Node Ports

Boots needs to access the same layer 2 network than the worker machine, and Hegel needs to be on the same layer 3 network. It is achieved using hostNetwork. Moreover, these services must run on ports ranging from 67 to 50061, which requires to to setup Kubelet's service-node-port-range accordingly.

k8s-sandbox's Issues

Archive k8s-sandbox repo

With the new Kubernetes backends in each service and the creation of the https://github.com/tinkerbell/charts repo, we should archive this repo.

which version of k8s should be used?

hi guys

i try to deploy tinkerbell in k8s v1.22.3
but it raise :

no matches for kind "MutatingWebhookConfiguration" in version "admissionregistration.k8s.io/v1beta1"

[root@k8s-master kubernetes]# kk api-resources |grep admissionregistration
mutatingwebhookconfigurations admissionregistration.k8s.io/v1 false MutatingWebhookConfiguration
validatingwebhookconfigurations admissionregistration.k8s.io/v1 false ValidatingWebhookConfiguration

what version should i used? thanks .

cainjector pod goes into CrashLoopBackOff status: MutatingWebhookConfiguration

Hi guys:
my kubernetes version is v1.22.2
i pull this repo, and ready to exec setup.sh

kubectl apply -f ./

deployment.apps/boots created
service/boots created
customresourcedefinition.apiextensions.k8s.io/certificaterequests.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/certificates.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/challenges.acme.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/clusterissuers.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/issuers.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/orders.acme.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/certificaterequests.cert-manager.io configured
customresourcedefinition.apiextensions.k8s.io/certificates.cert-manager.io configured
customresourcedefinition.apiextensions.k8s.io/challenges.acme.cert-manager.io configured
customresourcedefinition.apiextensions.k8s.io/clusterissuers.cert-manager.io configured
customresourcedefinition.apiextensions.k8s.io/issuers.cert-manager.io configured
customresourcedefinition.apiextensions.k8s.io/orders.acme.cert-manager.io configured
namespace/cert-manager created
serviceaccount/cert-manager-cainjector created
serviceaccount/cert-manager created
serviceaccount/cert-manager-webhook created
clusterrole.rbac.authorization.k8s.io/cert-manager-cainjector created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-issuers created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-clusterissuers created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-certificates created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-orders created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-challenges created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-ingress-shim created
clusterrole.rbac.authorization.k8s.io/cert-manager-view created
clusterrole.rbac.authorization.k8s.io/cert-manager-edit created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-cainjector created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-issuers created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-clusterissuers created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-certificates created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-orders created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-challenges created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-ingress-shim created
role.rbac.authorization.k8s.io/cert-manager-cainjector:leaderelection created
role.rbac.authorization.k8s.io/cert-manager:leaderelection created
role.rbac.authorization.k8s.io/cert-manager-webhook:dynamic-serving created
rolebinding.rbac.authorization.k8s.io/cert-manager-cainjector:leaderelection created
rolebinding.rbac.authorization.k8s.io/cert-manager:leaderelection created
rolebinding.rbac.authorization.k8s.io/cert-manager-webhook:dynamic-serving created
service/cert-manager created
service/cert-manager-webhook created
deployment.apps/cert-manager-cainjector created
deployment.apps/cert-manager created
deployment.apps/cert-manager-webhook created
mutatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook created
validatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook created
deployment.apps/db created
service/db created
persistentvolumeclaim/postgres-data created
deployment.apps/dhcrelay created
service/dhcrelay created
configmap/boots created
configmap/dhcrelay created
configmap/tink-client created
configmap/tink-init created
secret/db created
secret/packet created
secret/registry created
secret/tink-auth created
deployment.apps/hegel created
service/hegel created
configmap/my-root-ca.crt created
deployment.apps/nginx created
service/nginx created
persistentvolumeclaim/nginx-data created
deployment.apps/registry created
service/registry created
persistentvolumeclaim/registry-data created
deployment.apps/tink-cli created
job.batch/tink-init created
deployment.apps/tink-server created
service/tink-server created
Error from server (InternalError): error when creating "certs.yaml": Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s": dial tcp 10.245.222.235:443: connect: connection refused
Error from server (InternalError): error when creating "certs.yaml": Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s": dial tcp 10.245.222.235:443: connect: connection refused
Error from server (InternalError): error when creating "certs.yaml": Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s": dial tcp 10.245.222.235:443: connect: connection refused
Error from server (InternalError): error when creating "certs.yaml": Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s": dial tcp 10.245.222.235:443: connect: connection refused

kubectl get all -n cert-manager

NAME READY STATUS RESTARTS AGE
pod/cert-manager-5597cff495-lgz6f 1/1 Running 0 2d21h
pod/cert-manager-cainjector-bd5f9c764-lb6lj 0/1 CrashLoopBackOff 8 ( ago) 20m
pod/cert-manager-webhook-c4b5687dc-f25ft 1/1 Running 0 2d21h

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/cert-manager ClusterIP 10.245.90.139 9402/TCP 2d21h
service/cert-manager-webhook ClusterIP 10.245.188.82 443/TCP 2d21h

NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/cert-manager 1/1 1 1 2d21h
deployment.apps/cert-manager-cainjector 0/1 1 0 2d21h
deployment.apps/cert-manager-webhook 1/1 1 1 2d21h

NAME DESIRED CURRENT READY AGE
replicaset.apps/cert-manager-5597cff495 1 1 1 2d21h
replicaset.apps/cert-manager-cainjector-bd5f9c764 1 1 0 2d21h
replicaset.apps/cert-manager-webhook-c4b5687dc 1 1 1 2d21h

kubectl logs -f -n cert-manager cert-manager-cainjector-bd5f9c764-lb6lj

I1115 09:02:34.178462 1 start.go:91] "starting" version="v1.1.0" revision="7fbdd6487646e812fe74c0c05503805b5d9d4751"
I1115 09:02:34.737378 1 leaderelection.go:243] attempting to acquire leader lease kube-system/cert-manager-cainjector-leader-election...
I1115 09:02:49.822512 1 leaderelection.go:253] successfully acquired lease kube-system/cert-manager-cainjector-leader-election
I1115 09:02:49.822750 1 recorder.go:52] cert-manager/controller-runtime/manager/events "msg"="Normal" "message"="cert-manager-cainjector-bd5f9c764-lb6lj_e6cd18ac-3ed4-401b-b841-9ff39bef119f became leader" "object"={"kind":"ConfigMap","namespace":"kube-system","name":"cert-manager-cainjector-leader-election","uid":"80007a3f-b9d4-44a1-8ea3-f387d5e47045","apiVersion":"v1","resourceVersion":"6995575"} "reason"="LeaderElection"
E1115 09:02:50.379565 1 start.go:119] cert-manager/ca-injector "msg"="manager goroutine exited" "error"=null
E1115 09:02:50.379611 1 internal.go:521] cert-manager/controller-runtime/manager "msg"="error received after stop sequence was engaged" "error"="leader election lost"
I1115 09:02:51.525023 1 request.go:645] Throttling request took 1.04315391s, request: GET:https://10.245.0.1:443/apis/storage.k8s.io/v1?timeout=32s
E1115 09:02:52.427055 1 start.go:151] cert-manager/ca-injector "msg"="Error registering certificate based controllers. Retrying after 5 seconds." "error"="no matches for kind "MutatingWebhookConfiguration" in version "admissionregistration.k8s.io/v1beta1""
Error: error registering secret controller: no matches for kind "MutatingWebhookConfiguration" in version "admissionregistration.k8s.io/v1beta1"
Usage:
ca-injector [flags]

Flags:
--add_dir_header If true, adds the file directory to the header of the log messages
--alsologtostderr log to standard error as well as files
-h, --help help for ca-injector
--kubeconfig string Paths to a kubeconfig. Only required if out-of-cluster.
--leader-elect If true, cainjector will perform leader election between instances to ensure no more than one instance of cainjector operates at a time (default true)
--leader-election-namespace string Namespace used to perform leader election (defaults to controller's namespace). Only used if leader election is enabled
--log-flush-frequency duration Maximum number of seconds between log flushes (default 5s)
--log_backtrace_at traceLocation when logging hits line file:N, emit a stack trace (default :0)
--log_dir string If non-empty, write log files in this directory
--log_file string If non-empty, use this log file
--log_file_max_size uint Defines the maximum size a log file can grow to. Unit is megabytes. If the value is 0, the maximum file size is unlimited. (default 1800)
--logtostderr log to standard error instead of files (default true)
--master --kubeconfig (Deprecated: switch to --kubeconfig) The address of the Kubernetes API server. Overrides any value in kubeconfig. Only required if out-of-cluster.
--namespace string If set, this limits the scope of cainjector to a single namespace. If set, cainjector will not update resources with certificates outside of the configured namespace.
--skip_headers If true, avoid header prefixes in the log messages
--skip_log_headers If true, avoid headers when opening log files
--stderrthreshold severity logs at or above this threshold go to stderr (default 2)
-v, --v Level number for the log level verbosity (default 0)
--vmodule moduleSpec comma-separated list of pattern=N settings for file-filtered logging

error registering secret controller: no matches for kind "MutatingWebhookConfiguration" in version "admissionregistration.k8s.io/v1beta1"

[root@k8s-node2 ~]# docker logs -f 20f6831a09c0
I1116 10:29:16.890958 1 start.go:91] "starting" version="v1.1.0" revision="7fbdd6487646e812fe74c0c05503805b5d9d4751"
I1116 10:29:17.346901 1 leaderelection.go:243] attempting to acquire leader lease kube-system/cert-manager-cainjector-leader-election...
I1116 10:29:33.351547 1 leaderelection.go:253] successfully acquired lease kube-system/cert-manager-cainjector-leader-election
I1116 10:29:33.352379 1 recorder.go:52] cert-manager/controller-runtime/manager/events "msg"="Normal" "message"="cert-manager-cainjector-bd5f9c764-qgtvx_8d896baa-ec4d-481a-85d5-466b09e6c66c became leader" "object"={"kind":"ConfigMap","namespace":"kube-system","name":"cert-manager-cainjector-leader-election","uid":"55682406-8a67-4577-ad1c-23c5043c543f","apiVersion":"v1","resourceVersion":"11846"} "reason"="LeaderElection"
E1116 10:29:33.804117 1 start.go:119] cert-manager/ca-injector "msg"="manager goroutine exited" "error"=null
I1116 10:29:34.952203 1 request.go:645] Throttling request took 1.048271738s, request: GET:https://10.96.0.1:443/apis/admissionregistration.k8s.io/v1?timeout=32s
E1116 10:29:35.754094 1 start.go:151] cert-manager/ca-injector "msg"="Error registering certificate based controllers. Retrying after 5 seconds." "error"="no matches for kind "MutatingWebhookConfiguration" in version "admissionregistration.k8s.io/v1beta1""
Error: error registering secret controller: no matches for kind "MutatingWebhookConfiguration" in version "admissionregistration.k8s.io/v1beta1"
Usage:
ca-injector [flags]

Flags:
--add_dir_header If true, adds the file directory to the header of the log messages
--alsologtostderr log to standard error as well as files
-h, --help help for ca-injector
--kubeconfig string Paths to a kubeconfig. Only required if out-of-cluster.
--leader-elect If true, cainjector will perform leader election between instances to ensure no more than one instance of cainjector operates at a time (default true)
--leader-election-namespace string Namespace used to perform leader election (defaults to controller's namespace). Only used if leader election is enabled
--log-flush-frequency duration Maximum number of seconds between log flushes (default 5s)
--log_backtrace_at traceLocation when logging hits line file:N, emit a stack trace (default :0)
--log_dir string If non-empty, write log files in this directory
--log_file string If non-empty, use this log file
--log_file_max_size uint Defines the maximum size a log file can grow to. Unit is megabytes. If the value is 0, the maximum file size is unlimited. (default 1800)
--logtostderr log to standard error instead of files (default true)
--master --kubeconfig (Deprecated: switch to --kubeconfig) The address of the Kubernetes API server. Overrides any value in kubeconfig. Only required if out-of-cluster.
--namespace string If set, this limits the scope of cainjector to a single namespace. If set, cainjector will not update resources with certificates outside of the configured namespace.
--skip_headers If true, avoid header prefixes in the log messages
--skip_log_headers If true, avoid headers when opening log files
--stderrthreshold severity logs at or above this threshold go to stderr (default 2)
-v, --v Level number for the log level verbosity (default 0)
--vmodule moduleSpec comma-separated list of pattern=N settings for file-filtered logging

error registering secret controller: no matches for kind "MutatingWebhookConfiguration" in version "admissionregistration.k8s.io/v1beta1"

any idea? with great appreciate.

dhcrelay drop the dhcp-ack from calixxxx

Hi guys
I am here again

i deploy tinkerbell in k8s, my k8s version is V1.22.3, my cni is Calico
my boots docker and dhcrelay docker is on same host.
when i pxe boot a machine, i can see the dhcp-discover packet arrived into boots and the boots give dhcp-offer

i can tcpdump both the dhcp-discover and dhcp-offer packet in calicxxxx device
but i can only tcpdump the dhcp-discover packet in host's nic, such as ens3

pxe-boot ---> host ens3 ---> dhcrelay ---> calixxxx ----> calixxxx ---> boots

boots ---> calixxxx ----> calixxxx -----> the packet is not arrived ens3

i can see the dhcrelay thread:
dhcrelay -d -id ens3 -iu calife3a10a63a6 boots

kubectl logs -f dhcrelay-6465784854-qv4f9 get the output:
Forwarded BOOTREQUEST for 52:54:00:48:6c:40 to 10.105.102.184
Discarding packet received on calife3a10a63a6 interface that has no IPv4 address assigned.

i think it's the dhcrelay drop the dhcp-ack
but why
and how can i resolve this ?

with great appreciate.