data-8 / kubeadm-bootstrap

Supporting code + documentation for bootstrapping a kubeadm installation on bare-metal-ish machinery

Kubeadm Bootstrapper

This repository contains a bunch of helper scripts to set up Kubernetes clusters using kubeadm. It is meant for use on bare-metal clusters, as well as VMs that are being treated like bare-metal clusters for various reasons.

This is just a wrapper around kubeadm to provide sane defaults.

Pre-requisites

Empty nodes

Kubernetes takes full control of all the nodes it runs on, so do not do this on nodes that are also being used for other things.

Swap turned off

Kubernetes does not support running on Nodes with Swap turned on. Make sure that swap is turned off on the nodes you are planning on using this on.

Operating System

This has been tested on Ubuntu 16.04 only. We would welcome patches to support CentOS / RHEL 7. The Overlay filesystem must be enabled in your kernel - it is by default, so if you didn't fiddle with it you are good!

Networking

All nodes in the cluster must have unrestricted outbound internet access. This is for pulling in Docker images & Debian packages.

At least one node in the cluster must have a public IP if you want to expose network services to the world (via Ingress).

Ideally traffic between the various nodes is unrestricted by any firewall rules. If you need list of specific ports to open, please open an issue and we'll figure it out.

ssh

You must have ssh access to all the nodes. You also need root :)

Setting up a cluster

Setting up a Master Node

1. Install the pre-requisites for starting the master:

   git clone https://github.com/data-8/kubeadm-bootstrap
   cd kubeadm-bootstrap
   sudo ./install-kubeadm.bash

   This installs kubeadm, a supported version of Docker and sets up the appropriate storage driver options for Docker.

2. Setup the master.

   sudo -E ./init-master.bash

   The -E after sudo is important.

   This will take a minute or two, but should set up and install the following:

   a. A Kubernetes Master with all the required components (etcd, apiserver, scheduler and controller-manager)

   b. Flannel with VXLAN backend for the Pod Network

   c. Helm for installing software on to the cluster.

   d. An Nginx ingress that is installed on all nodes - this is used to get network traffic into the cluster. This is installed via Helm.

   e. Credentials to access the Kubernetes cluster in the currently running user's ~/.kube/config directory.

   The master node is also marked as schedulable - this might not be ideal if you are running a large cluster, but is useful otherwise. This also means that if you only wanted a single node Kubernetes cluster, you are already done!

3. Test that everything is up!

   a. Run kubectl get node - you should see one node (your master node) marked as Ready.

   b. Run kubectl --namespace=kube-system get pod. Everything should be in Running state. If it's still Pending, give it a couple minutes. If they are in Error or CrashLoopBackoff state, something is wrong.

   c. Do curl localhost. It should output 404 Not Found. This means network

   traffic into the cluster is working. If your master node also has an external IP that is accessible from the internet, try hitting that too - it should also return the same thing. If not, you might be having firewall issues - check to make sure traffic can reach the master node from outside!

Congratulations, now you have a single node Kubernetes cluster that can also act as a Kubernetes master for other nodes!

Setting up a worker node

1. In your master node, run:

   sudo kubeadm token create --print-join-command
   

   This will print a command that like:

   kubeadm join --token <some-secret> <master-ip>:6443 --discovery-token-ca-cert-hash sha256:<another-secret>
   

   Running this command as sudo creates a token that can be used by another node to join the cluster. This token is valid for 24h by default. Treat it very securely, since leaking it can compromise your cluster.

2. On the worker node you want to join to the cluster, install the pre-requisites:

   git clone https://github.com/data-8/kubeadm-bootstrap
   cd kubeadm-bootstrap
   sudo ./install-kubeadm.bash

   This installs kubeadm, a supported version of docker and sets up the appropriate storage driver options for docker.

3. Copy the kubeadm join command you got as output of step (1) from the master, prefix with sudo and run it. This should take a few minutes.

4. Test that everything is up!

   a. On the master, run kubectl get node. It should list your new node in Ready state.

   b. On the master, run sudo kubectl --namespace=kube-system get pod -o wide. This should show you a kube-proxy, a kube-flannel and kube-controller pod running on your new node in Ready state. If it is in Pending state, give it a few minutes to get to Ready. If it's in Error or CrashLoopBackoff you have a problem.

   c. On the new worker node, do curl localhost. It should output 404 Not Found. This means network traffic into your cluster is working. If this worker node also has a public IP that is accessible from the internet, hit that too - you should get the same output. If not, you might be having firewall issues - check to make sure traffic can reach this worker node from outside!

Congratulations, you have a working multi-node Kubernetes cluster! You can repeat these steps to add as many new nodes as you want :)

Docker Storage Base Directory

By default, Docker puts all of the images and other work files in a directory on the boot volume of the instance, called /var/lib/docker. This is quite convenient for a simple trial system, but this directory can easily fill up causing disastrous results for your cluster.

It is highly recommended that you mount an external volume as /var/lib/docker on each host before running the bootstrap script. Don't forget to configure this mount to be restored upon reboot or else docker will quietly create a new directory and start storing the files on your boot volume as a ticking time bomb.

Next step?

1. If you want to install JupyterHub on this cluster, follow the instructions in the Zero to JupyterHub guide
2. You can look for other software to install from the official kubernetes charts repository.

kubeadm-bootstrap's Issues

get pod not showing nginx-controller

Under "Setting up a Work Node", in step 4b, it says to run:

kubectl --namespace=kube-system get pod -o wide

That is to be run on master, right? On my deployment, when I run on master, I don't see nginx-controller running, like the docs say, although I do get kube-controller-manager running. Is that okay?

$ sudo kubectl --namespace=kube-system get pod -o wide

NAME                                                     READY     STATUS    RESTARTS   AGE       IP              NODE
etcd-js-169-176.jetstream-cloud.org                      1/1       Running   0          1h        172.27.247.13   js-169-176.jetstream-cloud.org
kube-apiserver-js-169-176.jetstream-cloud.org            1/1       Running   0          1h        172.27.247.13   js-169-176.jetstream-cloud.org
kube-controller-manager-js-169-176.jetstream-cloud.org   1/1       Running   0          1h        172.27.247.13   js-169-176.jetstream-cloud.org
kube-dns-6f4fd4bdf-52pp5                                 3/3       Running   0          1h        10.244.0.2      js-169-176.jetstream-cloud.org
kube-flannel-ds-g9bwx                                    1/1       Running   1          1h        172.27.247.13   js-169-176.jetstream-cloud.org
kube-flannel-ds-sjsjv                                    1/1       Running   0          1h        172.27.247.15   js-169-117.jetstream-cloud.org
kube-proxy-gbhbv                                         1/1       Running   0          1h        172.27.247.13   js-169-176.jetstream-cloud.org
kube-proxy-kjkx6                                         1/1       Running   0          1h        172.27.247.15   js-169-117.jetstream-cloud.org
kube-scheduler-js-169-176.jetstream-cloud.org            1/1       Running   0          1h        172.27.247.13   js-169-176.jetstream-cloud.org
tiller-deploy-69cb6984f-zm7x7                            1/1       Running   0          1h        10.244.0.3      js-169-176.jetstream-cloud.org

Need to Update Kubernetes to Remove Vulnerability CVE-2018-1002105

A vulnerability has been identified in Kubernetes which impacts Kubernetes v1.0.x-1.9.x

The patch for this vulnerability is only available in Kubernetes 1.10, 1.11, and 1.12, there are no official patches for 1.9 (which is the version of Kubernetes installed by this script).

This vulnerability is only a risk if you allow remote access to kubectl. As a matter of course we don't allow that in our setups, however other users of this script could expose themselves to this.

Should we update this repo to install the new, patched version of Kubernetes 1.10?

Issues running on Jetstream

Reporting issues as I go installing this on a Jetstream Ubuntu 16.04 image Devel + Docker, version 1.6, small instance:

Master setup

• WARNING: docker version is greater than the most recently validated version. Docker version: 17.06.0-ce. Max validated version: 1.12 I guess I shouldn't worry about this, but wanted to note it here
• kubeadm doesn't start due to /var/lib/kubelet, fixed with kubeadm reset, see
  kubernetes/kubernetes#37063 (comment)
• Error: could not find a ready tiller pod, I guess sleep 1m was not enough, I commented out the first part of the script and reran sudo ./init-master.bash

Small issues, everything looks to be working fine!

Worker setup

No issues on the commands

Should I do curl localhost on master again or on the worker? on the worker gives "Connection refused"

504 Gateway Time-out

On the worker node

https://github.com/data-8/kubeadm-bootstrap/blob/master/README.md#L147

curl localhost

yields

<html>
<head><title>504 Gateway Time-out</title></head>
<body bgcolor="white">
<center><h1>504 Gateway Time-out</h1></center>
<hr><center>nginx/1.13.8</center>
</body>
</html>

for me

I am pretty sure I've followed the instructions quite carefully, including the opening of ports.

init-worker error on Jetstream

The first step went fine:

root@js-168-217:~/kubeadm-bootstrap# ./install-kubeadm.bash

But I got this error on the 2nd step::

root@js-168-217:~/kubeadm-bootstrap# ./init-worker.bash
[preflight] Running pre-flight checks.
        [WARNING FileExisting-crictl]: crictl not found in system path
discovery: Invalid value: "": using token-based discovery without DiscoveryTokenCACertHashes can be unsafe. set --discovery-token-unsafe-skip-ca-verification to continue

etcdserver related error while running init-master.bash

After first running ./init-kubeadm.bash which completes successfully I then run ./init-master.bash which gives this error message:

Tiller (the helm server side component) has been installed into your Kubernetes Cluster.
Happy Helming!
Error: release support failed: etcdserver: request timed out

System is:

# . /etc/os-release ; echo $PRETTY_NAME
Ubuntu 16.04.2 LTS