v1k0d3n / gantry

A container that bootstraps Kubernetes clusters using Kubeadm.

Gantry: a containerized kubeadm project

A container that bootstraps Kubernetes using Kubeadm (containerized).

WARNING: THIS REPO IS A WIP
This is just a working start, but not how the project will be used as an end state. The plan is to put all logic in the gantry initially, to determine distro (for required mounts and placement), state (bootstrap, clean, etc), and potentially considerations for some common plugins or options (Helm, IPVS, etc).

Basic Usage:

If you want to try this as its in early stages, you can use the container like this:

1. Builds the container like below. You can leverage --build-args to customize your image.

git clone https://github.com/v1k0d3n/gantry.git
cd gantry 
export KUBE_VERSION=v1.9.6
sudo docker build --build-arg VERSION_KUBEADM=${KUBE_VERSION} --build-arg VERSION_KUBECTL=${KUBE_VERSION} --build-arg VERSION_KUBELET=${KUBE_VERSION} -t gantry:${KUBE_VERSION} .

2. Then start the container with the following parameters (this is likely to change as the project is being tested):
   NOTE: for $(pwd) in the line -v $(pwd)/etc/kubeadm:/kubeadm/etc/kubeadm, this should be the location of your kubeadm MasterConfiguration yaml manifest. The docker run command in the example below assumes that you are in the gantry project folder.

sudo rm -rf /opt/kubeadm
sudo docker run -it \
   --privileged \
   --net=host \
   -v /etc/cni:/etc/cni \
   -v /var/lib/etcd:/var/lib/etcd \
   -v /etc/kubernetes:/etc/kubernetes \
   -v /usr/libexec/kubernetes:/usr/libexec/kubernetes \
   -v /var/lib/kubelet:/var/lib/kubelet \
   -v /usr/bin/systemctl:/usr/bin/systemctl \
   -v /etc/systemd/system:/etc/systemd/system \
   -v /var/run/docker.sock:/var/run/docker.sock \
   -v /lib/modules:/lib/modules \
   -v /var/run:/var/run \
   -v /usr/bin:/usr/bin \
   -v /boot:/boot \
   -v /opt:/opt \
   -v $(pwd)/etc/kubeadm:/kubeadm/etc/kubeadm \
   gantry:${KUBE_VERSION} gantry -d ubuntu -i --config /kubeadm/etc/kubeadm/config.yaml

Container images of Gantry are available on both DockerHub and Quay.

NOTE: The intention of Gantry is to declaratively bootstrap a Kubernetes cluster using a custom Kubeadm MasterConfiguration file. The Gantry image includes a sample config, but we recommend reading the documentation for bootstrapping kubeadm with configuration file.

3. You can destroy a previously bootstrapped cluster by using gantry -r. Please refer to the --help menu for any questions on how to use the Gantry image.

4. After bootstrapping a cluster with Gantry/Kubeadm, you will still need to configure kubectl and apply an SDN manifest:

# Configure kubectl:
mkdir -p $HOME/.kube
yes | sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

# If testing:
kubectl taint nodes --all node-role.kubernetes.io/master-

# Apply SDN (Calico example):
kubectl apply -f https://gist.githubusercontent.com/v1k0d3n/aa318f52399f5ebdd6043dd615ae07b4/raw/ed583598170d67bc8c6c91dc523ce100482958eb/networking-calico.yaml

Preparation:

Docker should be the only requirement to run the gantry image, since the intention is to also run on immutable distros such as Atomic and Container Linux. If you have a new or default installation (currently Ubuntu Xenial: 16.04), you can use the preparation script to install Docker.

Ubuntu: For Ubuntu systems (where this will be deployed), run the following from the main gantry directory to prepare the host (install Docker).

./bin/distro/ubuntu/start

Red Hat: Install git first, sudo yum install -y git, and then you can use the following (similar to Ubuntu).

./bin/distro/centos/start

Alternative Methods:

If you don't want to use Gantry to bootstrap your cluster, you can still use the Gantry image to distribute Kubernetes binaries (kubeadm, kubectl, and kubelet). All of the binaries are being downloaded directly from Kubernetes releases and they are located in /kubeadm/bin/. A Gantry image will be created for each Kubernetes release. Simply copy them directly to your host, and use them for your specific setup.

ubuntu@gantry-test:~$ sudo docker ps -a
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS                      PORTS               NAMES
97a68d1dd91b        gantry:v1.10.0      "gantry -h"         24 minutes ago      Exited (0) 2  minutes ago                       reverent_mccarthy
ubuntu@gantry-test:~$ docker cp 97a68d1dd91b:/kubeadm/bin/kubeadm ~
ubuntu@gantry-test:~$ ls -asl ~/kubeadm
152804 -rwxr-xr-x 1 ubuntu ubuntu 156467952 Mar 31 04:28 /home/ubuntu/kubeadm
ubuntu@gantry-test:~$

Future State:

I would really like to get to a future-state that Jessie Frazelle is promoting on her blog which builds images securely. We can try to improve the need to run full --privileged flags in the meantime. This isn't desired, but is easiest for now.

Contributing, Comments, Questions

Comments, suggestions and PR's are welcome!

gantry's Issues

need to really, really clean this repo up

Some of the code was handme-down and rushed to a basic working state. General cleanup is required before it can even be considered for anything beyond exploring. But I think it could be useful to others, so I figured I would take on the task of maintaining this effort.

add additional mutable distro: atomic/container linux

i want to start the work for adding additional mutable distros, beyond just ubuntu which is available today:

• atomic
• container linux (coreos)

This means that the following files need to be modified/added:

• gantry command/script needs modification.
• a new prep script needs to be created in the distro directory.
• potentially separate scripts for mutable/immutable distros in bin directory that gantry calls.

add ci and gating to the project

This work has already begun, and will most likely be completed this weekend. If I'm going to share out the project, I want people to be confident in using it. More to come soon.

v1.10.0: kubelet issues

There seems to be some kubelet issue with v1.10.0:

[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz' failed with error: Get http://localhost:10255/healthz: dial tcp 127.0.0.1:10255: getsockopt: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz' failed with error: Get http://localhost:10255/healthz: dial tcp 127.0.0.1:10255: getsockopt: connection refused.

So this will have to be explored before issuing a v1.10.0 branch. The other branches will be issued this evening (v1.9.3-current).

join needs to be explored

in beginning tests, the join functionality wasn't quite working. this needs to get resolved.

add additional mutable distro: fedora/centos

i want to start the work for adding additional mutable distros, beyond just ubuntu which is available today:

• centos
• fedora

This means that the following files need to be modified/added:

• gantry command/script needs modification.
• a new prep script needs to be created in the distro directory.
• potentially separate scripts for mutable/immutable distros in bin directory that gantry calls.

start working on clean argument

the clean argument is intended to completely wipe any traces of kubernets/kubeadm from the system, and bring it back to a state prior to launching the gantry container. this goes beyond kubeadm reset and is distro specific. this issue is intended to track this work.