nec-openstack / remora

Remora is command-line tool and library to manage container orchestration engines on OpenStack cloud.

remora

Remora is command-line tool and library to manage kubernetes cluster on any cloud and baremetal.

You can see Quick Start Guide which uses Vagrant to install and run Kubernetes cluster.

Prerequisite

You will need following softwares.

• Python 3.5 or later and pip
• kubectl 1.7.x or later

Supported Linux distribution

• Ubuntu 16.04
• CoreOS

Supported Container Runtime

• Docker 1.12

How to install Kubernetes

This document assumes that you have 4 machines on which Docker 1.12 is already installed. And finally you will build a cluster of the following composition.

• VIP(192.168.1.101): VIP for accessing Kubernetes
• Machine01(192.168.1.111): Kubernetes Master
• Machine02(192.168.1.112): Kubernetes Master
• Machine03(192.168.1.121): Kubernetes Worker
• Machine04(192.168.1.122): Kubernetes Worker

You will be working on a local machine that can ssh into each machines.

1. Clone this repository and write config file

Please clone this repository into your local machine.

$ git clone https://github.com/nec-openstack/remora.git
$ cd remora
$ cat <<-EOF > configs/cluster.yaml
---
# Username of target VMs
# If you use Ubuntu, change below line to like `user: ubuntu`.
user: core

local:
  assets_dir: "~/.kube/assets"

masters: &masters
  - 192.168.1.111
  - 192.168.1.112
workers: &workers
  - 192.168.1.121
  - 192.168.1.122

roledefs:
  bootstrap:
  - 192.168.1.111
  etcd: *masters
  master: *masters
  worker: *workers

etcd:
  hosts:
    "192.168.1.111": cow001.local
    "192.168.1.112": cow002.local

kubernetes:
  # Public kubernetes service address
  public_service_ip: 192.168.1.101
EOF

This config file specify bootstrap host which boot Kubernetes cluster and Kubernetes master/worker addresses. And you can see some example config files in configs directory. configs/openstack.yaml tells you how to setup single master node cluster on the top of OpenStack.

Notes: Remora uses fabric to setup kubernetes cluster. And these config files are used for specifying target cluster. If you put configs/hoge.yaml file under configs directory, Remora will create hoge sub-task to specify hoge cluster.

2. Install dependencies

Remora requires some dependencies, so it's needed to install.

$ pip3 install -r requirements.txt

And also Remora will use kubectl command. So please install kubectl.

3. Generate assets

Following command will create the assets which are needed to build Kubernetes cluster such as TLS certs, systemd unit files, Kubernetes manifests and install scripts.

$ fab cluster render

After this procedure, the assets are generated in local.assets directory. You can modify these assets if you want.

4. Install Kubelet

Kubelet manages all components which are installed by remora. So it's needed to install Kubelet first.

$ fab cluster install.kubelet

5. Install Etcd

Etcd is also essential component for Kubernetes, but installing Etcd is out of scope. So this script only install test grade etcd for testing purpose.

$ fab cluster install.etcd

6. Install Bootstrap Kubernetes

Remora will install Kubernetes cluster using Kubernetes itself, in other words, Remora needs Bootstrap Kubernetes cluster before installing. Following command create Bootstrap Kubernetes cluster.

$ fab cluster install.bootstrap

7. Install Kubernetes

It's the time to install Real Kubernetes cluster. Following command build Kubernetes cluster and cleanup Bootstrap Kubernetes cluster.

$ fab cluster install.kubernetes

Check you cluster

You can access Kubernetes cluster easily. Following command will setup you kubeconfig.

$ fab cluster config

Then you can use kubectl command.

$ kubectl version

That's all.

remora's Issues

RBAC mode isn't supported

api server needs --authorization-mode=RBAC to support RBAC.
flannel plugin also needs cluster role.

Support self-hosted k8s

We should change our architecture to self-hosted style.

See bootkube.

Support PriorityClass

Create 2 priority classes and set PriorityClassName to cluster service.

1. system-cluster-critical
2. system-node-critical

Support MasterConfiguration

Like this

kind: MasterConfiguration
apiVersion: kubeadm.k8s.io/v1alpha1
controllerManagerExtraArgs:
  horizontal-pod-autoscaler-use-rest-clients: "true"
  horizontal-pod-autoscaler-sync-period: "10s"
  node-monitor-grace-period: "10s"
apiServerExtraArgs:
  runtime-config: "api/all=true"
kubernetesVersion: "stable-1.8"

Support TLS bootstrap

cloud provider config should be delivered by configmap

Add document about `port` which is expected to be opened

Related: #14

Check kubectl before running scripts

ubuntu 16.04 doesn't provide docker-engine 1.12 at the default

Now the latest docker-engine is 1.11.2-0~xenial at ubuntu 16.04.
So do we recommend to install docker-engine manually without a package ?

Support multi kubernetes version

To check k8s version on deploying time and set valid configuration.

For example, k8s 1.6 and 1.7 supports different admission controls.
So we should set admission controls correctly for each versions.

defaultv16AdmissionControl = "NamespaceLifecycle,LimitRanger,ServiceAccount,PersistentVolumeLabel,DefaultStorageClass,DefaultTolerationSeconds,ResourceQuota"
defaultv17AdmissionControl = "Initializers,NamespaceLifecycle,LimitRanger,ServiceAccount,PersistentVolumeLabel,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,ResourceQuota"

Failed to run "fab cluster render"

$ fab cluster render
Traceback (most recent call last):
File "/home/ubuntu/.local/lib/python2.7/site-packages/fabric/main.py", line 672, in main
docstring, callables, default = load_fabfile(fabfile)
File "/home/ubuntu/.local/lib/python2.7/site-packages/fabric/main.py", line 173, in load_fabfile
imported = importer(os.path.splitext(fabfile)[0])
File "/home/ubuntu/remora/fabfile.py", line 18, in
from remora.fab import clean # noqa
File "/home/ubuntu/remora/remora/fab/clean/init.py", line 25, in
from remora.fab.clean import kubernetes
File "/home/ubuntu/remora/remora/fab/clean/kubernetes.py", line 20, in
from remora.fab.clean import utils
File "/home/ubuntu/remora/remora/fab/clean/utils.py", line 17, in
from remora.fab import helpers
File "/home/ubuntu/remora/remora/fab/helpers.py", line 140
def run_script(script_name, *options, local_env=[]):
^
SyntaxError: invalid syntax

Add Vagrantfile to test remora easily

Add helm chart to install kubernetes