project31 / ansible-kubernetes-openshift-pi3 Goto Github PK

Hello.

I'm in the process to install kubernetes on my raspberry pi from my Linux box. On this box, my user uses sudo, but sudo is not configured to connect without password.

So, ansible's local actions fail:

TASK [kubernetes : Check for an already generated token]

fatal: [192.168.0.20 -> localhost]: FAILED! => {"changed": false, "failed": true, "module_stderr": "sudo: a  password is required\n", "module_stdout": "", "msg": "MODULE FAILURE", "rc": 1}

Enabling sudo without password on the ansible box make it work.

This should be indicated in the readme (or I missed it).

Kubeadm 1.7.1 is broken, when the init would run one'll get an error:

$ kubeadm init --config /etc/kubernetes/kubeadm.yml                                                                                                                 
[kubeadm] WARNING: kubeadm is in beta, please do not use it for production clusters.            
[init] Using Kubernetes version: v1.7.1         
[init] Using Authorization modes: [Node RBAC]   
[preflight] Running pre-flight checks           
can not mix '--config' with other arguments  

The solution is to downgrade to 1.7.0 with the following change in roles/kubernetes/tasks/apt.yml:

- name: Install Packages
  apt:
    name: "{{ item }}"
    force: yes
    state: present
  with_items:
    - kubelet=1.7.0-00
    - kubeadm=1.7.0-00
    - kubectl=1.7.0-00
    - kubernetes-cni

After this it works.

The corresponding kubernetes bug is: kubernetes/kubeadm#345

This affects Kubernetes 1.7.1, should be resolved when that update hits the repos.

This holds true for 1.2.0 and 1.2.3:

root@k8s-master:# file /usr/bin/apt
/usr/bin/apt: ELF 32-bit LSB shared object, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-armhf.so.3, for GNU/Linux 2.6.32, BuildID[sha1]=ae8ca4dda2b5e13978f904378bd43e02759b41b6, stripped
root@k8s-master:# file /usr/bin/kubelet
/usr/bin/kubelet: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux.so.3, for GNU/Linux 2.6.32, BuildID[sha1]=8451bc098c7ecbda570a38b5e9884a54eb00103a, not stripped
root@k8s-master:~# file /usr/bin/kubectl
/usr/bin/kubectl: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), statically linked, not stripped

As you can see, the kubelet is linked to the non-ARM LD version. Not sure why this works for you. I fixed it in my system by creating a hard-link from the ARM version to the searched path, but I guess that's not what you did :)

Hi,

I have installed the kubernetes on 4 Pi's with the playbooks (works perfect !).
Also the management playbook has been installed.
In the kubernetes UI I can see CPU/Memory graphs, so I assume heapster is working.

Nut when I want to use the horizontal pod autoscaler, I get errors the the CPU resources cannot be found
"unable to get metrics for resource cpu: unable to fetch metrics from API: the server could not find the requested resource (get pods.metrics.k8s.io)"

I found some issues reported that the value "--horizontal-pod-autoscaler-use-rest-clients=false", but using that value is also not working.

Do I need to install something extra ?

When running base task it fails at https://github.com/Project31/ansible-kubernetes-openshift-pi3/blob/master/roles/base/tasks/system.yml#L31

The error is: => input file not found at /home/pi/ansible-kubernetes-openshift-pi3/roles/base/templates/hosts or /home/pi/ansible-kubernetes-openshift-pi3/hosts

TASK: [etcd | Download Binaries] **********************************************
fatal: [163.172.142.208 -> 127.0.0.1] => One or more undefined variables: 'dict object' has no attribute 'etcd'

also (before me copying the file to the right position)
[h32@UbuntuMATE ansible-kubernetes-openshift-pi3]$ ansible-playbook -vvvv -i hosts kubernetes.yml
ERROR: file could not read: /data/h32/ansible-kubernetes-openshift-pi3/roles/includes/install_binaries.yml

It seems to me that the state is a little broken for the kubernetes (not openshift) setup...

BTW are you doing tests on Scaleway's ARM servers? I'm currently trying to get your scripts running there...

Hey there,

I've been inspired by your work of using your ansible playbooks to provision a K8S cluster with 4 RPis. I tried to get a cluster up and running as well using your scripts. (with the example config and without wifi)

The problem is that I cannot reach other pods or external servers from within a pod. (wanted to put the gitlab runner on there)
Using nslookup kubernetes.default on hypriot/rpi-alpine:3.6 gives the following:

nslookup: can't resolve '(null)': Name does not resolve

Name:      kubernetes.default
Address 1: 10.96.0.1 kubernetes.default.svc.cluster.local

/etc/resolv.conf looks like this:

nameserver 10.96.0.10
search default.svc.cluster.local svc.cluster.local cluster.local routerbfb8b0.com
options ndots:5

I found out that there's a known issue with alpine up to version 3.3 but I don't use any of these old versions. I tried it with hypriot/rpi-alpine:3.6 and resin/rpi-raspbian:jessie and busybox.
https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/#known-issues

I also used an upgrade weave(2.0.5) but that did not help as well. I couldn't try with flannel since your scripts are not 100% finished there. kube-dns logs does not show any errors.
Do you have any suggestions? I don't know where else to look.

Thank you very much!

EDIT:
I found out that internal names can be resolved. So I assume kube-dns is basically working but I cannot get external names to be resolved.

EDIT 2:
Seems like I cannot access the internet at all with the following images:

• hypriot/rpi-alpine:3.6
• resin/rpi-raspbian:jessie
  busybox seems to only image which works.
  I can work around this "limitation" by specifying hostNetwork: true but this is not something I want to prefer as a solution. I see that the pod is then getting the node ip and is able to go through my router. :/ Also by using that I cannot resolve K8S related services anymore.
  Any ideas how to get around this setting?

Hi,

Thank you for your code. I was able to run the code following the doc until I ran:

ansible-playbook -i hosts kubernetes.yml

I have got the following errors:

TASK [kubernetes : iptables | Add FORWARD ACCEPT for cni0 (in)] ****************************************************************************************************************
fatal: [192.168.7.247]: FAILED! => {"changed": true, "cmd": ["/sbin/iptables", "-A", "FORWARD", "-i", "cni0", "-j", "ACCEPT", "-m", "comment", "--comment", "CNI-Forward-Fix"], "delta": "0:00:00.009288", "end": "2019-02-21 20:58:39.514717", "msg": "non-zero return code", "rc": 1, "start": "2019-02-21 20:58:39.505429", "stderr": "iptables: No chain/target/match by that name.", "stderr_lines": ["iptables: No chain/target/match by that name."], "stdout": "", "stdout_lines": []}
to retry, use: --limit @/Volumes/Fifth/Backup/Softwares/Raspberry Pi/k8s-pi/kubernetes.retry

PLAY RECAP *********************************************************************************************************************************************************************
192.168.7.247 : ok=12 changed=1 unreachable=0 failed=1

Thank you for your help.

To workaround the problem I commented out and run become: yes and become_method: sudo from setup.yml.

After that I ran ansible-playbook -i setup-host setup.yml -sk instead of ansible-playbook -i setup-host setup.yml -k

I am using Hypriot 11.2 image.

The ssh id_rsa.pub file is missing under Pi account.. how can I fix it?

task path: /home/g2david/k8s-pi/roles/base/tasks/user.yml:20
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: If you are using a module and expect the file to exist on the remote, see the remote_src option
fatal: [192.168.1.26]: FAILED! => {"changed": false, "msg": "Could not find or access '~/.ssh/id_rsa.pub' on the Ansible Controller.\nIf you are using a module and expect the file to exist on the remote, see the remote_src option"}
A

Hi,

I'm trying to run this great ansible scripts, but the docker downgrade always fails.

RUNNING HANDLER [kubernetes : restart docker] ***************************************************************************************************************************************
fatal: [192.168.1.200]: FAILED! => {"changed": false, "failed": true, "msg": "Unable to start service docker: Job for docker.service failed. See 'systemctl status docker.service' and 'journalctl -xn' for details.\n"}

Anyone else also facing this issue ?

Is it meant to point to includes/install_binaries.yml?

FYI - i love your repo. It's very clean & it helped proactively fix a lot of problems before they occurred (like the same machine_id on all nodes, etc...)

There are a few tasks that are no longer necessary with HypriotOS 1.5. Should I just list them for you, or would you like me to submit a PR?

Hello,
first of all, thank you for this repository: works great and straightforward.
I just have a question about the default storage driver, why the default is devicemapper? I had some bad experiences with it on other OS and I've seen in your code that we can override it with overlay.
Did you try overlay and you had problems? To override it I just have to put it in the config.yaml, right?

Thank you

Sito

Header of repo reads

Experimental Ansible playbook for setting up Kubernetes on Rasperry Pi 3

which should be

Experimental Ansible playbook for setting up Kubernetes on Raspberry Pi 3

also, would be good to tag the repo with some keywords for ease of location and further retrieval - I had partially lost this one from memory until another issue had been opened to bring it back to recall.

Hi @rhuss - congrats on keeping this repo updated and current! I had helped out a bit about 6 months ago before you had added support with Hypriot 1.5.

Hi,

I downloaded a fresh hypriot image, flashed my rpi 2 SD cards, downloaded the latest version of your ansible files, edited the ansible configs and executed the init machine-id command you mention in your howto. All working fine.

After executing the setup playbook and rebooting the rpis, their docker daemon does not start anymore.

Executing systemctl status docker.service generates the following error

● docker.service - Docker Application Container Engine
   Loaded: loaded (/lib/systemd/system/docker.service; enabled)
   Active: failed (Result: start-limit) since Sun 2017-06-04 10:25:13 CEST; 7min ago
     Docs: https://docs.docker.com
  Process: 969 ExecStart=/usr/bin/dockerd -H fd:// (code=exited, status=1/FAILURE)
 Main PID: 969 (code=exited, status=1/FAILURE)

Jun 04 10:25:11 n0 dockerd[969]: time="2017-06-04T10:25:11.907392501+02:00" level=info msg="libcontainerd: new containerd process, pid: 977"
Jun 04 10:25:12 n0 dockerd[969]: time="2017-06-04T10:25:12.983599830+02:00" level=warning msg="devmapper: Usage of loopback devices is strongly discouraged for production use. Please use `--storage-opt dm.thinpooldev` or use `man docker` to refer to dm.thinpooldev section."
Jun 04 10:25:13 n0 dockerd[969]: time="2017-06-04T10:25:13.106356942+02:00" level=warning msg="devmapper: Base device already exists and has filesystem ext4 on it. User specified filesystem  will be ignored."
Jun 04 10:25:13 n0 dockerd[969]: time="2017-06-04T10:25:13.221333633+02:00" level=fatal msg="Error starting daemon: error initializing graphdriver: \"/var/lib/docker\" contains several valid graphdrivers: devicemapper, overlay2; Please cleanup or explicitly choose storage driver (-s <DRIVER>)"
Jun 04 10:25:13 n0 systemd[1]: docker.service: main process exited, code=exited, status=1/FAILURE
Jun 04 10:25:13 n0 systemd[1]: Failed to start Docker Application Container Engine.
Jun 04 10:25:13 n0 systemd[1]: Unit docker.service entered failed state.
Jun 04 10:25:13 n0 systemd[1]: Starting Docker Application Container Engine...
Jun 04 10:25:13 n0 systemd[1]: docker.service start request repeated too quickly, refusing to start.
Jun 04 10:25:13 n0 systemd[1]: Failed to start Docker Application Container Engine.

Do you have any advise?

Thanks!

There's an extraneous '\n' in the system.yml file.

During the running of the playbook I get two simple errors:

TASK [etcd : Install etcd] *****************************************************
included: /home/harry/k8s-pi/includes/install_binaries.yml for 192.168.23.200

TASK [etcd : Download Binaries] ************************************************
fatal: [192.168.23.200 -> 127.0.0.1]: FAILED! => {"changed": false, "failed": true, "msg": "Destination /home/harry/k8s-pi/roles/etcd/files not writable"}

.... and .....

TASK [flannel : Install flanneld] **********************************************
included: /home/harry/k8s-pi/includes/install_binaries.yml for 192.168.23.200

TASK [flannel : Download Binaries] *********************************************
fatal: [192.168.23.200 -> 127.0.0.1]: FAILED! => {"changed": false, "failed": true, "msg": "Destination /home/harry/k8s-pi/roles/flannel/files not writable"}

I worked around this with:
mkdir /home/harry/k8s-pi/roles/etcd/files
mkdir /home/harry/k8s-pi/roles/flannel/files

Unbelievable, I just made a working Kubernetes cluster in two hours, including unpacking the hardware :-)

Iḿ going to have so much fun with this :-)

BTW, I just used the wireless router as a NAT gateway, because my W10 laptop cannot do nat :-)
And I used a Virtualbox VM with an Ubuntu VM to run Ansible.
And it worked straight away.

I trying build the cluster but I have the follow error when I tried launch "ansible-playbook -i hosts kubernetes.yml"

"item": [
"kubelet=1.8.1*",
"kubeadm=1.8.1*",
"kubectl=1.8.1*",
"kubernetes-cni"
],
"msg": [
"'/usr/bin/apt-get -y -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold" --force-yes install 'kubelet=1.8.1*' 'kubeadm=1.8.1*' 'kubectl=1.8.1*' 'kubernetes-cni'' failed: E: Unable to correct problems, you have held broken packages.",
""
],
"rc": "100",
"stderr": [
"E: Unable to correct problems, you have held broken packages.",
""
],
"stderr_lines": [
"E: Unable to correct problems, you have held broken packages."
],
"stdout": [
"Reading package lists...",
"Building dependency tree...",
"Reading state information...",
"Some packages could not be installed. This may mean that you have",
"requested an impossible situation or if you are using the unstable",
"distribution that some required packages have not yet been created",
"or been moved out of Incoming.",
"The following information may help to resolve the situation:",
"",
"The following packages have unmet dependencies:",
" kubelet : Depends: kubernetes-cni (= 0.5.1) but 0.6.0-00 is to be installed",
""
],

Could you kindly help me please?

Hello.

All the tasks succeed, but in dmesg, as seen in hypriot/image-builder-rpi#166 .

I know it's not specific to ansible scripts however. Is it worth trying with a 3.16 kernel?

[ 1334.159754] ------------[ cut here ]------------
[ 1334.164707] WARNING: CPU: 1 PID: 3942 at kernel/sched/core.c:2966 preempt_count_add+0xfc/0x118()
[ 1334.174007] DEBUG_LOCKS_WARN_ON((preempt_count() & PREEMPT_MASK) >= PREEMPT_MASK - 10)
[ 1334.182183] Modules linked in:
[ 1334.185466]  xt_comment xt_mark ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter ip_tables xt_conntrack x_tables nf_nat nf_conntrack br_netfilter bridge stp llc bnep hci_uart btbcm bluetooth dm_thin_pool dm_bio_prison dm_persistent_data dm_bufio dm_mod brcmfmac brcmutil cfg80211 rfkill snd_bcm2835 snd_pcm snd_timer snd bcm2835_gpiomem bcm2835_wdt uio_pdrv_genirq uio overlay ipv6
[ 1334.226879] CPU: 1 PID: 3942 Comm: kube-proxy Not tainted 4.4.50-hypriotos-v7+ #1
[ 1334.234710] Hardware name: BCM2709
[ 1334.238323] [<80019468>] (unwind_backtrace) from [<80014a14>] (show_stack+0x20/0x24)
[ 1334.246453] [<80014a14>] (show_stack) from [<803362dc>] (dump_stack+0xbc/0x108)
[ 1334.254147] [<803362dc>] (dump_stack) from [<8002672c>] (warn_slowpath_common+0x8c/0xc8)
[ 1334.262638] [<8002672c>] (warn_slowpath_common) from [<800267a8>] (warn_slowpath_fmt+0x40/0x48)
[ 1334.271759] [<800267a8>] (warn_slowpath_fmt) from [<8005005c>] (preempt_count_add+0xfc/0x118)
[ 1334.280698] [<8005005c>] (preempt_count_add) from [<805bf608>] (_raw_spin_lock+0x20/0x60)
[ 1334.289314] [<805bf608>] (_raw_spin_lock) from [<7f3cd45c>] (nf_conntrack_set_hashsize+0xa4/0x200 [nf_conntrack])
[ 1334.391752] [<7f3cd45c>] (nf_conntrack_set_hashsize [nf_conntrack]) from [<80043530>] (param_attr_store+0x6c/0xc4)
[ 1334.491015] [<80043530>] (param_attr_store) from [<80042864>] (module_attr_store+0x30/0x3c)
[ 1334.588651] [<80042864>] (module_attr_store) from [<801dfd1c>] (sysfs_kf_write+0x54/0x58)
[ 1334.685822] [<801dfd1c>] (sysfs_kf_write) from [<801df4f4>] (kernfs_fop_write+0xc8/0x1c8)
[ 1334.783030] [<801df4f4>] (kernfs_fop_write) from [<8016ad94>] (__vfs_write+0x34/0xe8)
[ 1334.879753] [<8016ad94>] (__vfs_write) from [<8016b658>] (vfs_write+0xa0/0x1a8)
[ 1334.976101] [<8016b658>] (vfs_write) from [<8016bf78>] (SyS_write+0x4c/0xa0)
[ 1335.027879] [<8016bf78>] (SyS_write) from [<8000fc40>] (ret_fast_syscall+0x0/0x1c)
[ 1335.124128] ---[ end trace 41b89ceb5c5e0202 ]---
[ 1335.176234] BUG: scheduling while atomic: kube-proxy/3942/0x00000401
[ 1335.226418] Modules linked in: xt_nat xt_recent ipt_REJECT nf_reject_ipv4 xt_tcpudp xt_comment xt_mark ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter ip_tables xt_conntrack x_tables nf_nat nf_conntrack br_netfilter bridge stp llc bnep hci_uart btbcm bluetooth dm_thin_pool dm_bio_prison dm_persistent_data dm_bufio dm_mod brcmfmac brcmutil cfg80211 rfkill snd_bcm2835 snd_pcm snd_timer snd bcm2835_gpiomem bcm2835_wdt uio_pdrv_genirq uio overlay ipv6
[ 1335.587102] Preemption disabled at:[<  (null)>]   (null)

[ 1335.679506] CPU: 1 PID: 3942 Comm: kube-proxy Tainted: G        W       4.4.50-hypriotos-v7+ #1
[ 1335.772295] Hardware name: BCM2709
[ 1335.816601] [<80019468>] (unwind_backtrace) from [<80014a14>] (show_stack+0x20/0x24)
[ 1335.906485] [<80014a14>] (show_stack) from [<803362dc>] (dump_stack+0xbc/0x108)
[ 1335.995536] [<803362dc>] (dump_stack) from [<8010c4d4>] (__schedule_bug+0xac/0xd0)
[ 1336.085542] [<8010c4d4>] (__schedule_bug) from [<805bbc60>] (__schedule+0x6a0/0x750)
[ 1336.176499] [<805bbc60>] (__schedule) from [<805bbf1c>] (schedule+0x58/0xb8)
[ 1336.225211] [<805bbf1c>] (schedule) from [<80014210>] (do_work_pending+0x3c/0xd4)
[ 1336.315047] [<80014210>] (do_work_pending) from [<8000fc68>] (slow_work_pending+0xc/0x20)
HypriotOS/armv7: root@master01 in ~

Hi,
is there a reason why the persistentvolume-binder is disabled by default?

roles/kubernetes/templates/kubeadm.yml:

controllers: "*,-persistentvolume-binder,bootstrapsigner,tokencleaner"

In my fork I'm using GlusterFS and I had to enable the persistentvolume-binder in order to bind PersistentVolumeClaims to PersistentVolumes. So I far I didn't have issues and my pod with MySQL is bound correctly.

Thank you

Sito

I get a fair way through the playbook when this error occurs:

TASK [kubernetes : Install kubelet service definition] *************************
fatal: [192.168.1.59]: FAILED! => {"changed": false, "failed": true, "msg": "AnsibleUndefinedVariable: 'dns' is undefined"}

I see dns.service_ip in the kubelet.service, but don't see where that should already have been defined.
Any assistance would be appreciated.

During ansible-playbook -i hosts kubernetes.yml:

ASK [kubernetes : Run kubeadm init on master] ************************************************************************************************************************************
fatal: [192.168.0.230]: FAILED! => {"changed": true, "cmd": ["kubeadm", "init", "--config", "/etc/kubernetes/kubeadm.yml"], "delta": "0:00:06.811351", "end": "2017-10-22 15:50:01.583502", "failed": true, "rc": 2, "start": "2017-10-22 15:49:54.772151", "stderr": "[preflight] Some fatal errors occurred:\n\tPort 10250 is in use\n\tPort 10251 is in use\n\tPort 10252 is in use\n\t/etc/kubernetes/manifests is not empty\n\tPort 2379 is in use\n\t/var/lib/etcd is not empty\n[preflight] If you know what you are doing, you can skip pre-flight checks with --skip-preflight-checks", "stdout": "[kubeadm] WARNING: kubeadm is in beta, please do not use it for production clusters.\n[init] Using Kubernetes version: v1.8.2-beta.0\n[init] Using Authorization modes: [Node RBAC]\n[preflight] Running pre-flight checks", "stdout_lines": ["[kubeadm] WARNING: kubeadm is in beta, please do not use it for production clusters.", "[init] Using Kubernetes version: v1.8.2-beta.0", "[init] Using Authorization modes: [Node RBAC]", "[preflight] Running pre-flight checks"], "warnings": []}
to retry, use: --limit @/root/k8s-pi/kubernetes.retry

I followed the instructions, and I've definitely got docker up and running on the nodes. However, the Kubernetes install seems incomplete or something (no kubernetes master binaries?)

When I run kubectl cluster-info I get this:

The connection to the server master:8080 was refused - did you specify the right host or port?

However I can ping master just fine.

When I run:

netstat -a | grep 8080 on the master node, I get nothing back, so it seems that something was supposed to be installed, but it didn't get installed. When I look at the ansible scripts however I don't see where anything like the API listener was included.

What am I missing here?