corneliusweig / kubernetes-lxd Goto Github PK

I was heading towards doing something like this when I stumbled upon yours. Sterling effort. Well done for this.

I have not read and understood enough yet, but I have seen indications that the use case is a single lxc container with a single-node k8s running within it. Is this assumption correct?

If it is, how to adapt it so we can have multiple lxc containers acting as nodes for a cluster? ... just generalised overview, no need to delve into detailed implementation aspects.

Thanks in advance.

https://github.com/corneliusweig/kubernetes-lxd/blame/413e6bfbb3f5dd07722a12902e42f691e15b50f3/README.md#L35
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 23.04
Release: 23.04
Codename: lunar

$ sudo systemctl restart lxd
Failed to restart lxd.service: Unit lxd.service not found.

$ sudo systemctl restart snap.lxd.daemon

$sudo systemctl status snap.lxd.daemon
● snap.lxd.daemon.service - Service for snap application lxd.daemon
Loaded: loaded (/etc/systemd/system/snap.lxd.daemon.service; static)
Active: active (running) since Fri 2023-05-26 22:00:30 EDT; 4min 4s ago
TriggeredBy: ● snap.lxd.daemon.unix.socket
Main PID: 39274 (daemon.start)
Tasks: 0 (limit: 6889)
Memory: 22.1M
CPU: 853ms
CGroup: /system.slice/snap.lxd.daemon.service
‣ 39274 /bin/sh /snap/lxd/24846/commands/daemon.start

On Ubuntu 18.04, the host OS has ip_forwarding by default. I had to enable ip_forwarding on the host to get the Traefic load balancer to work.

Without it I get cryptic errors from Traefic that didn't make any sense. Not sure if anyone else has experienced this, but doing a quick sysctl -w net.ipv4.ip_forward=1 on the host resolves the issue.

Hope that helps anyone giving this a whirl.

I'm following Kubernetes the hard way and made it all the way to configuring the worker nodes: https://github.com/kelseyhightower/kubernetes-the-hard-way/blob/master/docs/09-bootstrapping-kubernetes-workers.md#start-the-worker-services
At this point, the kube-proxy service fails because

5286 server.go:489] open /proc/sys/net/netfilter/nf_conntrack_max: no such file or directory

which led me to your guide here.
I have added

config:
  linux.kernel_modules: xt_conntrack, nf_conntrack

to my worker node, and conntrack -L yields output.
On the host machine I can ls /proc/sys/net/netfilter/nf_conntrack_max and the file is there.

And yet the kube-proxy service still fails because it cannot find this file. Any advice?

edit: on the worker node this is lsmod | grep conntrack output

root@worker-0:~# lsmod | grep conntrack
nf_conntrack_netlink    45056  0
nfnetlink              16384  10 nf_conntrack_netlink,nf_tables
xt_conntrack           16384  28
nf_conntrack          139264  5 xt_conntrack,nf_nat,xt_nat,nf_conntrack_netlink,xt_MASQUERADE
nf_defrag_ipv6         24576  1 nf_conntrack
nf_defrag_ipv4         16384  1 nf_conntrack
libcrc32c              16384  2 nf_conntrack,nf_nat
x_tables               40960  24 ebtables,ip6table_filter,xt_conntrack,iptable_filter,xt_LOG,xt_multiport,xt_tcpudp,xt_addrtype,xt_CHECKSUM,xt_nat,ip6t_rt,xt_comment,ip6_tables,ipt_REJECT,ipt_rpfilter,iptable_raw,ip_tables,xt_limit,xt_hl,ip6table_mangle,xt_MASQUERADE,ip6t_REJECT,iptable_mangle,xt_mark

I've tried editing the systemd unit file for kube-proxy to include --conntrack-max-per-core=0 which according to the kube-proxy binary, should disable setting the conntrack value,

root@worker-1:~# kube-proxy --help | grep conntrack
      --conntrack-max-per-core int32                 Maximum number of NAT connections to track per CPU core (0 to leave the limit as-is and ignore conntrack-min). (default 32768)
      --conntrack-min int32                          Minimum number of conntrack entries to allocate, regardless of conntrack-max-per-core (set conntrack-max-per-core=0 to leave the limit as-is). (default 131072)

but this setting seems to be ignored and it tried to write to the file anyway.

I'm following this for learning purposes. All is well until the first kubeadm command. I am receiving preflight errors regarding memory and swap:

[ERROR Swap]: running with swap on is not supported. Please disable swap 

Maybe add a note about this in the guide? And any pointers on how to go around this "properly"?

LXD is version 4.0 and Kubernetes is at 1.18.1.

Your repo is awsome as using lxd to test and learn kubernetes feels much more efficient than minikube. Tho I am just now only starting and wondering few things.

In arch linux it was recommended to use

/etc/subuid
root:100000:65536
/etc/subgid
root:100000:65536

but in your guide its
1000000:1000000000
Can you elaborate why?

See kubernetes/kubernetes#58610

When the CPU count is large (e.g. I had it at 12 which is the amount on my host), conntrack hashsize may need to be increased when starting kube-proxy during k8s boot. The problem in LXC setups seems to be that the /sys/.../conntrack/hashsize file cannot be edited in any way inside the container, leading to failure if it needs to be altered.

My fix was to limit the CPU count on the system to 4 cores, which resulted in no wanted changes to the hashsize value.

Maybe add a note about this into the guide?