Agent node IP not replaced with Tailscale VPN IP about k3s

[manuelbuil]

Oh sorry! I have already created a fix this morning :( ==> #10074

[npitsillos]

I would definitely attempt to implement this! If I get this right the Vagrant sets up multiple VMs which act as nodes in the cluster and the tests are then run on these? I am currently away so will attempt work on this next week! Thank you very much for your help looking forward to contributing!

[manuelbuil]

I would definitely attempt to implement this! If I get this right the Vagrant sets up multiple VMs which act as nodes in the cluster and the tests are then run on these? I am currently away so will attempt work on this next week! Thank you very much for your help looking forward to contributing!

Correct. You run the test by executing go test -v -timeout=30m ./tests/e2e/tailscale/, that triggers the vagrant creation and when that is ready the testing starts. Hit me up in slack for more details

[npitsillos]

After some more digging, using --vpn-auth name=tailscale,joiKey=<key> works as expected whereas the --vpn-auth-file argument does not work as expected. My assumption would be that the functionality to read the file when an agent is run might contain a bug. This issue only appears when vpn-auth-file is used to install k3s on the agent node. Running this on the server node works as expected.

[manuelbuil]

Thanks for the report! You are totally correct and there is a bug. I have just created a PR to address it and fix it

[npitsillos]

Thanks @manuelbuil! I would try to fix this myself if possible might take some more time than usual but I'd be interested in contributing. I am not really sure how to approach local development though.

[manuelbuil]

Thanks @manuelbuil! I would try to fix this myself if possible might take some more time than usual but I'd be interested in contributing. I am not really sure how to approach local development though.

If you'd like to help, there is one thing you could certainly do. Enhance our tailscale e2e testcase with one extra node which uses vpn-auth-file parameter, that way we will be able to catch bugs if this problem happens again. Here is the testcase: https://github.com/k3s-io/k3s/tree/master/tests/e2e/tailscale. If you are up for the challenge, I can help you.

Right now we are deploying both server and worker nodes with vpn-auth ===> https://github.com/k3s-io/k3s/blob/master/tests/e2e/tailscale/Vagrantfile#L38-L55

[VestigeJ]

##Environment Details
Reproduced using VERSION=v1.30.1+k3s1
Validated using COMMIT=f2e7c01acfdc5f51bfd007c44bfe6605e8864975

Infrastructure

• Cloud

Node(s) CPU architecture, OS, and version:

Linux 5.14.21-150500.53-default x86_64 GNU/Linux
PRETTY_NAME="SUSE Linux Enterprise Server 15 SP5"

Config.yaml:

node-external-ip: 1.1.1.13
server: https://1.1.1.16:6443
token: YOUR_TOKEN_HERE
vpn-auth-file: /etc/rancher/k3s/vpn

Reproduction

$ curl https://get.k3s.io --output install-"k3s".sh
$ sudo chmod +x install-"k3s".sh
$ sudo groupadd --system etcd && sudo useradd -s /sbin/nologin --system -g etcd etcd
$ sudo modprobe ip_vs_rr
$ sudo modprobe ip_vs_wrr
$ sudo modprobe ip_vs_sh
$ sudo printf "on_oovm.panic_on_oom=0 \nvm.overcommit_memory=1 \nkernel.panic=10 \nkernel.panic_ps=1 \nkernel.panic_on_oops=1 \n" > ~/90-kubelet.conf
$ sudo cp 90-kubelet.conf /etc/sysctl.d/
$ sudo systemctl restart systemd-sysctl
$ COMMIT=f2e7c01acfdc5f51bfd007c44bfe6605e8864975
$ get_tailscale //function pasted below
$ sudo INSTALL_K3S_COMMIT=$COMMIT INSTALL_K3S_EXEC=agent ./install-k3s.sh

Results:

Showing on existing release v1.30.1+k3s1 that the tailscale ip is not used when starting up agents.

$ kgn -o wide

NAME               STATUS   ROLES                       AGE     VERSION        INTERNAL-IP      EXTERNAL-IP      OS-IMAGE                              KERNEL-VERSION              CONTAINER-RUNTIME
ip-1-1-1-12        Ready    <none>                      29s     v1.30.1+k3s1   1.1.1.12         1.1.1.3     SUSE Linux Enterprise Server 15 SP5   5.14.21-150500.53-default   containerd://1.7.15-k3s1
ip-1-1-1-16        Ready    control-plane,etcd,master   3m38s   v1.30.1+k3s1   100.77.213.109   1.1.1.16    SUSE Linux Enterprise Server 15 SP5   5.14.21-150500.53-default   containerd://1.7.15-k3s1

Same cluster but a newer agent running the latest commit_id for branch attention on node 1-1-1-13 also showing the commit_id version f2e7c01

$ kgn -o wide

NAME               STATUS   ROLES                       AGE   VERSION                INTERNAL-IP      EXTERNAL-IP      OS-IMAGE                              KERNEL-VERSION              CONTAINER-RUNTIME
ip-1-1-1-12        Ready    <none>                      11m   v1.30.1+k3s1           1.1.1.12         1.1.1.3          SUSE Linux Enterprise Server 15 SP5   5.14.21-150500.53-default   containerd://1.7.15-k3s1
ip-1-1-1-16        Ready    control-plane,etcd,master   14m   v1.30.1+k3s1           100.77.213.109   1.1.1.23         SUSE Linux Enterprise Server 15 SP5   5.14.21-150500.53-default   containerd://1.7.15-k3s1
ip-1-1-1-13        Ready    <none>                      32s   v1.30.1+k3s-f2e7c01a   100.86.92.4      1.1.1.12         SUSE Linux Enterprise Server 15 SP5   5.14.21-150500.53-default   containerd://1.7.17-k3s1

get_tailscale() {
    curl -fsSL https://tailscale.com/install.sh -o install-ts.sh
    chmod +x install-ts.sh
    sudo ./install-ts.sh
}