k3s kube-system pods failure when using --bind-address about k3s

[shakibamoshiri]

version

k3s --version 
k3s version v1.30.3+k3s1 (f6466040)
go version go1.22.5

I did a new test with --bind-address and see the same error

kubectl get pods -n kube-system  
NAME                                      READY   STATUS             RESTARTS        AGE
coredns-576bfc4dc7-9k8m2                  0/1     Running            0               20m
helm-install-traefik-crd-glb9q            0/1     CrashLoopBackOff   8 (4m28s ago)   20m
helm-install-traefik-xsmml                0/1     CrashLoopBackOff   8 (4m42s ago)   20m
local-path-provisioner-6795b5f9d8-vqx5j   0/1     CrashLoopBackOff   8 (4m38s ago)   20m
metrics-server-557ff575fb-5k6w8           0/1     CrashLoopBackOff   8 (4m35s ago)   20m

further testing shows that for example this pod metrics-server-557ff575fb-5k6w8 error is because of TCP failure .
I can ping the pods IP but cannot netcat it

describe pod

Successfully assigned kube-system/metrics-server-557ff575fb-5k6w8 to 208
Pulling image "rancher/mirrored-metrics-server:v0.7.0"
Successfully pulled image "rancher/mirrored-metrics-server:v0.7.0" in 6.911s (6.911s including waiting). Image size: 19434712 bytes.
Readiness probe failed: Get "https://10.52.0.5:10250/readyz": dial tcp 10.52.0.5:10250: connect: connection refused
Readiness probe failed: Get "https://10.52.0.5:10250/readyz": read tcp 10.52.0.1:55902->10.52.0.5:10250: read: connection reset by peer
Created container metrics-server
Started container metrics-server
Container image "rancher/mirrored-metrics-server:v0.7.0" already present on machine
Readiness probe failed: Get "https://10.52.0.5:10250/readyz": read tcp 10.52.0.1:57652->10.52.0.5:10250: read: connection reset by peer
Back-off restarting failed container metrics-server in pod metrics-server-557ff575fb-5k6w8_kube-system(a8c21f81-0a9e-4df5-967b-899035bb3236)

ping (okay)

ping -c4 10.52.0.5
PING 10.52.0.5 (10.52.0.5) 56(84) bytes of data.
64 bytes from 10.52.0.5: icmp_seq=1 ttl=64 time=0.123 ms
64 bytes from 10.52.0.5: icmp_seq=2 ttl=64 time=0.108 ms
64 bytes from 10.52.0.5: icmp_seq=3 ttl=64 time=0.110 ms
64 bytes from 10.52.0.5: icmp_seq=4 ttl=64 time=0.097 ms

netcat (fails)

nc -v 10.52.0.5 10250
nc: connect to 10.52.0.5 port 10250 (tcp) failed: Connection refused

[brandond]

Have you tried just setting --node-ip and --flannel-iface instead of mucking about with --advertise-address and --bind-address?

[shakibamoshiri]

it is not MUCKING
When we are learning anything we have to go with trail-and-error

I am running k8s but recently I am looking for a lighter distro like yours to have multi-region multi-cluster dual-stack (did not do it before)
For security reason and using my own overlay network (VPN) I waned all services listen on private IPs, thus I tried --bind-address so the cluster be configured for cross node communication via VPN network and no port be opened on any other IPs or Interfaces (just on VPN IP), I do not want any services listen to public IPs and I thought --bind-address is for this purpose

Yes after seeing errors , I tried the other way. I tested --flannel-iface and --node-external-ip , Because my setup is on-perm (no cloud) this way I can have flannel running on a private network and ingress-controller working with public ip (set by --node-external-ip)
And for disallowing open ports of k3s, I tested nftables and faced some issues that I mentioned here k3s nftables rule of ingress controller bypasses host nftables #10693

Any other software I have in mind (nginx, haproxy, ssh, etc) , when we set an IP to bind to, they listen on that IP and working fine. This is a common expectation of any command or option to bind an address. Thus if I did it wrong and it is not a bug excuse me for such a silly thing, you can close the issue. If it is a bug , let it be open
Thank you

I have not tried --node-ip yet , but I will try and update here

[brandond]

I am looking for a lighter distro like yours to have multi-region multi-cluster dual-stack

If this is what you're going for, I would suggest reading the docs at https://docs.k3s.io/networking/distributed-multicloud#embedded-k3s-multicloud-solution - distributed clusters are hard to get right, and using the integrated wireguard/tailscale integration is your best bet.