k3s-agent Fails to start with with embedded registry and kill entire OS about k3s

[liyimeng]

I also experience the same :(
my guessing is that during key generation, the system boot for some reason, causing the generated key got no chance to write to disk, hence result in the following errors

[brandond]

During the startup process, the file /var/lib/rancher/k3s/agent/containerd/peer.key is generated but its content is empty
May 15 06:36:21 node7 k3s[1819]: time="2024-05-15T06:36:21Z" level=fatal msg="failed to start embedded registry: failed to load or generate p2p private key: error loading key from /var/lib/rancher/k3s/agent/containerd/peer.key: <nil>"

This error is coming from https://github.com/rancher/dynamiclistener/blob/e590d58b896cc8dd33dde7cec80c52e23ec08189/cert/io.go#L89 - the message suggests that the file was created by a previous startup of k3s, but for some reason the file contents have been lost. Your best bet is probably to just delete the file from disk and let it be recreated on startup. You might be able to find other errors in the logs to suggest why the file has no contents or its contents are corrupted, but given that this node is also rebooting unexpectedly, I suspect that you may have lost data from your filesystem when the system crashed.

When I attempt to delete the /var/lib/rancher/k3s/agent/containerd/peer.key file and then restart the k3s-agent, the system immediately freezes and then reboots.

That sounds like a problem with your node; K3s shouldn't be capable of doing anything that would cause it to panic and reboot. You'll need to figure that out on your own.

[liyimeng]

@brandon agree with you! I manage to switch to an openrc system and test the same k3s version, all work as expected. systemd seems playing devil here. :(

[liyimeng]

strange, when rolling back to 1.28.6, it runs ok with no issue.

[liyimeng]

I have found another potential cause. As I understand, when running with systemd, the cgroup driver should be systemd, however, I found k3s mistaken it as cgroupfs, not sure if this is the issue.

[brandond]

I'm not aware of any defect in k3s that would cause it to use cgroupfs instead of systemd, when using the embedded containerd on a systemd-based OS. You're not trying to use docker or another user-provided container runtime, are you?

[liyimeng]

no, I use kairos from https://github.com/kairos-io/kairos/, which should have no other runtime available. In addition to that, I add some additional printout and find

ARN[0002] isRunningInUserNS=false, cgroup controller map[cpu:true cpuset:true hugetlb:true io:true memory:true misc:true pids:true rdma:true], INVOCATION_ID= 

INVOCATION_ID is empty, something go wrong with systemd, it should set this value.

This is very likely systemd issue in their distribution, I will shout out loud there. :D