Failed to update correctly - ` validate CRI v1 image API for endpoint \"unix:///run/containerd-stargz-grpc/containerd-stargz-grpc.sock\"` about k3s

[brandond]

Check the containerd log file to see why the stargz image service failed to start?

[plsnotracking]

I'm not sure what exactly I should be looking for, I did 2 of the following things

1. Tried to look for containerd process in the systemctl list:

> sudo systemctl status c
calamares-firstboot.service
canberra-system-bootup.service
canberra-system-shutdown-reboot.service
canberra-system-shutdown.service
chronyd-restricted.service
chronyd.service
chrony-wait.service
colord.service
console-getty.service
cri-containerd-00929d54ad0b8a92ff5f054af2de926cfa3773a9398423605a4143482c4394cf.scope
cri-containerd-07ca53cd37b27531a4d8b183671d45c5c1e743d690a326912d18c4e544486a29.scope
cri-containerd-11e90b800025bc1b5a957667b201242e1114f3f7696961aff40c2b55d3c754ad.scope
cri-containerd-1275d534f038661b348508b7e0f351dc29a71bc50b197c1856692dd91b26a7da.scope
cri-containerd-3743f054fcc5c03cbc16c5c4735dc30e137163bab1741b9529419cd1250c5011.scope
cri-containerd-3c2eda63665777dbea4e695d44c4335a97b5dc4776d61caea512dc3b6dcadeca.scope
cri-containerd-3e21587a0351ab677d250a94e7e3b581732a7fd9f13514c80fffb9ffee6c0115.scope
cri-containerd-4cf0e915f7713c48f0bfd6c8ebcb867d30bf73acb4dcd0a9433f028bcd602a73.scope
cri-containerd-4d26eb29f9b80e309f03a55da87f40c8d8c12969eed852b9656a91f325d9c770.scope
cri-containerd-4d4fb1a0697a2c5fd470be019194ba77b16a2ccd0876ed4fd255f27ec8f87514.scope
cri-containerd-4da8a4bfc8875a1518d04f67f6861a9886aa6128b3dbaec71f97bfcb30f8cb4e.scope
cri-containerd-4dd232f35e0808936bc5c37b9c10252f27e15c8741552eac1878aa3fd45e1a27.scope
cri-containerd-5337ecd060d2544ec1c21a4db86239093276a1d3706af7d9937fd61e1af13288.scope
cri-containerd-5aa75578011ea56e22ab32d00a8f39377aafe0d1fac4e0f0fc754b6b3df7c38a.scope
cri-containerd-5d99140586c23b64516c52e59183f54e58aef9abf8c3793a6dc9176a8dc808cc.scope
cri-containerd-5ea8fbaa4c61db90cd72bca4394af526833304a0a4a47622b8e8ff8937940aa7.scope
cri-containerd-63333f2f2a4ef2f43cd39448ebb66848b70ecb2ca3d3713073a101a834f9cfc6.scope
cri-containerd-65b1ec7303ee5a676e718d51010c79640f808b13991b219778cc2f83550798c6.scope
cri-containerd-671509b5adce43c9235e32f83c076781d246116047bb06cb44153bfc6dc7a76c.scope
cri-containerd-811beae091d42a754dceee8ecb0057a4f718f72255ffef0099d3f00ffb01131e.scope
cri-containerd-87682967070ffb750df06f639ccf2a0978b809dcaa5b469cee7cb29d276af54f.scope
cri-containerd-8a7b30f9788b46be33cf33b1445a302521fdb3442b2737de5d35c404ec7fde46.scope
cri-containerd-96ef6b57c0e87a0bd24a8cb892e156a3899a8b7cb4ffe55bd47b15560ec7467d.scope
cri-containerd-9853a1ba17a782901badb8d4c91869feacb9f359f76971cadae0edfd5224d849.scope
cri-containerd-9865a3469562a240a41758270d13ae25e8793a536b8dd1d5e982c70d3859d4da.scope
cri-containerd-9a2aef3b732baabbcdd1a4214b77b1ff0f0ed17f486fd1e0a3d8bf13de1759d4.scope
cri-containerd-9b29000bcfae49553faca5928b64b34152391f455d7439efa51c0af9920d8bb3.scope
cri-containerd-9b49c96ab6f647de8a7898885fd879247c1aee96df5cd333d8644c7e8ef36192.scope
cri-containerd-9ffc976e76b78b7ec5267a90f420705342778f9f9c4e44f0d2bf1bf4339a4bb7.scope
cri-containerd-a062baf09a5d1568c79c40ce4ea98f36be485acabd91c847c6f489931a9bd38b.scope
cri-containerd-a75fca490cbb887003ff5a5ca2bcddb82d6ed4fee7bb8910f18981bd8a479071.scope
cri-containerd-b29cbbead966212a77ca0a9ce48cc67d106ab22e09f3a42629b9010b24bfcfc8.scope
cri-containerd-b921298245f162be15dd3ec27562c1a96deb3671a5995939a8ab5241715be258.scope
cri-containerd-b9a3a4c2e3cd44a9fb4bea646d29d3fcd394e0b5da3c026d7f5c810ade7e11cb.scope
cri-containerd-bbd1ca30e5b4c4cd551cd4cffb6b614b8c570413af2e3c173f85ad81564ea2dc.scope
cri-containerd-bd7305b5626337a772d7f0aecf0a37ce95cd3c05fbaa17dee288d8759f601627.scope
cri-containerd-c0b6c37c2512dc64d3476baf48fa2f78f132137c398bb6b803c0c04942199054.scope
cri-containerd-c24173bca03a715e77a4f9d32bec0df9724bcf99387d0bca3ce5cb68bb1d5588.scope
cri-containerd-ca72fd1ed8b61c406a25b6768161056bb3338dc6894780164a0e021948c50bdd.scope
cri-containerd-d278cc8d0064f6ee152c9c7d507bc5aca66a944d5e3e86a58c65214929adaf3a.scope
cri-containerd-d5213cca6a795e245ac5bf6f182db5ecece36b5467b1595c54706a1612d25010.scope
cri-containerd-d7bb219e5a3862f6e3f249e7345955ad087f5a784286822388142eb092650192.scope
cri-containerd-e5596a2c4633d685f668b56a0f5f4a9dcb3659217456563abf553c02f438569d.scope
cri-containerd-e899a38d9b60635359b3e2739a8ad1bf9f6cc0b87e0f7814f07a7417efecc6a9.scope
cri-containerd-f9c8324d7be54748689de44f5e3b9d2d4808495a096de1e0a3a368092965988c.scope
crond.service
cryptsetup-pre.target
cryptsetup.target
ctrl-alt-del.target
cups-browsed.service
cups.path
cups.service
cups.socket

2. The other thing I tried was cat /var/log/co, but there is containerd app/process to cat logs from.

Any other ways to see what I messed up?

[brandond]

The log is at /var/lib/rancher/k3s/agent/containerd/containerd.log

[plsnotracking]

Saw a bunch of these but it eventually came back up by itself.

time="2024-04-02T11:53:26.897694415-07:00" level=error msg="post event" error="failed to connect: dial unix /run/k3s/containerd/containerd.sock.ttrpc: connect: connection refused"
time="2024-04-02T11:53:26.956829149-07:00" level=error msg="post event" error="failed to connect: dial unix /run/k3s/containerd/containerd.sock.ttrpc: connect: connection refused"
time="2024-04-02T11:53:27.011435217-07:00" level=error msg="post event" error="failed to connect: dial unix /run/k3s/containerd/containerd.sock.ttrpc: connect: connection refused"
time="2024-04-02T11:53:41.963038517-07:00" level=error msg="forward event" error="failed to connect: dial unix /run/k3s/containerd/containerd.sock.ttrpc: connect: connection refused"
time="2024-04-02T11:53:42.016557210-07:00" level=error msg="forward event" error="failed to connect: dial unix /run/k3s/containerd/containerd.sock.ttrpc: connect: connection refused"
time="2024-04-02T11:54:35.420320769-07:00" level=error msg="post event" error="failed to connect: dial unix /run/k3s/containerd/containerd.sock.ttrpc: connect: connection refused"
time="2024-04-02T12:37:38.474185393-07:00" level=error msg="ContainerStatus for \"34a3c7bac19e967dcde9a4e1e0d730762c37558665c1d4819dda51c1857c1362\" failed" error="rpc error: code = NotFound desc = an error occurred when try to find container \"34a3c7bac19e967dcde9a4e1e0d730762c37558665c1d4819dda51c1857c1362\": not found"
time="2024-04-02T14:00:00.820014557-07:00" level=error msg="ContainerStatus for \"8a7b30f9788b46be33cf33b1445a302521fdb3442b2737de5d35c404ec7fde46\" failed" error="rpc error: code = NotFound desc = an error occurred when try to find container \"8a7b30f9788b46be33cf33b1445a302521fdb3442b2737de5d35c404ec7fde46\": not found"
time="2024-04-02T14:00:00.820471057-07:00" level=error msg="ContainerStatus for \"162c0819119565a84d7bddfff0b177ef64ec75960c0dd4e23ad594dabf5649c6\" failed" error="rpc error: code = NotFound desc = an error occurred when try to find container \"162c0819119565a84d7bddfff0b177ef64ec75960c0dd4e23ad594dabf5649c6\": not found"
time="2024-04-02T14:00:37.045066423-07:00" level=error msg="ContainerStatus for \"bbd1ca30e5b4c4cd551cd4cffb6b614b8c570413af2e3c173f85ad81564ea2dc\" failed" error="rpc error: code = NotFound desc = an error occurred when try to find container \"bbd1ca30e5b4c4cd551cd4cffb6b614b8c570413af2e3c173f85ad81564ea2dc\": not found"
time="2024-04-02T14:00:37.045322506-07:00" level=error msg="ContainerStatus for \"0a02945c89438cd4b63fbbe1bb420ff8fd89ed6bbb1799f35e887fb17035e418\" failed" error="rpc error: code = NotFound desc = an error occurred when try to find container \"0a02945c89438cd4b63fbbe1bb420ff8fd89ed6bbb1799f35e887fb17035e418\": not found"
time="2024-04-02T14:00:51.179059858-07:00" level=error msg="ContainerStatus for \"9ffc976e76b78b7ec5267a90f420705342778f9f9c4e44f0d2bf1bf4339a4bb7\" failed" error="rpc error: code = NotFound desc = an error occurred when try to find container \"9ffc976e76b78b7ec5267a90f420705342778f9f9c4e44f0d2bf1bf4339a4bb7\": not found"
time="2024-04-02T14:00:51.179641066-07:00" level=error msg="ContainerStatus for \"077691ba4fc266debd3d78828a8b36adc918a4cd9f40395e7a32b1f669379833\" failed" error="rpc error: code = NotFound desc = an error occurred when try to find container \"077691ba4fc266debd3d78828a8b36adc918a4cd9f40395e7a32b1f669379833\": not found"
time="2024-04-02T14:01:04.229868050-07:00" level=error msg="ContainerStatus for \"671509b5adce43c9235e32f83c076781d246116047bb06cb44153bfc6dc7a76c\" failed" error="rpc error: code = NotFound desc = an error occurred when try to find container \"671509b5adce43c9235e32f83c076781d246116047bb06cb44153bfc6dc7a76c\": not found"
time="2024-04-02T14:01:04.230341008-07:00" level=error msg="ContainerStatus for \"b4658df241469fd9cd79031aae0b0661e213aafd2220766211d017fffb8a9d70\" failed" error="rpc error: code = NotFound desc = an error occurred when try to find container \"b4658df241469fd9cd79031aae0b0661e213aafd2220766211d017fffb8a9d70\": not found"
time="2024-04-02T14:01:21.327287402-07:00" level=error msg="ContainerStatus for \"3743f054fcc5c03cbc16c5c4735dc30e137163bab1741b9529419cd1250c5011\" failed" error="rpc error: code = NotFound desc = an error occurred when try to find container \"3743f054fcc5c03cbc16c5c4735dc30e137163bab1741b9529419cd1250c5011\": not found"
time="2024-04-02T14:01:21.327883152-07:00" level=error msg="ContainerStatus for \"3db686d251bad3e560b61f6bc9bc84f0402f01cf9415adde13696b6e4744d3b6\" failed" error="rpc error: code = NotFound desc = an error occurred when try to find container \"3db686d251bad3e560b61f6bc9bc84f0402f01cf9415adde13696b6e4744d3b6\": not found"
time="2024-04-02T18:02:35.232330615-07:00" level=error msg="ContainerStatus for \"4de91c29e3ce8d21c4fb2ca66b44cbc05a172d2ce2615dd56aee93ea2777a5cf\" failed" error="rpc error: code = NotFound desc = an error occurred when try to find container \"4de91c29e3ce8d21c4fb2ca66b44cbc05a172d2ce2615dd56aee93ea2777a5cf\": not found"

Thanks for all the help.