Hi, I think we found a bug maybe. Or I made a misstake with the configuration and would be thankful for help.
Generel the ovs-cni works great. But if I hard kill a Kubernetes Node (or do a reboot without draining the node), on which are pods running with the ovs-cni, the pods does not restart propably.
I startet the samplepod from the documentation and than reboot the ubuntu system which runs the kubernetes node (with reboot
, no drain before).
Before the reboot, the pod is running well.
After the reboot kubectl get pods says:
samplepod-1 0/1 Error 0 8m1s <none>
Feb 18 11:34:35 HOSTNAME kubelet[1219]: 2020/02/18 11:34:35 CNI DEL was called for container ID: bc79277db8b567a2864807a9d310eb240bffed01d840df0135ea34198ffd017b, network namespace , interface name
Feb 18 11:34:35 HOSTNAME kubelet[1219]: panic: This should never happen, if it does, it means caller does not pass container network namespace as a parameter and therefore OVS port cleanup will not
Feb 18 11:34:35 HOSTNAME kubelet[1219]: goroutine 1 [running, locked to thread]:
Feb 18 11:34:35 HOSTNAME kubelet[1219]: github.com/kubevirt/ovs-cni/pkg/plugin.CmdDel(0xc0000f2620, 0xc000014500, 0x5)
Feb 18 11:34:35 HOSTNAME kubelet[1219]: /home/travis/gopath/src/github.com/kubevirt/ovs-cni/pkg/plugin/plugin.go:276 +0x792
Feb 18 11:34:35 HOSTNAME kubelet[1219]: github.com/containernetworking/cni/pkg/skel.(*dispatcher).checkVersionAndCall(0xc00009bed8, 0xc0000f2620, 0x71a080, 0xc0000cc4e0, 0x6d0d30, 0x0, 0xc00009bec0)
Feb 18 11:34:35 HOSTNAME kubelet[1219]: /home/travis/gopath/src/github.com/kubevirt/ovs-cni/vendor/github.com/containernetworking/cni/pkg/skel/skel.go:185 +0x259
Feb 18 11:34:35 HOSTNAME kubelet[1219]: github.com/containernetworking/cni/pkg/skel.(*dispatcher).pluginMain(0xc00009bed8, 0x6d0d20, 0x6d0d28, 0x6d0d30, 0x71a080, 0xc0000cc4e0, 0xc00001c060, 0x25, 0
Feb 18 11:34:35 HOSTNAME kubelet[1219]: /home/travis/gopath/src/github.com/kubevirt/ovs-cni/vendor/github.com/containernetworking/cni/pkg/skel/skel.go:251 +0x3d4
Feb 18 11:34:35 HOSTNAME kubelet[1219]: github.com/containernetworking/cni/pkg/skel.PluginMainWithError(...)
Feb 18 11:34:35 HOSTNAME kubelet[1219]: /home/travis/gopath/src/github.com/kubevirt/ovs-cni/vendor/github.com/containernetworking/cni/pkg/skel/skel.go:286
Feb 18 11:34:35 HOSTNAME kubelet[1219]: github.com/containernetworking/cni/pkg/skel.PluginMain(0x6d0d20, 0x6d0d28, 0x6d0d30, 0x71a080, 0xc0000cc4e0, 0xc00001c060, 0x25)
Feb 18 11:34:35 HOSTNAME kubelet[1219]: /home/travis/gopath/src/github.com/kubevirt/ovs-cni/vendor/github.com/containernetworking/cni/pkg/skel/skel.go:301 +0x129
Feb 18 11:34:35 HOSTNAME kubelet[1219]: main.main()
Feb 18 11:34:35 HOSTNAME kubelet[1219]: /home/travis/gopath/src/github.com/kubevirt/ovs-cni/cmd/plugin/main.go:26 +0x11c
Feb 18 11:34:35 HOSTNAME kubelet[1219]: E0218 11:34:35.943468 1219 cni.go:379] Error deleting loadbalancer_samplepod-1/bc79277db8b567a2864807a9d310eb240bffed01d840df0135ea34198ffd017b from networ
Feb 18 11:34:35 HOSTNAME kubelet[1219]: E0218 11:34:35.944454 1219 remote_runtime.go:128] StopPodSandbox "bc79277db8b567a2864807a9d310eb240bffed01d840df0135ea34198ffd017b" from runtime service fa
Feb 18 11:34:35 HOSTNAME kubelet[1219]: E0218 11:34:35.944527 1219 kuberuntime_manager.go:878] Failed to stop sandbox {"docker" "bc79277db8b567a2864807a9d310eb240bffed01d840df0135ea34198ffd017b"}
Feb 18 11:34:35 HOSTNAME kubelet[1219]: E0218 11:34:35.944618 1219 kuberuntime_manager.go:658] killPodWithSyncResult failed: failed to "KillPodSandbox" for "08155d12-cdcd-4c2d-a7cc-ad8f8f14761e"
Feb 18 11:34:35 HOSTNAME kubelet[1219]: E0218 11:34:35.944663 1219 pod_workers.go:191] Error syncing pod 08155d12-cdcd-4c2d-a7cc-ad8f8f14761e ("samplepod-1_loadbalancer(08155d12-cdcd-4c2d-a7cc-ad
Feb 18 11:34:38 HOSTNAME kubelet[1219]: E0218 11:34:38.621370 1219 summary_sys_containers.go:47] Failed to get system container stats for "/system.slice/docker.service": failed to get cgroup stat
Feb 18 11:34:38 HOSTNAME kubelet[1219]: W0218 11:34:38.854130 1219 cni.go:328] CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container "c12faf953
Feb 18 11:34:38 HOSTNAME kubelet[1219]: 2020/02/18 11:34:38 CNI DEL was called for container ID: c12faf95367d396da18c3b90b1099ebce6b372bc073622bba1d1b88bb621e2c6, network namespace , interface name
Feb 18 11:34:38 HOSTNAME kubelet[1219]: panic: This should never happen, if it does, it means caller does not pass container network namespace as a parameter and therefore OVS port cleanup will not
Feb 18 11:34:38 HOSTNAME kubelet[1219]: goroutine 1 [running, locked to thread]:
Feb 18 11:34:38 HOSTNAME kubelet[1219]: github.com/kubevirt/ovs-cni/pkg/plugin.CmdDel(0xc000106690, 0xc0000ea4a8, 0x5)
Feb 18 11:34:38 HOSTNAME kubelet[1219]: /home/travis/gopath/src/github.com/kubevirt/ovs-cni/pkg/plugin/plugin.go:276 +0x792
Feb 18 11:34:38 HOSTNAME kubelet[1219]: github.com/containernetworking/cni/pkg/skel.(*dispatcher).checkVersionAndCall(0xc0000cbed8, 0xc000106690, 0x71a080, 0xc0000d04e0, 0x6d0d30, 0x0, 0xc0000cbec0)
Feb 18 11:34:38 HOSTNAME kubelet[1219]: /home/travis/gopath/src/github.com/kubevirt/ovs-cni/vendor/github.com/containernetworking/cni/pkg/skel/skel.go:185 +0x259
Feb 18 11:34:38 HOSTNAME kubelet[1219]: github.com/containernetworking/cni/pkg/skel.(*dispatcher).pluginMain(0xc0000cbed8, 0x6d0d20, 0x6d0d28, 0x6d0d30, 0x71a080, 0xc0000d04e0, 0xc00012a000, 0x25, 0
Feb 18 11:34:38 HOSTNAME kubelet[1219]: /home/travis/gopath/src/github.com/kubevirt/ovs-cni/vendor/github.com/containernetworking/cni/pkg/skel/skel.go:251 +0x3d4
Feb 18 11:34:38 HOSTNAME kubelet[1219]: github.com/containernetworking/cni/pkg/skel.PluginMainWithError(...)
Feb 18 11:34:38 HOSTNAME kubelet[1219]: /home/travis/gopath/src/github.com/kubevirt/ovs-cni/vendor/github.com/containernetworking/cni/pkg/skel/skel.go:286
Feb 18 11:34:38 HOSTNAME kubelet[1219]: github.com/containernetworking/cni/pkg/skel.PluginMain(0x6d0d20, 0x6d0d28, 0x6d0d30, 0x71a080, 0xc0000d04e0, 0xc00012a000, 0x25)
Feb 18 11:34:38 HOSTNAME kubelet[1219]: /home/travis/gopath/src/github.com/kubevirt/ovs-cni/vendor/github.com/containernetworking/cni/pkg/skel/skel.go:301 +0x129
Feb 18 11:34:38 HOSTNAME kubelet[1219]: main.main()
Feb 18 11:34:38 HOSTNAME kubelet[1219]: /home/travis/gopath/src/github.com/kubevirt/ovs-cni/cmd/plugin/main.go:26 +0x11c
...
Port "vethb0617177"
Interface "vethb0617177"
error: "could not open network device vethb0617177 (No such device)"
Port "veth5f25970b"
Interface "veth5f25970b"
error: "could not open network device veth5f25970b (No such device)"
Port "veth62e7faab"
Interface "veth62e7faab"
error: "could not open network device veth62e7faab (No such device)"
Port "vethc5110e9d"
Interface "vethc5110e9d"
error: "could not open network device vethc5110e9d (No such device)"
Port "vethef8643f7"
Interface "vethef8643f7"
error: "could not open network device vethef8643f7 (No such device)"
Port "veth12efe3d7"
Interface "veth12efe3d7"
error: "could not open network device veth12efe3d7 (No such device)"
Thanks for your help. And let me know, if you need more inforamtions.