Can I disable the device plugin pod for an individual node? about k8s-device-plugin

[flx42]

@RenaudWasTaken didn't we say that we should do an infinite loop in this case?

[dredwilliams]

Just to see if it would make a difference, I installed the drivers, cuda libs, and container runtime on the non-GPU nodes. The error changed to: "nvidia-container-cli: initialization error: cuda error: unknown error"

@flx42 I'd suggest an infinite sleep ... we don't want to consume CPU resources unnecessarily.

Thanks!

[flx42]

Sure sure, that's what I meant. Like this: https://github.com/NVIDIA/k8s-device-plugin/blob/v1.9/main.go#L26

[dredwilliams]

Unless I'm mistaken, that error message is coming from the container runtime when it's trying to set up before it runs the plugin. This might need to be a mod to the nvidia-container runtime?

[flx42]

No, you just shouldn't install the runtime and the driver on a non-GPU node.
And then we will just need to modify this line:
https://github.com/NVIDIA/k8s-device-plugin/blob/v1.9/main.go#L19

[dredwilliams]

@flx42 I replaced the exit() with the select {} as you suggested and it now behaves nicely. Thanks!

[RenaudWasTaken]

@dredwilliams thanks for reporting this! Fixed in the latest version :)
Images will pushed to the Docker Hub shortly