Device plugin does not start on MIG-enabled host due to insufficient permissions about k8s-device-plugin

[klueska]

This definitely points to NVIDIA_MIG_MONITOR_DEVICES not being set correctly. Can you verify that this setting is actually being picked up in the container? Meaning, exec into the container can run export to observe the envvars set.

[elezar]

@yunfeng-scale since you mention the GPU Operator being used, could you please confirm the GPU Operator version that is being used to deploy the v0.14.0 version of the device plugin?

[yunfeng-scale]

GPU Operator version is 22.9.1, driver version is 470.161.03. will circle back on checking NVIDIA_MIG_MONITOR_DEVICES next week

[yunfeng-scale]

This definitely points to NVIDIA_MIG_MONITOR_DEVICES not being set correctly. Can you verify that this setting is actually being picked up in the container? Meaning, exec into the container can run export to observe the envvars set.

sorry for the late reply. @klueska yes i can confirm this set correctly by getting into a container and grep env vars.

also using the latest GPU operator 23.9.2 the problem persists

[yunfeng-scale]

tried to install gpu-feature-discovery from its own helm chart (removing it from gpu operator) and that didn't work either

also can you help me understand what sets the permissions based on env var NVIDIA_MIG_MONITOR_DEVICES, so I may able to do some investigations?

[yunfeng-scale]

for others encountering the same issue: we upgraded EKS from 1.23 to 1.29 and the permission issue is resolved.