node should reboot once MIG config is enabled or disabled via node label about mig-parted

[klueska]

Reboot is only necessary on CSPs with passhthrough virtualization but no GPU reset capabilities (e.g. GCP). Setting the envvar WITH_REBOOT=true on the k8s-mig-manager container should give you the behavior you want.

[dogra-gopal]

It does. not work. I have tried setting envvar WITH_REBOOT=true. Code does not seem to be right. https://github.com/NVIDIA/mig-parted/blob/main/deployments/gpu-operator/reconfigure-mig.sh#L469 currently, iwith WITH_REBOOT true will reboot node only if, this apply command fails.. but this succeeds, so we need manual reboot.

These are the logs: you can see that it fails in the end, but code is checking only if mig-configuration apply fails, which passes, but in the end mig-apply fails and it needs reboot of node to get fixed.

Applying the MIG mode change from the selected config to the node
If the -r option was passed, the node will be automatically rebooted if this is not successful
time="2022-04-25T05:42:13-11:00" level=debug msg="Parsing config file..."
time="2022-04-25T05:42:13-11:00" level=debug msg="Selecting specific MIG config..."
time="2022-04-25T05:42:13-11:00" level=debug msg="Running apply-start hook"
time="2022-04-25T05:42:13-11:00" level=debug msg="Checking current MIG mode..."
time="2022-04-25T05:42:13-11:00" level=debug msg="Walking MigConfig for (devices=all)"
time="2022-04-25T05:42:13-11:00" level=debug msg=" GPU 0: 0x20B510DE"
time="2022-04-25T05:42:13-11:00" level=debug msg=" Asserting MIG mode: Enabled"
time="2022-04-25T05:42:13-11:00" level=debug msg=" MIG capable: true\n"
time="2022-04-25T05:42:13-11:00" level=debug msg=" Current MIG mode: Disabled"
time="2022-04-25T05:42:13-11:00" level=debug msg="Running pre-apply-mode hook"
time="2022-04-25T05:42:13-11:00" level=debug msg="Applying MIG mode change..."
time="2022-04-25T05:42:13-11:00" level=debug msg="Walking MigConfig for (devices=all)"
time="2022-04-25T05:42:13-11:00" level=debug msg=" GPU 0: 0x20B510DE"
time="2022-04-25T05:42:13-11:00" level=debug msg=" MIG capable: true\n"
time="2022-04-25T05:42:13-11:00" level=debug msg=" Current MIG mode: Disabled"
time="2022-04-25T05:42:13-11:00" level=debug msg=" Updating MIG mode: Enabled"
time="2022-04-25T05:42:16-11:00" level=debug msg=" Mode change pending: true"
time="2022-04-25T05:42:16-11:00" level=debug msg="At least one mode change pending"
time="2022-04-25T05:42:16-11:00" level=debug msg="Resetting GPUs..."
time="2022-04-25T05:42:16-11:00" level=debug msg=" NVIDIA kernel module loaded"
time="2022-04-25T05:42:16-11:00" level=debug msg=" Using nvidia-smi to perform GPU reset"
time="2022-04-25T05:42:20-11:00" level=debug msg="Running apply-exit hook"
MIG configuration applied successfully
Applying the selected MIG config to the node
time="2022-04-25T05:42:20-11:00" level=debug msg="Parsing config file..."
time="2022-04-25T05:42:20-11:00" level=debug msg="Selecting specific MIG config..."
time="2022-04-25T05:42:20-11:00" level=debug msg="Running apply-start hook"
time="2022-04-25T05:42:20-11:00" level=debug msg="Checking current MIG mode..."
time="2022-04-25T05:42:20-11:00" level=debug msg="Walking MigConfig for (devices=all)"
time="2022-04-25T05:42:20-11:00" level=debug msg=" GPU 0: 0x20B510DE"
time="2022-04-25T05:42:20-11:00" level=debug msg=" Asserting MIG mode: Enabled"
time="2022-04-25T05:42:20-11:00" level=debug msg=" MIG capable: true\n"
time="2022-04-25T05:42:20-11:00" level=debug msg=" Current MIG mode: Disabled"
time="2022-04-25T05:42:20-11:00" level=debug msg="Running pre-apply-mode hook"
time="2022-04-25T05:42:20-11:00" level=debug msg="Applying MIG mode change..."
time="2022-04-25T05:42:20-11:00" level=debug msg="Walking MigConfig for (devices=all)"
time="2022-04-25T05:42:20-11:00" level=debug msg=" GPU 0: 0x20B510DE"
time="2022-04-25T05:42:20-11:00" level=debug msg=" MIG capable: true\n"
time="2022-04-25T05:42:20-11:00" level=debug msg=" Current MIG mode: Disabled"
time="2022-04-25T05:42:20-11:00" level=debug msg=" Updating MIG mode: Enabled"
time="2022-04-25T05:42:23-11:00" level=debug msg=" Mode change pending: true"
time="2022-04-25T05:42:23-11:00" level=debug msg="At least one mode change pending"
time="2022-04-25T05:42:23-11:00" level=debug msg="Resetting GPUs..."
time="2022-04-25T05:42:23-11:00" level=debug msg=" NVIDIA kernel module loaded"
time="2022-04-25T05:42:23-11:00" level=debug msg=" Using nvidia-smi to perform GPU reset"
time="2022-04-25T05:42:26-11:00" level=debug msg="Checking current MIG device configuration..."
time="2022-04-25T05:42:26-11:00" level=debug msg="Walking MigConfig for (devices=all)"
time="2022-04-25T05:42:26-11:00" level=debug msg=" GPU 0: 0x20B510DE"
time="2022-04-25T05:42:26-11:00" level=debug msg="Running pre-apply-config hook"
time="2022-04-25T05:42:26-11:00" level=debug msg="Applying MIG device configuration..."
time="2022-04-25T05:42:26-11:00" level=debug msg="Walking MigConfig for (devices=all)"
time="2022-04-25T05:42:26-11:00" level=debug msg=" GPU 0: 0x20B510DE"
time="2022-04-25T05:42:26-11:00" level=debug msg=" MIG capable: true\n"
time="2022-04-25T05:42:26-11:00" level=debug msg="Running apply-exit hook"
time="2022-04-25T05:42:26-11:00" level=fatal msg="Unable to apply MIG config with MIG mode disabled"
Restarting any GPU clients previously shutdown on the host by restarting their systemd services
Starting kubelet.service
Restarting any GPU clients previously shutdown in Kubernetes by reenabling their component-specific nodeSelector labels
node/worker-1 unlabeled
Changing the 'nvidia.com/mig.config.state' node label to 'failed'
node/worker-1 unlabeled
time="2022-04-25T16:42:26Z" level=error msg="Error: exit status 1"
time="2022-04-25T16:42:26Z" level=info msg="Waiting for change to 'nvidia.com/mig.config' label"

[dogra-gopal]

The above issue is seen while both enabling and disabling MIG. It needs node reboot to proceed further.

[klueska]

If the apply command is succeeding then (for whatever reason) it thinks the GPU reset completed successfully, when it didn’t actually reset the GPU.

What environment are you running this in? As I mentioned before, a reboot is only necessary if the reset fails, which in your setup appears to be silently failing and returning a successful error code.

[dogra-gopal]

HX 3.0 VMware vCenter environment.

[klueska]

I have a vague memory of this coming up in vsphere before and there was some incorrect vbios settings or something.

@shivamerla, @supertetelman, @kpouget
Does this ring a bell for you?

[supertetelman]

I do recall seeing this as some sort of misconfiguration, but I thought that had been addressed in updated documentation. I will dig around and try to find an answer.

@dogra-gopal, are you using GPU passthrough in this environment or is this vGPU? If you are using vGPU, MIG configuration changes need to be done through VMware and cannot be controlled through the mig-parted tool.

[dogra-gopal]

@supertetelman I am using GPU passthrough.

[dogra-gopal]

Verified that with latest version v0.4.2, this issue is fixed. So closing this. Thanks @klueska

[klueska]

Yes, we added this to work around this issue:
e761afb

It's still unclear why the GPU reset silently fails, but at least we are able to get past it with this workaround.