gpustat reports only the first program on nv driver 535 about gpustat

[wookayin]

This bug is due to the breaking changes in NVIDIA Driver R535.xx series (affected versions are >= 535.43, < 535.98.

TL;DR)

• Avoid NVIDIA Drivers between 535.43 and 535.86. These are broken. If you must use this driver version, use pip install nvidia-ml-py == 12.535.77 as a workaround.
• If you use NVIDIA Drivers 535.104.05+ and pynvml 12.535.108+, process information will be OK.

NVIDIA Driver Changes:

• 535.43.xx (NVIDIA/nvidia-settings@39c3e28) added a field usedGpuCcProtectedMemory to nvmlProcessInfo_st, which breaks the process information API. The only compatible pynvml version is 12.535.77.

• 535.54.xx (still affected)

• 535.86.xx (still affected)

• 535.98.xx (NVIDIA/nvidia-settings@0cb3bef) reverts the change, removing the field usedGpuCcProtectedMemory from nvmlProcessInfo_st (v2 API).

• 535.104.05 (NVIDIA/nvidia-settings@74cae7f) everything is now fixed. Adds nvmlProcessInfo_v2_st again without usedGpuCcProtectedMemory (which is correct). Needs pynvml >= 12.535.108.

Cross-ref: XuehaiPan/nvitop#88 (comment)

[wookayin]

We won't be adding monkey-patching because it is extremely complex to manage all the combinations. The buggy versions of nvidia drivers (535.43 and 535.86) and nvidia-ml-py 12.535.77 should be avoided, but there is a working workaround. I've added a warning message shown when such incompatible versions of driver/pynvml are found.

[wookayin]

I see, thanks for the report. I will make an update to support the new nvidia driver. Probably the same issue as #157.

[rxqy]

Hi, I'll be using nvidia-ml-py 12.535.77 for now. Many thanks for the help.