[CLI]: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf8 in position 0: invalid start byte about wandb

[j-v-n]

Update: I fixed my issue by rolling back my Nvidia driver to 552.44 and restarting my machine.

[rjavierch]

Update: I fixed my issue by rolling back my Nvidia driver to 552.44 and restarting my machine.

I second this, downgraded to the Nvidia 552.44 studio driver, it and now is working again.

[Aricept094]

I've recently started experiencing the same exact issue that I suspect is related to a recent NVIDIA driver update. Yesterday, I updated my NVIDIA driver to version 555.85, and since then, I've been encountering errors in both Ray Tune and wand.

Initially, I encountered the error in Ray Tune, but after modifying the nvidia_gpu.py file in python3.11/site-packages/ray/_private/accelerators/ to use Latin-1 encoding instead of UTF-8, I was able to get my Ray Tune project working again. The modified code is as follows:

try:
pynvml.nvmlInit()
except pynvml.NVMLError:
return None # pynvml init failed
device_count = pynvml.nvmlDeviceGetCount()
cuda_device_type = None
if device_count > 0:
handle = pynvml.nvmlDeviceGetHandleByIndex(0)
device_name = pynvml.nvmlDeviceGetName(handle)
if isinstance(device_name, bytes):
device_name = device_name.decode("latin1") # Changed from "utf-8" to "latin1"
cuda_device_type = (
NvidiaGPUAcceleratorManager._gpu_name_to_accelerator_type(device_name)
)
pynvml.nvmlShutdown()
return cuda_device_type

However, I'm still experiencing issues with W&B, where I'm receiving errors and my metrics are not being monitored as intended.

[Aricept094]

same here , worked.

[j-v-n]

Has anyone tested out if it's safe to update the driver now?