cudart.cudaSetDevice allocates memory on GPU other than target about cuda-python

[vzhurba01]

Thanks for the report! I've pushed a fix a6511d5 to main branch and it can be installed from source.

The PyPi/Conda packages will receive the fix in the next release.

FYI for the code snippet you likely want to pass the device_num instead of device. Here the device_num is a device ordinal whereas device is a device handle. Their integer representation may not always match.
(in fact cudaSetDevice internally calls cuDeviceGet on the passed deviceOrdinal to get a device handle)

[QuiteAFoxtrot]

Great, thank you! A question regarding your device_num advice - does that hold up under CUDA_VISIBLE_DEVICES? For example, if CUDA_VISIBLE_DEVICES = 4,5,6,7 and I set device_num=2, presumably that actually gives me a pointer to device 6 - so if I passed a "2" into cuda.cuCtxCreate, will that also properly map to device 6?

I've also noticed some strange behavior with cuda.cuDeviceCanAccessPeer - namely if you pass in the same device twice it reports that access is not possible (which I interpret as, it can't map its own memory?). Is that intentional behavior? If so, would you like me to open another issue?

[vzhurba01]

does that hold up under CUDA_VISIBLE_DEVICES?

It holds up. Your following example works just like that.

which I interpret as, it can't map its own memory?

That's expected, peer-to-self is disallowed.