Distributed MPI simulation: cudaErrorInvalidResourceHandle about cuquantum

[jagunnels]

Given the effort that you put in to creating a minimal reproducible example, I would suggest that you contact support for this system. Generally, on a large system, the support team would be thrilled to see a ~10 line reproducer. Please let us know what they say (or if that path is not fruitful).

[mtjrider]

Hi @danlkv ... you're using Polaris, right? I believe this system runs SS11 with a special CUDA-aware MPICH. Referencing this.

I'd make sure that the correct MPI implementation is being loaded by mpi4py.

I'm making a few assumptions about which compute resource you're using, but keep us looped in with what their support teams say.

[danlkv]

Thanks for getting back to me. Yes, I'm using a single node of Polaris for these tests. I did verify that the data after allgather is correct. Does ipcOpenMemHandle use MPI? I thought this was a custom Nvidia p2p communication protocol.

[DmitryLyakh]

@danlkv No, ipcOpenMemHandle does not rely on MPI. Can you build and run the standard CUDA IPC sample on Polaris? https://github.com/NVIDIA/cuda-samples/tree/master/Samples/0_Introduction/simpleIPC

[mtjrider]

Thanks @DmitryLyakh

@danlkv there are two things happening:

1. You're trying to create IPC handles.
2. You're passing the IPC handles to MPI.

(2) would only work if the MPI implementation loaded by mpi4py is CUDA aware (unless data is being moved to the host prior to the actual allgather collective).

Regarding (1), you'd need to confirm that all GPUs are actually peer accessible, and that there isn't an issue with your cupy installation.

[danlkv]

@DmitryLyakh I can build the sample using cudatoolkit 11.8, but not smaller versions.

nvcc fatal   : Unsupported gpu architecture 'compute_89'

I did not previously use the 11.8 cuda, since I'm not sure it is compatible with the device (Driver cuda version 11.4). I checked compute capability with deviceQuery example compiled with 11.8 cuda:

-> % ./deviceQuery 
./deviceQuery Starting...

 CUDA Device Query (Runtime API) version (CUDART static linking)

Detected 1 CUDA Capable device(s)

Device 0: "NVIDIA A100 80GB PCIe"
  CUDA Driver Version / Runtime Version          11.4 / 11.8
  CUDA Capability Major/Minor version number:    8.0
  Total amount of global memory:                 80995 MBytes (84929216512 bytes)
[...]

Does 8.0 means that it will not be compatible with 8.9? Should I try to run everything with 11.8 or try to change the Makefile so it uses a smaller compute capability?

P.S. samples make doesn't look in PATH for nvcc, so I had to export CUDA_PATH to make it work.

[DmitryLyakh]

If you run on A100 GPU, you only need compute capability 8.0, and you can use CUDA-11.8. You do not need compute capability 89 in this case (you can just remove those flags). I would recommend trying with CUDA-11.8 regardless.

[danlkv]

Update: the problem just magically disappeared and everything now works on all cuda versions. I now use 11.4 and checked it up to 64 nodes. I guess there was some problem on ALCF side.

[mtjrider]

Update: the problem just magically disappeared and everything now works on all cuda versions. I now use 11.4 and checked it up to 64 nodes. I guess there was some problem on ALCF side.

Thanks for keeping us posted.