mpi + nccl hangs in muti-process scenarios about nccl

[Yangqing]

FWIW - we found when developing Caffe2 that if you happen to have multi-threading going on and one thread tries to call cudaMallocHost() or cudaFree, then that causes nccl to freeze. Essentially, what helped us solving the problem is to guard all malloc, free and nccl calls so they don't overlap. That needed some good care (thanks to @akyrola ). An example can be seen here:

https://github.com/caffe2/caffe2/blob/master/caffe2/core/context_gpu.cu#L284

[sjeaugey]

First, are you running all the processes on the same node ?

In any case, you should not need a barrier in the case of MPI, only with multiple threads. A small code sample to reproduce could help.

[hiyijian]

Yes,it happens on a single node.
The hang project is caffe with ncclallreduce during backward. I will try to reproduce it with a small demo.
thanks.

[hiyijian]

hi @Yangqing. Thank you very much. I am working on our own branch of a very old version of caffe, with OpenMPI Integrated. In other words, our case is multi-process on a single node, one GPU per process. I think no thread-based lock is needed. Am I right?

[hiyijian]

ps: it hangs every 1 day or 1.5 days. I have to keep training with snapshot after freezon. The trained moel performance is quite good

[sjeaugey]

Closing old issue. Please re-open if you still have issues with 2.3.