Can ranks use different group patterns when calling P2P communication? about nccl

[sjeaugey]

Ah. Good point. Indeed, in that case, as we're creating connections lazily, one side will try to connect both the send and recv connection, while the other side will only try to connect one, causing a mismatch in the amount of metadata they send to each other.

[sjeaugey]

Yes it should be considered as undefined.

This would probably work (at this small scale), but in general, it is not safe to assume that all calls within a group will progress without blocking each other.

If rank 0 was communicating with hundreds or other ranks, then its recv operation could be blocked by the send operation, which would block waiting for the Recv on rank 1.

[constroy]

Thanks for the explanation!
Actually, in my experiment, not only is blocking observed, but also NCCL error.
(the experiment is conducted using PyTorch)

torch.distributed.DistBackendError: NCCL error in: /root/pytorch/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:3608, internal error - please report this issue to the NCCL developers, NCCL version 2.20.5
ncclInternalError: Internal check failed.
Last error:
Message truncated : received 4096 bytes instead of 2048

I guess that the two ranks are using different communication protocols (or schemes) according to group patterns. Is that right?

[constroy]

Very clear explanation. Thank you very much.