Why does redecescatter Ring algo's result is better than NVLS's result in one H100 server? about nccl

[KaimingOuyang]

Because algorithm-wise, NVLS reduce_scatter transfers one more chunk to itself which causes lower perf compared with Ring.

[haobaba5353]

Thanks for your reply. From my understanding, NVLS can accelerate the collective communication and the message should be sent to the NVSwitch only once. So why NVLS reduce_scatter transfers one more chunk to itself?

[KaimingOuyang]

because load-reduce involves all ranks on the node, a rank needs to send its own data to switch so the total data each rank sends is nRanks * count; however, Ring only sends (nRanks - 1) * count in total which makes it faster.

[yanminjia]

From the kernel function in reduce_scatter.h, the collective communication ReduceScatter is performed by scatter (prims.scatter) and recv (prims.recv). I guess the data on each GPU/Rank is scattered and reduced in different intra-node NVSwitches and then sent back to the GPU/Ranks. If my logic is correct, to complete ReduceScatter, with respect to the GPU/Rank, NVLS needs only once send and once receive. By the way, could you please let me know how nvls->down and nvls->up is used in the kernnel function? It looks the values of the elements in nvls->down and nvls->up are greater than comm->nranks. Thanks.

because load-reduce involves all ranks on the node, a rank needs to send its own data to switch so the total data each rank sends is nRanks * count; however, Ring only sends (nRanks - 1) * count in total which makes it faster.