How to set the environment variables for all_to_all_perf profiling? about npkit

[yzygitzh]

Hi,

For alltoall, I would recommend using

-DENABLE_NPKIT_EVENT_SEND_ENTRY
-DENABLE_NPKIT_EVENT_SEND_EXIT
-DENABLE_NPKIT_EVENT_RECV_ENTRY
-DENABLE_NPKIT_EVENT_RECV_EXIT

for GPU events, and

-DENABLE_NPKIT_EVENT_NET_SEND_ENTRY
-DENABLE_NPKIT_EVENT_NET_SEND_EXIT
-DENABLE_NPKIT_EVENT_NET_RECV_ENTRY
-DENABLE_NPKIT_EVENT_NET_RECV_EXIT

for net events.

[Chen-Chang]

I have tested the settings for GPU events, but the json output is "{"traceEvents": [], "displayTimeUnit": "ns"}" and npkit_dump dir is empty.
Should I change the NCCL_PROTO and NCCL_ALGO settings?

[yzygitzh]

There should always be files in NPKIT_DUMP_DIR. Which NPKit version did you use?

[Chen-Chang]

NPKit_NCCL

[yzygitzh]

Got it. That version only LL and LL128 protocol are supported for those events.

We recommend using the NPKit for MSCCL instead of NPKit for NCCL because the latter is no longer actively maintained. MSCCL and NPKit for it are being actively developed, and MSCCL in terms of functionality is a superset of NCCL.

[Chen-Chang]

Thanks a lot! Can I profile the trace file of the original nccl implementation in msccl framework?

[yzygitzh]

It’s better to process the trace files using their corresponding scripts.

[Chen-Chang]

I have changed the NCCL_PROTO from "Simple" to "LL". The output trace of NET events is normal, but there are some errors when generating GPU events.