Add DeepSpeed Example with MPI Operator about training-operator

[tenzen-y]

Sure, that sound great @tenzen-y! It would be great to see the benchmarks for mpirun and torchrun to run DeepSpeed on Kubernetes.

It sounds great, but I guess that there are no significant performance differences between both approaches since the deepspeed uses the NCCL backend even if we use mpirun.

[google-oss-prow]

@andreyvelich:
This request has been marked as suitable for new contributors.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-good-first-issue command.

In response to this:

Related: #2040

As we discussed multiple times, Kubeflow community are looking for examples on how to use MPI Operator and DeepSpeed.

We should add some example to the MPI Operator: https://github.com/kubeflow/mpi-operator/tree/master/examples/v2beta1 or Training Operator: https://github.com/kubeflow/training-operator/tree/master/examples.

Some pending PRs can be found here as reference:

• kubeflow/mpi-operator#610
• kubeflow/mpi-operator#567

/good-first-issue
/help
/area example

/cc @alculquicondor @kubeflow/wg-training-leads @kuizhiqing

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[tenzen-y]

I believe that both (training-operator and mpi-operator) examples would be worth it. But, I think that we should add each example for PyTorchJob with deepspeed and torchrun, and MPIJob v2 with deepspeed and mpirun.

[andreyvelich]

Sure, that sound great @tenzen-y!
It would be great to see the benchmarks for mpirun and torchrun to run DeepSpeed on Kubernetes.

[vsoch]

I'm working on an equivalent example for the Flux Operator - but quick question. Will it work OK to test without GPU? I've been trying to get just 3 nodes, each with one nvidia GPU on Google Cloud, and I never get the allocation.

[vsoch]

Ah - this looks more promising. https://github.com/kubeflow/mpi-operator/pull/567/files

[andreyvelich]

@tenzen-y Does DeepSpeed only support nccl backend ? E.g. we can't run it with CPUs ?

[tenzen-y]

@tenzen-y Does DeepSpeed only support nccl backend ? E.g. we can't run it with CPUs ?

TBH, I don't have any experience only with CPU. But at the first glance, the deepspeed seems to support PyTorch without GPU: https://github.com/microsoft/DeepSpeed/blob/master/.github/workflows/cpu-torch-latest.yml

[kuizhiqing]

that there are no significant performance differences between both approaches since the deepspeed uses the NCCL backend even if we use mpirun.

This statement is generally correct in almost all cases with NCCL context. Though, I've two few experiences to share for those using mpi-style set-up and suffering performance issue.

• Avoid using mpi library(mpi4py e.g.) to do any collective communication, for some metrics collection etc., even if they only using TCP network.
• DO NOT produce too much logs in mpirun scenario, the mpirun would collect logs from all the workers which may hurt the performance

Overall, mpirun and torchrun should have no performance difference.