Simple PyTorch Distributed Training (Multiple Nodes)
Preparations
Do the following steps on all nodes:
Clone Repo
git clone https://github.com/lambdal/pytorch_ddp
cd pytorch_ddp
Download the dataset on each node before starting distributed training
mkdir -p data
cd data
wget -c --quiet https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz
tar -xvzf cifar-10-python.tar.gz
rm cifar-10-python.tar.gz
cd ..
Creating directories for saving models before starting distributed training
mkdir -p saved_models
Run
Do the following steps to run 2x Nodes distributed training (3xGPUs per node)
Node One
NCCL_DEBUG=INFO NCCL_ALGO=Ring NCCL_NET_GDR_LEVEL=4 python3 -m torch.distributed.launch \
--nproc_per_node=3 --nnodes=2 --node_rank=0 \
--master_addr="xxx.xxx.xxx.xxx" --master_port=1234 \
resnet_ddp.py \
--backend=nccl
Node Two
NCCL_DEBUG=INFO NCCL_ALGO=Ring NCCL_NET_GDR_LEVEL=4 python3 -m torch.distributed.launch \
--nproc_per_node=3 --nnodes=2 --node_rank=1 \
--master_addr="xxx.xxx.xxx.xxx" --master_port=1234 \
resnet_ddp.py \
--backend=nccl
Note: backend
options are nccl
, gloo
, and mpi
. "By default for Linux, the Gloo and NCCL backends are built and included in PyTorch distributed (NCCL only when building with CUDA). MPI is an optional backend that can only be included if you build PyTorch from source". More details can be found here
Credit
The resnet_ddp.py
script is written by Lei Mao.