Multi GPU training problem about di-engine

[zqh0253]

The file demo/simple_rl/ppo_train.py is in DI-drive repo

[PaParaZz1]

You can refer to this demo and add DistContext in your program

[zqh0253]

Thanks for your suggestion. But I add DistContext and get the following error:

WARNING:root:If you want to use numba to speed up segment tree, please install numba first
pygame 1.9.6
Hello from the pygame community. https://www.pygame.org/contribute.html
Traceback (most recent call last):
File "imgppo_train.py", line 189, in
with DistContext():
File "/home/qhzhang/code/DI-engine/ding/utils/pytorch_ddp_dist_helper.py", line 114, in enter
dist_init()
File "/home/qhzhang/code/DI-engine/ding/utils/pytorch_ddp_dist_helper.py", line 81, in dist_init
raise RuntimeError("please indicate rank explicitly in dist_init method")
RuntimeError: please indicate rank explicitly in dist_init method

[PaParaZz1]

Are you run multi-gpu program in your own machine? And I guess you should prepare related environment variables here.

We test this parts in slurm before so it would be a bit inconvenient in local machine. And you can add your own advice for multi-gpu training in local machine.

[zqh0253]

I am running multi-gpu program in my own machine so your code based on slurm environment does not work.
I have gone through the relevant code about multi-gpu part. I think using torch.distributed.launch may help user to use your multi-gpu code in local machine.
But another problem for me is that I am actually using DI-drive and one di-drive program will lauch several (like 8) carla clients. When using torch.distributed.launch, the carla clients will be launched multiple times (like 8*N, the N is gpu num).
So I find a quick fix using torch.DataParallel to do multi-gpu training without your API.

[PaParaZz1]

OK, we will support torch.DataParallel in next version, for some cases like you encountered. And you can pay attention to roadmap for further update.

[zxzzz0]

We have this same error today when running on a single machine with 2 GPUs.
raise RuntimeError("please indicate rank explicitly in dist_init method")

@PaParaZz1

[WangJuan6]

Hi, @PaParaZz1
I have the same error, can you provide a demo of how to use multi-GPU in the custom environment?
Thanks