OOM when training the same size of gpt2(2.6B) with mp=2 dp=8 with 64*V100(32GB) about megatron-lm

[Hanlard]

I have used following techniches to save memory...
--fp16
--checkpoint-activations
--distribute-checkpointed-activations
--fp16-lm-cross-entropy
--use-cpu-initialization

[Hanlard]

This is 16 * V100(32G) for debug...

[Hanlard]

I find that if add "torch.cuda.empty_cahce()" at every train_step then paragram will not OOM ...

[eric-haibin-lin]

Did you intentionally choose MP=2? Would increasing MP help? (e.g. MP=4, DP=4)

[Hanlard]

Did you intentionally choose MP=2? Would increasing MP help? (e.g. MP=4, DP=4)
increasing MP will help...

[Hanlard]

Did you intentionally choose MP=2? Would increasing MP help? (e.g. MP=4, DP=4)

I solve the problem by changing DDP_impl = “local” to ”torch” ...

[gbxu]

In the readme it says that "2.5 billion parameters using 2-way model parallel and 1.2 billion parameters with no model parallel".
It's confusing that splitting the model into 2 shards takes more GPU memory. Assuming that the original model size as M, after 2-way model parallel each GPU should hold M/2.
Does Anyone know what other GPU footprint comes from with setting DDP_impl “local” and ”torch” respectively? Dose the communication library pin M or more GPU memory?
Thanks.

[ShivanshuPurohit]

I am getting out of memory errors with mp>1, which all disappear if I just use mp=1, with the deepspeed megatron enabled.

[github-actions]

Marking as stale. No activity in 60 days. Remove stale label or comment or this will be closed in 7 days.

[github-actions]

Marking as stale. No activity in 60 days.