Error when using `Trainer.compile_config={}` in DDP mode about composer

[Ghelfi]

On torch 2.3, adding torch._dynamo.config.optimize_ddp = False at the start of the file seems to fix it.

I am having issue with DDP and torch.compile on other leads also. I'll keep investigating.

[mvpatel2000]

@Skylion007 do you think this is a torch error or something we can do differently?

[mvpatel2000]

@Ghelfi do you know if this works for you elsewhere, e.g. if you compile outside Composer? Will help us narrow down if its a Composer issue or PyTorch issue, as the trace looks more like a Pytorch issue to me

[Ghelfi]

This is not clear to me. The provided example above works if you remove the BlurPool algorithm, which is only on the composer side.

I'll try to redefine some model layer before feeding it to the trainer to mimic the behaviour outside of any composer scope.