Can BN be applied within DistributedDataParallel (DDP)? about moco

[KaimingHe]

The running_mean and running_std are not used by training; they are only used for inference. The motivation of having running_mean and running_std is to have an approximate estimation of the statistics of the training samples. This estimation can happen in any sequence of random samples, and when the model converges, this estimation should tend to be the same regardless of the sequence. So it is not necessary to sync the running_mean and running_std; it is sufficient to use them from any of the GPUs.

[KaimingHe]

What you described is Sync SGD; it has nothing to do with SyncBN. BN can be applied with DDP training. DDP training is by default Sync SGD training.

[Ze-Yang]

As I know, the original BN only compute the mean and variance for each process (GPU) separately and does not communicate accross GPUs. That's why SyncBN arises to enable the mean and variance being computed on all batch samples.

In detectron2, defaults.py L457-L458 allow us to choose BN from (FrozenBN, GN, "SyncBN", "BN"). Here I focus on "BN" case. As you mention that Sync SGD can ensure the gradients of the affine weights and bias will be synchronized across GPUs. But what about the running_mean and running_var? Note that they do not belong to model weights and have nothing to do with SGD. Will they also be synchronized in original "BN"? Thanks.

[Ze-Yang]

Thanks a lot for rectifying my misunderstanding of BN. Originally, I thought that the running_mean and running_var were also used in training phase, which confused me a lot.