Question on `sqrt(max(var, eps))` about synchronized-batchnorm-pytorch

[vacancy]

That's a really good question. I think I have tested both. My test result was (based on a very old pytorch implementation ~0.3 I think),
using max(var, eps) outputs closer results (the output or the gradient) to the PyTorch's batchnorm implementation than var + eps.

Maybe it's time to do some new tests.

[nmhkahn]

Thanks for sharing your experience.
I observed that switching to sqrt(var+eps) results in slightly better performance on some image-to-image translation methods. But not 100% sure since I didn't fix the random seed nor deeply dug in the result.

Close this issue but let me know if you get some new results 👍