The weight update results of torch.nn.BatchNorm1d on different hardware are inconsistent about pytorch

[ZhaoqiongZ]

Hi @PhyllisJi , I tried with 2.4.0.dev20240526+cu118 and not reproducible
here is what I got:

conv1_mutated.weight
3.1925738e-06
conv1_mutated.bias
5.509591e-08
bn1.weight
8.106232e-06
bn1.bias
1.758337e-05
conv2_mutated.weight
3.2205135e-06
conv2_mutated.bias
0.000114182476
bn2.weight
7.212162e-06
bn2.bias
1.000613e-05
conv3_mutated.weight
4.3958426e-06
conv3_mutated.bias
2.1457672e-06
bn3.weight
2.1976884e-06
bn3.bias
1.821667e-06
conv4_mutated.weight
1.5348196e-06
conv4_mutated.bias
1.937151e-07
bn4.weight
1.3131648e-06
bn4.bias
1.7601997e-07
conv5_mutated.weight
7.171184e-07
conv5_mutated.bias
4.0512532e-08
bn5.weight
1.1497177e-06
bn5.bias
3.5203993e-07
linear1.weight
4.172325e-07
linear1.bias
2.719462e-07
bn6.weight
3.3900142e-07
bn6.bias
1.1175871e-08
tail_fc.weight
3.4831464e-07
tail_fc.bias
2.2351742e-08

[colesbury]

Please see https://pytorch.org/docs/stable/notes/numerical_accuracy.html