Abnormal Loss curve about zero123

[ruoshiliu]

I can't really help you without any additional information... Can you give more details?

[cwchenwang]

Thanks for your reply. I just trained with the default config except setting gradient accumulation to 2. Now I have trained for about 20k iterations, and the loss is about 0.021. Just not sure if this is normal since the loss curve is fluctuating like the image I show above

[ruoshiliu]

I think loss step is supposed to pretty unstable but loss_epoch should consistently decrease with much smaller variance than loss_step.