Train loss increasing during training about zero-transformer

[fattorib]

As a first debugging step, will train a much smaller model (~50M) param in multihost setting to see if same issues arise. In theory, if everything is working this model shouldn't overfit

[fattorib]

50M run for 4 epochs on pod completed, this training curve looks as expected. Given that this run only took around 12 hours, I didn't have to restart training at any point, Need to investigate code for resuming runs next.

For reference, commit 80825b39bb20256f7df48f70537b74efc47e1c67.

[fattorib]

Could also be that this is the result of some issues with the data/model scale? Following OPT, will try and log l2 norm of final layer activations, they found this to be good indicator for loss divergences. I'm a bit skeptical this is what it would be, but working on this is still good practice

[fattorib]

Made a small change to how data resuming is handled:

• Splitting out rng keys while resuming in dataset. Before, rng keys weren't split out when resuming from a previous ckpt, meaning rng state was reused.

Confirmed that this works properly by training a 125M model and forcing a resume about 50% through training. Model trained properly from this ckpt and loss didnt diverge. Considering this issue closed for now

[fattorib]

Turns out there was an odd duplication bug with the webtext data. Remade dataset from scratch, will see how new models perform