training resumption reverts to `bert-base-cased` rather than the provided checkpoint about colbert

[okhat]

There are three relevant flags:

• --resume (continue from data point)
• --checkpoint (continue from model parameters)
• --resume_optimizer (continue from optimizer parameters)

When you pass the first two, the model is in fact resumed but not the optimizer. This leads to seemingly high loss for a number of steps until the optimizer catches up.

[okhat]

The --resume_optimizer flag is NOT implemented in the public repository (here), but I have an internal implementation that I can merge if you need it. Let me know.

[JamesDeAntonis]

that would be great, thanks!

[okhat]

Great --- I think I can do this Friday or Saturday.

[wonderingalways]

Hey!
Is the above optimizer reload code merged? If so I am unable to see the changes? If it wont be official merged can I get the internal implementation through mail?

[tdieu29]

Even after loading the optimizer state dict from my previous run, resuming training still did not give me the average loss that I ended up with from the previous run. @JamesDeAntonis If you resolved this problem, could you please share how you did it? Thanks a lot!

[tdieu29]

Found the solution! For anyone who might encounter this problem, what I did was saving train_loss (in addition to saving the model state dict, optimizer state dict and batch idx) when saving the checkpoints! You need the train_loss from the previous run when you resume training if you want the average loss from the current run to start at where you left off.