EC pretraining prediction accuracy collapses after achieving 99+% accuracy. about ecnmt

[yaoyiran]

Hi, here is a simple solution: (1) set the learning rate as 1e-4 rather than the default 1e-3; (2) set a minimum message length in models.py, e.g. replacing line 26 "if idx_ == 0:" with , say, "if idx_ <= 5:", "if idx_ <= 10:" or "if idx_ >= 0:".

PS. To reproduce the results in our paper, you can just use the default hyper-parameters and stop the training process when you get a model whose EC accuracy is satisfactory.

[soumye]

I was indeed able to get the EC accuracy to a satisfactory 99%, which I guess is good enough for pre-training. But the instability of the model to maintain that and the eventual drift I think highlights the brittleness of the training. I was wondering if there was a fix to prevent the model from collapsing later on.

[yaoyiran]

I've already mentioned that such a "fix" really exists: you could just set the lr = 1e-4 and set a minimum message length. Here is my training log with the new hyper-parameters, where the EC prediction accuracy on our valid set is >= 99.95% and super stable from the beginning to epoch 251000 and I expect it to be stable afterwards. Please feel free to leave a comment if you still cannot get a stable training process.

log.train.txt