Can't reproduce Cometinho model scores about comet

[ricardorei]

Hi @Rexhaif I think your problem is on the models you are using. Have you tried these ones?

• eamt22-cometinho-da
• eamt22-prune-comet-da

The confusion is that the "first" versions of cometinho were actually trained for WMT 21. They are not distilled versions of larger COMET models but rather smaller encoders trained with the same data. Then for EAMT Conference we experimented with distillation of larger models (which allowed us to train the smaller encoder in much more data) those are the results presented in COMETINHO paper.

Sorry about the confusing names. I hope with this you are able to reproduce the results

[Rexhaif]

Hi, thanks for the clarification and the model names!

Are there any distilled/pruned COMETINHO models available that were trained on the MQM scores rather than the DA scores?

[ricardorei]

Unfortunately no.