[QUESTION] About HTER models in download list. about comet

[ricardorei]

Hey @ElliottYan
I was planning to remove those as they were outperformed by the DA models but I can add them back!

For now, you will have to resort to the previous version... Those checkpoints still depend on fairseq but I have a script to convert the models into the new codebase (with transformers)

[ElliottYan]

I see. Thanks for your reply!

As I need a bounded score, I will try the previous version for now, but I would really appreciate it if you add them back to the current one.

In addition, I see you explain about HTER models are not as good as DA ones. Can I ask how bad are we talking about? Is there any result reported that I missed?

Thanks

[ricardorei]

we are not talking about a huge performance decrease but its usually a few points behind. You can look for example at the WMT20 Metrics task findins where we submitted several COMET versions. I also know that the authors of the To Ship or Not to Ship compared the DA model with the HTER model. The HTER model was even outperformed by the reference-less DA model (wmt20-comet-qe-da)

[ricardorei]

Btw the wmt20-comet-qe-da is also bounded with a sigmoid. If you only care about having a bounded score this one might be useful.

I'll add the HTER models meanwhile.

[ElliottYan]

Oh, great! The "wmt20-comet-qe-da" model also performs very well in the paper, and I will try that.
Thanks!