Reimplementing results in COMET EMNLP'20 paper about comet

[ricardorei]

Hi @wanyu2018umac,

The naming of the file test19-relative-ranks is a bit misleading. That file is just a random split that I did to validate data from 2019 without having to run the entire corpus from that year. To reproduce the results you have to use the entire en-de data from 2019.

The emnlp-base-da-ranker is trained on data from 2017 and 2018 only and the reported results use the entire 2019 data.

We do have a script to reproduce the result https://github.com/Unbabel/COMET/blob/master/wmt-shared-task/segment-level/segment_level_comet.py

You just need to download the data as you did and download the checkpoint locally also. Then it should be easy to run

[ricardorei]

Please send me an email if you have any problems reproducing the results (I am usually faster at replying to emails).

[wanyu2018umac]

Ah I see it. It's a little misleading though. Thanks for your response!