[QUESTION] Keep getting scores of '0' no matter what input used about comet

[awaelchli]

I left a reply in Lightning-AI/pytorch-lightning#19537 (comment) with a suggestion. I hope it provides some useful insights.

[ricardorei]

Hey @Brecony76. I am not able to replicate this error. I just tried it and I get the following scores:

Prediction([('scores', [0.8417137265205383, 0.7745385766029358]), ('system_score', 0.8081261515617371)])

[clang88]

Hi @Brecony76 I'm observing the same issue.

• OS: Windows 10
• unbabel-comet 2.2.1
• pip 23.3.1
• Python 3.10.13
• torch 2.1.2+cu121
• Geforce 250MX (Driver Version: 537.79 CUDA Version: 12.2) (Yeah... it's my work laptop)

The behavior is particularly odd, because sometimes it does actually return a score, with no change in code or data... I'm not sure how to reproduce the 0.0 scores, nor the proper scores. Sometimes it just works, sometimes it doesn't. I will retest this tomorrow, to see if I can make any sense of it. For now I completed my task of evaluating some translations with Comet (thanks to the devs and researchers for making this so intuitive!)

[BramVanroy]

I can confirm that this issue exists on Windows. It might be related to this CUDA warning:

[W CudaIPCTypes.cpp:16] Producer process has been terminated before all shared CUDA tensors released. See Note [Sharing CUDA tensors]

But I am not sure and do not have time to dig into this deeper. It is a shame though, as this makes COMET unfortunately unreliable on Windows.

[BramVanroy]

I've done some digging but haven't found a solution, although I have pinpointed the place in the PL Trainer where something goes wrong. The model weights are turned to zero but I do not know why.

To put this into higher priority, feel free to comment on the issue that I raised over at PyTorch Lightning to indicate that you are also experiencing this problem. Lightning-AI/pytorch-lightning#19537