Mismatch between error span and offsets about comet

[danieldeutsch]

I did find examples where the offsets and text differ beyond leading whitespace, and I think it's due to weird characters in the input:

{
    "src": "高龄老人坐着面包车从南方到东北看雪？？",
    "mt": "The elderly are watching the snow from the south to the northeast in a van? ？",
    "ref": "An elderly goes to watch the snow in the northeast from the south in a minibus? ?",
    "COMET": 0.8957364559173584,
    "errors": [
        {
            "text": "ly are watching",
            "confidence": 0.39867228269577026,
            "severity": "minor",
            "start": 9,
            "end": 24
        },
        {
            "text": "to",
            "confidence": 0.3982390761375427,
            "severity": "minor",
            "start": 48,
            "end": 51
        },
        {
            "text": "van??",
            "confidence": 0.4210003614425659,
            "severity": "minor",
            "start": 70,
            "end": 77
        }
    ]
}

The second question mark in the mt text is a weird character. It gets normalized in the span text to a normal quote.

{
            "src": "There's a circularity to it...",
            "mt": "Darin besteht eine Zirkularität …",
            "ref": "Das ist ein richtiger Kreislauf...",
            "COMET": 0.9912000298500061,
            "errors": [
                {
                    "text": "Darin besteht eine Zirkularität...",
                    "confidence": 0.5221754312515259,
                    "severity": "minor",
                    "start": 0,
                    "end": 33
                }
            ]
        }

In the mt text, there's a space followed by 3 periods as a single character, but the span text removes the whitespace and uses 3 separate period characters.

I assume the offsets are ok to use. I only ran into this because I was verifying that COMET doesn't change the source text when it makes span predictions. This is important when you evaluate the spans so you can directly compare the predicted spans to the MQM spans. If the text is edited, the mapping might not be correct.

[ricardorei]

Hi @danieldeutsch, you are right! the offsets are correct and the "text" field is more informative. I get the text field by detokenizing the token ids belonging to a span. Yet, if you detokenize just a part of the original input, you might get slightly different output. Whitespaces are good examples as they are sometimes encoded with the suffix _