Script crashes when evaluating merged discontinuous mentions about craft-shared-tasks

[bill-baumgartner]

Dear Khoa,
Thank you for your very complete error report. I will look into this and get back to you with a fix.
Best,
Bill

[bill-baumgartner]

Dear Khoa,

Thank you again for pointing out this issue. I was able to reproduce the error thanks to your very detailed issue report. As you observed, the issue is a results of some improperly formatted discontinuous spans. They are a result of errors in the original coreference annotation. I have updated the evaluation code to address this issue. There is now a validation step at the beginning of the coreference evaluation process that will check both the gold standard and test files for proper discontinuous spans.

Discontinuous spans are validated in two ways. First, if a discontinuous span contains adjacent component spans, e.g. [35..43][44..52], or component spans are are separated by only whitespace, then the component spans are combined, e.g. [35..52]. Second, if the discontinuous span contains a component span that is nested in another component span, e.g. [78..92][88..92], then the nested span is removed, e.g. [78..92].

While fixing issues with the discontinuous spans, I was also able to address an issue whereby annotations of type "nonreferential pronoun" were being output as part of the CoNLL-Coref file format. These annotation types should not have been included, and are now excluded from the output.

I've released an update to the CRAFT corpus that includes the fix to address the issue you reported. Please update to CRAFT v3.1.3, and to the 0.1.2 version of this project. Or, if you are running the evaluations via Docker, please use the ucdenverccp/craft-eval:3.1.3_0.1.2 container which is now available on DockerHub.

Let me know if you continue to experience this issue, or if you have further questions.

Best,
Bill

[dnanhkhoa]

Dear Bill,

Thank you so much for your time and effort to fix this issue. I will rerun our model on the latest evaluation script and will let you know if there is any problem. Thank you.

Best regards,
Khoa

[dnanhkhoa]

Now, the latest version works perfectly. Thanks.