ucdenver-ccp / craft-shared-tasks Goto Github PK
View Code? Open in Web Editor NEWThis repository hosts evaluation scripts for evaluating annotations of various types against the CRAFT corpus.
License: Eclipse Public License 1.0
craft-shared-tasks's People
Contributors
Stargazers
Watchers
craft-shared-tasks's Issues
Coreference prediction evaluation in sanity checking your results by self-evaluation via Docker
Hi, when I try to evaluate my coreference prediction result in test dataset via docker(option 2), I get a strange output, as shown below. The metrics are either 0 or 1. I have that confusion because surely my prediction is not 100% correct... Do you know why would that happen? Thank you so much for your help.
#document-id :bcub-mention-tp :bcub-mention-fp :bcub-mention-fn :bcub-mention-p :bcub-mention-r :bcub-mention-f :bcub-coref-tp :bcub-coref-fp :bcub-coref-fn :bcub-coref-p :bcub-coref-r :bcub-coref-f :bcub-allow_partial-mention-tp :bcub-allow_partial-mention-fp :bcub-allow_partial-mention-fn :bcub-allow_partial-mention-p :bcub-allow_partial-mention-r :bcub-allow_partial-mention-f :bcub-allow_partial-coref-tp :bcub-allow_partial-coref-fp :bcub-allow_partial-coref-fn :bcub-allow_partial-coref-p :bcub-allow_partial-coref-r :bcub-allow_partial-coref-f :blanc-mention-tp :blanc-mention-fp :blanc-mention-fn :blanc-mention-p :blanc-mention-r :blanc-mention-f :blanc-coref-tp :blanc-coref-fp :blanc-coref-fn :blanc-coref-p :blanc-coref-r :blanc-coref-f :blanc-non-coref-tp :blanc-non-coref-fp :blanc-non-coref-fn :blanc-non-coref-p :blanc-non-coref-r :blanc-non-coref-f :blanc-score :blanc-allow_partial-mention-tp :blanc-allow_partial-mention-fp :blanc-allow_partial-mention-fn :blanc-allow_partial-mention-p :blanc-allow_partial-mention-r :blanc-allow_partial-mention-f :blanc-allow_partial-coref-tp :blanc-allow_partial-coref-fp :blanc-allow_partial-coref-fn :blanc-allow_partial-coref-p :blanc-allow_partial-coref-r :blanc-allow_partial-coref-f :blanc-allow_partial-non-coref-tp :blanc-allow_partial-non-coref-fp :blanc-allow_partial-non-coref-fn :blanc-allow_partial-non-coref-p :blanc-allow_partial-non-coref-r :blanc-allow_partial-non-coref-f :blanc-allow_partial-score :ceafe-mention-tp :ceafe-mention-fp :ceafe-mention-fn :ceafe-mention-p :ceafe-mention-r :ceafe-mention-f :ceafe-coref-tp :ceafe-coref-fp :ceafe-coref-fn :ceafe-coref-p :ceafe-coref-r :ceafe-coref-f :ceafe-allow_partial-mention-tp :ceafe-allow_partial-mention-fp :ceafe-allow_partial-mention-fn :ceafe-allow_partial-mention-p :ceafe-allow_partial-mention-r :ceafe-allow_partial-mention-f :ceafe-allow_partial-coref-tp :ceafe-allow_partial-coref-fp :ceafe-allow_partial-coref-fn :ceafe-allow_partial-coref-p :ceafe-allow_partial-coref-r :ceafe-allow_partial-coref-f :ceafm-mention-tp :ceafm-mention-fp :ceafm-mention-fn :ceafm-mention-p :ceafm-mention-r :ceafm-mention-f :ceafm-coref-tp :ceafm-coref-fp :ceafm-coref-fn :ceafm-coref-p :ceafm-coref-r :ceafm-coref-f :ceafm-allow_partial-mention-tp :ceafm-allow_partial-mention-fp :ceafm-allow_partial-mention-fn :ceafm-allow_partial-mention-p :ceafm-allow_partial-mention-r :ceafm-allow_partial-mention-f :ceafm-allow_partial-coref-tp :ceafm-allow_partial-coref-fp :ceafm-allow_partial-coref-fn :ceafm-allow_partial-coref-p :ceafm-allow_partial-coref-r :ceafm-allow_partial-coref-f :lea-mention-tp :lea-mention-fp :lea-mention-fn :lea-mention-p :lea-mention-r :lea-mention-f :lea-coref-tp :lea-coref-fp :lea-coref-fn :lea-coref-p :lea-coref-r :lea-coref-f :lea-allow_partial-mention-tp :lea-allow_partial-mention-fp :lea-allow_partial-mention-fn :lea-allow_partial-mention-p :lea-allow_partial-mention-r :lea-allow_partial-mention-f :lea-allow_partial-coref-tp :lea-allow_partial-coref-fp :lea-allow_partial-coref-fn :lea-allow_partial-coref-p :lea-allow_partial-coref-r :lea-allow_partial-coref-f :muc-mention-tp :muc-mention-fp :muc-mention-fn :muc-mention-p :muc-mention-r :muc-mention-f :muc-coref-tp :muc-coref-fp :muc-coref-fn :muc-coref-p :muc-coref-r :muc-coref-f :muc-allow_partial-mention-tp :muc-allow_partial-mention-fp :muc-allow_partial-mention-fn :muc-allow_partial-mention-p :muc-allow_partial-mention-r :muc-allow_partial-mention-f :muc-allow_partial-coref-tp :muc-allow_partial-coref-fp :muc-allow_partial-coref-fn :muc-allow_partial-coref-p :muc-allow_partial-coref-r :muc-allow_partial-coref-f
11319941 121 0 0 1 1 1 121 0 0 1 1 1 121 0 0 1 1 1 121 0 0 1 1 1 121 0 0 1 1 1 1253 0 0 1 1 1 6007 0 0 1 1 1 1 121 0 0 1 1 1 1253 0 0 1 1 1 6007 0 0 1 1 1 1 121 0 0 1 1 1 16 0 0 1 1 1 121 0 0 1 1 1 16 0 0 1 1 1 121 0 0 1 1 1 121 0 0 1 1 1 121 0 0 1 1 1 121 0 0 1 1 1 121 0 0 1 1 1 121 0 0 1 1 1 121 0 0 1 1 1 121 0 0 1 1 1 121 0 0 1 1 1 105 0 0 1 1 1 121 0 0 1 1 1 105 0 0 1 1 1
11604102 62 0 0 1 1 1 62 0 0 1 1 1 62 0 0 1 1 1 62 0 0 1 1 1 62 0 0 1 1 1 314 0 0 1 1 1 1577 0 0 1 1 1 1 62 0 0 1 1 1 314 0 0 1 1 1 1577 0 0 1 1 1 1 62 0 0 1 1 1 15 0 0 1 1 1 62 0 0 1 1 1 15 0 0 1 1 1 62 0 0 1 1 1 62 0 0 1 1 1 62 0 0 1 1 1 62 0 0 1 1 1 62 0 0 1 1 1 62 0 0 1 1 1 62 0 0 1 1 1 62 0 0 1 1 1 62 0 0 1 1 1 47 0 0 1 1 1 62 0 0 1 1 1 47 0 0 1 1 1
14624252 232 0 0 1 1 1 232 0 0 1 1 1 232 0 0 1 1 1 232 0 0 1 1 1 232 0 0 1 1 1 5339 0 0 1 1 1 21457 0 0 1 1 1 1 232 0 0 1 1 1 5339 0 0 1 1 1 21457 0 0 1 1 1 1 232 0 0 1 1 1 22 0 0 1 1 1 232 0 0 1 1 1 22 0 0 1 1 1 232 0 0 1 1 1 232 0 0 1 1 1 232 0 0 1 1 1 232 0 0 1 1 1 232 0 0 1 1 1 232 0 0 1 1 1 232 0 0 1 1 1 232 0 0 1 1 1 232 0 0 1 1 1 210 0 0 1 1 1 232 0 0 1 1 1 210 0 0 1 1 1
14675480 215 0 0 1 1 1 215 0 0 1 1 1 215 0 0 1 1 1 215 0 0 1 1 1 215 0 0 1 1 1 6541 0 0 1 1 1 16464 0 0 1 1 1 1 215 0 0 1 1 1 6541 0 0 1 1 1 16464 0 0 1 1 1 1 215 0 0 1 1 1 14 0 0 1 1 1 215 0 0 1 1 1 14 0 0 1 1 1 215 0 0 1 1 1 215 0 0 1 1 1 215 0 0 1 1 1 215 0 0 1 1 1 215 0 0 1 1 1 215 0 0 1 1 1 215 0 0 1 1 1 215 0 0 1 1 1 215 0 0 1 1 1 201 0 0 1 1 1 215 0 0 1 1 1 201 0 0 1 1 1
14691534 234 0 0 1 1 1 234 0 0 1 1 1 234 0 0 1 1 1 234 0 0 1 1 1 234 0 0 1 1 1 5535 0 0 1 1 1 21726 0 0 1 1 1 1 234 0 0 1 1 1 5535 0 0 1 1 1 21726 0 0 1 1 1 1 234 0 0 1 1 1 21 0 0 1 1 1 234 0 0 1 1 1 21 0 0 1 1 1 234 0 0 1 1 1 234 0 0 1 1 1 234 0 0 1 1 1 234 0 0 1 1 1 234 0 0 1 1 1 234 0 0 1 1 1 234 0 0 1 1 1 234 0 0 1 1 1 234 0 0 1 1 1 213 0 0 1 1 1 234 0 0 1 1 1 213 0 0 1 1 1
15018652 49 0 0 1 1 1 49 0 0 1 1 1 49 0 0 1 1 1 49 0 0 1 1 1 49 0 0 1 1 1 239 0 0 1 1 1 937 0 0 1 1 1 1 49 0 0 1 1 1 239 0 0 1 1 1 937 0 0 1 1 1 1 49 0 0 1 1 1 8 0 0 1 1 1 49 0 0 1 1 1 8 0 0 1 1 1 49 0 0 1 1 1 49 0 0 1 1 1 49 0 0 1 1 1 49 0 0 1 1 1 49 0 0 1 1 1 49 0 0 1 1 1 49 0 0 1 1 1 49 0 0 1 1 1 49 0 0 1 1 1 41 0 0 1 1 1 49 0 0 1 1 1 41 0 0 1 1 1
15070402 149 0 0 1 1 1 149 0 0 1 1 1 149 0 0 1 1 1 149 0 0 1 1 1 149 0 0 1 1 1 2623 0 0 1 1 1 8403 0 0 1 1 1 1 149 0 0 1 1 1 2623 0 0 1 1 1 8403 0 0 1 1 1 1 149 0 0 1 1 1 10 0 0 1 1 1 149 0 0 1 1 1 10 0 0 1 1 1 149 0 0 1 1 1 149 0 0 1 1 1 149 0 0 1 1 1 149 0 0 1 1 1 149 0 0 1 1 1 149 0 0 1 1 1 149 0 0 1 1 1 149 0 0 1 1 1 149 0 0 1 1 1 139 0 0 1 1 1 149 0 0 1 1 1 139 0 0 1 1 1
15238161 247 0 0 1 1 1 247 0 0 1 1 1 247 0 0 1 1 1 247 0 0 1 1 1 247 0 0 1 1 1 10408 0 0 1 1 1 19973 0 0 1 1 1 1 247 0 0 1 1 1 10408 0 0 1 1 1 19973 0 0 1 1 1 1 247 0 0 1 1 1 13 0 0 1 1 1 247 0 0 1 1 1 13 0 0 1 1 1 247 0 0 1 1 1 247 0 0 1 1 1 247 0 0 1 1 1 247 0 0 1 1 1 247 0 0 1 1 1 247 0 0 1 1 1 247 0 0 1 1 1 247 0 0 1 1 1 247 0 0 1 1 1 234 0 0 1 1 1 247 0 0 1 1 1 234 0 0 1 1 1
15328538 158 0 0 1 1 1 158 0 0 1 1 1 158 0 0 1 1 1 158 0 0 1 1 1 158 0 0 1 1 1 2020 0 0 1 1 1 10383 0 0 1 1 1 1 158 0 0 1 1 1 2020 0 0 1 1 1 10383 0 0 1 1 1 1 158 0 0 1 1 1 19 0 0 1 1 1 158 0 0 1 1 1 19 0 0 1 1 1 158 0 0 1 1 1 158 0 0 1 1 1 158 0 0 1 1 1 158 0 0 1 1 1 158 0 0 1 1 1 158 0 0 1 1 1 158 0 0 1 1 1 158 0 0 1 1 1 158 0 0 1 1 1 139 0 0 1 1 1 158 0 0 1 1 1 139 0 0 1 1 1
15560850 48 0 0 1 1 1 48 0 0 1 1 1 48 0 0 1 1 1 48 0 0 1 1 1 48 0 0 1 1 1 142 0 0 1 1 1 986 0 0 1 1 1 1 48 0 0 1 1 1 142 0 0 1 1 1 986 0 0 1 1 1 1 48 0 0 1 1 1 11 0 0 1 1 1 48 0 0 1 1 1 11 0 0 1 1 1 48 0 0 1 1 1 48 0 0 1 1 1 48 0 0 1 1 1 48 0 0 1 1 1 48 0 0 1 1 1 48 0 0 1 1 1 48 0 0 1 1 1 48 0 0 1 1 1 48 0 0 1 1 1 37 0 0 1 1 1 48 0 0 1 1 1 37 0 0 1 1 1
15615595 131 0 0 1 1 1 131 0 0 1 1 1 131 0 0 1 1 1 131 0 0 1 1 1 131 0 0 1 1 1 2543 0 0 1 1 1 5972 0 0 1 1 1 1 131 0 0 1 1 1 2543 0 0 1 1 1 5972 0 0 1 1 1 1 131 0 0 1 1 1 6 0 0 1 1 1 131 0 0 1 1 1 6 0 0 1 1 1 131 0 0 1 1 1 131 0 0 1 1 1 131 0 0 1 1 1 131 0 0 1 1 1 131 0 0 1 1 1 131 0 0 1 1 1 131 0 0 1 1 1 131 0 0 1 1 1 131 0 0 1 1 1 125 0 0 1 1 1 131 0 0 1 1 1 125 0 0 1 1 1
15619330 147 0 0 1 1 1 147 0 0 1 1 1 147 0 0 1 1 1 147 0 0 1 1 1 147 0 0 1 1 1 1899 0 0 1 1 1 8832 0 0 1 1 1 1 147 0 0 1 1 1 1899 0 0 1 1 1 8832 0 0 1 1 1 1 147 0 0 1 1 1 19 0 0 1 1 1 147 0 0 1 1 1 19 0 0 1 1 1 147 0 0 1 1 1 147 0 0 1 1 1 147 0 0 1 1 1 147 0 0 1 1 1 147 0 0 1 1 1 147 0 0 1 1 1 147 0 0 1 1 1 147 0 0 1 1 1 147 0 0 1 1 1 128 0 0 1 1 1 147 0 0 1 1 1 128 0 0 1 1 1
15784609 107 0 0 1 1 1 107 0 0 1 1 1 107 0 0 1 1 1 107 0 0 1 1 1 107 0 0 1 1 1 1328 0 0 1 1 1 4343 0 0 1 1 1 1 107 0 0 1 1 1 1328 0 0 1 1 1 4343 0 0 1 1 1 1 107 0 0 1 1 1 11 0 0 1 1 1 107 0 0 1 1 1 11 0 0 1 1 1 107 0 0 1 1 1 107 0 0 1 1 1 107 0 0 1 1 1 107 0 0 1 1 1 107 0 0 1 1 1 107 0 0 1 1 1 107 0 0 1 1 1 107 0 0 1 1 1 107 0 0 1 1 1 96 0 0 1 1 1 107 0 0 1 1 1 96 0 0 1 1 1
15850489 95 0 0 1 1 1 95 0 0 1 1 1 95 0 0 1 1 1 95 0 0 1 1 1 95 0 0 1 1 1 1508 0 0 1 1 1 2957 0 0 1 1 1 1 95 0 0 1 1 1 1508 0 0 1 1 1 2957 0 0 1 1 1 1 95 0 0 1 1 1 13 0 0 1 1 1 95 0 0 1 1 1 13 0 0 1 1 1 95 0 0 1 1 1 95 0 0 1 1 1 95 0 0 1 1 1 95 0 0 1 1 1 95 0 0 1 1 1 95 0 0 1 1 1 95 0 0 1 1 1 95 0 0 1 1 1 95 0 0 1 1 1 82 0 0 1 1 1 95 0 0 1 1 1 82 0 0 1 1 1
15882093 218 0 0 1 1 1 218 0 0 1 1 1 218 0 0 1 1 1 218 0 0 1 1 1 218 0 0 1 1 1 5629 0 0 1 1 1 18024 0 0 1 1 1 1 218 0 0 1 1 1 5629 0 0 1 1 1 18024 0 0 1 1 1 1 218 0 0 1 1 1 15 0 0 1 1 1 218 0 0 1 1 1 15 0 0 1 1 1 218 0 0 1 1 1 218 0 0 1 1 1 218 0 0 1 1 1 218 0 0 1 1 1 218 0 0 1 1 1 218 0 0 1 1 1 218 0 0 1 1 1 218 0 0 1 1 1 218 0 0 1 1 1 203 0 0 1 1 1 218 0 0 1 1 1 203 0 0 1 1 1
16026622 78 0 0 1 1 1 78 0 0 1 1 1 78 0 0 1 1 1 78 0 0 1 1 1 78 0 0 1 1 1 643 0 0 1 1 1 2360 0 0 1 1 1 1 78 0 0 1 1 1 643 0 0 1 1 1 2360 0 0 1 1 1 1 78 0 0 1 1 1 17 0 0 1 1 1 78 0 0 1 1 1 17 0 0 1 1 1 78 0 0 1 1 1 78 0 0 1 1 1 78 0 0 1 1 1 78 0 0 1 1 1 78 0 0 1 1 1 78 0 0 1 1 1 78 0 0 1 1 1 78 0 0 1 1 1 78 0 0 1 1 1 61 0 0 1 1 1 78 0 0 1 1 1 61 0 0 1 1 1
16027110 138 0 0 1 1 1 138 0 0 1 1 1 138 0 0 1 1 1 138 0 0 1 1 1 138 0 0 1 1 1 1999 0 0 1 1 1 7454 0 0 1 1 1 1 138 0 0 1 1 1 1999 0 0 1 1 1 7454 0 0 1 1 1 1 138 0 0 1 1 1 18 0 0 1 1 1 138 0 0 1 1 1 18 0 0 1 1 1 138 0 0 1 1 1 138 0 0 1 1 1 138 0 0 1 1 1 138 0 0 1 1 1 138 0 0 1 1 1 138 0 0 1 1 1 138 0 0 1 1 1 138 0 0 1 1 1 138 0 0 1 1 1 120 0 0 1 1 1 138 0 0 1 1 1 120 0 0 1 1 1
16410827 251 0 0 1 1 1 251 0 0 1 1 1 251 0 0 1 1 1 251 0 0 1 1 1 251 0 0 1 1 1 11107 0 0 1 1 1 20268 0 0 1 1 1 1 251 0 0 1 1 1 11107 0 0 1 1 1 20268 0 0 1 1 1 1 251 0 0 1 1 1 16 0 0 1 1 1 251 0 0 1 1 1 16 0 0 1 1 1 251 0 0 1 1 1 251 0 0 1 1 1 251 0 0 1 1 1 251 0 0 1 1 1 251 0 0 1 1 1 251 0 0 1 1 1 251 0 0 1 1 1 251 0 0 1 1 1 251 0 0 1 1 1 235 0 0 1 1 1 251 0 0 1 1 1 235 0 0 1 1 1
16517939 102 0 0 1 1 1 102 0 0 1 1 1 102 0 0 1 1 1 102 0 0 1 1 1 102 0 0 1 1 1 1112 0 0 1 1 1 4039 0 0 1 1 1 1 102 0 0 1 1 1 1112 0 0 1 1 1 4039 0 0 1 1 1 1 102 0 0 1 1 1 12 0 0 1 1 1 102 0 0 1 1 1 12 0 0 1 1 1 102 0 0 1 1 1 102 0 0 1 1 1 102 0 0 1 1 1 102 0 0 1 1 1 102 0 0 1 1 1 102 0 0 1 1 1 102 0 0 1 1 1 102 0 0 1 1 1 102 0 0 1 1 1 90 0 0 1 1 1 102 0 0 1 1 1 90 0 0 1 1 1
16611361 70 0 0 1 1 1 70 0 0 1 1 1 70 0 0 1 1 1 70 0 0 1 1 1 70 0 0 1 1 1 514 0 0 1 1 1 1901 0 0 1 1 1 1 70 0 0 1 1 1 514 0 0 1 1 1 1901 0 0 1 1 1 1 70 0 0 1 1 1 8 0 0 1 1 1 70 0 0 1 1 1 8 0 0 1 1 1 70 0 0 1 1 1 70 0 0 1 1 1 70 0 0 1 1 1 70 0 0 1 1 1 70 0 0 1 1 1 70 0 0 1 1 1 70 0 0 1 1 1 70 0 0 1 1 1 70 0 0 1 1 1 62 0 0 1 1 1 70 0 0 1 1 1 62 0 0 1 1 1
16787536 144 0 0 1 1 1 144 0 0 1 1 1 144 0 0 1 1 1 144 0 0 1 1 1 144 0 0 1 1 1 2551 0 0 1 1 1 7745 0 0 1 1 1 1 144 0 0 1 1 1 2551 0 0 1 1 1 7745 0 0 1 1 1 1 144 0 0 1 1 1 11 0 0 1 1 1 144 0 0 1 1 1 11 0 0 1 1 1 144 0 0 1 1 1 144 0 0 1 1 1 144 0 0 1 1 1 144 0 0 1 1 1 144 0 0 1 1 1 144 0 0 1 1 1 144 0 0 1 1 1 144 0 0 1 1 1 144 0 0 1 1 1 133 0 0 1 1 1 144 0 0 1 1 1 133 0 0 1 1 1
16800892 157 0 0 1 1 1 157 0 0 1 1 1 157 0 0 1 1 1 157 0 0 1 1 1 157 0 0 1 1 1 2605 0 0 1 1 1 9641 0 0 1 1 1 1 157 0 0 1 1 1 2605 0 0 1 1 1 9641 0 0 1 1 1 1 157 0 0 1 1 1 18 0 0 1 1 1 157 0 0 1 1 1 18 0 0 1 1 1 157 0 0 1 1 1 157 0 0 1 1 1 157 0 0 1 1 1 157 0 0 1 1 1 157 0 0 1 1 1 157 0 0 1 1 1 157 0 0 1 1 1 157 0 0 1 1 1 157 0 0 1 1 1 139 0 0 1 1 1 157 0 0 1 1 1 139 0 0 1 1 1
16968134 319 0 0 1 1 1 319 0 0 1 1 1 319 0 0 1 1 1 319 0 0 1 1 1 319 0 0 1 1 1 8426 0 0 1 1 1 42295 0 0 1 1 1 1 319 0 0 1 1 1 8426 0 0 1 1 1 42295 0 0 1 1 1 1 319 0 0 1 1 1 21 0 0 1 1 1 319 0 0 1 1 1 21 0 0 1 1 1 319 0 0 1 1 1 319 0 0 1 1 1 319 0 0 1 1 1 319 0 0 1 1 1 319 0 0 1 1 1 319 0 0 1 1 1 319 0 0 1 1 1 319 0 0 1 1 1 319 0 0 1 1 1 298 0 0 1 1 1 319 0 0 1 1 1 298 0 0 1 1 1
17029558 149 0 0 1 1 1 149 0 0 1 1 1 149 0 0 1 1 1 149 0 0 1 1 1 149 0 0 1 1 1 2117 0 0 1 1 1 8909 0 0 1 1 1 1 149 0 0 1 1 1 2117 0 0 1 1 1 8909 0 0 1 1 1 1 149 0 0 1 1 1 16 0 0 1 1 1 149 0 0 1 1 1 16 0 0 1 1 1 149 0 0 1 1 1 149 0 0 1 1 1 149 0 0 1 1 1 149 0 0 1 1 1 149 0 0 1 1 1 149 0 0 1 1 1 149 0 0 1 1 1 149 0 0 1 1 1 149 0 0 1 1 1 133 0 0 1 1 1 149 0 0 1 1 1 133 0 0 1 1 1
17201918 182 0 0 1 1 1 182 0 0 1 1 1 182 0 0 1 1 1 182 0 0 1 1 1 182 0 0 1 1 1 2537 0 0 1 1 1 13934 0 0 1 1 1 1 182 0 0 1 1 1 2537 0 0 1 1 1 13934 0 0 1 1 1 1 182 0 0 1 1 1 15 0 0 1 1 1 182 0 0 1 1 1 15 0 0 1 1 1 182 0 0 1 1 1 182 0 0 1 1 1 182 0 0 1 1 1 182 0 0 1 1 1 182 0 0 1 1 1 182 0 0 1 1 1 182 0 0 1 1 1 182 0 0 1 1 1 182 0 0 1 1 1 167 0 0 1 1 1 182 0 0 1 1 1 167 0 0 1 1 1
17206865 186 0 0 1 1 1 186 0 0 1 1 1 186 0 0 1 1 1 186 0 0 1 1 1 186 0 0 1 1 1 3199 0 0 1 1 1 14006 0 0 1 1 1 1 186 0 0 1 1 1 3199 0 0 1 1 1 14006 0 0 1 1 1 1 186 0 0 1 1 1 19 0 0 1 1 1 186 0 0 1 1 1 19 0 0 1 1 1 186 0 0 1 1 1 186 0 0 1 1 1 186 0 0 1 1 1 186 0 0 1 1 1 186 0 0 1 1 1 186 0 0 1 1 1 186 0 0 1 1 1 186 0 0 1 1 1 186 0 0 1 1 1 167 0 0 1 1 1 186 0 0 1 1 1 167 0 0 1 1 1
17465682 237 0 0 1 1 1 237 0 0 1 1 1 237 0 0 1 1 1 237 0 0 1 1 1 237 0 0 1 1 1 3933 0 0 1 1 1 24033 0 0 1 1 1 1 237 0 0 1 1 1 3933 0 0 1 1 1 24033 0 0 1 1 1 1 237 0 0 1 1 1 33 0 0 1 1 1 237 0 0 1 1 1 33 0 0 1 1 1 237 0 0 1 1 1 237 0 0 1 1 1 237 0 0 1 1 1 237 0 0 1 1 1 237 0 0 1 1 1 237 0 0 1 1 1 237 0 0 1 1 1 237 0 0 1 1 1 237 0 0 1 1 1 204 0 0 1 1 1 237 0 0 1 1 1 204 0 0 1 1 1
17503968 172 0 0 1 1 1 172 0 0 1 1 1 172 0 0 1 1 1 172 0 0 1 1 1 172 0 0 1 1 1 1848 0 0 1 1 1 12858 0 0 1 1 1 1 172 0 0 1 1 1 1848 0 0 1 1 1 12858 0 0 1 1 1 1 172 0 0 1 1 1 19 0 0 1 1 1 172 0 0 1 1 1 19 0 0 1 1 1 172 0 0 1 1 1 172 0 0 1 1 1 172 0 0 1 1 1 172 0 0 1 1 1 172 0 0 1 1 1 172 0 0 1 1 1 172 0 0 1 1 1 172 0 0 1 1 1 172 0 0 1 1 1 153 0 0 1 1 1 172 0 0 1 1 1 153 0 0 1 1 1
17565376 228 0 0 1 1 1 228 0 0 1 1 1 228 0 0 1 1 1 228 0 0 1 1 1 228 0 0 1 1 1 7008 0 0 1 1 1 18870 0 0 1 1 1 1 228 0 0 1 1 1 7008 0 0 1 1 1 18870 0 0 1 1 1 1 228 0 0 1 1 1 17 0 0 1 1 1 228 0 0 1 1 1 17 0 0 1 1 1 228 0 0 1 1 1 228 0 0 1 1 1 228 0 0 1 1 1 228 0 0 1 1 1 228 0 0 1 1 1 228 0 0 1 1 1 228 0 0 1 1 1 228 0 0 1 1 1 228 0 0 1 1 1 211 0 0 1 1 1 228 0 0 1 1 1 211 0 0 1 1 1
17677002 150 0 0 1 1 1 150 0 0 1 1 1 150 0 0 1 1 1 150 0 0 1 1 1 150 0 0 1 1 1 2766 0 0 1 1 1 8409 0 0 1 1 1 1 150 0 0 1 1 1 2766 0 0 1 1 1 8409 0 0 1 1 1 1 150 0 0 1 1 1 13 0 0 1 1 1 150 0 0 1 1 1 13 0 0 1 1 1 150 0 0 1 1 1 150 0 0 1 1 1 150 0 0 1 1 1 150 0 0 1 1 1 150 0 0 1 1 1 150 0 0 1 1 1 150 0 0 1 1 1 150 0 0 1 1 1 150 0 0 1 1 1 137 0 0 1 1 1 150 0 0 1 1 1 137 0 0 1 1 1
TOTAL 4776 0 0 1 1 1 4776 0 0 1 1 1 4776 0 0 1 1 1 4776 0 0 1 1 1 4776 0 0 1 1 1 99686 0 0 1 1 1 344763 0 0 1 1 1 1 4776 0 0 1 1 1 99686 0 0 1 1 1 344763 0 0 1 1 1 1 4776 0 0 1 1 1 466 0 0 1 1 1 4776 0 0 1 1 1 466 0 0 1 1 1 4776 0 0 1 1 1 4776 0 0 1 1 1 4776 0 0 1 1 1 4776 0 0 1 1 1 4776 0 0 1 1 1 4776 0 0 1 1 1 4776 0 0 1 1 1 4776 0 0 1 1 1 4776 0 0 1 1 1 4310 0 0 1 1 1 4776 0 0 1 1 1 4310 0 0 1 1 1
Script crashes when evaluating merged discontinuous mentions
Hi,
Thank you for creating this evaluation script.
Currently, I am using your script for evaluating coreference resolution share task. I have downloaded the latest version (v3.1.2_0) from DockerHub and do the evaluation on that. The context is that I was trying to read gold CoNLL files and write it down as predicted ones. I expected that it could obtain 100% F-score, but got an error like this:
craft@8f98280572df:~/evaluation$ boot eval-coreference
Classpath conflict: org.clojure/clojure version 1.8.0 already loaded, NOT loading version 1.10.1
Running coreference evaluation. Metric = muc
Running coreference evaluation. Metric = muc
Running coreference evaluation. Metric = bcub
Running coreference evaluation. Metric = bcub
Running coreference evaluation. Metric = ceafm
Running coreference evaluation. Metric = ceafm
Running coreference evaluation. Metric = ceafe
Running coreference evaluation. Metric = ceafe
Running coreference evaluation. Metric = blanc
Running coreference evaluation. Metric = blanc
Running coreference evaluation. Metric = lea
Running coreference evaluation. Metric = lea
Processing coref metric log file: /home/craft/evaluation/.intermediate-results/coref/muc.log
Processing coref metric log file: /home/craft/evaluation/.intermediate-results/coref/muc.allow_partial.log
Processing coref metric log file: /home/craft/evaluation/.intermediate-results/coref/lea.log
Processing coref metric log file: /home/craft/evaluation/.intermediate-results/coref/lea.allow_partial.log
Processing coref metric log file: /home/craft/evaluation/.intermediate-results/coref/bcub.allow_partial.log
Processing coref metric log file: /home/craft/evaluation/.intermediate-results/coref/bcub.log
Processing BLANC metric log file: /home/craft/evaluation/.intermediate-results/coref/blanc.allow_partial.log
Processing coref metric log file: /home/craft/evaluation/.intermediate-results/coref/ceafe.allow_partial.log
Processing coref metric log file: /home/craft/evaluation/.intermediate-results/coref/ceafm.log
Processing BLANC metric log file: /home/craft/evaluation/.intermediate-results/coref/blanc.log
Processing coref metric log file: /home/craft/evaluation/.intermediate-results/coref/ceafm.allow_partial.log
Processing coref metric log file: /home/craft/evaluation/.intermediate-results/coref/ceafe.log
java.lang.Thread.run Thread.java: 748
java.util.concurrent.ThreadPoolExecutor$Worker.run ThreadPoolExecutor.java: 624
java.util.concurrent.ThreadPoolExecutor.runWorker ThreadPoolExecutor.java: 1149
java.util.concurrent.FutureTask.run FutureTask.java: 266
...
clojure.core/binding-conveyor-fn/fn core.clj: 1938
boot.core/boot/fn core.clj: 1032
boot.core/run-tasks core.clj: 1022
boot.user$eval363$fn__364$fn__369$fn__370.invoke : 57
boot.user$eval432$fn__433$fn__438$fn__439.invoke : 84 (repeats 12 times)
boot.user$eval469$fn__470$fn__475$fn__476.invoke : 103
boot.user$eval469$fn__470$fn__475$fn__476$fn__477.invoke : 104
craft-eval.coref/serialize-coref-results coref.clj: 251
clojure.core/reduce-kv core.clj: 6573
clojure.core.protocols/fn/G protocols.clj: 174
clojure.core/fn core.clj: 6547
clojure.core/reduce core.clj: 6545
clojure.core.protocols/fn/G protocols.clj: 13
clojure.core.protocols/fn protocols.clj: 75
clojure.core.protocols/iter-reduce protocols.clj: 49
clojure.core/fn/fn core.clj: 6557
craft-eval.coref/serialize-coref-results/fn coref.clj: 263
...
clojure.core/merge-with core.clj: 2942
clojure.core/merge-with core.clj: 2950
clojure.core/reduce1 core.clj: 915
clojure.core/reduce1 core.clj: 925
clojure.core/merge-with/merge2 core.clj: 2957
clojure.core/reduce1 core.clj: 925
clojure.core/merge-with/merge-entry core.clj: 2954
clojure.core/+ core.clj: 973
...
java.lang.NullPointerException:
clojure.lang.ExceptionInfo:
line: 409
craft@8f98280572df:~/evaluation$
Then, I have opened all logs reported on the screen to see what happened and found that some of metrics, for example BCUB, show #TP counts as a decimal number like log below, maybe, this number violates your regex patterns to extract the score from log:
====== TOTALS =======
Identification of Mentions: Recall: (1853 / 1854) 99.94% Precision: (1853 / 1853) 100% F1: 99.97%
--------------------------------------------------------------------------
Coreference: Recall: (1852.33333333333 / 1854) 99.91% Precision: (1853 / 1853) 100% F1: 99.95%
--------------------------------------------------------------------------
------------------ end /home/craft/eval-data/coreference/conllcoref/16628246.conll
I'm not sure but in my reading/writing CoNLL code, I do an additional step, namely, I correct (merge) continuous mentions annotated as discontinuous mentions.
For example:
Gold data:
File: 17069463.conll
Line: 4931
ID: 266a
1626108 0 5 inbred JJ - - - - - - - (266a
1626108 0 6 RanBP2 NN - - - - - - - -
1626108 0 7 + SYM - - - - - - - -
1626108 0 8 / HYPH - - - - - - - -
1626108 0 9 − SYM - - - - - - - -
1626108 0 10 mice NNS - - - - - - - -
1626108 0 11 on IN - - - - - - - -
1626108 0 12 high JJ - - - - - - - (267
1626108 0 13 - HYPH - - - - - - - -
1626108 0 14 fat NN - - - - - - - -
1626108 0 15 ( -LRB- - - - - - - - -
1626108 0 16 ~ SYM - - - - - - - (268
1626108 0 17 10 CD - - - - - - - -
1626108 0 18 % NN - - - - - - - -
1626108 0 19 fat NN - - - - - - - 268)
1626108 0 20 ) -RRB- - - - - - - - -
1626108 0 21 diet NN - - - - - - - (266a)|267)|266a)
Obviously, mention at word 21 (266a) is a substring of a mention from word 5 (266a to word 21 266a). I understand that there is only one discontinuous mention with offset from [(5, 21), (21, 21)]. That's it.
In my predicted data, I normalized as follows:
1626108 0 5 inbred JJ - - - - - - - (248
1626108 0 6 RanBP2 NN - - - - - - - -
1626108 0 7 + SYM - - - - - - - -
1626108 0 8 / HYPH - - - - - - - -
1626108 0 9 − SYM - - - - - - - -
1626108 0 10 mice NNS - - - - - - - -
1626108 0 11 on IN - - - - - - - -
1626108 0 12 high JJ - - - - - - - (244
1626108 0 13 - HYPH - - - - - - - -
1626108 0 14 fat NN - - - - - - - -
1626108 0 15 ( -LRB- - - - - - - - -
1626108 0 16 ~ SYM - - - - - - - (243
1626108 0 17 10 CD - - - - - - - -
1626108 0 18 % NN - - - - - - - -
1626108 0 19 fat NN - - - - - - - 243)
1626108 0 20 ) -RRB- - - - - - - - -
1626108 0 21 diet NN - - - - - - - 244)|248)
Can see that there is only mention from word 5 to word 21 too (248) corresponding to (266a) above.
I have just applied this additional step for discontinuous mentions only. But according to the definition for discontinuous mentions, I think that in this case, predicting a mention with offset (5, 21) must be the same with [(5, 21), (21, 21)] above. Is it right?
And after doing statistics, I found that there are 2 files causing this error. 16628246.conll 391a and 17069463.conll (266a) (I have also attached these files here) error.zip.
I think we should fix this issue because we don't know how to predict "enough spans" for a mention to avoid crashing evaluation script. I mean we can treat the above example as a wrong prediction but the script must run without any error.
Thank you in advance,
Khoa
Divide by zero error when there is no coreference prediction
Hi,
First of all, thank you for the CRAFT corpus and your extensive work that makes it easier to use.
While using the evaluation tool (version 4.0.1_0.1.2 from docker), we came across an error: java.lang.ArithmeticException: Divide by zero.
The command we use is:
sudo docker run --rm -v /home/nursima/Desktop/491/real_output:/files-to-evaluate -v /home/nursima/Desktop/491/CRAFT/articles/txt/:/corpus-distribution -v /home/nursima/Desktop/491/output:/gold ucdenverccp/craft-eval:4.0.1_0.1.2 sh -c '(cd /home/craft/evaluation && boot eval-coreference -c /corpus-distribution -i /files-to-evaluate -g /gold -b /home/craft/evaluation -s /home/craft/evaluation/coreference/reference-coreference-scorers.git)'
In the files-to-evaluate (real_output) directory, we give a prediction file for one article, with no coreference prediction.
11319941.conll.zip
In the gold (output) directory, there is gold file for that article.
11319941.conll.zip
But we get this error:
...
clojure.core/map/fn core.clj: 2646
craft-eval.coref/compile-coref-results-for-document/fn coref.clj: 185
craft-eval.coref/prf coref.clj: 146
...
java.lang.ArithmeticException: Divide by zero
clojure.lang.ExceptionInfo: Divide by zero
line: 437
It seems that the prf function in coref.clj line 145 causing the problem. It doesn't check whether denominators are zero. (tp + fp is zero in our case)
We would appreciate any help. Thank you.
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.