tommccoy1 / hans Goto Github PK

View Code? Open in Web Editor NEW

124.0 124.0 13.0 8.94 MB

Heuristic Analysis for NLI Systems

License: MIT License

Python 100.00%

hans's People

Contributors

Stargazers

Watchers

Forkers

jabalazs little1tow herobring alexisjihyeross linktopast1990 rosafish shacharosn hryang06 caleblu yuxiang-wu mathisall dongfangyixi dezibelkarate

hans's Issues

Some different results

I trained a BERT on MNLI, and evaluated it on HANS dataset. However, I got the results different from the results in your paper.

Model	Lexical(en)	Subseq(en)	Const(en)	Lexical(non-en)	Subseq(non-en)	Const(non-en)
BERT(yours)	0.95	0.99	0.98	0.16	0.04	0.16
BERT(mine)	0.99	1.00	1.00	0.52	0.04	0.19

Why my acc in lexical(non-entailment) is much higher than yours?

License

Would you be willing to add some kind of open source/CC license to HANS? I'd like to use it in a setting where that'd be necessary.

I trained BERT on MNLI and evaluated it on HANS data. I got some results which are different from what was given in the paper. I am getting all the predictions as non entailment. I tried both fine-tuning BERT and pretraining BERT.

Why are my results drastically different? Could you please help me with this. Thank you

Meaning of some abbreviations

Hi, can you please share the meaning of some abbreviations used as subcategories. In particular, what do "NP/S" and "NP/Z" mean? Thank you in advance.