microsoft / lonli

Testing Diverse Reasoning of NLI Systems

LoNLI v1.0

We release CheckList-based behavioral testsuite LoNLI dataset associated with Trusting RoBERTa over BERT: Insights from CheckListing the Natural Language Inference Task . Please note that the ArXiv version is outdated. Our updated version is under review and will soon update the version once the anonymity period is over.

We create a semi-synthetic large test-bench (363 templates, 363k examples) and an associated framework that offers following utilities:

1. individually test and analyze reasoning capabilities along 17 reasoning dimensions (including pragmatic reasoning),
2. design experiments to study cross-capability information content (leave one out or bring one in); and
3. the synthetic nature enable us to control for artifacts and biases.

Files

1. data/data_v2.zip - Unzip it to find 363 files. Each files is named with the corresponding capability (borrowed from TaxiNLI).
   • For example boolean_1.tsv contains the examples form boolean_1 template (found in data/checklist_master.tsv).
   • Each file contains three columns "premise", "hypothesis" and "label"
2. data/checklist_master.tsv - The list of all 363 templates and corresponding generated file-names.
3. templates.ipynb - The python notebook is a single-stop shop for creating data for all templates. It uses the lexicon.py
   • The notebook needs the following packages to run: copy, num2words, checklist
   • To run the notebook, please install using pip install -r requirements.txt
   • Please also checkout the CheckList repository.

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.

When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

Trademarks

This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft trademarks or logos is subject to and must follow Microsoft's Trademark & Brand Guidelines. Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. Any use of third-party trademarks or logos are subject to those third-party's policies.

lonli's Issues

Exact model names to reproduce your error rate

Thanks for the great set of templates for NLI model testing! I noticed you provided error rates for five models in data/checklist_master.tsv (shown below)

BERT	DistilBERT	RoBERTa-large	DeBERTa	RoBERTa-SNLI-MNLI-FEVER-ANLI

I am trying to reproduce some of the errors for a smaller subset of your dataset. However, I am not sure what exactly these models correspond to. Are they models you find-tuned from scratch or are they the checkpoints already fine-tuned and hosted on HuggingFace model hub?

If you fine-tuned you models from scratch, would it be possible to share your fine-tuning scripts so I could start in the exactly same setting (mostly hyperparameters) as you did.

Thank you in advance!

This repo is missing important files

There are important files that Microsoft projects should all have that are not present in this repository. A pull request has been opened to add the missing file(s). When the pr is merged this issue will be closed automatically.

Microsoft teams can learn more about this effort and share feedback within the open source guidance available internally.

Merge this pull request