marcotcr / sears

Code for "Semantically Equivalent Adversarial Rules for Debugging NLP Models"

sears

Code for Semantically Equivalent Adversarial Rules for Debugging NLP Models

Installation

Run the following:

git clone [email protected]:marcotcr/sears.git
cd sears
virtualenv -p python3 ENV
source ENV/bin/activate
pip install editdistance keras numpy jupyter tensorflow-gpu==1.3.0 torchtext==0.1.1 spacy==1.9.0
pip install http://download.pytorch.org/whl/cu80/torch-0.2.0.post3-cp35-cp35m-manylinux1_x86_64.whl
python -m ipykernel install --user --name=sears
python -m spacy download en
git clone https://github.com/marcotcr/OpenNMT-py
cd OpenNMT-py/
python setup.py install
cd ..

Download and unpack the translation models into the translation_models folder.

Run jupyter notebook, change kernel to sears, and run the notebook :)

sears's Issues

Using AllenNLP model in SEARs computation

I'm trying to compute SEARs for machine comprehension task, and need to load the pretrained AllenNLP model for this purpose. However, the AllenNLP project has different dependency than your sears project. For example, AllenNLP requires python 3.6 with pytorch 0.4, while sears uses python 3.5 with pytorch 0.2.

How do I use make them working at the same time? I tried to set up a local server for AllenNLP and send http requests from sears, but it seems really slow. Did you make your own implementation? Any help will be really appreciated!

Ziyao

AttributeError: module 'onmt' has no attribute 'Translator'

mldl@ub1604:/ub16_prj/sears$ python3 Computing_SEARs_for_sentiment_analysis.py
GPU ID 0
Traceback (most recent call last):
File "Computing_SEARs_for_sentiment_analysis.py", line 32, in
ps = paraphrase_scorer.ParaphraseScorer(gpu_id=0)
File "sears/paraphrase_scorer.py", line 64, in init
translator = onmt_model.OnmtModel(f, gpu_id)
File "sears/onmt_model.py", line 103, in init
self.translator = onmt.Translator(opt)
AttributeError: module 'onmt' has no attribute 'Translator'
mldl@ub1604:/ub16_prj/sears$

Missing models in the repository link

Hi,

Thank you for the code! I just followed the instructions under the repository and found that the link "translation models" has no files linked and I could not access it. Can you help reupload it or directly send it to me? That is very helpful for the topic I am working on. Thank you!

training and creating adversarial examples on another dataset/language

I am interested in trying this method for another dataset at another language other than English. Is there some instruction on how to create such attack?

Thank you so much!

Notebook for computing SEARs for machine comprehension

Hi,

Thank you for publishing this code! Right now the only notebook is for sentiment analysis. I'm wondering if there's another notebook available to compute SEARs for machine comprehension? I imagine there might be some differences between these two tasks. If so, that would be great if you could provide that notebook as well.

Otherwise, I can try to modify the existing notebook to compute SEARs for machine comprehension myself. In this case, I have several questions:

1. What dataset did you use to compute SEAs for machine comprehension task? Is it SQUAD? Does it matter which dataset to use at all?
2. How did you define an adversarial question for machine comprehension problem? For sentiment analysis it's easy because the prediction is either positive or negative, and flip of correct prediction means that machine make it wrong. However, machine comprehension problem can have multiple reasonable answers. So how did you define a flip for machine comprehension?

Thank you so much for your time!
Ziyao

LICENCE?

Hi,

Could you add a LICENCE file to this repo so we know under what terms we can reuse the code?

The MIT licence is pretty standard if you are not used to these questions (ex: https://github.com/rowanz/swagaf/blob/master/LICENSE, https://github.com/uclmr/adversarial-nli/blob/master/LICENSE) but you may opt for different ways of licencing of course.

Best,

Thomas

Issues about notebook

Hi,

Thank you for publishing this code! I'm interested in using your notebook to generate some semantic equivalent adversarial samples so that I can test robustness of our model. However I can't get even a single rule when training in on our own dataset. I guess it may because of hyperparameters. Could you give me suggestions on how to decide the best number for these parameters? You know it is time consuming to debug the model because it takes almost one day to run one experiment completely. Thank you