microsoft / transformer-xh

Microsoft Open Source Code of Conduct

This project has adopted the Microsoft Open Source Code of Conduct.

Resources:

• Microsoft Open Source Code of Conduct
• Microsoft Code of Conduct FAQ
• Contact [email protected] with questions or concerns

Transformer-XH

The source codes of the paper "Transformer-XH: Multi-evidence Reasoning with Extra Hop Attention (ICLR 2020)".

Dependency Installation

First, Run python setup.py develop to install required dependencies for transformer-xh. Also install apex (for distributed training) following official documentation here.

Data and trained model Download

You can run bash script download.sh

For Hotpot QA, we provide processed graph (Transformer-XH) input here, after downloading, unzip it and put into ./data folder We also provide trained model here, unzip the downloaded model and put into ./experiments folder

Similarly, we provide processed graph in fever here, and trained model here.

Run Your Models

Use hotpot_train.sh for training on hotpot QA task, hotpot_eval.sh for evaluation (default fp16 training).

Similarly, fever_train.sh for training on FEVER task, fever_eval.sh for evaluation (default fp16 training).

Contact

If you have questions, suggestions and bug reports, please email [email protected] and/or [email protected].

transformer-xh's Issues

Fever score in eval script

Do you also calculate Fever score in Evaluator.py along with classification accuracy?

data processing

hello, i have already your code and paper, but can you show me your data processing code with fully wiki data? thanks

Adding specific version of pytorch_transformers as setup

Hi - I'm currently running evaluation of Transformer-XH and ran into a small problem that was resolved.

There seems to be a breaking change with pytorch_transformers==1.2.0 where it would not compile the model as is:

Traceback (most recent call last):
  File "transformer-xh/main.py", line 129, in <module>
    model = Model_Hotpot(args, config)
  File "/content/Transformer-XH/transformer-xh/model/model_hotpotqa.py", line 38, in __init__
    super(Model_Hotpot, self).__init__(args, config)
  File "/content/Transformer-XH/transformer-xh/model/model.py", line 229, in __init__
    self.network= ModelHelper(self.bert_node_encoder, self.args, self.bert_config, self.config_model)
  File "/content/Transformer-XH/transformer-xh/model/model.py", line 212, in __init__
    self.final_layer.apply(self.init_weights)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 290, in apply
    fn(self)
TypeError: init_weights() takes 1 positional argument but 2 were given

This is resolved by having pytorch_transformers==1.0.0 installed rather than pytorch_transformers==1.2.0.

The dev result file have only 5904 samples

I ran the codes, but I found there are only 5904 samples in hotpot_dev_graph.json as well as in the final results, not the total 7405 samples. What's wrong with it，is there any thing I should change besides default settings? I read your paper and the result reported in your paper is on the total dev dataset.
I am looking forward to your reply and thank you very much!

issues with apex

Thanks a lot for providing the code.

Would someone please give more information on how to install apex properly, and also possibly which apex version and pytorch version to ise.

I followed the steps here: https://github.com/NVIDIA/apex
I am using docker with image: pytorch/pytorch:1.6.0-cuda10.1-cudnn7-devel

Then I installed apex, as in the quick start instructions of apex, using:

pip install -v --disable-pip-version-check --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext" ./

and it seems to have been installed successfully.
Currently, I am facing these issues, when attempting to train on fever (bash fever_train.sh):

1-line 167 from 'transformer-xh/main.py'
from apex.optimizers import FP16_Optimizer, gives:

ImportError: cannot import name 'FP16_Optimizer' from 'apex.optimizers'.

I tried to solve this by doing: from apex.contrib.optimizers import FP16_Optimizer instead, based on NVIDIA/apex#593

2- With the previous fix, I still face an error at line 176:

ModuleNotFoundError: No module named 'fused_adam_cuda'

Would someone please point at what I am doing wrong here?

It seems that batch_fever in data/fever.py only return the first sample

Graph processing for new claims in Fever Task

Can you provide your script for graph processing or some references to reproduce it? I think I can utilize top 5 sentence retrieval from KGAT, but I think there's another step of graph construction after that.
I wanted to run it for some new claim examples.

Reproducing results on FEVER test

Hi,

Thanks for open sourcing the code!
I was trying to reproduce the results on the FEVER task.
Ran the scripts as directed in the Readme and changed the test set path in config_fever.json from data/fever_dev_graph.json to data/fever_test_graph.json.

The label accuracy I'm getting with the shared pre-trained weights in this setting is 39.73% .
Could you please highlight if there's something else that needs to be changed to get the reported results?