ibm / techqa

The TechQA dataset -- http://ibm.biz/Tech_QA

Building a Baseline Model for the TechQA dataset

This repo contains the code to train and run a baseline model against the TechQA dataset released in the ACL 2020 paper: The TechQA dataset

The TechQA train and dev data along with 800,000 TechNotes are available here. Note: You need to go to "Files and versions" -> then download the tar gz file. The download may take some time depending on your internet speed. There are several large files in the tar ball. Also, note, that the TechQA leaderboard has been sunset. However, you're welcome to use the data. Please cite the paper in your publications/ experiments.

Installation

After cloning this repo, install dependencies using

pip install -r requirements.txt

If you want to run with fp16, you need to install Apex

Training a model

In order to train a model on TechQA, use the script below.

Note: Since TechQA is smaller dataset, it is better to start with a model that is already trained on a bigger QA dataset. Here, we start with BERT-Large trained on Squad.

python run_techqa.py \
    --model_type bert \
    --model_name_or_path bert-large-uncased-whole-word-masking-finetuned-squad \
    --do_lower_case \
    --learning_rate 5.5e-6 \ 
    --do_train \
    --num_train_epochs 20 \
    --train_file <PATH TO training_Q_A.json> \
    --do_eval \
    --predict_file <PATH TO dev_Q_A.json> \
    --input_corpus_file <PATH TO training_dev_technotes.json> \
    --overwrite_output_dir \
    --output_dir <PATH TO OUTPUT FOLDER> \ 
    --add_doc_title_to_passage 

You can add the --fp16 flag if you have apex installed.

To evaluate a model, you can run:

python run_techqa.py \
    --model_type bert \
    --model_name_or_path <PATH TO TRAINED MODEL FOLDER> \
    --do_lower_case \
    --do_eval \
    --predict_file <PATH TO dev_Q_A.json> \
    --input_corpus_file <PATH TO training_dev_technotes.json> \
    --overwrite_output_dir \
    --output_dir <PATH TO OUTPUT FOLDER> \ 
    --add_doc_title_to_passage 

Contact

For help or issues, please submit a GitHub issue.

For direct communication, please contact Avi Sil ([email protected]).

techqa's Issues

Unable to register for acquiring the TechQA Dataset

There is no Sign up link for registering for getting a login in order to get the TechQA Dataset. Could you please how I could get the dataset?

Can't access http://ibm.biz/Tech_QA

Hi, I tried to access http://ibm.biz/Tech_QA, but it's not working. Is the website accessible now? Or, where can I get the dataset?

pre-trained model weights

Hi, can i get weight of model pre-trained at techqa for reporting performance at techQA paper?

I got this error AttributeError: Can't get attribute 'TechQaInputFeature' on <module '__main__' from 'run_techqa.py'> while training the model

Code for the bert on NQ baseline and TAP method used in the paper

Could you share code for the BERT on NQ baseline and TAP method used in the paper?

How to set threshold for Test docker image for leaderboard submission

I was wondering how we can really set the threshold (to determine if the question has an answer or not) after we tuned it on our dev_set in the docker image creation as the submission to the leaderboard?

How can we keep training from checkpoint

To continue training from checkpoints, do we only need to change --model_name_or_path to where a specific checkpoint folder is? Do we have to keep --overwrite_output_dir or so? Thank you!