nextplusplus / tat-qa Goto Github PK

I used the official evaluation script with below code line

!python tatqa_eval.py --gold_path=/content/tatqa_dataset_dev.json --pred_path=/content/sample_prediction.json

and I got the following output

Exact-match accuracy 45.92
F1 score 58.88
Scale score 90.95
45.92 & 58.88

---- raw detail ---
em ...
answer_from table ... text
answer_type ...
arithmetic 497.0 ... 16.0
count 12.0 ... 0.0
multi-span 92.0 ... 24.0
span 171.0 ... 349.0

[4 rows x 3 columns]
---- em detail ---
em ...
answer_from table ... text
answer_type ...
arithmetic 0.531187 ... 0.000000
count 0.500000 ... 0.000000
multi-span 0.684783 ... 0.083333
span 0.584795 ... 0.045845

[4 rows x 3 columns]
---- f1 detail ---
f1 ...
answer_from table ... text
answer_type ...
arithmetic 0.531187 ... 0.000
count 0.500000 ... 0.000
multi-span 0.833696 ... 0.255
span 0.584795 ... 0.540

[4 rows x 3 columns]

Why does it not show the answer_from table-text

It is a great job! But I have a question here: there is no tatqa_dataset_test.json under the dataset_tagop directory. I think we need it for prediction. How can we generate it?

dear author,
there is no answer in test set.
And I found it is validated in dev set and evaluated also in dev set
And I am so confused about it

Hi do you happen to have the CIK or the company name and the year for each sample in the dataset. i.e

{
   uid: ...,
   cik: ...,
   company_name: ...,
   year: ...
}

Thank you very much for any help you can offer.

I run the code on Goggle Colab GPU. When I run the !CUDA_VISIBLE_DEVICES=2 PYTHONPATH=$PYTHONPATH:$(pwd) python tag_op/trainer.py --data_dir tag_op/cache/ \ --save_dir ./checkpoint --batch_size 4 --eval_batch_size 2 --max_epoch 3 --warmup 0.06 --optimizer adam --learning_rate 5e-4 \ --weight_decay 5e-5 --seed 123 --gradient_accumulation_steps 4 --bert_learning_rate 1.5e-5 --bert_weight_decay 0.01 \ --log_per_updates 50 --eps 1e-6 --encoder roberta command. I get the following error:
10/12/2021 10:32:46 At epoch 1
Traceback (most recent call last):
File "tag_op/trainer.py", line 130, in
main()
File "tag_op/trainer.py", line 109, in main
model.update(batch)
File "/content/TatQA/tag_op/tagop/model.py", line 98, in update
output_dict = self.mnetwork(**tasks)
File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "/content/TatQA/tag_op/tagop/modeling_tagop.py", line 221, in forward
reduce_max_index_get_vector(table_tag_prediction[:, :, 1], table_sequence_output, table_cell_index)
File "/content/TatQA/tag_op/tagop/modeling_tagop.py", line 849, in reduce_max_index_get_vector
return _index_reduce_max_get_vector(values_for_reduce, values_for_reference, index, max_length, name)
File "/content/TatQA/tag_op/tagop/modeling_tagop.py", line 1047, in _index_reduce_max_get_vector
reduce_values, reduce_index = scatter_max(
NameError: name 'scatter_max' is not defined

Please help over it.

When I used the below command line
python tag_op/prepare_dataset.py --mode [train/dev/test]
I used the “tatqa_dataset_test.json” from the folder “dataset_raw”
The generated file “tagop_roberta_cached_test.pkl” is empty. How can I generate the right “tagop_roberta_cached_test.pkl”?
Thanks for your help!

I am looking for pretrained model weights (best_checkpoint.pt) of the tagop model to re-create the results. Could you provide it?

@fengbinzhu

hi, guys👋 Could you please upload the dataset to HuggingFace so that it can be easily downloaded?😊