Issues in setting up demo for SQuAD 1.1 data about denspi

[alphaf52]

Hi,

I am facing a similar issue when I downloaded the Pretrained Model and then tried to eval on dev-v1.1 using: "python run_piqa.py --do_predict --output_dir tmp --do_load --load_dir model --predict_file dev-v1.1.json --do_eval --gt_file dev-v1.1.json --metadata_dir bert"

The predicted answer seems to be random span, resulting in a metric like: {"exact_match": 0.47303689687795647, "f1": 4.43806570152543}. 0.47% EM means something is totally wrong.

Did I do anything incorrectly? Thanks in advance.

[jhyuklee]

Hi, the model at gs://denspi/v1-0/model should work fine. I think the model wasn't loaded properly somehow. Did you try using the full dump provided at gs://denspi/v1-0/dump and run the demo?

[pragnakalpdev]

Hi,
I have not used the full dump provided at gs://denspi/v1-0/dump, I have created my own custom phrase index. Used the created custom phrase index with the model given at gs://denspi/v1-0/model. I am not getting the correct answers, answers seems to be very random.

[jhyuklee]

Hi, i think it's related to #10. Have you tried giving --parallel option when making the dump?

[mittalpatel]

Hello @jhyuklee Thanks for your comment. We are in the same team. So consider this fixed as well. 👍

[jhyuklee]

Hi @mittalpatel, great! Thank you.