VQA-CPv2 about murel.bootstrap.pytorch

[Zzmonica]

Sorry to bother: How to test VQA-CP v2 ? Does it share the same test dataset with VQA v2 ? And the results on VQA-CP v2 is 39.54%. Which dataset to be tested on? Should it submit on the website to get results?
Hope for your respond, thanks~

[nicholasly]

Test datasets are different. You can visit https://www.cc.gatech.edu/~aagrawal307/vqa-cp/ for more details about VQA-CP v2.

[Cadene]

Hi @Zzmonica and @nicholasly,
Thanks for your interest!

You can run experiments on VQACPv2 using this command line:

python -m bootstrap.run \
-o murel/options/vqacp2/murel.yaml \
--exp.dir logs/vqacp2/murel

And then you can display a comparison of several experiments with:

python -m murel.compare_vqa_val -d logs/vqacp2/murel logs/vqacp2/murel_exp2

VQACPv2 comes with a train set and a test set (with annotations):

$ ls data/vqa/vqacp2/raw/annotations/
mscoco_train2014_annotations.json  OpenEnded_mscoco_train2014_questions.json  vqacp_v2_test_annotations.json  vqacp_v2_train_annotations.json
mscoco_val2014_annotations.json    OpenEnded_mscoco_val2014_questions.json    vqacp_v2_test_questions.json    vqacp_v2_train_questions.json

As the test set comes with answer annotations (not like in VQA 2.0 dataset), we consider the test set as a val set. See some preprocessing of the data.

[Zzmonica]

@Cadene Thanks for your respond !

• Oh I know, the results of VQA-CP v2 is actually testing on the val set (for that it has annotations).

• I was also wondering that what type of GPU did you use to train the VQA v2 ? And some details of the training setup?

• About the training results: I have the same experiment setup as your repository, didn't change anything, but the results on VQA v2 val dataset is lower than yours, like the overall result is ~0.6% lower than baseline's 65.14%.

Any suggestions? Hope for your respond, Thanks!

[Cadene]

@Zzmonica

I was also wondering that what type of GPU did you use to train the VQA v2 ? And some details of the training setup?

pytorch 0.3 and 1 * Titan X Pascal and 4 * Titan X Pascal for the biggest model (pairwise + n_step=3)

About the training results: I have the same experiment setup as your repository, didn't change anything, but the results on VQA v2 val dataset is lower than yours, like the overall result is ~0.6% lower than baseline's 65.14%. Any suggestions?

Unfortunately, I am not able to reproduce this result on pytorch 1.1 with the current code... It is largely due to an issue with the backward pass. That's a shame and I am working on it right now. I should be able to push a fix and upload pretrained models that reach/surpass 65.14% before the end of this week. I am so sorry for the inconvenience.

For more info about the issue:
https://gist.github.com/Cadene/a00eba2db81f44b4492dc8941f6abe45

(pytorch.0.30) $ python bug_pytorch03.py
1.8359577655792236

(pytorch.1.1) $ python bug_pytorch11_slow.py
3.8738763332366943

(pytorch.1.1) $ python bug_pytorch11_nograd.py
0.9558875560760498

(pytorch.1.1) $ python bug_pytorch11_detach.py
0.955887556076049

[Cadene]

@Zzmonica
7c9eaeb

FIXED!!!!

YEEAAAAAHAHAHDHASHUASDHSYHDS

[Zzmonica]

Thank you very much!!! I'll try that.

[Zzmonica]

I retrained the murel model, and the results of the modified version is even lower than the past version. So confused.

[xywhat]

can i get your log ?the result of VQACPV2