Comments (9)
Sorry to bother: How to test VQA-CP v2 ? Does it share the same test dataset with VQA v2 ? And the results on VQA-CP v2 is 39.54%. Which dataset to be tested on? Should it submit on the website to get results?
Hope for your respond, thanks~
from murel.bootstrap.pytorch.
Test datasets are different. You can visit https://www.cc.gatech.edu/~aagrawal307/vqa-cp/ for more details about VQA-CP v2.
from murel.bootstrap.pytorch.
Hi @Zzmonica and @nicholasly,
Thanks for your interest!
You can run experiments on VQACPv2 using this command line:
python -m bootstrap.run \
-o murel/options/vqacp2/murel.yaml \
--exp.dir logs/vqacp2/murel
And then you can display a comparison of several experiments with:
python -m murel.compare_vqa_val -d logs/vqacp2/murel logs/vqacp2/murel_exp2
VQACPv2 comes with a train set and a test set (with annotations):
$ ls data/vqa/vqacp2/raw/annotations/
mscoco_train2014_annotations.json OpenEnded_mscoco_train2014_questions.json vqacp_v2_test_annotations.json vqacp_v2_train_annotations.json
mscoco_val2014_annotations.json OpenEnded_mscoco_val2014_questions.json vqacp_v2_test_questions.json vqacp_v2_train_questions.json
As the test set comes with answer annotations (not like in VQA 2.0 dataset), we consider the test set as a val set. See some preprocessing of the data.
from murel.bootstrap.pytorch.
@Cadene Thanks for your respond !
-
Oh I know, the results of VQA-CP v2 is actually testing on the val set (for that it has annotations).
-
I was also wondering that what type of GPU did you use to train the VQA v2 ? And some details of the training setup?
-
About the training results: I have the same experiment setup as your repository, didn't change anything, but the results on VQA v2 val dataset is lower than yours, like the
overall
result is ~0.6% lower than baseline's 65.14%.
Any suggestions? Hope for your respond, Thanks!
from murel.bootstrap.pytorch.
I was also wondering that what type of GPU did you use to train the VQA v2 ? And some details of the training setup?
pytorch 0.3 and 1 * Titan X Pascal and 4 * Titan X Pascal for the biggest model (pairwise + n_step=3)
About the training results: I have the same experiment setup as your repository, didn't change anything, but the results on VQA v2 val dataset is lower than yours, like the overall result is ~0.6% lower than baseline's 65.14%. Any suggestions?
Unfortunately, I am not able to reproduce this result on pytorch 1.1 with the current code... It is largely due to an issue with the backward pass. That's a shame and I am working on it right now. I should be able to push a fix and upload pretrained models that reach/surpass 65.14% before the end of this week. I am so sorry for the inconvenience.
For more info about the issue:
https://gist.github.com/Cadene/a00eba2db81f44b4492dc8941f6abe45
(pytorch.0.30) $ python bug_pytorch03.py
1.8359577655792236
(pytorch.1.1) $ python bug_pytorch11_slow.py
3.8738763332366943
(pytorch.1.1) $ python bug_pytorch11_nograd.py
0.9558875560760498
(pytorch.1.1) $ python bug_pytorch11_detach.py
0.955887556076049
from murel.bootstrap.pytorch.
FIXED!!!!
YEEAAAAAHAHAHDHASHUASDHSYHDS
from murel.bootstrap.pytorch.
Thank you very much!!! I'll try that.
from murel.bootstrap.pytorch.
I retrained the murel model, and the results of the modified version is even lower than the past version. So confused.
from murel.bootstrap.pytorch.
can i get your log ?the result of VQACPV2
from murel.bootstrap.pytorch.
Related Issues (20)
- a question about training with/out VGenome HOT 4
- code question HOT 1
- result reproduce
- Hello
- How to visualize the loss per batch by tensorboard?
- Hello, I need to extract features from a pretrained Faster-RCNN with caffe because i want to use different dataset
- About the `2018-04-27_bottom-up-attention_fixed_36` features HOT 1
- What does the self_q_att attribute do in murel.yaml?
- Answer vocabulary HOT 3
- inference
- FileNotFoundError HOT 1
- Access to Murel predictions on test set HOT 3
- unknown_params = parameters[dictionary['UNK']] KeyError: 'UNK' HOT 1
- Overfitting!!!
- 该项目是借助 block.bootstrap.pytorch.实现的吗
- VQACP2 test
- Hello, I just want to use your model to test directly on other data sets. Could you provide the model that is finally used for testing?
- SSL Certificate expired HOT 1
- Evaluation on test set HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from murel.bootstrap.pytorch.