cyanogenoid / pytorch-vqa Goto Github PK
View Code? Open in Web Editor NEWStrong baseline for visual question answering
Strong baseline for visual question answering
The paper said questions are capped at 15 words, but I can't find any mention of 15 in the code.
The answer normalization code is different from the official VQA code.
Can we assume this?
"normalizations is not needed, assuming that the human answers are already normalized."
The official code removes the articles (a, the) but this code doesn't.
There are actually "the" in answers.
Hello,
Thanks for the implementation. The paper does not detail how the LSTM encoding and the feature maps are fused altogether but only provides the Figure 2 where it says "the concatenated image features and the final state of the LSTM are then used to compute multiple attention distributions over image features". They also draw a Concat
box in the diagram that receives the tiled LSTM state and the spatial feature maps as input.
I was experimenting with this idea for another task where I did the fusion with concatenation across the channel dimension but looking at your code I see that after tiling the q
vector, you simply do a self.relu(v + q)
. Did you have some insight about this step, maybe some discussions with the authors?
Thanks!
I have this issue:
OSError: Unable to create file (unable to open file: name = '/ssd/resnet-14x14.h5', errno = 2, error message = 'No such file or directory', flags = 15, o_flags = c2)
Is there a way to convert the preprocess-images.py to a version that doesnt require CUDA?
Attempting to run this code without any modifications sometimes results in this error:
will save to logs/2017-09-26_17:30:02.pth
train E000: 0% 0/3396 [00:00<?, ?it/s]Traceback (most recent call last):
File "train.py", line 128, in <module>
main()
File "train.py", line 109, in main
_ = run(net, train_loader, optimizer, tracker, train=True, prefix='train', epoch=i)
File "train.py", line 55, in run
out = net(v, q, q_len)
File "/usr/lib/python3.6/site-packages/torch/nn/modules/module.py", line 224, in __call__
result = self.forward(*input, **kwargs)
File "/usr/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 58, in forward
return self.module(*inputs[0], **kwargs[0])
File "/usr/lib/python3.6/site-packages/torch/nn/modules/module.py", line 224, in __call__
result = self.forward(*input, **kwargs)
File "/home/peter/Documents/Repositories/pytorch-vqa/model.py", line 51, in forward
v = v / (v.norm(p=2, dim=1).expand_as(v) + 1e-8)
File "/usr/lib/python3.6/site-packages/torch/autograd/variable.py", line 725, in expand_as
return Expand.apply(self, (tensor.size(),))
File "/usr/lib/python3.6/site-packages/torch/autograd/_functions/tensor.py", line 111, in forward
result = i.expand(*new_size)
RuntimeError: The expanded size of the tensor (2048) must match the existing size (128) at non-singleton dimension 1. at /tmp/yaourt-tmp-peter/aur-python-pytorch/src/pytorch-0.2.0/torch/lib/THC/generic/THCTensor.c:323
Hi,
Thank you so much for providing the code. Can you please give me some indication about the training time. On my google VM with Tesla k80 its takes around 35 mins for one train epoch. Is that what you faced?
Thanks
in model.py line 114: x = self.relu(v + q)
in the forward method of Class 'Attention',
why not use '*' ?
According to the code from official VQA api , accuracy is accounted only when agreeing answer is not same as discarded because in line 98, it is computing other GT answers which are not same as discarded one. But it varies from what you have done. you are assuming that it will be matched to all but discarded answer while in their API, they are computing it as if it will be matched to all but discarded and other which are same as discarded answers.
So, seems like it will (10-agreeing)*min(agreeing/3,1).
Please correct me if I am wrong.
Can you please provide some information regarding the training time (with the hardware specifications)?
Thanks!
Can you specify how exactly can i test the model i.e given an image with question the model is expected to return answers with confidence.
Hi,
I read your code from start to finish and found it very clear and modular.
Based on your implementation, with some minor modifications, an overall accuracy result of 60.16 (original paper is 59.67) on VQA v2.0 validation set as well as 64.99 (original paper is 64.5) on VQA v1.0 test-dev were observed! It's very impressive and promising which means we can easily design our own algorithm on this code.
But there is a small issue confusing me (maybe stupid). That the computation of two metrics (loss and acc) in the training phase doesn't make much sense. What does the momentum do? How about just using the batch loss and acc?
Thanks.
The Error Log is as following
/home/mmc_xhma/software/anconda3/bin/python3.6 /home/mmc_xhma/code/TMM_2017/pytorch-vqa-master/preprocess-images.py
/home/mmc_xhma/software/anconda3/lib/python3.6/site-packages/h5py/init.py:36: FutureWarning: Conversion of the second argument of issubdtype fromfloat
tonp.floating
is deprecated. In future, it will be treated asnp.float64 == np.dtype(float).type
.
from ._conv import register_converters as _register_converters
/home/mmc_xhma/software/anconda3/lib/python3.6/site-packages/torchvision-0.2.0-py3.6.egg/torchvision/transforms/transforms.py:156: UserWarning: The use of the transforms.Scale transform is deprecated, please use transforms.Resize instead.
found 82783 images in mscoco/train2014
found 40504 images in mscoco/val2014
0%| | 0/123287 [00:00<?, ?it/s]
Traceback (most recent call last):
File "/home/mmc_xhma/code/TMM_2017/pytorch-vqa-master/preprocess-images.py", line 79, in
main()
File "/home/mmc_xhma/code/TMM_2017/pytorch-vqa-master/preprocess-images.py", line 70, in main
out = net(imgs)
File "/home/mmc_xhma/software/anconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 357, in call
result = self.forward(*input, **kwargs)
File "/home/mmc_xhma/code/TMM_2017/pytorch-vqa-master/preprocess-images.py", line 31, in forward
self.model(x)
File "/home/mmc_xhma/software/anconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 357, in call
result = self.forward(*input, **kwargs)
File "/home/mmc_xhma/software/anconda3/lib/python3.6/site-packages/torchvision-0.2.0-py3.6.egg/torchvision/models/resnet.py", line 151, in forward
File "/home/mmc_xhma/software/anconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 357, in call
result = self.forward(*input, **kwargs)
File "/home/mmc_xhma/software/anconda3/lib/python3.6/site-packages/torch/nn/modules/linear.py", line 55, in forward
return F.linear(input, self.weight, self.bias)
File "/home/mmc_xhma/software/anconda3/lib/python3.6/site-packages/torch/nn/functional.py", line 835, in linear
return torch.addmm(bias, input, weight.t())
RuntimeError: size mismatch at
/opt/conda/conda-bld/pytorch_1518243271935/work/torch/lib/THC/generic/THCTensorMathBlas.cu:247
config.py
preprocess_batch_size = 64
image_size =448 # scale shorter end of image to this size and centre crop
output_size = image_size // 32 # size of the feature maps after processing through a network
output_features = 2048 #2048 # number of feature maps thereof
central_fraction = 0.875 # only take this much of the centre when scaling and centre
when the parameters are set as default , the error occured.
I have checked the input of the last fc layer in resnet152 , the input shape is [64 131072], however the weight martix shape is [2048 1000] and bias is none.
File "/home/mmc_xhma/software/anconda3/lib/python3.6/site-packages/torch/nn/functional.py", line 835, in linear return torch.addmm(bias, input, weight.t())
Obviously , the size mismatch.
How can I fix the error.
In place of MSCOCO images, would the code work as is with the abstract scene images? We will place the images in the same folders as mentioned in the config.py folder.
I am getting error when running preprocess-images.py file because I don't have resnet-14x14.h5 and vocab.json files. How can I get these files?
Thanks in advance!!
Hi
I got this error running your code
AttributeError: module ‘torchvision.transforms’ has no attribute ‘Scale’
I'm using torchvision 0.12, and Scale
was deprecated in favour of Resize
Modifying this line of code fixes it.
Hi, I got the following issue when I ran train.py. Could you please help me to fix it?
will save to logs\2019-02-21_13:49:31.pth
D:\blondie\pytorch-vqa\model.py:90: UserWarning: nn.init.xavier_uniform is now deprecated in favor of nn.init.xavier_uniform_.
init.xavier_uniform(w)
D:\blondie\pytorch-vqa\model.py:86: UserWarning: nn.init.xavier_uniform is now deprecated in favor of nn.init.xavier_uniform_.
init.xavier_uniform(self.embedding.weight)
D:\blondie\pytorch-vqa\model.py:44: UserWarning: nn.init.xavier_uniform is now deprecated in favor of nn.init.xavier_uniform_.
init.xavier_uniform(m.weight)
train E000: 0% 0/3396 [00:00<?, ?it/s]Traceback (most recent call last):
File "train.py", line 134, in
main()
File "train.py", line 115, in main
_ = run(net, train_loader, optimizer, tracker, train=True, prefix='train', epoch=i)
File "train.py", line 45, in run
for v, q, a, idx, q_len in tq:
File "C:\Users\Konstantinos\Miniconda3\lib\site-packages\tqdm_tqdm.py", line 1002, in iter
for obj in iterable:
File "C:\Users\Konstantinos\Miniconda3\lib\site-packages\torch\utils\data\dataloader.py", line 819, in iter
return _DataLoaderIter(self)
File "C:\Users\Konstantinos\Miniconda3\lib\site-packages\torch\utils\data\dataloader.py", line 560, in init
w.start()
File "C:\Users\Konstantinos\Miniconda3\lib\multiprocessing\process.py", line 112, in start
self._popen = self._Popen(self)
File "C:\Users\Konstantinos\Miniconda3\lib\multiprocessing\context.py", line 223, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "C:\Users\Konstantinos\Miniconda3\lib\multiprocessing\context.py", line 322, in _Popen
return Popen(process_obj)
File "C:\Users\Konstantinos\Miniconda3\lib\multiprocessing\popen_spawn_win32.py", line 65, in init
reduction.dump(process_obj, to_child)
File "C:\Users\Konstantinos\Miniconda3\lib\multiprocessing\reduction.py", line 60, in dump
ForkingPickler(file, protocol).dump(obj)
File "C:\Users\Konstantinos\Miniconda3\lib\site-packages\torch\multiprocessing\reductions.py", line 286, in reduce_storage
metadata = storage.share_filename()
RuntimeError: Couldn't map view of shared file <torch_3844_2531614511>, error code: <5>
Traceback (most recent call last):
File "", line 1, in
File "C:\Users\Konstantinos\Miniconda3\lib\multiprocessing\spawn.py", line 105, in spawn_main
exitcode = _main(fd)
File "C:\Users\Konstantinos\Miniconda3\lib\multiprocessing\spawn.py", line 115, in _main
self = reduction.pickle.load(from_parent)
EOFError: Ran out of input
I have such warning pytorch-vqa/model.py:96: UserWarning: RNN module weights are not part of single contiguous chunk of memory. This means they need to be compacted at every call, possibly greatly increasing memory usage. To compact weights again call flatten_parameters().
Even though I use self.lstm.flatten_parameters()
before _, (_, c) = self.lstm(packed)
, the programme consume almost all of my memory (16G), which is abnormal. In the issues before, you states that you can run epoch in 7 min, which I guess is because you have SSD.
Let's check the code to see what causes memory leak :)
Hi, I am trying to replicate your work on Jupyter notebook with Pytorch. Bu when I tried to run preprocess-image, it keeps telling me this error.
Really don't know what's wrong. I am following exactly as the instructions in README except I used Resnet from torch instead of caffe. But I don't think that is the reason.
Hello,
I wanted a way to look at the images in the main training/validation loop of train.py
I wasn't able to do this.
On the one hand, the images from tq.iterable.dataset.coco_ids
does not sync with the question and answers (which do sync).
On the other hand, the dictionary tq.iterable.dataset.coco_id_to_index
contains 123287 keys, which means that it contains both train and validation sets (82783+40504).
When I "reverse" this dictionary, the mapping from idx
(where idx
comes from for v, q, a, idx, q_len in tq:
) to coco_id isn't correct (for example, idx from the val_loader maps to coco_ids in the training set).
Could you take a look at this?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.