deepinsight-pcalab / compactbilinearpooling-pytorch

A Pytorch Implementation for Compact Bilinear Pooling.

CompactBilinearPooling-Pytorch

A Pytorch Implementation for Compact Bilinear Pooling. Adapted from tensorflow_compact_bilinear_pooling

Prerequisites

Install pytorch_fft by

pip install pytorch_fft

Usage

from torch import nn
from torch.autograd import Variable
from CompactBilinearPooling import CompactBilinearPooling

bottom1 = Variable(torch.randn(128, 512, 14, 14)).cuda()
bottom2 = Variable(torch.randn(128, 512, 14, 14)).cuda()

layer = CompactBilinearPooling(512, 512, 8000)
layer.cuda()
layer.train()

out = layer(bottom1, bottom2)

Reference

Yang Gao, et al. "Compact Bilinear Pooling." in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (2016).
Akira Fukui, et al. "Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding." arXiv preprint arXiv:1606.01847 (2016).

compactbilinearpooling-pytorch's Issues

Compact Bilinear Pooling in pytorch1.0+

Could you please add Compact Bilinear Pooling in pytorch1.0+
THS

Does the current implementation include back propagation

RT

NotImplementedError in fft

I am getting this particular error while running sample test given in readme.

  File "CompactBilinearPooling/test.py", line 14, in <module>
    out = layer(bottom1, bottom2) 
  File "anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in __call__
    result = self.forward(*input, **kwargs)
  File "CompactBilinearPooling/CompactBilinearPooling.py", line 91, in forward
    fft1_real, fft1_imag = afft.Fft()(sketch_1, Variable(torch.zeros(sketch_1.size())).cuda())
  File "anaconda3/lib/python3.6/site-packages/pytorch_fft/fft/autograd.py", line 16, in forward
    return fft(X_re, X_im)
  File "anaconda3/lib/python3.6/site-packages/pytorch_fft/fft/fft.py", line 25, in fft
    raise NotImplementedError
NotImplementedError

Now this is due to this function fft.py:

    if 'Float' in type(X_re).__name__ :
        f = th_fft.th_Float_fft1
    elif 'Double' in type(X_re).__name__: 
        f = th_fft.th_Double_fft1
    else: 
        raise NotImplementedError
    return _fft(X_re, X_im, f, 1)

Because inputs to fft() are tensors, type(sketch_1) does not include 'Float' or 'Double'.
Any help is appreciated.

No module named _th_fft

Hi, thanks for your job!
lt looks like there is no module named _th_fft when follow the command as you said.
Do you have any idea about it?
Thanks a lot~
`
dl@dl:~/wxptest$ sudo python wxp.py

Traceback (most recent call last):
File "wxp.py", line 3, in
from CompactBilinearPooling import CompactBilinearPooling
File "/home/dl/wxptest/CompactBilinearPooling.py", line 6, in
import pytorch_fft.fft.autograd as afft
File "/home/dl/wxptest/pytorch_fft/init.py", line 1, in
from . import fft
File "/home/dl/wxptest/pytorch_fft/fft/init.py", line 1, in
from .fft import *
File "/home/dl/wxptest/pytorch_fft/fft/fft.py", line 3, in
from .._ext import th_fft
File "/home/dl/wxptest/pytorch_fft/_ext/th_fft/init.py", line 3, in
from ._th_fft import lib as _lib, ffi as _ffi
ImportError: No module named _th_fft
`

No parameters for optmizer

Thanks for your working, when I apply your code, I meet the situation that model.parameters() is empty and fail to creat optimizer. Model is compactbilinearpooling(1536,1536,400). Am I wrong? Thanks for answering

Complex product should be used in eltwise product

I was investigating differences in results of this package and of the original Caffe version. Things found so far:

1. Rfft can be used directly, it is provided by pytorch_fft
2. Complex product should be used here: https://github.com/DeepInsight-PCALab/CompactBilinearPooling-Pytorch/blob/master/CompactBilinearPooling.py#L91, the tf.multiply does complex product, original caffe version does complex product
3. Output should be mutliplied by output_dim in order to achieve full equivalence with original caffe version

pytorch_fft install

when I install pytorch_fft,the problem is：
error in pytorch_fft setup command: 'C:\Users\14375\AppData\Local\Temp\pip-install-6vh12s8w\pytorch-fft\build.py:ffi' must be of the form 'path/build.py:ffi_va
riable'
Including CUDA code.
----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

Possible error in computation

According to the existing code beginning here

CompactBilinearPooling-Pytorch/CompactBilinearPooling.py

Line 92 in 56e23d0

fft_product_real = temp_rr - temp_ii

the product of two complex numbers(a+ib, x+iy) is {(ax-by) + i(ax+by)}.

   `temp_rr, temp_ii = fft1_real.mul(fft2_real), fft1_imag.mul(fft2_imag)
    fft_product_real = temp_rr - temp_ii
    fft_product_imag = temp_rr + temp_ii

    cbp_flat = afft.Ifft()(fft_product_real, fft_product_imag)[0]`

However the correct product is {(ax-by) + i(ay+bx)} in which case the following product is wrong.

Correct me if I am wrong.

How's the output

Thank you for your job! I have some question, if x = (4,512,64,64), y = (4,512,64,64), a=CompactBilinearPooling(x,y) ， what the shape of a ? And How I can make the a‘s shape is (batch_size,channels,height,width) which the same as x and y？

multi GPUs error: RuntimeError: arguments are located on different GPUs

For multi GPU, it outputs:

RuntimeError: arguments are located on different GPUs at /pytorch/torch/lib/THC/generic/THCTensorMathBlas.cu:236

How to fix it?

In the code seem to be different from the setting of the paper.

rand_h_1 = np.random.randint(output_dim, size=self.input_dim1) and rand_s_1 = 2 * np.random.randint(2, size=self.input_dim1) - 1 in the code seem to be different from the setting of the paper. In the paper, hk is {1,2,...,k}, sk is {+1 ,-1}.

Update fft using pytorch 0.4

Hi,

Is it possible for you to upgrade the code using pytorch fft. As you own this code, it will be easy for you to do so.

Regards,

gender recognition based on the whole body

Can I use gender recognition based on the whole body?

out of cuda memory

I just run the test, and it run out of my cuda memory, which is 20G. Have I miss something?