1zb / deformable-convolution-pytorch Goto Github PK

First, thanks for the great work.
I found there's possible memory leak in the code.
To reproduce the problem, change the test.py as below:

import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.autograd import Variable
from torch.autograd import gradcheck
from torch.nn.modules.utils import _single, _pair

from modules import ConvOffset2d

num_deformable_group = 1

N, inC, inH, inW = 1, 3, 512, 512
outC, outH, outW = 4, 512, 512
kH, kW = 3, 3

conv = nn.Conv2d(inC, num_deformable_group * 2 * kH * kW, kernel_size=(kH, kW), stride=(1,1), padding=(1,1), bias=False).cuda()
conv_offset2d = ConvOffset2d(inC, outC, (kH, kW), stride=1, padding=1).cuda()

for epoch in range(500):
    inputs = Variable(torch.randn(N, inC, inH, inW).cuda())
    offset = conv(inputs)
    output = conv_offset2d(inputs, offset)
    output.backward(output.data)
    print(output.size())

Code will crash for OOM while doing the backward()

• 我使用 gradcheck 提供的梯度测试方法，测试梯度不通过。比较数值梯度和解析梯度，其每一个非零数（非边界位置）的符号，位置等都是一样的，但就是数值差了一点。
• 另外，在cifar10上简单测试，没有效果，它在使用上有什么要注意的吗，可不可以交流一下。

Hi, @1zb, Thanks for your great work! Can you add a license to this repository such that this code can be used in our own projects?

An excellent you have done! But I don't know why it runs very slow when it is embeded into my network. I also refer to the implementation of https://github.com/ChunhuanLin/deform_conv_pytorch, pure pytorch without c, cu files, which is faster than yours but costs more memory.

I want to use 3D version of deformable convolution, but it seems like your code only support 2D. What change should be made to adapt to 3D convolution? Is it possible to just change the number of offsets?

请问pytorch版本的performance和mxnet相比如何呢？

感谢您开源代码　但是ｔｅｓｔ出问题了　能麻烦您给一个类似ｍnist的测试例子吗

sun@sunwin:~/0newcodedown/deformable-convolution-pytorch-master$ python test.py
Traceback (most recent call last):
File "test.py", line 6, in
from modules import ConvOffset2d
File "/home/sun/0newcodedown/deformable-convolution-pytorch-master/modules/init.py", line 1, in
from .deform_conv import ConvOffset2d
File "/home/sun/0newcodedown/deformable-convolution-pytorch-master/modules/deform_conv.py", line 7, in
from functions import conv_offset2d
File "/home/sun/0newcodedown/deformable-convolution-pytorch-master/functions/init.py", line 1, in
from .deform_conv import conv_offset2d
File "/home/sun/0newcodedown/deformable-convolution-pytorch-master/functions/deform_conv.py", line 5, in
from _ext import deform_conv
ImportError: No module named _ext

hello,is it useful to use dcn on 1x1conv?

#According to test.py, the line 'offset=conv(inputs)' is to learn the offset of neighbor domain of each pixel, so the size of offsets is 118H*W. if the size of input is 1x3xHxW and the size of output is 1x4xHxW, does it means that there has 4 kernels，and each kernel shares the same offsets？

run nvcc -c -o deform_conv_cuda_kernel.cu.o deform_conv_cuda_kernel.cu -x cu -Xcompiler -fPIC -std=c++11 happens error，nvcc fatal : Cannot find compiler 'cl.exe' in PATH

When the size of input image is larger than 1024, below error is occurred.

error in deformable_im2col: invalid argument

This error need to be fixed.

First off, thank you for your amazing port, @1zb!

How does num_deformable_groups work? I was wondering if it is splitting the offsets, thus each group of input channels will have a given set of offsets (which may be different from other groups). For instance, if we have an input of 6 channels (1, 6, H, W), and num_deformable_groups=2, then we would have our input split between two tensors of (1, 3, H, W) and (1, 3, H, W), each with its respective instance of offset mask (1, 3, kH, kW). Is that correct?

I'm trying to extend ConvOffset2d to 1d convolution as well.

While this can be easily done by simply consider a length-L 1D tensor as a dimension [1xL] 2D tensor, the problem is that the offset now is for 2D deformation (you can go up or down, left or right); whereas in 1D, I only want the kernel connection to move by left/right.

I'm seeking to change the .c and .cu code so that I no longer need to worry about the up/down offset, and in offset = Conv2d(...) I only need output dimension of num_deformable_groups * kernel_size, instead of num_deformable_groups * 2 * kernel_size`.

Any suggestion on which parts of the .c and .cu that I should change? Thank :-)

Thinking about deformable convolutions, some things I found different interpretations of:
-) Does the offset change for each individual k x k kernel or is it fixed for the whole image? Would this mean that pixels could potentially overlap?
-) Is the same offset then applied for each input layer, ie. AxBxC where C might be any number of filters.
-) During inference, keeping the offset generating layers in the network, each k x k kernel would experience an individual offset, or would the offset be the same for the whole image?

Hi, @1zb, great work! Did you try to measure performance/loss difference of your implementation compared to ordinary convolutions?

Thanks for your code firstly.
In this implementation, the spatial resolution of offset should be the same with output feature map.
But in the original paper, the spatial resolution of offset is the same with input feature map.
Does this have an effect on the performance?

I ran the sh make.sh and then CC=g++ python build.py. It seems that It is building fine, however after running the test.py I have got error. Please note that in the _ext/deform_conv I have only an init.py and _deform_conv.
I really appreciate your help.

sh make.sh
CC=g++ python build.py
Including CUDA code.
/home/mika/wd/deformable-convolution-pytorch-master
generating /tmp/tmp_sneiacp/_deform_conv.c
setting the current directory to '/tmp/tmp_sneiacp'
running build_ext
building '_deform_conv' extension
creating home
creating home/mika
creating home/mika/wd
creating home/mika/wd/deformable-convolution-pytorch-master
creating home/mika/wd/deformable-convolution-pytorch-master/src
g++ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -DWITH_CUDA -I/home/mika/.conda/envs/py36/lib/python3.6/site-packages/torch/utils/ffi/../../lib/include -I/home/mika/.conda/envs/py36/lib/python3.6/site-packages/torch/utils/ffi/../../lib/include/TH -I/home/v/.conda/envs/py36/lib/python3.6/site-packages/torch/utils/ffi/../../lib/include/THC -I/usr/local/cuda/include -I/home/mika/.conda/envs/py36/include/python3.6m -c _deform_conv.c -o ./_deform_conv.o
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
g++ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -DWITH_CUDA -I/home/mika/.conda/envs/py36/lib/python3.6/site-packages/torch/utils/ffi/../../lib/include -I/home/mika/.conda/envs/py36/lib/python3.6/site-packages/torch/utils/ffi/../../lib/include/TH -I/home/mika/.conda/envs/py36/lib/python3.6/site-packages/torch/utils/ffi/../../lib/include/THC -I/usr/local/cuda/include -I/home/mika/.conda/envs/py36/include/python3.6m -c /home/mika/wd/deformable-convolution-pytorch-master/src/deform_conv.c -o ./home/mika/wd/deformable-convolution-pytorch-master/src/deform_conv.o
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
g++ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -DWITH_CUDA -I/home/mika/.conda/envs/py36/lib/python3.6/site-packages/torch/utils/ffi/../../lib/include -I/home/mika/.conda/envs/py36/lib/python3.6/site-packages/torch/utils/ffi/../../lib/include/TH -I/home/mika/.conda/envs/py36/lib/python3.6/site-packages/torch/utils/ffi/../../lib/include/THC -I/usr/local/cuda/include -I/home/mika/.conda/envs/py36/include/python3.6m -c /home/mika/wd/deformable-convolution-pytorch-master/src/deform_conv_cuda.c -o ./home/mika/wd/deformable-convolution-pytorch-master/src/deform_conv_cuda.o
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
gcc -pthread -shared -B /home/mika/.conda/envs/py36/compiler_compat -L/home/mika/.conda/envs/py36/lib -Wl,-rpath=/home/mika/.conda/envs/py36/lib -Wl,--no-as-needed -Wl,--sysroot=/ ./_deform_conv.o ./home/mika/wd/deformable-convolution-pytorch-master/src/deform_conv.o ./home/mika/wd/deformable-convolution-pytorch-master/src/deform_conv_cuda.o /home/mika/wd/deformable-convolution-pytorch-master/src/deform_conv_cuda_kernel.cu.o -o ./_deform_conv.so

python test.py
Traceback (most recent call last):
File "test.py", line 6, in
from modules import ConvOffset2d
File "/home/v/wd/deformable-convolution-pytorch-master/modules/init.py", line 1, in
from .deform_conv import ConvOffset2d
File "/home/mika/wd/deformable-convolution-pytorch-master/modules/deform_conv.py", line 7, in
from functions import conv_offset2d
File "/home/mika/wd/deformable-convolution-pytorch-master/functions/init.py", line 1, in
from .deform_conv import conv_offset2d
File "/home/mika/wd/deformable-convolution-pytorch-master/functions/deform_conv.py", line 5, in
from _ext import deform_conv
File "/home/mika/wd/deformable-convolution-pytorch-master/_ext/deform_conv/init.py", line 3, in
from ._deform_conv import lib as _lib, ffi as _ffi
ImportError: /home/mika/wd/deformable-convolution-pytorch-master/_ext/deform_conv/_deform_conv.so: undefined symbol: __cudaPopCallConfiguration

why offset numbers are kW x kH x 2?
In the paper ,offset= 2 x N, the N is number of channel。

Hi:
When I ran CC=g++ python build.py, I got errors of cffi.error.VerificationError: CompileError: command 'g++' failed with exit status 1. Could you give me some suggestions? Thank you. @1zb
The error details are as follows:

Traceback (most recent call last):
  File "/home/sean/anaconda3/lib/python3.6/distutils/unixccompiler.py", line 118, in _compile
    extra_postargs)
  File "/home/sean/anaconda3/lib/python3.6/distutils/ccompiler.py", line 909, in spawn
    spawn(cmd, dry_run=self.dry_run)
  File "/home/sean/anaconda3/lib/python3.6/distutils/spawn.py", line 36, in spawn
    _spawn_posix(cmd, search_path, dry_run=dry_run)
  File "/home/sean/anaconda3/lib/python3.6/distutils/spawn.py", line 159, in _spawn_posix
    % (cmd, exit_status))
distutils.errors.DistutilsExecError: command 'g++' failed with exit status 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/sean/anaconda3/lib/python3.6/site-packages/cffi/ffiplatform.py", line 51, in _build
    dist.run_command('build_ext')
  File "/home/sean/anaconda3/lib/python3.6/distutils/dist.py", line 974, in run_command
    cmd_obj.run()
  File "/home/sean/anaconda3/lib/python3.6/distutils/command/build_ext.py", line 339, in run
    self.build_extensions()
  File "/home/sean/anaconda3/lib/python3.6/distutils/command/build_ext.py", line 448, in build_extensions
    self._build_extensions_serial()
  File "/home/sean/anaconda3/lib/python3.6/distutils/command/build_ext.py", line 473, in _build_extensions_serial
    self.build_extension(ext)
  File "/home/sean/anaconda3/lib/python3.6/distutils/command/build_ext.py", line 533, in build_extension
    depends=ext.depends)
  File "/home/sean/anaconda3/lib/python3.6/distutils/ccompiler.py", line 574, in compile
    self._compile(obj, src, ext, cc_args, extra_postargs, pp_opts)
  File "/home/sean/anaconda3/lib/python3.6/distutils/unixccompiler.py", line 120, in _compile
    raise CompileError(msg)
distutils.errors.CompileError: command 'g++' failed with exit status 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "build.py", line 36, in <module>
    ffi.build()
  File "/home/sean/anaconda3/lib/python3.6/site-packages/torch/utils/ffi/__init__.py", line 189, in build
    _build_extension(ffi, cffi_wrapper_name, target_dir, verbose)
  File "/home/sean/anaconda3/lib/python3.6/site-packages/torch/utils/ffi/__init__.py", line 111, in _build_extension
    outfile = ffi.compile(tmpdir=tmpdir, verbose=verbose, target=libname)
  File "/home/sean/anaconda3/lib/python3.6/site-packages/cffi/api.py", line 697, in compile
    compiler_verbose=verbose, debug=debug, **kwds)
  File "/home/sean/anaconda3/lib/python3.6/site-packages/cffi/recompiler.py", line 1520, in recompile
    compiler_verbose, debug)
  File "/home/sean/anaconda3/lib/python3.6/site-packages/cffi/ffiplatform.py", line 22, in compile
    outputfilename = _build(tmpdir, ext, compiler_verbose, debug)
  File "/home/sean/anaconda3/lib/python3.6/site-packages/cffi/ffiplatform.py", line 58, in _build
    raise VerificationError('%s: %s' % (e.__class__.__name__, e))
cffi.error.VerificationError: CompileError: command 'g++' failed with exit status 1

请问我可以把这个模型嵌入到某个网络结构中吗？比如TSN。
您有这样的例子吗？
我想用这个实现一下一个完整的数据训练过程。
谢谢

I have the following network:

class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.conv1 = nn.Conv2d(1,8,3,padding=1,bias=False)
        self.batch1 = nn.BatchNorm2d(8,affine=False)
        #self.conv2 = nn.Conv2d(8,16,3,padding=0,bias=False)
        self.conv2offset = nn.Conv2d(8,2*3*3,3,padding=0,bias=False)
        self.deform_conv2 = ConvOffset2d(8,16,3,padding=0,num_deformable_groups=1)
        self.batch2 = nn.BatchNorm2d(16,affine=False)
        self.pooling = nn.MaxPool2d(2)
        self.fc1 = nn.Linear(6*6*16,10)
        self.activation = nn.ReLU()

    def forward(self,x):
        x = self.conv1(x)
        x = self.pooling(x)
        x = self.batch1(x)
        x = self.activation(x)
        #x = self.conv2(x)
        offset = self.conv2offset(x)
        x = self.deform_conv2(x,offset)
        x = self.pooling(x)
        x = self.batch2(x)
        x = self.activation(x)
        logits = self.fc1(x.view(-1,6*6*16))
        probas = F.softmax(logits, dim=1)
        return logits, probas

I train it on MNIST for 2 batches. It takes 327 seconds to run (97.64% accuracy on test set).

Now if I remove deform conv and replace it with normal convolution (commented in the code above), it takes 19 seconds for 2 batches (97.54% accuracy on test set).

What do you think is the cause?

Pytorch v0.3.0
Python v3.6.1

请问哪里有Pytorch的C扩展相关文档或样例，官方给的GitHub例子太简单了，你是直接根据那个写出来的吗？

It is not working on pytorch 0.4.1.

Any way to get the offset_grad gradient from the .cu files?