sniklaus / pytorch-pwc

a reimplementation of PWC-Net in PyTorch that matches the official Caffe version

pytorch-pwc

This is a personal reimplementation of PWC-Net [1] using PyTorch. Should you be making use of this work, please cite the paper accordingly. Also, make sure to adhere to the licensing terms of the authors. Should you be making use of this particular implementation, please acknowledge it appropriately [2].

For the original version of this work, please see: https://github.com/NVlabs/PWC-Net
Other optical flow implementations from me: pytorch-unflow, pytorch-spynet, pytorch-liteflownet

background

The authors of PWC-Net are thankfully already providing a reference implementation in PyTorch. However, its initial version did not reach the performance of the original Caffe version. This is why I created this repositroy, in which I replicated the performance of the official Caffe version by utilizing its weights.

The official PyTorch implementation has adopted my approach of using the Caffe weights since then, which is why they are all performing equally well now. Many people have reported issues with CUDA when trying to get the official PyTorch version to run though, while my reimplementaiton does not seem to be subject to such problems.

setup

The correlation layer is implemented in CUDA using CuPy, which is why CuPy is a required dependency. It can be installed using pip install cupy or alternatively using one of the provided binary packages as outlined in the CuPy repository.

usage

To run it on your own pair of images, use the following command. You can choose between two models, please make sure to see their paper / the code for more details.

python run.py --model default --one ./images/one.png --two ./images/two.png --out ./out.flo

I am afraid that I cannot guarantee that this reimplementation is correct. However, it produced results identical to the Caffe implementation of the original authors in the examples that I tried. Please feel free to contribute to this repository by submitting issues and pull requests.

comparison

license

As stated in the licensing terms of the authors of the paper, the models are free for non-commercial share-alike purpose. Please make sure to further consult their licensing terms.

references

[1]  @inproceedings{Sun_CVPR_2018,
         author = {Deqing Sun and Xiaodong Yang and Ming-Yu Liu and Jan Kautz},
         title = {{PWC-Net}: {CNNs} for Optical Flow Using Pyramid, Warping, and Cost Volume},
         booktitle = {IEEE Conference on Computer Vision and Pattern Recognition},
         year = {2018}
     }

[2]  @misc{pytorch-pwc,
         author = {Simon Niklaus},
         title = {A Reimplementation of {PWC-Net} Using {PyTorch}},
         year = {2018},
         howpublished = {\url{https://github.com/sniklaus/pytorch-pwc}}
    }

pytorch-pwc's Issues

can you provide a pytorch version for DDFlow model please

An error regarding the. flo file

Hello,@sniklaus
I installed pytorch-pwc according to your instructions. I successfully generated the. flo file, but I encountered a problem. After I visualized the. flo file and converted it to png file, I found the lower part of the picture was covered. I am not sure what this is. Can you help me?

Generalization to unseen data

Hi,

Thanks for this work. I was having issues setting up the official version, but yours was very simple and easy.

I'm wondering how well would this model generalize to unseen data. I tried on a couple of frames from UCF-101 dataset and the results were not great. Among the other models you've, do any of them perform better on UCF-101 dataset?

Can it test in CPU device?

In the project, in the file correlation/correlation.py, it uses a module called cupy. However, cupy is based on cuda, which means that it needs to contain GPU. However, in most macOS computers, cuda and cupy cannot be supported.
So, if I want to test this project without cuda, is there any way to achieve？Or is there any method that does not use cupy module to implement the correlation function?

How to do forward warping using im1 and flow?

Given the first image and the forward flow, how can I perform forward warping to get im2?

module 'correlation' has no attribute 'ModuleCorrelation'

When I try to run your code, errors happened like this:

I uese python3.6, pytorch0.4。So, what should I do to solve the problem. Looking forward to your reply. Thank you very much.

Do you have a model trained on Kitti dataset?

Thank you for the re-implementation! It is so helpful. Do you have a model trained on the Kitti flow 2015 dataset?

A word of advice

Hi @sniklaus,

What do you propose as the best way to change your correlation kernel to correlate along the X and Y axis (rather than the C axis which AFAIU is what is currently done).

Thanks!

what‘s the version of python

INSTALLING CUPY

While Installing Cupy we encounter issue so please update this link to install cupy based on cuda version:
https://docs.cupy.dev/en/stable/install.html?highlight=install#requirements

a runtime error

hey, thank you fro sharing! When I ran the code follow the command, there occured an error.
File "run_pwc.py", line 207, in forward tenFeat = torch.cat([ self.netOne(tenFeat), tenFeat ], 1) File "D:\Anaconda3\envs\torch\lib\site-packages\torch\nn\modules\module.py", line 722, in _call_impl result = self.forward(*input, **kwargs) File "D:\Anaconda3\envs\torch\lib\site-packages\torch\nn\modules\container.py", line 117, in forward input = module(input) File "D:\Anaconda3\envs\torch\lib\site-packages\torch\nn\modules\module.py", line 722, in _call_impl result = self.forward(*input, **kwargs) File "D:\Anaconda3\envs\torch\lib\site-packages\torch\nn\modules\conv.py", line 419, in forward return self._conv_forward(input, self.weight) File "D:\Anaconda3\envs\torch\lib\site-packages\torch\nn\modules\conv.py", line 416, in _conv_forward self.padding, self.dilation, self.groups) RuntimeError: Given groups=1, weight of size [128, 81, 3, 3], expected input[1, 49, 4, 8] to have 81 channels, but got 49 channels instead
It seems like the dimension is wrong in somewhere. Could you help me to figure out it? I will be really appreciate.

output of the run.py

how do you check the output of the run.py?
line no:32 of run.py

arguments_strOut = './out.flo'

the final optical flow is multiply by 20,Why multiply by 20，not 40？

tensorFlow = 20.0 * torch.nn.functional.interpolate(input=moduleNetwork(tensorPreprocessedFirst, tensorPreprocessedSecond), size=(intHeight, intWidth), mode='bilinear', align_corners=False)

in the paper ，the author say "We scale the ground truth flow by 20 and downsample it to obtain the supervision signals at different levels.",

and " our model outputs a quarter resolution optical flow and uses bilinear interpolation to obtain the full-resolution optical flow.",

the output flow interpolation to full-resolution optical flow ,why it not multiply by 2?

Save flow as image (RGB)

Hello there,

I was wondering if there is a way to save the flow between two images as an image as well instead of saving it as a .flo file. Do you have any code for that or any recommendation ?

Thank you

about the pre-trained model

I can't find a pre-training model in your project,Could you provide the pre-trained model?thank you very much.

Is it possible to process bigger image size?

Current code has limitation of 1024x436. Can it change to handle large size image, like 1920x1080?

Thanks,

About the pre-trained model

Hi! Thank you very much for the code. I just have a question about the pre-trained model. Is the network-chairs-things model pre-trained on the Flyingchairs dataset and Flyingthings3D dataset? How about the default model? What dataset is it pre-trained on? Looking forward to your reply! Many thanks!

Training code?

How do you re-implement the performance of the official caffe? May you release your training code?
Very thanks.

optical flow upsampling

hi,
I would like to ask why the optical flow upsampling is multiplied by 0.65, 1.25, 2.5, 5. The original paper does not seem to mention it.
Thank.

Comparison with official's implementation

Hi @sniklaus ,

Taking no account of modifications which have no impact on precision, such as coding style, what's the main factor that results in your implementation's improvement? Is it because of the correlation layer or something else? Thanks a lot.

about the result

Hi, thanks for convert the model from caffe to pytorch. But the weights "network-default.pytorch" you uploaded is for which dataset? KITTI or Sintel?

Besides, the EPE calculated from "network-default.pytorch" is quite far from the original paper. I am not sure that the EPE should be computed between the estimated flow and ground truth flow, or between estimated flow and 20*ground truth flow?

Thanks!

What type of loss did you use during training?

I try to train this network using multiscale loss described in the paper. But it failed. It works when I use L2 loss. So what type of loss did you use during training? Could you please explain why it does not work with multiscale loss? Thank you!

Performance on Synthetic Driving Dataset?

Hi @sniklaus ,

Thank you for your great work. I am just wondering if you have tested PWC's performance on synthetic driving dataset(https://lmb.informatik.uni-freiburg.de/resources/datasets/SceneFlowDatasets.en.html).
In theory, PWC's performance should be good on this dataset, given that initial training of PWC is on FlyingThings3D. However, on this dataset, performance on syntheic driving is about 28, which is very large.
Do you have any clue?

I am looking forward to your reply. Thank you very much.

Best,

align_corners function parameter is inconsistent between grid_sample and interpolate method

In the current code, grid_sample() is not passed the align_corners parameter so it is set to True as described here

pytorch-pwc/run.py

Line 61 in 1a5f196

tensorOutput = torch.nn.functional.grid_sample(input=tensorInput, grid=(Backward_tensorGrid[str(tensorFlow.size())] + tensorFlow).permute(0, 2, 3, 1), mode='bilinear', padding_mode='zeros')

However, interpolate() has align_corners set to False.

pytorch-pwc/run.py

Line 299 in 1a5f196

tensorFlow = 20.0 * torch.nn.functional.interpolate(input=moduleNetwork(tensorPreprocessedFirst, tensorPreprocessedSecond), size=(intHeight, intWidth), mode='bilinear', align_corners=False)

From the pytorch docs:
The align_corners option parallels the align_corners option in interpolate(), and so whichever option is used here should also be used there to resize the input image before grid sampling.

Is this intended? If not, then align_corners=False needs to be passed in to the grid_sample() function.

about training

Hi, is it possible to train this network with your code on a dataset like Sintel?

Data Parallel for PWC-Net

Hi,
Thank you for your implementation, it is really helpful since the original version used the older Python version and need some extra functions in language C.
However, presently I am trying to use your model into my work, and I found this network cannot be warped in the data parallel module. Therefore, it can not profit multi-GPUs. I think the error is caused by 'torch.nn.functional.grid_sample' in the 'Backward' function. I'm wondering if you have any clue on this problem? Thank you in advance!

A backward issue

Hi sniklaus. I suffered a problem when I tried to use your model as an optical flow prediction component of my model. I simplify the issue as below:

t1 = torch.randn((16,3,256,256))
t2 = torch.randn((16,3,256,256))
input = Network()(t1, t2)
input.backward()

It will raise "RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn" on the backward line. Does it mean there is somewhere non-differentiable inside the network? How can I deal with this if I just want to use the output to train other models?

KITTI finetune model

Hi Niklaus,
Thanks for providing a clean implementaion of pwc-net on pytorch.
I am working with your code on KITTI dataset. However, its performance did not match PWC-Net and PWC-Net+. I assume the model is not finetuned on KITTI and not using the training strategy of PWC-Net+.
Refer https://github.com/NVlabs/PWC-Net/tree/master/Caffe/model/PWC-Net_plus
Could you please also convert the model of the above link or share your conversion script?

argument 'second' in '_FunctionCorrelation' is not contiguous if batch size bigger than 1

Hi @sniklaus,

thanks a lot for your work! I managed to get the model overfit to one sample, using a batch size of 1. But when moving on to fine tuning the pretrained models and using a batch size different than 1, argument second in _FunctionCorrelation is not contiguous. I guess one option is to do second.contiguous(). But I was curious as to why this is happening. I've roughly spotted that function Backward returns a non-contiguous tensor (when batch size != 1). Let me know if this had happened to you before, or if maybe it's some issue from my side.
Cheers!

EDIT: in particular, it's this line:

tensorMask = tensorOutput[:, -1:, :, :]

that is causing the output of Backward to not be contiguous when the batch size is not 1, which makes sense, since slicing can cause this (see this thread)

Regarding CPU implementation of correlation function.

Hi, thanks for the implementation. In my use case, I need to perform the inference on CPU. Inspecting your code in the file correlation.py, I kind of get that for that, we need to call the extern C functions ourselves instead of invoking CuPy functions to do it for us. In your code, each C function is called like this:
cupy_launch('kernel_Correlation_rearrange', cupy_kernel('kernel_Correlation_rearrange', { 'input': second, 'output': rbot1 }))( grid=tuple([ int((n + 16 - 1) / 16), second.shape[1], second.shape[0] ]), block=tuple([ 16, 1, 1 ]), args=[ n, second.data_ptr(), rbot1.data_ptr() ] )

I am not familiar with CuPy code, so it will be helpful if you could explain these function calls a bit and give any clue about how to do the equivalent stuff on CPU. I understand that the args in each call are the arguments passed to the C function, but I am not sure what grid and block signify here. Probably, they may not be needed when CuPy is not used. As I only need to run on CPU at test time, I guess I don't need to care about the updateGrad functions.

I will appreciate your help/suggestion regarding this.

I want to fine-tune PWC on ChairSDHom data-set

Hi sniklaus.I am vivasvan.
I am a beginner.
Can you guide me on how to this fine-tune PWC-Net on ChairSDHom dataset?
Your help will be really appreciated.
Thank you so much

Can you offer the .flo to .png code?

You have done a good job. But I find that the output files did not include the .png file, so can you offer the .flo to .png file code? Thank you very much.

Use correlation as loss

correlation(image1, image2) outputs a multidimensional tensor, would it be possible to produce a scalar that can be used as a loss. I.e deform image 1 and minimise negative correlation between image1, image2.

summing up over all dimensions doesn't work.

Convert pwc_net.caffemodel to network-default.pytorch

Thanks for your wonderful code!

I try to convert the official pwc_net.caffemodel to pytorch. And I find a interesing problem, the bias of the FeatureUpsample layers within all the decoder in this repo are not consistent with official model.

I compare my pytorch weights (converted from official caffemodel) with network-default.pytorch in this repo as follows.

data1 = torch.load('network-default.pytorch')  
data2 = torch.load('my-network.pytorch')  
for key in data1.keys():
    print(key, torch.max(data1[key] - data2[key]), torch.min(data1[key] - data2[key])

moduleExtractor.moduleOne.0.weight tensor(0.) tensor(0.)
moduleExtractor.moduleOne.0.bias tensor(0.) tensor(0.)
moduleExtractor.moduleOne.2.weight tensor(0.) tensor(0.)
moduleExtractor.moduleOne.2.bias tensor(0.) tensor(0.)
moduleExtractor.moduleOne.4.weight tensor(0.) tensor(0.)
moduleExtractor.moduleOne.4.bias tensor(0.) tensor(0.)
moduleExtractor.moduleTwo.0.weight tensor(0.) tensor(0.)
moduleExtractor.moduleTwo.0.bias tensor(0.) tensor(0.)
moduleExtractor.moduleTwo.2.weight tensor(0.) tensor(0.)
moduleExtractor.moduleTwo.2.bias tensor(0.) tensor(0.)
moduleExtractor.moduleTwo.4.weight tensor(0.) tensor(0.)
moduleExtractor.moduleTwo.4.bias tensor(0.) tensor(0.)
moduleExtractor.moduleThr.0.weight tensor(0.) tensor(0.)
moduleExtractor.moduleThr.0.bias tensor(0.) tensor(0.)
moduleExtractor.moduleThr.2.weight tensor(0.) tensor(0.)
moduleExtractor.moduleThr.2.bias tensor(0.) tensor(0.)
moduleExtractor.moduleThr.4.weight tensor(0.) tensor(0.)
moduleExtractor.moduleThr.4.bias tensor(0.) tensor(0.)
moduleExtractor.moduleFou.0.weight tensor(0.) tensor(0.)
moduleExtractor.moduleFou.0.bias tensor(0.) tensor(0.)
moduleExtractor.moduleFou.2.weight tensor(0.) tensor(0.)
moduleExtractor.moduleFou.2.bias tensor(0.) tensor(0.)
moduleExtractor.moduleFou.4.weight tensor(0.) tensor(0.)
moduleExtractor.moduleFou.4.bias tensor(0.) tensor(0.)
moduleExtractor.moduleFiv.0.weight tensor(0.) tensor(0.)
moduleExtractor.moduleFiv.0.bias tensor(0.) tensor(0.)
moduleExtractor.moduleFiv.2.weight tensor(0.) tensor(0.)
moduleExtractor.moduleFiv.2.bias tensor(0.) tensor(0.)
moduleExtractor.moduleFiv.4.weight tensor(0.) tensor(0.)
moduleExtractor.moduleFiv.4.bias tensor(0.) tensor(0.)
moduleExtractor.moduleSix.0.weight tensor(0.) tensor(0.)
moduleExtractor.moduleSix.0.bias tensor(0.) tensor(0.)
moduleExtractor.moduleSix.2.weight tensor(0.) tensor(0.)
moduleExtractor.moduleSix.2.bias tensor(0.) tensor(0.)
moduleExtractor.moduleSix.4.weight tensor(0.) tensor(0.)
moduleExtractor.moduleSix.4.bias tensor(0.) tensor(0.)
moduleTwo.moduleUpflow.weight tensor(0.) tensor(0.)
moduleTwo.moduleUpflow.bias tensor(0.) tensor(0.)
moduleTwo.moduleUpfeat.weight tensor(0.) tensor(0.)
moduleTwo.moduleUpfeat.bias tensor(0.0873) tensor(0.0774)
moduleTwo.moduleOne.0.weight tensor(0.) tensor(0.)
moduleTwo.moduleOne.0.bias tensor(0.) tensor(0.)
moduleTwo.moduleTwo.0.weight tensor(0.) tensor(0.)
moduleTwo.moduleTwo.0.bias tensor(0.) tensor(0.)
moduleTwo.moduleThr.0.weight tensor(0.) tensor(0.)
moduleTwo.moduleThr.0.bias tensor(0.) tensor(0.)
moduleTwo.moduleFou.0.weight tensor(0.) tensor(0.)
moduleTwo.moduleFou.0.bias tensor(0.) tensor(0.)
moduleTwo.moduleFiv.0.weight tensor(0.) tensor(0.)
moduleTwo.moduleFiv.0.bias tensor(0.) tensor(0.)
moduleTwo.moduleSix.0.weight tensor(0.) tensor(0.)
moduleTwo.moduleSix.0.bias tensor(0.) tensor(0.)
moduleThr.moduleUpflow.weight tensor(0.) tensor(0.)
moduleThr.moduleUpflow.bias tensor(0.) tensor(0.)
moduleThr.moduleUpfeat.weight tensor(0.) tensor(0.)
moduleThr.moduleUpfeat.bias tensor(-0.1421) tensor(-0.1514)
moduleThr.moduleOne.0.weight tensor(0.) tensor(0.)
moduleThr.moduleOne.0.bias tensor(0.) tensor(0.)
moduleThr.moduleTwo.0.weight tensor(0.) tensor(0.)
moduleThr.moduleTwo.0.bias tensor(0.) tensor(0.)
moduleThr.moduleThr.0.weight tensor(0.) tensor(0.)
moduleThr.moduleThr.0.bias tensor(0.) tensor(0.)
moduleThr.moduleFou.0.weight tensor(0.) tensor(0.)
moduleThr.moduleFou.0.bias tensor(0.) tensor(0.)
moduleThr.moduleFiv.0.weight tensor(0.) tensor(0.)
moduleThr.moduleFiv.0.bias tensor(0.) tensor(0.)
moduleThr.moduleSix.0.weight tensor(0.) tensor(0.)
moduleThr.moduleSix.0.bias tensor(0.) tensor(0.)
moduleFou.moduleUpflow.weight tensor(0.) tensor(0.)
moduleFou.moduleUpflow.bias tensor(0.) tensor(0.)
moduleFou.moduleUpfeat.weight tensor(0.) tensor(0.)
moduleFou.moduleUpfeat.bias tensor(0.1554) tensor(-0.1341)
moduleFou.moduleOne.0.weight tensor(0.) tensor(0.)
moduleFou.moduleOne.0.bias tensor(0.) tensor(0.)
moduleFou.moduleTwo.0.weight tensor(0.) tensor(0.)
moduleFou.moduleTwo.0.bias tensor(0.) tensor(0.)
moduleFou.moduleThr.0.weight tensor(0.) tensor(0.)
moduleFou.moduleThr.0.bias tensor(0.) tensor(0.)
moduleFou.moduleFou.0.weight tensor(0.) tensor(0.)
moduleFou.moduleFou.0.bias tensor(0.) tensor(0.)
moduleFou.moduleFiv.0.weight tensor(0.) tensor(0.)
moduleFou.moduleFiv.0.bias tensor(0.) tensor(0.)
moduleFou.moduleSix.0.weight tensor(0.) tensor(0.)
moduleFou.moduleSix.0.bias tensor(0.) tensor(0.)
moduleFiv.moduleUpflow.weight tensor(0.) tensor(0.)
moduleFiv.moduleUpflow.bias tensor(0.) tensor(0.)
moduleFiv.moduleUpfeat.weight tensor(0.) tensor(0.)
moduleFiv.moduleUpfeat.bias tensor(0.0935) tensor(-0.1689)
moduleFiv.moduleOne.0.weight tensor(0.) tensor(0.)
moduleFiv.moduleOne.0.bias tensor(0.) tensor(0.)
moduleFiv.moduleTwo.0.weight tensor(0.) tensor(0.)
moduleFiv.moduleTwo.0.bias tensor(0.) tensor(0.)
moduleFiv.moduleThr.0.weight tensor(0.) tensor(0.)
moduleFiv.moduleThr.0.bias tensor(0.) tensor(0.)
moduleFiv.moduleFou.0.weight tensor(0.) tensor(0.)
moduleFiv.moduleFou.0.bias tensor(0.) tensor(0.)
moduleFiv.moduleFiv.0.weight tensor(0.) tensor(0.)
moduleFiv.moduleFiv.0.bias tensor(0.) tensor(0.)
moduleFiv.moduleSix.0.weight tensor(0.) tensor(0.)
moduleFiv.moduleSix.0.bias tensor(0.) tensor(0.)
moduleSix.moduleOne.0.weight tensor(0.) tensor(0.)
moduleSix.moduleOne.0.bias tensor(0.) tensor(0.)
moduleSix.moduleTwo.0.weight tensor(0.) tensor(0.)
moduleSix.moduleTwo.0.bias tensor(0.) tensor(0.)
moduleSix.moduleThr.0.weight tensor(0.) tensor(0.)
moduleSix.moduleThr.0.bias tensor(0.) tensor(0.)
moduleSix.moduleFou.0.weight tensor(0.) tensor(0.)
moduleSix.moduleFou.0.bias tensor(0.) tensor(0.)
moduleSix.moduleFiv.0.weight tensor(0.) tensor(0.)
moduleSix.moduleFiv.0.bias tensor(0.) tensor(0.)
moduleSix.moduleSix.0.weight tensor(0.) tensor(0.)
moduleSix.moduleSix.0.bias tensor(0.) tensor(0.)
moduleRefiner.moduleMain.0.weight tensor(0.) tensor(0.)
moduleRefiner.moduleMain.0.bias tensor(0.) tensor(0.)
moduleRefiner.moduleMain.2.weight tensor(0.) tensor(0.)
moduleRefiner.moduleMain.2.bias tensor(0.) tensor(0.)
moduleRefiner.moduleMain.4.weight tensor(0.) tensor(0.)
moduleRefiner.moduleMain.4.bias tensor(0.) tensor(0.)
moduleRefiner.moduleMain.6.weight tensor(0.) tensor(0.)
moduleRefiner.moduleMain.6.bias tensor(0.) tensor(0.)
moduleRefiner.moduleMain.8.weight tensor(0.) tensor(0.)
moduleRefiner.moduleMain.8.bias tensor(0.) tensor(0.)
moduleRefiner.moduleMain.10.weight tensor(0.) tensor(0.)
moduleRefiner.moduleMain.10.bias tensor(0.) tensor(0.)
moduleRefiner.moduleMain.12.weight tensor(0.) tensor(0.)
moduleRefiner.moduleMain.12.bias tensor(0.) tensor(0.)

All the parameters are actually same but the bias in FeatureUpsample as aforementioned. And I find the bias in my converted pytorch model are 0 and in the network-default.pytorch they are not 0. I look into the official trianing prototxt file and find the lr_mult for all deconvolution layers are 0 and so does the initializer(constant means 0 also by default). So the real value of bias should be 0.

I also check the sintel.pytorch in an older version of this repo. I compare sintel.pytorch with network-default.pytorch and find the same phenomenon, all the parameters are actually same but the bias in FeatureUpsample. There may be some randomness during your conversion within the FeatureUpsample layers. I guess the bias are initialized differently everytime.

@sniklaus

AttributeError: module 'correlation' has no attribute 'FunctionCorrelation'

Trying to run:
python run.py --model default --first ./images/first.png --second ./images/second.png --out ./out.flo

Gives me the following output:

  File "run.py", line 315, in <module>
    tensorOutput = estimate(tensorFirst, tensorSecond)
  File "run.py", line 301, in estimate
    tensorFlow = 20.0 * torch.nn.functional.interpolate(input=moduleNetwork(tensorPreprocessedFirst, tensorPreprocessedSecond), size=(intHeight, intWidth), mode='bilinear', align_corners=False)
  File "/home/fabio/anaconda3/envs/ai/lib/python3.7/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "run.py", line 268, in forward
    objectEstimate = self.moduleSix(tensorFirst[-1], tensorSecond[-1], None)
  File "/home/fabio/anaconda3/envs/ai/lib/python3.7/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "run.py", line 196, in forward
    tensorVolume = torch.nn.functional.leaky_relu(input=correlation.FunctionCorrelation(tensorFirst=tensorFirst, tensorSecond=tensorSecond), negative_slope=0.1, inplace=False)
AttributeError: module 'correlation' has no attribute 'FunctionCorrelation' 

Using Ubunutu 18.10
Pytorch 1.0.1
CUDA 9.1
Python 3.7.2
GCC 7.3.0

Any ideias?

Is it difficult to make train code?

I think your reproducible implementation is very nice.
However, there is no train code as it is said in various issues.
So, I am wondering why you do not write train code.

Performance compared to Flownet2.0

I found the flow produced by pwcnet is more blurry than that of flownet2.0. Is it supposed to perform worse?
Image:


Flow by Flownet2.0:

Image warped by Flownet2.0:

Flow by PWCnet:

Image warped by PWCnet:

Performance issue

Hi @sniklaus ,

In the original PWC-net paper, PWC-net's processing time is about 28ms.

But when I measured the time passed in run.py's estimate function and moduleNetwork, they cost about 330ms for estimate function and 173ms for moduleNetwork, which is too large to ignore.

I want to know that is there a reason that causes this issue. Thanks!

what is the range of optical flow value in an image?

Hi,
Thanks for this work! I use this pwcnet to estimate optical flow. I have a question regarding the value range of optical flow. Is it range from (-w+1, w-1) (-h+1, h-1)? If I want to normalise optical flow to (-1,1), should I follow this:
optical_flow[0,:,:] = optical_flow[0,:,:]/ (w-1)
optical_flow[1,:,:] = optical_flow[1,:,:]/ (h-1)?
(w and h is the image size. optical_flow_shape = (2, h, w))

AttributeError: module 'correlation' has no attribute 'ModuleCorrelation'

Hi man, I am using pythorch with python 3.6 cuda 8.0,
when I run

python run.py

I got this error:

Traceback (most recent call last):
File "run.py", line 293, in
moduleNetwork = Network().cuda().eval()
File "run.py", line 268, in init
self.moduleTwo = Decoder(2)
File "run.py", line 170, in init
self.moduleCorrelation = correlation.ModuleCorrelation()
AttributeError: module 'correlation' has no attribute 'ModuleCorrelation'

Could you help me ?

ImportError: cannot import name correlation

What's the solution for "ImportError: cannot import name correlation"?
I didn't find the def correlation in correlation.py.

how to compute optical flow for small size imgs?

Hi,
Thanks for providing easy way of running pwcnet! I want to use pwc-net in my own dataset, inside which, all images are in 48*48 gray scale. Can you give some tips on how to preprocess these images? Do I need to resize them to (436, 1024) first? Do I need to convert them to rgb imgs? Thanks!

Training settings?

Thanks for sharing this nice piece of code.
Was wondering if you are planning to share the training settings / scripts?

Also in a related question, what do you think was the source of discrepancy in the two original implementations (caffe vs. pytorch) and how do you think it got solved?

Best,

Cannot select the GPU index

Hi.

It seems that I cannot make the code to run on another GPU rather than 0-th. The following line doesn't make any difference in my case

pytorch-pwc/run.py

Line 25 in f613890

torch.cuda.device(1) # change this if you have a multiple graphics cards and you want to utilize them

I managed to solve it by introducing the device variable instead and adding torch.cuda.set_device(device) right after the FunctionCorrelation signature

pytorch-pwc/correlation/correlation.py

Lines 392 to 393 in f613890

	def FunctionCorrelation(tensorFirst, tensorSecond):
	return _FunctionCorrelation.apply(tensorFirst, tensorSecond)

and also passing this variable as an argument here

pytorch-pwc/run.py

Line 196 in f613890

tensorVolume = torch.nn.functional.leaky_relu(input=correlation.FunctionCorrelation(tensorFirst=tensorFirst, tensorSecond=tensorSecond), negative_slope=0.1, inplace=False)

and here

pytorch-pwc/run.py

Line 204 in f613890

tensorVolume = torch.nn.functional.leaky_relu(input=correlation.FunctionCorrelation(tensorFirst=tensorFirst, tensorSecond=Backward(tensorInput=tensorSecond, tensorFlow=tensorFlow * self.dblBackward)), negative_slope=0.1, inplace=False)

If this is a bug do you want me to form a Pull Request?

Anyway, thanks a lot for sharing the code. It is really helpful when the official version is on CUDA 8 and PyTorch 0.2!

Assertion Error (using images of a different size)

It seems like using images with a different size throws this error. I would like to double check that modifying that part of the code to make it accept images with other size will not create an issue in calculating the flow. Also. if there is any specific part of the code I need to be careful with when modifying.

Thank you

EPE in sintel clean lower han the original

Dear @sniklaus thanks for this nice implementation. I find it really useful I don't have to worry about cuda versioning.
I am writing a paper which uses this implementation and to be sure of I have checked the EPE on sintel training clean. I got 1.81, which is much lower than what stated from the authors.
Is this network fine tuned on Sintel ? Does my result make sense?

Thanks,
Stefano

Cannot inspect the model using TensorBoard

I want to visualize the model using tensorboard. And some error happens:

C:\Users\Administrator\anaconda3\envs\pwc\lib\site-packages\torch\tensor.py:455: RuntimeWarning: Iterating over a tensor might cause the trace to be incorrect. Passing a tensor of different shape won't change the number of iterations executed (and might lead to errors or silently give incorrect results).
  'incorrect results).', category=RuntimeWarning)
Traceback (most recent call last):
  File "C:\Users\Administrator\anaconda3\envs\pwc\lib\site-packages\cupy\cuda\compiler.py", line 549, in compile
    nvrtc.compileProgram(self.ptr, options)
  File "cupy_backends\cuda\libs\nvrtc.pyx", line 112, in cupy_backends.cuda.libs.nvrtc.compileProgram
  File "cupy_backends\cuda\libs\nvrtc.pyx", line 124, in cupy_backends.cuda.libs.nvrtc.compileProgram
  File "cupy_backends\cuda\libs\nvrtc.pyx", line 58, in cupy_backends.cuda.libs.nvrtc.check_status
cupy_backends.cuda.libs.nvrtc.NVRTCError: NVRTC_ERROR_COMPILATION (6)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "D:/workspace/pycharm/pytorch-pwc-master/check_model.py", line 354, in <module>
    writer.add_graph(netNetwork, (tenPreprocessedFirst, tenPreprocessedSecond))
  File "C:\Users\Administrator\anaconda3\envs\pwc\lib\site-packages\torch\utils\tensorboard\writer.py", line 714, in add_graph
    self._get_file_writer().add_graph(graph(model, input_to_model, verbose))
  File "C:\Users\Administrator\anaconda3\envs\pwc\lib\site-packages\torch\utils\tensorboard\_pytorch_graph.py", line 285, in graph
    trace = torch.jit.trace(model, args)
  File "C:\Users\Administrator\anaconda3\envs\pwc\lib\site-packages\torch\jit\__init__.py", line 955, in trace
    check_tolerance, strict, _force_outplace, _module_class)
  File "C:\Users\Administrator\anaconda3\envs\pwc\lib\site-packages\torch\jit\__init__.py", line 1109, in trace_module
    module._c._create_method_from_trace(method_name, func, example_inputs, var_lookup_fn, strict, _force_outplace)
  File "C:\Users\Administrator\anaconda3\envs\pwc\lib\site-packages\torch\nn\modules\module.py", line 720, in _call_impl
    result = self._slow_forward(*input, **kwargs)
  File "C:\Users\Administrator\anaconda3\envs\pwc\lib\site-packages\torch\nn\modules\module.py", line 704, in _slow_forward
    result = self.forward(*input, **kwargs)
  File "D:/workspace/pycharm/pytorch-pwc-master/check_model.py", line 307, in forward
    objEstimate = self.netSix(tenFirst[-1], tenSecond[-1], None)
  File "C:\Users\Administrator\anaconda3\envs\pwc\lib\site-packages\torch\nn\modules\module.py", line 720, in _call_impl
    result = self._slow_forward(*input, **kwargs)
  File "C:\Users\Administrator\anaconda3\envs\pwc\lib\site-packages\torch\nn\modules\module.py", line 704, in _slow_forward
    result = self.forward(*input, **kwargs)
  File "D:/workspace/pycharm/pytorch-pwc-master/check_model.py", line 215, in forward
    input=correlation.FunctionCorrelation(tenFirst=tenFirst, tenSecond=tenSecond),
  File "./correlation\correlation.py", line 386, in FunctionCorrelation
    return _FunctionCorrelation.apply(tenFirst, tenSecond)
  File "./correlation\correlation.py", line 295, in forward
    'output': rbot0
  File "cupy\_util.pyx", line 59, in cupy._util.memoize.decorator.ret
  File "./correlation\correlation.py", line 275, in cupy_launch
    return cupy.cuda.compile_with_cache(strKernel).get_function(strFunction)
  File "C:\Users\Administrator\anaconda3\envs\pwc\lib\site-packages\cupy\cuda\compiler.py", line 373, in compile_with_cache
    cache_in_memory)
  File "C:\Users\Administrator\anaconda3\envs\pwc\lib\site-packages\cupy\cuda\compiler.py", line 437, in _compile_with_cache_cuda
    log_stream, cache_in_memory)
  File "C:\Users\Administrator\anaconda3\envs\pwc\lib\site-packages\cupy\cuda\compiler.py", line 212, in compile_using_nvrtc
    name_expressions, log_stream)
  File "C:\Users\Administrator\anaconda3\envs\pwc\lib\site-packages\cupy\cuda\compiler.py", line 195, in _compile
    ptx, mapping = prog.compile(options, log_stream)
  File "C:\Users\Administrator\anaconda3\envs\pwc\lib\site-packages\cupy\cuda\compiler.py", line 561, in compile
    'nvrtc' if not runtime.is_hip else 'hiprtc')
cupy.cuda.compiler.CompileException: C:\Users\ADMINI~1\AppData\Local\Temp\tmpthz_eabd\256b2afebe918404edfb6592392d441b_2.cubin.cu(16): error: identifier "tensor" is undefined

1 error detected in the compilation of "C:\Users\ADMINI~1\AppData\Local\Temp\tmpthz_eabd\256b2afebe918404edfb6592392d441b_2.cubin.cu".

Here is some of the code I used.

from torch.utils.tensorboard import SummaryWriter

writer = SummaryWriter('runs/test')
writer.add_graph(netNetwork, (tenPreprocessedFirst, tenPreprocessedSecond))
writer.close()

I have tested the run.py, it works well. Please let me know if I missed something. Thank you.

[probably basic question] How can inference a tensor of frames in one batch?

Hi @sniklaus, thanks for your support. I have one more (probably newbbie) question.
How can I inference a batch of frames ? The model expects two filepaths. Can I give it a list like in caffe?
Currently the model must be loaded and it takes around 30s per frame.

Thanks,
Stefano

How can I add backward support for correlation package?

Hey, I'm running some experiment on video generation, and I would like to use the L1-loss between optical flow tensors of real video and fake video as a temporal constraint. This means that I need to back-propagate gradients of the flow w.r.t fake video frames.

However I found in your correlation package the backward function is not implemented:

def backward(self, gradOutput):
		first, second = self.saved_tensors

		assert(gradOutput.is_contiguous() == True)

		gradFirst = first.new_zeros(first.size(0), first.size(1), first.size(2), first.size(3)) if self.needs_input_grad[0] == True else None
		gradSecond = first.new_zeros(first.size(0), first.size(1), first.size(2), first.size(3)) if self.needs_input_grad[1] == True else None

		if first.is_cuda == True:
			raise NotImplementedError()

		elif first.is_cuda == False:
			raise NotImplementedError()

		# end

		return gradFirst, gradSecond
	# end

I find that the officical PWC-Net implementation provided a complete correlation package, but I'm not quite sure how to incorporated it to your project. Can you give me a hint on what to do?

Thanks!

Training code?

Hello, I know that the parameters is converted from official Caffe. But, could you please provide the training code if you have? Thanks!