How to build your extra folder files in Pytorch? about spynet

[anuragranj]

Actually, we want to warp image2 to image1 given the optical flow. The warping here is also called inverse warping. Here is why this is better (slides 1-6): https://www.cs.unc.edu/~lazebnik/research/fall08/lec08_faces.pdf

[EnQing626]

@anuragranj Very thanks for your replying. It helps me to understand warping and inverse warping better.

[anuragranj]

The files in extra implement two layers:

1. warping: this can be implemented using pytorch/torch.nn.functional.grid_sample. Note that spynet's warping takes optical flow (u,v) as input, and grid_sample takes (2((x+u)/w -0.5), 2((y+v)/h -0.5)) as input, where (w,h) are the image width and height, and (x,y) are the image coordinates. So you would need to scale the flows accordingly while using grid_sample.
2. Scale: This can be implemented as bilinear upsampling using pytorch/torch.nn.function.upsample(mode='bilinear')

[anuragranj]

This is equivalent flow warper in pytorch.

class FlowWarper(nn.Module):
    def __init__(self):
        super(FlowWarper, self).__init__(w,h)
        x = np.arange(0,w)
        y = np.arange(0,h)
        gx, gy = np.meshgrid(x,y)
        self.w = w
        self.h = h
        self.grid_x = Variable(torch.Tensor(gx), requires_grad=False)
        self.grid_y = Variable(torch.Tensor(gy), requires_grad=False)

    def forward(self, img, uv):
        u = uv[:,0,:,:]
        v = uv[:,1,:,:]
        X = self.grid_x.unsqueeze(0).expand_as(u) + u
        Y = self.grid_y.unsqueeze(0).expand_as(v) + v
        X = 2*(X/self.w - 0.5)
        Y = 2*(Y/self.h - 0.5)
        grid_tf = torch.stack((X,Y), dim=3)
        img_tf = F.grid_sample(img, grid_tf)
        return img_tf

[sniklaus]

You may also have a look at: https://github.com/sniklaus/pytorch-spynet

Thank you Anurag for you great work!

[PkuRainBow]

@sniklaus Great! I am wondering could you share your reproduced performance?

[sniklaus]

All necessary code is in the linked repository, what do you need in particular?

[anuragranj]

Thanks @sniklaus for the pytorch version.

[PkuRainBow]

@sniklaus Thanks. Nothing particular is needed.

[EnQing626]

hi @anuragranj , It seems that if you want to warp an image from image1 to image2 and the optical flow is generated from image1 to image2, you should set "u = -uv[:,0,:,:] and v = -uv[:,1,:,:]" instead of "u = uv[:,0,:,:] v = uv[:,1,:,:]". Is that correct?