prs-eth / pixtransform

[ICCV 2019] Guided Super-Resolution as a Learned Pixel-to-Pixel Transformation

PixTransform

PyTorch implementation of the algorithm presented in our paper [1]. The algorithm can be used to perform guided super-resolution, for instance:

Have a look at our blog post or publication if you want to learn more about the method.

The function PixTransform takes as input two images, the source image of size M x M and a guide image of size C x N x N, and returns an upsampled version of the source image with size N x N. The upsampling factor D, equal to N/M, must be an integer.

predicted_target = PixTransform(source,guide)

additional variables can be passed to change the default parameters. For further details about the algorithm see [1]

Getting started

Installation

clone this git repository and make sure that the following packages are installed:

• numpy
• matplotlib
• scipy
• Pytorch
• ProxTV (optional)
• tqdm

Example script

To run the algorithm on some sample images check the Jupyter Notebook file process_examples.ipynb.

Other methods

We compare the results of our algorithm with the results of the guided filter [2], the fast bilateral solver [3] (authors' implementation), the static-dynamic filter [4] (authors' implementation) and the multi-scale guided network [5] (authors' implementation).

References

[1] R. de Lutio, S. D'Aronco, J. D. Wegner, K. Schindler. "Guided Super-Resolution as Pixel-to-Pixel Transformation", ICCV, 2019.

[2] K. He, J. Sun, X. Tang. "Guided image filtering", TPAMI, 2013.

[3] J. T. Barron, B. Poole. "The fast bilateral solver", ECCV, 2016.

[4] B. Ham, M. Cho, J. Ponce. "Robust guided image filtering using nonconvex potentials", TPAMI, 2018.

[5] T.-W. Hui, C. C. Loy, X. Tang. "Depth map super-resolution by deep multi-scale guidance", ECCV, 2016.

pixtransform's Issues

Some question about the paper

As mentioned in the issue openned by @codyshen0000, the proposed network uses the source image(low reso depth image) only at the training stage(to construct the loss func), and it directly transform the guide image(color image/gray image) to the target image(depth) without the source image at the inference stage, so I think the network should be a Monocular-Depth-Estimation network rather than a Super-Resolution network.

The size does not match in pix_transform?

Hi!
in your infer code
predicted_target_img = PixTransform(guide_img=guide_img,source_img=source_img,params=params,target_img=target_img)

source_img is N*N < M*M

while in your code Pixtransform/pix_transform.py, line91
for i in range(0, M): for j in range(0, M): guide_patches[j + i * M, :, :, :] = guide_img[:, i * D:(i + 1) * D, j * D:(j + 1) * D] **source_pixels[j + i * M] = source_img[i:(i + 1), j:(j + 1)]**

I am confused because the source_img is of N*N, while the index can be larger than N!

Besides, the guide_path can be a writer with torch.unfold(), which is fast than for..for..

How to do SR on the whole picture

Thank you for your work. As shown in your paper，the results are all patches operations. So i padding zero around the pictures to n*n resolution, then i run the program. However, the results are not good.


The ratio of downsampling is 8. Is it due to padding zero?

Question for PixTransform

Hello author(s), as mentioned in your paper, the guide image and the target image are directly mapped, Does this mean we can directly transform the guide image to the target image without the source image?