jingyunliang / fkp

Official PyTorch code for Flow-based Kernel Prior with Application to Blind Super-Resolution (FKP, CVPR2021)

Jingyun Liang

Email / Homepage / Google Scholar / Github

I am currently a PhD Student at Computer Vision Lab, ETH Zürich, Switzerland. I am co-supervised by Prof. Luc Van Gool and Prof. Radu Timofte. I also work closely with Dr. Kai Zhang. I mainly focus on low-level vision research, especially on image and video restoration, such as

• image/video super-resolution (SR)
• image/video deblurring
• image/video denoising
• ...

🚀 News

• 2022-10-04: Our new paper RVRT, NeurlPS2022 achieves SOTA video restoration results with balanced size, memory and runtime.
• 2022-08-30: See our papers on real-world image denoising (SCUNet) and video denoising (ReViD).
• 2022-07-30: Three papers, including EFNet (event-based image deblurring, oral), DATSR (reference image SR) and DAVSR (video SR), accepted by ECCV2022.
• 2022-01-28: Our new paper VRT outperforms previous Video SR/ deblurring/ denoising/ frame interpolation/ space-time video SR methods by up to 😍 2.16dB. 😍
• 2021-10-20: SwinIR is awarded the best paper prize in ICCV-AIM2021.
• 2021-08-01: Three papers (HCFlow, MANet and BSRGAN) accepted by ICCV2021.
• 2021-03-29: One paper (FKP) accepted by CVPR2021.

🌱 Repositories

Topic	Title	Badge
real-world video denoising	Practical Real Video Denoising with Realistic Degradation Model
event-based image deblurring	Event-based Fusion for Motion Deblurring with Cross-modal Attention, ECCV2022
reference image SR	Reference-based Image Super-Resolution with Deformable Attention Transformer, ECCV2022
interpretable video restoration	Towards Interpretable Video Super-Resolution via Alternating Optimization, ECCV2022
transformer-based video restoration	Recurrent Video Restoration Transformer with Guided Deformable Attention
transformer-based video restoration	VRT: A Video Restoration Transformer
transformer-based image restoration	SwinIR: Image Restoration Using Swin Transformer
real-world image denoising	Practical Blind Denoising via Swin-Conv-UNet and Data Synthesis
real-world image SR	Designing a Practical Degradation Model for Deep Blind Image Super-Resolution, ICCV2021
blind image SR	Mutual Affine Network for Spatially Variant Kernel Estimation in Blind Image Super-Resolution, ICCV2021
blind image SR	Flow-based Kernel Prior with Application to Blind Super-Resolution, CVPR2021
normalizing flow-based image SR and image rescaling	Hierarchical Conditional Flow: A Unified Framework for Image Super-Resolution and Image Rescaling, ICCV2021
image/ video restoration	Image/ Video Restoration Toolbox

fkp's Issues

How to train FKP?(no input datasets)

Thanks for your work. I have a question about how to train FKP. Because in your code, there are no input datasets. I am very confused about it. Could you help me?

Can not get the correct result of KernelGAN + USRNet

Use the given code of KernelGAN+USRNet, the PSNR is only 15.84 on DIV2K valid dateset of scale 4.
But the result of scale 2 is same with your paper.

Benchmark Evaluation on KernelGANFKPx4

Hello,

Thanks for uploading the code.

I am trying to evaluate KernelGANFKP on Set5 & Set14 & B100 for scaling factor x4:
FKP/KernelGANFKP$ python main.py --SR --sf 4 --dataset {dataset}

However, I am hitting an error during the forward pass of the discriminator if the input patch has a height or width < 27. Is KernelGANFKPx4 not supported for Set5, Set14, and B100 datasets?

Pixel Offset Problem

Hi, thank you for sharing the code.

I have an LR image and corresponding HR image.

We use the LR image as the input to the KernelGAN-FKP, and obtain a super-resolved image without ground-truth kernels, .

It seems that there is pixel offset between the super-resolved image and the original HR image, leading to a very low PSNR of the super-resolved image.

Do you know how to solve this problem?

Training on pre-defined kernels

Hello! I have a kernel-pool which has 1k kernels of size 33x33 for x4 upscaling. How should the code be modified in order to make sure that we obtain proper estimations? Cause right now in order to obtain the x4 kernel you use FKP_x2 to obtain an estimation and then you do a kernel shift to obtain the x4 one. But this should not work if we would have to bring our own kernels of a different size.

What is Zk, please

The Zk in this paper is also the kernel_code in the code. What is Zk,please

New Super-Resolution Benchmarks

Hello,

MSU Graphics & Media Lab Video Group has recently launched two new Super-Resolution Benchmarks.

• Video Upscalers Benchmark: Quality Enhancement determines the best upscaling methods for increasing video resolution and improving visual quality.
• Super-Resolution for Video Compression benchmark aims to test Super-Resolution methods on compressed videos and select the best model for each video codec standard.

If you are interested in participating, you can add your algorithm following the submission steps:

• Submit for Video Upscalers Benchmark: Quality Enhancement
• Submit for Super-Resolution for Video Compression benchmark

We would be grateful for your feedback on our work!

image alignment issue

It seems that the alignment between HR and LR generated by your program is off and it could be related to the following code (different from original version). But somehow after USRNet, the alignment is back to normal. This is quite confusing.

FKP/data/prepare_dataset.py

Line 24 in 5ca3f7c

wanted_center_of_mass = (np.array(kernel.shape) - sf) / 2.

about random kernel generator

In the following code, lamda_1&2 (referred as kernel width) are used to calculate the COV matrix. But shouldn't the square of lambda be used here for covariance?

FKP/data/prepare_dataset.py

Line 50 in 5ca3f7c

def gen_kernel_fixed(k_size, scale_factor, lambda_1, lambda_2, theta, noise):

how to test on my own images?

hi Jingyun,
Thanks for your work. There are too many test code in the project which make me confuesd.
I just want to test on my own image, what should do ?
Thanks again.