cszn / dpir

Plug-and-Play Image Restoration with Deep Denoiser Prior (IEEE TPAMI 2021) (PyTorch)

Kai Zhang

[Homepage] [Google Scholar] [ResearchGate] [知乎]

I am currently a postdoctoral researcher at Computer Vision Lab, ETH Zurich, Switzerland.

⚡ News:

Our new work SCUNet for practical image denoising.

⚡ News

We released the testing code of BSRGAN!

🌱 My Repositories

SCUNet	BSRGAN	KAIR	USRNet	DPIR	IRCNN	SRMD	DPSR	DnCNN	FFDNet

The pytorch training and testing codes of SRMD, DnCNN, FFDNet can be found in KAIR.

dpir's Issues

main_dipr_sisr throws input type RuntimeError

Hi, I was trying to run the main_dipr_sisr.py(inconsistent naming, the rest say dpir) in colab like this:

!python main_dipr_sisr.py

it gives me this output:

`LogHandlers setup!

20-09-28 07:21:02.662 : model_name:drunet_color, image sigma:0.000, model sigma:0.000
20-09-28 07:21:02.664 : Model path: model_zoo/drunet_color.pth
20-09-28 07:21:02.664 : testsets/srbsd68
20-09-28 07:21:02.669 : --------- sf:2 --k: 0 ---------
Traceback (most recent call last):
File "main_dipr_sisr.py", line 301, in
main()
File "main_dipr_sisr.py", line 222, in main
x = utils_model.test_mode(model, x, mode=3, refield=32, min_size=256, modulo=16)
File "/content/drive/My Drive/dpir/DPIR/utils/utils_model.py", line 36, in test_mode
E = test_x8(model, L, modulo)
File "/content/drive/My Drive/dpir/DPIR/utils/utils_model.py", line 164, in test_x8
E_list = [test_pad(model, util.augment_img_tensor(L, mode=i), modulo=modulo) for i in range(8)]
File "/content/drive/My Drive/dpir/DPIR/utils/utils_model.py", line 164, in
E_list = [test_pad(model, util.augment_img_tensor(L, mode=i), modulo=modulo) for i in range(8)]
File "/content/drive/My Drive/dpir/DPIR/utils/utils_model.py", line 68, in test_pad
E = model(L)
File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 722, in _call_impl
result = self.forward(*input, **kwargs)
File "/content/drive/My Drive/dpir/DPIR/models/network_unet.py", line 106, in forward
x1 = self.m_head(x0)
File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 722, in _call_impl
result = self.forward(*input, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/conv.py", line 419, in forward
return self._conv_forward(input, self.weight)
File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/conv.py", line 416, in _conv_forward
self.padding, self.dilation, self.groups)
RuntimeError: Input type (torch.cuda.DoubleTensor) and weight type (torch.cuda.FloatTensor) should be the same
`

I got it to run by adding this

L = L.float()

to the function

test_mode(model, L, mode=0, refield=32, min_size=256, sf=1, modulo=1)

in DPIR/utils/utils_model.py right before the if statements, however I do not know enough to tell if using a simple cast like that causes any internal problems. Thought I'd put this here in case anyone else runs into the issue.

Using main_dpir_sisr_real_applications.py throws AttributeError

I have duplicated main_dpir_sisr_real_applications.py and pointed it to my images folder. Running this script produces the following error:

k = sr.shift_pixel(k, sf)  # shift the kernel
AttributeError: module 'utils.utils_sisr' has no attribute 'shift_pixel'

I'm running this in CPU mode.

Any way to fix this error or is there an updated file?

Thanks

can you share your training code? thank you

How to train DPIR on different CFA patterns for demosaicing task ?

Hi, author,

The default network only offer RGGB pattern for demosaicing, can i use this network to train with other patterns like X-Trans, or CMY patterns?

Can not find drunet_gray.pth in google driver

Hi, thanks for sharing this awesome work, but I can' t find the drunet_gray.pth in the google driver. Did it go by any other names?

Hello there! ! Is there a public training code or loss function plan?

The DPIR is consist of USRNet (training) and inference codes of several type?

Hello Sir..

I don't know what DPIR is.

The DPIR is consist of USRNet (training) and inference codes of several type. Right??

Thanks,
Edward Cho.

Can you add the version of each required library? Cause some seem to be deprecated

It would be awesome if you share the training code as well

torch.fft() related functions are outdated in utils_sisr.py

torch.fft() related functions were removed since v1.8.0. Instead a torch.fft module was introduced. Please consider updating the codes in utils_sisr.py to adapt to the changes. Thank you very much for your hard effort.

Fits format

Hello,

thank you for this awesome package I would like to train a database with a fits format which folder should I change. Any recommendation please.

Thanks

How to realize the ZOOM IN function to compare the results in the paper?

in many images restoration papers and image super resolution papers,
the images showed like below, are there any code provided in python or matlab there to do the ZOOM IN job?
I am a rookie in this area, so I would like to thank you all for giving advice or responces

Super Resolution using DPIR

Thank you for the code.

Which model is used to generate the SR results as shown in the "single image super resolutions, visual comparison figure". In the model zoo (google drive link), I cannot find the DPIR model.

Am I missing something?

There is no drunet_xxxxx.pth file

In the Google download address given in the model_zoor file, there is no drunet_gray.pth and drunet_color.pth, and the link to Tencent Weiyun is invalid. When I checked it on May 9, it was still valid.
Teacher, if you still have these two documents, can you send a copy? I would be very grateful.

License for code and models

Hi Kai,

Thanks for sharing the code and pretrained models online. It's an amazing work!

Would you issue a license for the public to use your work?

FYI, pretrained models in Detectron2 are licensed under the Creative Commons Attribution-ShareAlike 3.0 license, while the code is licensed under Apache License 2.0.

Best,
-Weilin

What methods are suitable for estimating the noise level or as a noise_level_model?

This is an outstanding job, but I still have a question regarding noise_level_model.
Now there are various noise level estimation methods, including those in the spatial domain, frequency domain, and based on deep learning, and so on. The scale of the results obtained by applying different methods to estimate the noise level on the same image is greatly different, In practical applications,using them as noise level estimation models can make the effect vastly different.; so what method should be used to estimate the noise level in DPIR(noise_level_model) is the most suitable?

Bad results when testing on real world images

Hi, thanks for sharing this work, looks great!

I was impressed by the initial sample test results until I started testing on real world images.
Please see the attached input + output, I've tested on many real world cases where the SR just fails and image quality of original image looks superior although it has 15% the size.

input - "input-sd-test.jpg" - 1200 × 819 resolution 0.2MB
Output1 - "Output_x2_k1_drunet_color.png" - 1200x818 resolution 1.6MB
Output2 - "Output_x2_k0_drunet_color.png" - 1200x818 resolution 1.5MB

Please advise.

input

Output1

Output2

Can I get the installation environment information?

I have environment info

docker - pytorch 1.6
python 3.7.7

run : 'python main_dpir_sisr.py'
I know that pytorch-fft is necessary.

pip install pytorch-fft
Collecting pytorch-fft
Using cached pytorch_fft-0.15.tar.gz (7.8 kB)
ERROR: Command errored out with exit status 1:
command: /opt/conda/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-0qpqfjyv/pytorch-fft/setup.py'"'"'; file='"'"'/tmp/pip-install-0qpqfjyv/pytorch-fft/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-install-0qpqfjyv/pytorch-fft/pip-egg-info
cwd: /tmp/pip-install-0qpqfjyv/pytorch-fft/
Complete output (9 lines):
Traceback (most recent call last):
File "", line 1, in
File "/tmp/pip-install-0qpqfjyv/pytorch-fft/setup.py", line 6, in
import build
File "/tmp/pip-install-0qpqfjyv/pytorch-fft/build.py", line 3, in
from torch.utils.ffi import create_extension
File "/opt/conda/lib/python3.7/site-packages/torch/utils/ffi/init.py", line 1, in
raise ImportError("torch.utils.ffi is deprecated. Please use cpp extensions instead.")
ImportError: torch.utils.ffi is deprecated. Please use cpp extensions instead.
----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

Is it right to install this?
Thank you for letting me know the wrong part.

How to determine the iteration nums?

There are different iter nums for different tasks? How to decide it ? And whether the larger the iter num, the better the reults?

drunet for denoisin isn't in google drive link

Hi, I want to use your cnn for denoising, but drunet_gray and drunet_color isn't in google drive link

I got errors in load_state_dict in IRCNN

I downloaded icrnn gray pth and loaded to model ircnn but it didn't work. How can I solve it?
(What I erased is my private information)

Number of parameters much larger than your previous work?

The number of parameters of this UNetRes is much larger than your previous work like DnCNN?

Performance on other image restoration tasks

Hello,

Thanks for the amazing work and paper! I know from the paper that the denoiser can be used for regularization of general image restoration tasks. I am curious whether you have also done experiments on image inpainting and random missing pixels? If so, can you show some examples? Thank you very much:)