tengfei-wang / external-internal-inpainting

[CVPR 2021] EII: Image Inpainting with External-Internal Learning and Monochromic Bottleneck

EII: Image Inpainting with External-Internal Learning and Monochromic Bottleneck

Image Inpainting with External-Internal Learning and Monochromic Bottleneck
Tengfei Wang*, Hao Ouyang*, Qifeng Chen
CVPR 2021

paper | project website | video

Introduction

The proposed method can be applied to improve the color consistency of leaning-based image inpainting results. The progressive internal color propagation shows strong performance even with large mask ratios.

Prerequisites

• Python 3.6
• Pytorch 1.6
• Numpy

Installation

git clone https://github.com/Tengfei-Wang/external-internal-inpainting.git
cd external-internal-inpainting

Quick Start

Colorization

To try our internal colorization method:

python main.py  --img_path images/input2.png --gray_path images/gray2.png  --mask_path images/mask2.png  --pyramid_height 3

The colorization results are placed in ./results.
In case the colorization results are unsatisfactory, you may consider changing the pyramid_height (2~5 work well for most cases).

Reconstruction

For the monochromic reconstruction stage, multiple inpainting networks can be applied as backbones by modifying the original input image, like:

input_new = torch.concat([input_RGB, input_gray],1) #input_new is 4-channel
output = backbone_model(input_new, mask) #output is single-channel
loss = criterion(output, input_gray)

Citation

If you find this work useful for your research, please cite:

@inproceedings{wang2021image,
  title={Image Inpainting with External-internal Learning and Monochromic Bottleneck},
  author={Wang, Tengfei and Ouyang, Hao and Chen, Qifeng},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={5120--5129},
  year={2021}
}

Contact

Please send emails to [email protected] or [email protected] if there is any question

Acknowledgement

We thank the authors of DIP for sharing their codes.

external-internal-inpainting's Issues

pre-trained weights

Hi,

Thanks for the GitHub repo. Can I check if the pre-trained weights will be released anywhere?

Image preprocessing problems

Hi, this is a good job. According to the gray processing method mentioned in your paper: 0.11 * b + 0.59 * g + 0.30 * R for mask gray processing. The image completed by GatedConv is first gray processed and then converted into RGB image. When I put the uncompleted image, mask and processed completed gray image into the network, there will be white uncompleted area in the generated image. Is there a mistake in my image processing?

questions about the net code

Dear author
Is there any code about network structure
The code on github is incomplete
best wishes!

Red artifact

Hey, nice work!

In your project website examples, is it possible that the resulting red artifacts are caused by the red mask?

Pre-trained model?

Hey there! Loved the paper.

Is there access to the pre-trained model?

Thanks!

When running main.py encountered an error while testing its own dataset

Traceback (most recent call last):
File "main.py", line 111, in
main()
File "main.py", line 27, in main
train(opt.pyramid_height, mask, img, img_gray, torch.cuda.FloatTensor, opt)
File "main.py", line 77, in train
mask_var_pyr, img_var_pyr, img_gray_var_pyr = get_pyramids(mask, img, img_gray, pyramid_height, dtype)
File "main.py", line 67, in get_pyramids
img_gray_var_pyr.append(get_pyramid_image(img_gray, w_n, h_n)[:,:1,:,:].type(dtype))
File "main.py", line 33, in get_pyramid_image
img = np.array(img).transpose(2,0,1)/ 255.
ValueError: axes don't match array

I spent a long time trying to solve the problem, but I still failed. I hope I can get your help. Thank you and look forward to your reply！

Is it possible to use for higher resolution images?

Great paper! Loved it.

For real user-case scenarios, is it possible to use it for HQ images (Full HD, for example)?

Thanks!