shepnerd / inpainting_gmcnn

Image Inpainting via Generative Multi-column Convolutional Neural Networks, NeurIPS2018

Image Inpainting via Generative Multi-column Convolutional Neural Networks

by Yi Wang, Xin Tao, Xiaojuan Qi, Xiaoyong Shen, Jiaya Jia.

Results on Places2, CelebA-HQ, and Paris streetview with rectangle masks.

Results on Places2 and CelebA-HQ with random strokes.

Introduction

This repository is for the NeurIPS 2018 paper, 'Image Inpainting via Generative Multi-column Convolutional Neural Networks'.

If our method is useful for your research, please consider citing:

@inproceedings{wang2018image,
  title={Image Inpainting via Generative Multi-column Convolutional Neural Networks},
  author={Wang, Yi and Tao, Xin and Qi, Xiaojuan and Shen, Xiaoyong and Jia, Jiaya},
  booktitle={Advances in Neural Information Processing Systems},
  pages={331--340},
  year={2018}
}

Our framework

Partial Results

    

    

More results

Prerequisites

• Python3.5 (or higher)
• Tensorflow 1.4 (or later versions, excluding 2.x) with NVIDIA GPU or CPU
• OpenCV
• numpy
• scipy
• easydict
• Pytorch 1.0 with NVIDIA GPU or CPU
• tensorboardX

Installation

git clone https://github.com/shepnerd/inpainting_gmcnn.git
cd inpainting_gmcnn/tensorflow

or

cd inpainting_gmcnn/pytorch

For tensorflow implementations

Testing

Download pretrained models through the following links (paris_streetview, CelebA-HQ_256, CelebA-HQ_512, Places2), and unzip and put them into checkpoints/. To test images in a folder, you can specify the folder address by the opinion --dataset_path, and set the pretrained model path by --load_model_dir when calling test.py.

For example:

python test.py --dataset paris_streetview --data_file ./imgs/paris-streetview_256x256/ --load_model_dir ./checkpoints/paris-streetview_256x256_rect --random_mask 0

or

sh ./script/test.sh

Training

For a given dataset, the training is formed of two stages. We pretrain the whole network with only confidence-driven reconstruction loss first, and finetune this network using adversarial and ID-MRF loss along with the reconstruction loss after the previous phase converges.

To pretrain the network,

python train.py --dataset [DATASET_NAME] --data_file [DATASET_TRAININGFILE] --gpu_ids [NUM] --pretrain_network 1 --batch_size 16

where [DATASET_TRAININGFILE] indicates a file storing the full paths of the training images.

Then finetune the network,

python train.py --dataset [DATASET_NAME] --data_file [DATASET_TRAININGFILE] --gpu_ids [NUM] --pretrain_network 0 --load_model_dir [PRETRAINED_MODEL_PATH] --batch_size 8

We provide both random stroke and rectangle masks in the training and testing phase. The used mask type is indicated by specifying --mask_type [rect(default)|stroke] option when calling train.py or test.py.

A simple interactive inpainting GUI

A GUI written using tkinter is given in `painter_gmcnn.py`. Start it by calling ```shell sh ./script/vis_tool.sh ```

Other pretrained models

CelebA-HQ_512 trained with stroke masks.

For pytorch implementations

The testing and training procedures are similar to these in the tensorflow version except some parameters are with different names.

Testing

A pretrained model: CelebA-HQ_256.

Training

Compared with the tensorflow version, this pytorch version would expect a relatively smaller batch size for training.

Other versions

Checkout the keras implementation of our paper by Tomasz Latkowski here.

Disclaimer

• For the provided pretrained models, their performance would degrade obviously when they are evaluated by a mask whose unknown areas are too large.
• As claimed in the paper, for the large datasets with thousands of categories, the model performance is unstable. Recent GAN using large-scale techniques may ease this problem.
• We did not give the full implementation of ID-MRF (in this repo) described in our original paper. The step of excluding s is omitted for computational efficiency.
• In the pytorch version, a different GAN loss (wgan hinge loss with spectral normalization) is adopted.

Acknowledgments

Our code is partially based on Generative Image Inpainting with Contextual Attention and pix2pixHD. The implementation of id-mrf loss is borrowed from contextual loss.

Contact

Please send email to [email protected].

inpainting_gmcnn's Issues

Bad results for 512 images

I followed your instructions on issue #9 by setting learning rate to 1e-5 and by changing "mask_priority = priority_loss_mask(mask)" to "mask_priority = priority_loss_mask(mask, hsize=128, sigma=1.0 / 60, iters=16)" in network.py (L178). See image bellow for my results.
(Top left- pretrain results, Top right- fine tuning at 100 epochs, bottom left- fine tuning at 500 epochs and Bottom right- fine tuning at 1000 epochs.)

Bellow is a screenshot of my training loss.

am I doing anything wrong? are this results normal? #5

Bad results for large masks

I tried testing your code on masks of large sizes and the output is quite bad. Any idea for some workaround that can be done for this?

IndexError: list index out of range

Following is the full stack trace of the PyTorch version...

C:\inpainting_gmcnn\pytorch>python test.py --dataset paris_streetview --data_file imgs/paris-streetview_256x256/ --load_model_dir checkpoints/paris-streetview_256x256_rect --random_mask 0
------------ Options -------------
d_cnum: 32
data_file: imgs/paris-streetview_256x256/
dataset: paris_streetview
dataset_path: imgs/paris-streetview_256x256/
date_str: test_20191211-232248
g_cnum: 32
gpu_ids: 0
img_shapes: [256, 256, 3]
load_model_dir: checkpoints/paris-streetview_256x256_rect
mask_shapes: [128, 128]
mask_type: rect
mode: save
model: gmcnn
model_folder: test_20191211-232248_paris_streetview_gmcnn_s256x256_gc32
phase: test
random_mask: False
saving_path: ./test_results\test_20191211-232248_paris_streetview_gmcnn_s256x256_gc32
seed: 1
test_dir: ./test_results
test_num: -1
-------------- End ----------------
The total number of testing images is 4, and we take 4 for test.
configuring model..
initialize network with normal
---------- Networks initialized -------------
GMCNN(
(EB1): ModuleList(
(0): Conv2d(4, 32, kernel_size=(7, 7), stride=(1, 1))
(1): Conv2d(32, 64, kernel_size=(7, 7), stride=(2, 2))
(2): Conv2d(64, 64, kernel_size=(7, 7), stride=(1, 1))
(3): Conv2d(64, 128, kernel_size=(7, 7), stride=(2, 2))
(4): Conv2d(128, 128, kernel_size=(7, 7), stride=(1, 1))
(5): Conv2d(128, 128, kernel_size=(7, 7), stride=(1, 1))
(6): Conv2d(128, 128, kernel_size=(7, 7), stride=(1, 1), dilation=(2, 2))
(7): Conv2d(128, 128, kernel_size=(7, 7), stride=(1, 1), dilation=(4, 4))
(8): Conv2d(128, 128, kernel_size=(7, 7), stride=(1, 1), dilation=(8, 8))
(9): Conv2d(128, 128, kernel_size=(7, 7), stride=(1, 1), dilation=(16, 16))
(10): Conv2d(128, 128, kernel_size=(7, 7), stride=(1, 1))
(11): Conv2d(128, 128, kernel_size=(7, 7), stride=(1, 1))
(12): PureUpsampling()
)
(EB2): ModuleList(
(0): Conv2d(4, 32, kernel_size=(5, 5), stride=(1, 1))
(1): Conv2d(32, 64, kernel_size=(5, 5), stride=(2, 2))
(2): Conv2d(64, 64, kernel_size=(5, 5), stride=(1, 1))
(3): Conv2d(64, 128, kernel_size=(5, 5), stride=(2, 2))
(4): Conv2d(128, 128, kernel_size=(5, 5), stride=(1, 1))
(5): Conv2d(128, 128, kernel_size=(5, 5), stride=(1, 1))
(6): Conv2d(128, 128, kernel_size=(5, 5), stride=(1, 1), dilation=(2, 2))
(7): Conv2d(128, 128, kernel_size=(5, 5), stride=(1, 1), dilation=(4, 4))
(8): Conv2d(128, 128, kernel_size=(5, 5), stride=(1, 1), dilation=(8, 8))
(9): Conv2d(128, 128, kernel_size=(5, 5), stride=(1, 1), dilation=(16, 16))
(10): Conv2d(128, 128, kernel_size=(5, 5), stride=(1, 1))
(11): Conv2d(128, 128, kernel_size=(5, 5), stride=(1, 1))
(12): PureUpsampling()
(13): Conv2d(128, 64, kernel_size=(5, 5), stride=(1, 1))
(14): Conv2d(64, 64, kernel_size=(5, 5), stride=(1, 1))
(15): PureUpsampling()
)
(EB3): ModuleList(
(0): Conv2d(4, 32, kernel_size=(3, 3), stride=(1, 1))
(1): Conv2d(32, 64, kernel_size=(3, 3), stride=(2, 2))
(2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1))
(3): Conv2d(64, 128, kernel_size=(3, 3), stride=(2, 2))
(4): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1))
(5): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1))
(6): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), dilation=(2, 2))
(7): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), dilation=(4, 4))
(8): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), dilation=(8, 8))
(9): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), dilation=(16, 16))
(10): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1))
(11): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1))
(12): PureUpsampling()
(13): Conv2d(128, 64, kernel_size=(3, 3), stride=(1, 1))
(14): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1))
(15): PureUpsampling()
(16): Conv2d(64, 32, kernel_size=(3, 3), stride=(1, 1))
(17): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1))
)
(decoding_layers): ModuleList(
(0): Conv2d(224, 16, kernel_size=(3, 3), stride=(1, 1))
(1): Conv2d(16, 3, kernel_size=(3, 3), stride=(1, 1))
)
(pads): ModuleList(
(0): ReflectionPad2d((0, 0, 0, 0))
(1): ReflectionPad2d((1, 1, 1, 1))
(2): ReflectionPad2d((2, 2, 2, 2))
(3): ReflectionPad2d((3, 3, 3, 3))
(4): ReflectionPad2d((4, 4, 4, 4))
(5): ReflectionPad2d((5, 5, 5, 5))
(6): ReflectionPad2d((6, 6, 6, 6))
(7): ReflectionPad2d((7, 7, 7, 7))
(8): ReflectionPad2d((8, 8, 8, 8))
(9): ReflectionPad2d((9, 9, 9, 9))
(10): ReflectionPad2d((10, 10, 10, 10))
(11): ReflectionPad2d((11, 11, 11, 11))
(12): ReflectionPad2d((12, 12, 12, 12))
(13): ReflectionPad2d((13, 13, 13, 13))
(14): ReflectionPad2d((14, 14, 14, 14))
(15): ReflectionPad2d((15, 15, 15, 15))
(16): ReflectionPad2d((16, 16, 16, 16))
(17): ReflectionPad2d((17, 17, 17, 17))
(18): ReflectionPad2d((18, 18, 18, 18))
(19): ReflectionPad2d((19, 19, 19, 19))
(20): ReflectionPad2d((20, 20, 20, 20))
(21): ReflectionPad2d((21, 21, 21, 21))
(22): ReflectionPad2d((22, 22, 22, 22))
(23): ReflectionPad2d((23, 23, 23, 23))
(24): ReflectionPad2d((24, 24, 24, 24))
(25): ReflectionPad2d((25, 25, 25, 25))
(26): ReflectionPad2d((26, 26, 26, 26))
(27): ReflectionPad2d((27, 27, 27, 27))
(28): ReflectionPad2d((28, 28, 28, 28))
(29): ReflectionPad2d((29, 29, 29, 29))
(30): ReflectionPad2d((30, 30, 30, 30))
(31): ReflectionPad2d((31, 31, 31, 31))
(32): ReflectionPad2d((32, 32, 32, 32))
(33): ReflectionPad2d((33, 33, 33, 33))
(34): ReflectionPad2d((34, 34, 34, 34))
(35): ReflectionPad2d((35, 35, 35, 35))
(36): ReflectionPad2d((36, 36, 36, 36))
(37): ReflectionPad2d((37, 37, 37, 37))
(38): ReflectionPad2d((38, 38, 38, 38))
(39): ReflectionPad2d((39, 39, 39, 39))
(40): ReflectionPad2d((40, 40, 40, 40))
(41): ReflectionPad2d((41, 41, 41, 41))
(42): ReflectionPad2d((42, 42, 42, 42))
(43): ReflectionPad2d((43, 43, 43, 43))
(44): ReflectionPad2d((44, 44, 44, 44))
(45): ReflectionPad2d((45, 45, 45, 45))
(46): ReflectionPad2d((46, 46, 46, 46))
(47): ReflectionPad2d((47, 47, 47, 47))
(48): ReflectionPad2d((48, 48, 48, 48))
)
)
[Network GM] Total number of parameters : 12.562 M

Loading pretrained model from checkpoints/paris-streetview_256x256_rect
Traceback (most recent call last):
File "test.py", line 32, in
ourModel.load_networks(getLatest(os.path.join(config.load_model_dir, '*.pth')))
File "C:\Projects\inpainting_gmcnn\pytorch\util\utils.py", line 84, in getLatest
return files[sorted(range(len(file_times)), key=lambda x: file_times[x])[-1]]
IndexError: list index out of range

The design of upsampling

I notice that you use different strategies to upsample the components. Bilinear is for 7x7 branch, deconvolution is for 3x3 branch and both are used for 5x5 branch. Is there any difference between bilinear and deconvolution? Why do you arrange upsample strategy like this?
Thank you!

ID-MRF loss

I did not modify any parameters during my training. Why did the ID-MRF loss increase during the second phase of training?

Issues regarding input images with already present holes

I have tested your Tensorflow implementattion code on your already present data set in imgs/places2_256x256 folder. Now I want to test with my own image. My image already has some hole present. It is given in this link

Accordingly I tried modifying your code by eliminating the following portion from your test.py file
if h >= config.img_shapes[0] and w >= config.img_shapes[1]:
h_start = (h-config.img_shapes[0]) // 2
w_start = (w-config.img_shapes[1]) // 2
image = image[h_start: h_start+config.img_shapes[0], w_start: w_start+config.img_shapes[1], :]

else:
t = min(h, w)
image = image[(h-t)//2:(h-t)//2+t, (w-t)//2:(w-t)//2+t, :]
image = cv2.resize(image, (config.img_shapes[1], config.img_shapes[0]))
image = image * (1-mask) + 255 * mask

I want only to fill the holes in the images through your code. I already have images in which holes are present . Kindly say how to do the same?

I can visualize that I need to apply mask on the portion to be refilled and then run the session, but how to do the same?
Here is the portion I guess I need to change but I wonder how?
for i in range(test_num):
if config.mask_type == 'rect':
mask = generate_mask_rect(config.img_shapes, config.mask_shapes, config.random_mask)
else:
mask = generate_mask_stroke(im_size=(config.img_shapes[0], config.img_shapes[1]), parts=8, maxBrushWidth=24, maxLength=100, maxVertex=20)

THANKS in advance. Waiting for your reply.

Questions about implementation

@shepnerd Thank you for the great work. After reading the paper and code, I can not understand the following code quite well. Need your help.

When computing relative similarity between v and s:

inpainting_gmcnn/tensorflow/net/ops.py

Line 330 in fe5295e

def calc_relative_distances(self, axis=3):

    def calc_relative_distances(self, axis=3):
        epsilon = 1e-5
        div = tf.reduce_min(self.raw_distances, axis=axis, keep_dims=True)
        relative_dist = self.raw_distances / (div + epsilon)
        return relative_dist

The paper said:

In my understanding:

1. max(r) in paper is the div in the code.
2. When s is not equal to value in div, the code is OK.
3. When s is equal to the value in div, as the paper said, div should exclude this value first, then find another suitable value.

But the code didn't handle the condition when s is equal to the value in div and didn't exclude it. Is this a small bug in the code or something wrong in my understanding ?

Hope I described it clearly.

the bad result on rect mask

@shepnerd thanks for your excellent work！I notice you release the stroke model on places2，it works well on stoke mask。 but when I finetune the model with rect mask from your released model， the test result often occurs heavily artifacts。for example：

is it a normal result with rect mask on places2? Can you release your trained model on places2 with rect mask?

TypeError: __init__() got multiple values for argument 'filters'

python test.py --dataset paris_streetview --data_file imgs/paris-streetview_256x256/ --load_model_dir checkpoints/paris-streetview_256x256_rect/ --random_mask 0 --g_cnum 32

Traceback (most recent call last):
File "test.py", line 37, in
output = model.evaluate(input_image_tf, input_mask_tf, config=config, reuse=reuse)
File "C:\Projects\Image_inpainting\net\network.py", line 300, in evaluate
batch_predict = self.build_generator(im, mask, reuse = reuse)
File "C:\Projects\Image_inpainting\net\network.py", line 37, in build_generator
x = conv_7( x_w_mask, filters = cnum, strides = 1, name = b_names[0] + 'conv1' )
TypeError: init() got multiple values for argument 'filters'

help me

Excuse me,I want to run prtrained models of CelebA-HQ-256,but have a error as follows :‘
return files[sorted(range(len(file_times)), key=lambda x: file_times[x])[-1]]
IndexError: list index out of range’ I can't solve it.

Trying to test and receiving error

Hi! I'm trying to test the pre-trained model on CelebA-HQ. I downloaded the checkpoint, put 1 image in the folder and provided a text file with its path. I type this:
python /content/inpainting_gmcnn/tensorflow/test.py --data_file /content/1.txt --load_model_dir '/content/drive/My Drive/Multicolumn' --random_mask 0 but receiving this error:

Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py", line 1607, in _create_c_op
c_op = c_api.TF_FinishOperation(op_desc)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Dimension 3 in both shapes must be equal, but are 32 and 64. Shapes are [7,7,5,32] and [7,7,5,64]. for 'Assign' (op: 'Assign') with input shapes: [7,7,5,32], [7,7,5,64].

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/content/inpainting_gmcnn/tensorflow/test.py", line 45, in
vars_list))
File "/content/inpainting_gmcnn/tensorflow/test.py", line 44, in
assign_ops = list(map(lambda x: tf.assign(x, tf.contrib.framework.load_variable(config.load_model_dir, x.name)),
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/state_ops.py", line 227, in assign
validate_shape=validate_shape)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/gen_state_ops.py", line 66, in assign
use_locking=use_locking, name=name)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/op_def_library.py", line 794, in _apply_op_helper
op_def=op_def)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/util/deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py", line 3357, in create_op
attrs, op_def, compute_device)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py", line 3426, in _create_op_internal
op_def=op_def)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py", line 1770, in init
control_input_ops)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py", line 1610, in _create_c_op
raise ValueError(str(e))
ValueError: Dimension 3 in both shapes must be equal, but are 32 and 64. Shapes are [7,7,5,32] and [7,7,5,64]. for 'Assign' (op: 'Assign') with input shapes: [7,7,5,32], [7,7,5,64].

What may be the problem?

Batch normalization layers

Hi @shepnerd ,
What do you thing of applying additional batch normalization layers? Have you tried to utilize batch normalization to stabilize learning in generator and discriminators?

dataset

can you give me your dataset(celeba-hq) used for train the model celeba-hq-256.my email is [email protected] ,thanks again.

Hard-code error

Here

inpainting_gmcnn/pytorch/model/net.py

Line 258 in 512acf3

self.netD = GlobalLocalDiscriminator(3, cnum=opt.d_cnum, act=act,

, the code hard-code the final-size of the feature to the Discriminator, 16, which is suitable for input size of 256. But if you are using input size of 512, it should be 32*32*some_other_value.

trained model conversion to coreml

Hi, is it possible to convert your trained pytorch or tensorflow model into coreml without implementing custom layer?

when i run test.py, a problem occured,and i didn't change anything

Traceback (most recent call last):
File "/media/inpainting_gmcnn-master/pytorch/test.py", line 32, in
ourModel.load_networks(getLatest(os.path.join(config.load_model_dir, '*.pth')))
File "/media/inpainting_gmcnn-master/pytorch/util/utils.py", line 84, in getLatest
return files[sorted(range(len(file_times)), key=lambda x: file_times[x])[-1]]
IndexError: list index out of range

The error of training

I used my dataset "celeba" which contains 1000 images to pretrain the GMCNN in Pycharm, Windows. I set the "--data_file" as "./imgs/celeba",but the error happened:

How should i solve this problem? Thanks very much!

ValueError: attempt to get argmax of an empty sequence

$ python3 test.py --dataset paris_streetview --data_file ./imgs/paris-streetview_256x256/ --load_model_dir ./checkpoints/paris-streetview_256x256_rect --random_mask 0
/home/slothdemon/.local/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/slothdemon/.local/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/home/slothdemon/.local/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint16 = np.dtype([("qint16", np.int16, 1)])
/home/slothdemon/.local/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/home/slothdemon/.local/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint32 = np.dtype([("qint32", np.int32, 1)])
/home/slothdemon/.local/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
np_resource = np.dtype([("resource", np.ubyte, 1)])
/home/slothdemon/.local/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/slothdemon/.local/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/home/slothdemon/.local/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint16 = np.dtype([("qint16", np.int16, 1)])
/home/slothdemon/.local/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/home/slothdemon/.local/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint32 = np.dtype([("qint32", np.int32, 1)])
/home/slothdemon/.local/lib/python3.7/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
np_resource = np.dtype([("resource", np.ubyte, 1)])
Traceback (most recent call last):
File "test.py", line 12, in
"nvidia-smi -q -d Memory | grep -A4 GPU | grep Free", shell=True, stdout=subprocess.PIPE).stdout.readlines()]
File "<array_function internals>", line 6, in argmax
File "/home/slothdemon/.local/lib/python3.7/site-packages/numpy/core/fromnumeric.py", line 1153, in argmax
return _wrapfunc(a, 'argmax', axis=axis, out=out)
File "/home/slothdemon/.local/lib/python3.7/site-packages/numpy/core/fromnumeric.py", line 58, in _wrapfunc
return _wrapit(obj, method, *args, **kwds)
File "/home/slothdemon/.local/lib/python3.7/site-packages/numpy/core/fromnumeric.py", line 47, in _wrapit
result = getattr(asarray(obj), method)(*args, **kwds)
ValueError: attempt to get argmax of an empty sequence

Bad results on 512*680 image

Hi , i was testing a random image on 512680 pretrained model but the results seems to be not reflecting what they should be. Do you have any suggestions on how to improve it? Will they improve if i retrain your 512x680 model using 512x512 images? My assumption is that 256x256 images were used in training 512x680 model(might be wrong, kindly correct) , because when i set input tensor to size 256256 given the same image (after resizing of-course) for inpainting then results are good.

Image chosen from internet randomly

Inpainting results

inpainting result when input tensor is set to 256*256, on the same resized image

Question about training time

@shepnerd ,hello. Thanks for your great work and also looking forward to paper ' Semantic Regeneration Network' .
Since gmcnn has a huge amount of parameters, i wonder how much time you spend on training the model on different dataset?Thx!

finetune problem

I pretrained the model with my own data about 10 epochs. and the result does not converge. then i want to try finetune step.but failed. any one can help me? thanks. @shepnerd (my image size is 512X512, about 1000 pics in trainingset)
RuntimeError: size mismatch, m1: [4 x 4096], m2: [16384 x 1] at C:/w/1/s/tmp_conda_3.7_055457/conda/conda-bld/pytorch_1565416617654/work/aten/src\THC/generic/THCTens
orMathBlas.cu:273

Question on the order of parameters

@shepnerd Thanks for your well written code. my question is about the order of parameters of the following function:

inpainting_gmcnn/tensorflow/net/ops.py

Line 472 in fe5295e

def id_mrf_reg_feat(feat_A, feat_B, config):

From the place this function is invoked:

inpainting_gmcnn/tensorflow/net/ops.py

Line 507 in fe5295e

mrf_style_loss = [w * id_mrf_reg_feat(src_vgg[layer], dst_vgg[layer], mrf_config)

and

inpainting_gmcnn/tensorflow/net/ops.py

Line 511 in fe5295e

mrf_content_loss = [w * id_mrf_reg_feat(src_vgg[layer], dst_vgg[layer], mrf_config)

I can figure out:

1. feat_A is the feature computed from predicted data
2. feat_B is the feature computed from ground truth data

Then this function invokes mrf_loss:

mrf_loss(feat_A, feat_B, distance=config.Dist, nnsigma=config.nn_stretch_sigma)

In the definition of mrf_loss function:

inpainting_gmcnn/tensorflow/net/ops.py

Line 409 in fe5295e

def mrf_loss(T_features, I_features, distance=Distance.DotProduct, nnsigma=float(1.0)):

I can figure out:

1. T_features should be feature computed from ground truth data
2. I_features should be feature computed from predicted data

So the order of parameters is not right? May be I misunderstand something. Could you help me check this?

Some questions about testing on paris_street

Thanks for your excellent work！I have some doubts about testing on paris_street. When I test the image with your realesed trained weight, I found the result is different from that in your published paper.

I used the following command for testing:
python test.py --dataset paris_streetview
--data_file ./imgs/paris-streetview_256x256/
--load_model_dir ./checkpoints/paris-streetview_256x256_rect/ --random_mask 0

Is there anything I am missing?

Paris streetview dataset

I want to do a repair training, I want to use Paris streetview, but it is not public, can you send the link of this dataset to me? Thank you very much, my email address : [email protected]

training pytorch model on celebAHQ

When I training the pytorch model on celebAHQ from the pretrain model you give, I watch the tensorboard result seems to be blurred like this.

I train the model on 1080ti with batch size 4. could you give me some sugestion to get more real results?

Test failure with gui

ValueError: Cannot feed value of shape (1, 512, 512, 3) for Tensor 'Placeholder:0', which has shape '(1, 512, 680, 3)'

I used a 512*512 color picture provided by your project.

error of train.py

i just use 20 images for experiment .and train.txt includes 20 images path like /dataset/celeba_256X256/000001.jpg .....dataset/celeba_256X256/0000020.jpg

python train.py --dataset [./dataset/celeba_256X256] --data_file [./dataset/train.txt] --gpu_ids [0] --pretrain_network 1 --batch_size 8
and the problem Traceback (most recent call last):
File "train.py", line 11, in
config = TrainOptions().parse()
File "/media//inpainting_gmcnn-master/pytorch/options/train_options.py", line 74, in parse
id = int(str_id)
ValueError: invalid literal for int() with base 10: '[0]'
by the way ,i just have one gpu 1060

question about Figure 6 in the paper

I read the paper and have a question about the results in Figure 6.

It seems that the results w/o ID-MRF are better than ID-MRF.

Want to train on a 512-size image, which parameters need to be modified?

@shepnerd Thanks for your great work. However, when I want to train on 512 size celebahq images, the results are bad. Which parameters need to be modified? I just modified the following parameters:
img_shapes=256
mask_shapes=128
g_cnum=64

And the results are bad.

could you provide paris streetview dataset for me

sorry for open this issue, I am a graduate student in image inpainting.
I have searched for paris streetview dataset in many ways, even email to pathak.
I will appreciate it if you can send a link for dataset to [email protected]
I think your work is meaningful, and sincerely thanks.

not training

When I tried to train it looks like it freezes, i do not see any progress after hours. I am using a 1080ti and i am training with the following command:

python train.py --dataset coco_2014 --data_file /home/jkim/Documents/datasets/coco_2014/ --gpu_ids 0

Explain discriminator loss?

Hi @shepnerd, thanks for the interesting work!

I have a hard time understanding the logic of discriminator. Could you please explain the motivation for the value range of discriminator outputs and respective loss?

Here is my understanding:
The step of evaluation of 'realism' for inpainted and ground truth images is (here, excluding the local terms for brevity):

self.completed_logit, _ = self.netD(self.completed.detach(), self.completed_local.detach())
self.gt_logit, _ = self.netD(self.gt, self.gt_local)

where self.completed_logit and self.gt_logit are outputs from the last linear layer of the discriminator and these contain a single value for each image in the batch.

So the discriminator does mapping [b, c, h, w] --> [b, 1]. This part is clear, except for the range of output values which might be arbitrary due to the linear activation in the last layer of discriminator. So it might be -3, 5, 0.3, etc.

The loss term is defined as follows:
self.D_loss = nn.ReLU()(1.0 - self.gt_logit).mean() + nn.ReLU()(1.0 + self.completed_logit).mean()

As I understand, a perfect discriminator should assign anything above 1 to real image, self.gt_logit (identifies the ground truth as real) and anything less than -1 to fake image, self.completed_logit (identifies the inpainted image as fake).

Is that correct? If so, could you explain what is the practical difference in using these compared to regular 0 and 1 adversarial ground truths like here?

Thanks!

How to continue training?

So how do I carry on a previously stopped training? It seems that if I try to load a previous model it starts with a new folder.

Question about the way of calculating PSNR and SSIM

Hello~
I used the metrics.py of paper"Edge-connect:……" to calculate PSNR and SSIM on validation set of celeba-HQ-256, but the result is different from yours especially on SSIM, so could u share the file used for calculating PSNR and SSIM?

And i want to make clear somethings which need your help:
When i make celeba-HQ-256 dataset from celeba, there is a .zip about delta ,when i used it,there will be noise in the photos,like colorful points in the black area, if not, the photos are clean, how about the celeba-HQ-256 you used? And what's the format（.jpg or .png)? It would be helpful if you share the celeba-HQ-256 dataset.
THX !~

When training on 512-size celeba-hq images

@shepnerd Thank you for your reply. I am very sorry, I made some mistakes in the previous issue. When training on 512-size celeba-hq images. I modified the following parameters in train_options.py：

self.parser.add_argument('--img_shapes', type=str, default='512,512,3',
help='given shape parameters: h,w,c or h,w')
self.parser.add_argument('--mask_shapes', type=str, default='256,256',
help='given mask parameters: h,w')
self.parser.add_argument('--g_cnum', type=int, default=64,
help='# of generator filters in first conv layer')

But it still not work. Are there any other parameters that need to be modified?

i'm sorry,pytorch unable to realize GUI right ?

train the celebaHQ with tensorflow version

thanks for your excellent work! when i train the tensorflow version on celebAHQ. I find the train data is crop from the original images not resize.

could i need to change some parameters to make the training data resize from the original one?

Where does the script saves the model?

Hello!
I'm trying to finetune the model for my data using the model pre-trained on Places2.
For training, I have specified the parameter --load_model_dir to the directory where I have saved the Places2 weights that you shared (Thanks a lot!) and other parameters that were required in the README.
After overnight training, I found that the only output was the folder checkpoints with events.out.tfevents.1576231107.PM1 file.
The files in the Places2 directory were not changed.
What I am doing wrong?

Training with custom images

Hi,

I tested the pre trained model, and results were amazing,

Can please please tell us how to re-train the model with custom image data set.?
What should be resolution of images.?
And is there any pre processing needed on images before keeping them in training

Thanks @shepnerd

Problem in running gui

where's code?

rt

Input to generator

Hi @shepnerd ,

I've analysed your code and I have a question regarding the input to the generator.
While making the input you concatenate the masked image with mask and matrix of ones. I understand pros of using mask as additional input but what is the intuition behind the usage of one more channel filled with 1?

pretrained model for random mask on celeba-HQ

Hi, can you share the model trained on random mask of celeba? I test the released model (celebahq256_rect) on random mask, the result is really bad. Just shown below.

the code for the quantitative analysis

Hello, is there a tensorflow version of the code for the quantitative analysis part of the paper?

RuntimeError: An attempt has been made to start a new process before the current process has finished

Hi, I selected 10000 images from celebA and try to train. I prepared dir and files, and ran the following:
python train.py --dataset CelebA --data_file ./training_data_list/data_list.txt --gpu_ids 0 --pretrain_network 1
--batch_size 8
but something went wrong. It seems the dada was loaded twice and the code was trying to start a second training initiallization...... any one can help me? @shepnerd
------------ Options -------------
D_max_iters: 5
batch_size: 1
checkpoint_dir: ./checkpoints
d_cnum: 64
data_file: ./training_data_list/data_list.txt
dataset: CelebA
dataset_path: ./training_data_list/data_list.txt
date_str: 20191006-131434
epochs: 40
g_cnum: 32
gpu_ids: ['0']
img_shapes: [256, 256, 3]
lambda_adv: 0.001
lambda_ae: 1.2
lambda_gp: 10
lambda_mrf: 0.05
lambda_rec: 1.4
load_model_dir:
lr: 1e-05
margins: [0, 0]
mask_shapes: [128, 128]
mask_type: rect
max_delta_shapes: [32, 32]
model_folder: ./checkpoints\20191006-131434_GMCNN_CelebA_b1_s256x256_gc32_dc64_randmask-rect_pretrain
model_name: GMCNN
padding: SAME
phase: train
pretrain_network: True
random_crop: True
random_mask: True
random_seed: False
spectral_norm: True
train_spe: 1000
vgg19_path: vgg19_weights/imagenet-vgg-verydeep-19.mat
viz_steps: 5
-------------- End ----------------
loading data..
data loaded..
configuring model..
initialize network with normal
---------- Networks initialized -------------
GMCNN(
(EB1): ModuleList(
(0): Conv2d(4, 32, kernel_size=(7, 7), stride=(1, 1))
(1): Conv2d(32, 64, kernel_size=(7, 7), stride=(2, 2))
(2): Conv2d(64, 64, kernel_size=(7, 7), stride=(1, 1))
(3): Conv2d(64, 128, kernel_size=(7, 7), stride=(2, 2))
(4): Conv2d(128, 128, kernel_size=(7, 7), stride=(1, 1))
(5): Conv2d(128, 128, kernel_size=(7, 7), stride=(1, 1))
(6): Conv2d(128, 128, kernel_size=(7, 7), stride=(1, 1), dilation=(2, 2))
(7): Conv2d(128, 128, kernel_size=(7, 7), stride=(1, 1), dilation=(4, 4))
(8): Conv2d(128, 128, kernel_size=(7, 7), stride=(1, 1), dilation=(8, 8))
(9): Conv2d(128, 128, kernel_size=(7, 7), stride=(1, 1), dilation=(16, 16))
(10): Conv2d(128, 128, kernel_size=(7, 7), stride=(1, 1))
(11): Conv2d(128, 128, kernel_size=(7, 7), stride=(1, 1))
(12): PureUpsampling()
)
(EB2): ModuleList(
(0): Conv2d(4, 32, kernel_size=(5, 5), stride=(1, 1))
(1): Conv2d(32, 64, kernel_size=(5, 5), stride=(2, 2))
(2): Conv2d(64, 64, kernel_size=(5, 5), stride=(1, 1))
(3): Conv2d(64, 128, kernel_size=(5, 5), stride=(2, 2))
(4): Conv2d(128, 128, kernel_size=(5, 5), stride=(1, 1))
(5): Conv2d(128, 128, kernel_size=(5, 5), stride=(1, 1))
(6): Conv2d(128, 128, kernel_size=(5, 5), stride=(1, 1), dilation=(2, 2))
(7): Conv2d(128, 128, kernel_size=(5, 5), stride=(1, 1), dilation=(4, 4))
(8): Conv2d(128, 128, kernel_size=(5, 5), stride=(1, 1), dilation=(8, 8))
(9): Conv2d(128, 128, kernel_size=(5, 5), stride=(1, 1), dilation=(16, 16))
(10): Conv2d(128, 128, kernel_size=(5, 5), stride=(1, 1))
(11): Conv2d(128, 128, kernel_size=(5, 5), stride=(1, 1))
(12): PureUpsampling()
(13): Conv2d(128, 64, kernel_size=(5, 5), stride=(1, 1))
(14): Conv2d(64, 64, kernel_size=(5, 5), stride=(1, 1))
(15): PureUpsampling()
)
(EB3): ModuleList(
(0): Conv2d(4, 32, kernel_size=(3, 3), stride=(1, 1))
(1): Conv2d(32, 64, kernel_size=(3, 3), stride=(2, 2))
(2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1))
(3): Conv2d(64, 128, kernel_size=(3, 3), stride=(2, 2))
(4): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1))
(5): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1))
(6): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), dilation=(2, 2))
(7): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), dilation=(4, 4))
(8): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), dilation=(8, 8))
(9): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), dilation=(16, 16))
(10): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1))
(11): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1))
(12): PureUpsampling()
(13): Conv2d(128, 64, kernel_size=(3, 3), stride=(1, 1))
(14): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1))
(15): PureUpsampling()
(16): Conv2d(64, 32, kernel_size=(3, 3), stride=(1, 1))
(17): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1))
)
(decoding_layers): ModuleList(
(0): Conv2d(224, 16, kernel_size=(3, 3), stride=(1, 1))
(1): Conv2d(16, 3, kernel_size=(3, 3), stride=(1, 1))
)
(pads): ModuleList(
(0): ReflectionPad2d((0, 0, 0, 0))
(1): ReflectionPad2d((1, 1, 1, 1))
(2): ReflectionPad2d((2, 2, 2, 2))
(3): ReflectionPad2d((3, 3, 3, 3))
(4): ReflectionPad2d((4, 4, 4, 4))
(5): ReflectionPad2d((5, 5, 5, 5))
(6): ReflectionPad2d((6, 6, 6, 6))
(7): ReflectionPad2d((7, 7, 7, 7))
(8): ReflectionPad2d((8, 8, 8, 8))
(9): ReflectionPad2d((9, 9, 9, 9))
(10): ReflectionPad2d((10, 10, 10, 10))
(11): ReflectionPad2d((11, 11, 11, 11))
(12): ReflectionPad2d((12, 12, 12, 12))
(13): ReflectionPad2d((13, 13, 13, 13))
(14): ReflectionPad2d((14, 14, 14, 14))
(15): ReflectionPad2d((15, 15, 15, 15))
(16): ReflectionPad2d((16, 16, 16, 16))
(17): ReflectionPad2d((17, 17, 17, 17))
(18): ReflectionPad2d((18, 18, 18, 18))
(19): ReflectionPad2d((19, 19, 19, 19))
(20): ReflectionPad2d((20, 20, 20, 20))
(21): ReflectionPad2d((21, 21, 21, 21))
(22): ReflectionPad2d((22, 22, 22, 22))
(23): ReflectionPad2d((23, 23, 23, 23))
(24): ReflectionPad2d((24, 24, 24, 24))
(25): ReflectionPad2d((25, 25, 25, 25))
(26): ReflectionPad2d((26, 26, 26, 26))
(27): ReflectionPad2d((27, 27, 27, 27))
(28): ReflectionPad2d((28, 28, 28, 28))
(29): ReflectionPad2d((29, 29, 29, 29))
(30): ReflectionPad2d((30, 30, 30, 30))
(31): ReflectionPad2d((31, 31, 31, 31))
(32): ReflectionPad2d((32, 32, 32, 32))
(33): ReflectionPad2d((33, 33, 33, 33))
(34): ReflectionPad2d((34, 34, 34, 34))
(35): ReflectionPad2d((35, 35, 35, 35))
(36): ReflectionPad2d((36, 36, 36, 36))
(37): ReflectionPad2d((37, 37, 37, 37))
(38): ReflectionPad2d((38, 38, 38, 38))
(39): ReflectionPad2d((39, 39, 39, 39))
(40): ReflectionPad2d((40, 40, 40, 40))
(41): ReflectionPad2d((41, 41, 41, 41))
(42): ReflectionPad2d((42, 42, 42, 42))
(43): ReflectionPad2d((43, 43, 43, 43))
(44): ReflectionPad2d((44, 44, 44, 44))
(45): ReflectionPad2d((45, 45, 45, 45))
(46): ReflectionPad2d((46, 46, 46, 46))
(47): ReflectionPad2d((47, 47, 47, 47))
(48): ReflectionPad2d((48, 48, 48, 48))
)
)
[Network GM] Total number of parameters : 12.562 M

model setting up..
training initializing..
------------ Options -------------
D_max_iters: 5
batch_size: 1
checkpoint_dir: ./checkpoints
d_cnum: 64
data_file: ./training_data_list/data_list.txt
dataset: CelebA
dataset_path: ./training_data_list/data_list.txt
date_str: 20191006-131438
epochs: 40
g_cnum: 32
gpu_ids: ['0']
img_shapes: [256, 256, 3]
lambda_adv: 0.001
lambda_ae: 1.2
lambda_gp: 10
lambda_mrf: 0.05
lambda_rec: 1.4
load_model_dir:
lr: 1e-05
margins: [0, 0]
mask_shapes: [128, 128]
mask_type: rect
max_delta_shapes: [32, 32]
model_folder: ./checkpoints\20191006-131438_GMCNN_CelebA_b1_s256x256_gc32_dc64_randmask-rect_pretrain
model_name: GMCNN
padding: SAME
phase: train
pretrain_network: True
random_crop: True
random_mask: True
random_seed: False
spectral_norm: True
train_spe: 1000
vgg19_path: vgg19_weights/imagenet-vgg-verydeep-19.mat
viz_steps: 5
-------------- End ----------------
loading data..
data loaded..
configuring model..
initialize network with normal
---------- Networks initialized -------------
GMCNN(
(EB1): ModuleList(
(0): Conv2d(4, 32, kernel_size=(7, 7), stride=(1, 1))
(1): Conv2d(32, 64, kernel_size=(7, 7), stride=(2, 2))
(2): Conv2d(64, 64, kernel_size=(7, 7), stride=(1, 1))
(3): Conv2d(64, 128, kernel_size=(7, 7), stride=(2, 2))
(4): Conv2d(128, 128, kernel_size=(7, 7), stride=(1, 1))
(5): Conv2d(128, 128, kernel_size=(7, 7), stride=(1, 1))
(6): Conv2d(128, 128, kernel_size=(7, 7), stride=(1, 1), dilation=(2, 2))
(7): Conv2d(128, 128, kernel_size=(7, 7), stride=(1, 1), dilation=(4, 4))
(8): Conv2d(128, 128, kernel_size=(7, 7), stride=(1, 1), dilation=(8, 8))
(9): Conv2d(128, 128, kernel_size=(7, 7), stride=(1, 1), dilation=(16, 16))
(10): Conv2d(128, 128, kernel_size=(7, 7), stride=(1, 1))
(11): Conv2d(128, 128, kernel_size=(7, 7), stride=(1, 1))
(12): PureUpsampling()
)
(EB2): ModuleList(
(0): Conv2d(4, 32, kernel_size=(5, 5), stride=(1, 1))
(1): Conv2d(32, 64, kernel_size=(5, 5), stride=(2, 2))
(2): Conv2d(64, 64, kernel_size=(5, 5), stride=(1, 1))
(3): Conv2d(64, 128, kernel_size=(5, 5), stride=(2, 2))
(4): Conv2d(128, 128, kernel_size=(5, 5), stride=(1, 1))
(5): Conv2d(128, 128, kernel_size=(5, 5), stride=(1, 1))
(6): Conv2d(128, 128, kernel_size=(5, 5), stride=(1, 1), dilation=(2, 2))
(7): Conv2d(128, 128, kernel_size=(5, 5), stride=(1, 1), dilation=(4, 4))
(8): Conv2d(128, 128, kernel_size=(5, 5), stride=(1, 1), dilation=(8, 8))
(9): Conv2d(128, 128, kernel_size=(5, 5), stride=(1, 1), dilation=(16, 16))
(10): Conv2d(128, 128, kernel_size=(5, 5), stride=(1, 1))
(11): Conv2d(128, 128, kernel_size=(5, 5), stride=(1, 1))
(12): PureUpsampling()
(13): Conv2d(128, 64, kernel_size=(5, 5), stride=(1, 1))
(14): Conv2d(64, 64, kernel_size=(5, 5), stride=(1, 1))
(15): PureUpsampling()
)
(EB3): ModuleList(
(0): Conv2d(4, 32, kernel_size=(3, 3), stride=(1, 1))
(1): Conv2d(32, 64, kernel_size=(3, 3), stride=(2, 2))
(2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1))
(3): Conv2d(64, 128, kernel_size=(3, 3), stride=(2, 2))
(4): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1))
(5): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1))
(6): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), dilation=(2, 2))
(7): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), dilation=(4, 4))
(8): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), dilation=(8, 8))
(9): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), dilation=(16, 16))
(10): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1))
(11): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1))
(12): PureUpsampling()
(13): Conv2d(128, 64, kernel_size=(3, 3), stride=(1, 1))
(14): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1))
(15): PureUpsampling()
(16): Conv2d(64, 32, kernel_size=(3, 3), stride=(1, 1))
(17): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1))
)
(decoding_layers): ModuleList(
(0): Conv2d(224, 16, kernel_size=(3, 3), stride=(1, 1))
(1): Conv2d(16, 3, kernel_size=(3, 3), stride=(1, 1))
)
(pads): ModuleList(
(0): ReflectionPad2d((0, 0, 0, 0))
(1): ReflectionPad2d((1, 1, 1, 1))
(2): ReflectionPad2d((2, 2, 2, 2))
(3): ReflectionPad2d((3, 3, 3, 3))
(4): ReflectionPad2d((4, 4, 4, 4))
(5): ReflectionPad2d((5, 5, 5, 5))
(6): ReflectionPad2d((6, 6, 6, 6))
(7): ReflectionPad2d((7, 7, 7, 7))
(8): ReflectionPad2d((8, 8, 8, 8))
(9): ReflectionPad2d((9, 9, 9, 9))
(10): ReflectionPad2d((10, 10, 10, 10))
(11): ReflectionPad2d((11, 11, 11, 11))
(12): ReflectionPad2d((12, 12, 12, 12))
(13): ReflectionPad2d((13, 13, 13, 13))
(14): ReflectionPad2d((14, 14, 14, 14))
(15): ReflectionPad2d((15, 15, 15, 15))
(16): ReflectionPad2d((16, 16, 16, 16))
(17): ReflectionPad2d((17, 17, 17, 17))
(18): ReflectionPad2d((18, 18, 18, 18))
(19): ReflectionPad2d((19, 19, 19, 19))
(20): ReflectionPad2d((20, 20, 20, 20))
(21): ReflectionPad2d((21, 21, 21, 21))
(22): ReflectionPad2d((22, 22, 22, 22))
(23): ReflectionPad2d((23, 23, 23, 23))
(24): ReflectionPad2d((24, 24, 24, 24))
(25): ReflectionPad2d((25, 25, 25, 25))
(26): ReflectionPad2d((26, 26, 26, 26))
(27): ReflectionPad2d((27, 27, 27, 27))
(28): ReflectionPad2d((28, 28, 28, 28))
(29): ReflectionPad2d((29, 29, 29, 29))
(30): ReflectionPad2d((30, 30, 30, 30))
(31): ReflectionPad2d((31, 31, 31, 31))
(32): ReflectionPad2d((32, 32, 32, 32))
(33): ReflectionPad2d((33, 33, 33, 33))
(34): ReflectionPad2d((34, 34, 34, 34))
(35): ReflectionPad2d((35, 35, 35, 35))
(36): ReflectionPad2d((36, 36, 36, 36))
(37): ReflectionPad2d((37, 37, 37, 37))
(38): ReflectionPad2d((38, 38, 38, 38))
(39): ReflectionPad2d((39, 39, 39, 39))
(40): ReflectionPad2d((40, 40, 40, 40))
(41): ReflectionPad2d((41, 41, 41, 41))
(42): ReflectionPad2d((42, 42, 42, 42))
(43): ReflectionPad2d((43, 43, 43, 43))
(44): ReflectionPad2d((44, 44, 44, 44))
(45): ReflectionPad2d((45, 45, 45, 45))
(46): ReflectionPad2d((46, 46, 46, 46))
(47): ReflectionPad2d((47, 47, 47, 47))
(48): ReflectionPad2d((48, 48, 48, 48))
)
)
[Network GM] Total number of parameters : 12.562 M

model setting up..
training initializing..
Traceback (most recent call last):
Traceback (most recent call last):
File "", line 1, in
File "train.py", line 34, in
File "D:\Anaconda3\lib\multiprocessing\spawn.py", line 105, in spawn_main
for i, data in enumerate(dataloader):
exitcode = _main(fd) File "D:\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py", line 278, in iter

File "D:\Anaconda3\lib\multiprocessing\spawn.py", line 114, in _main
return _MultiProcessingDataLoaderIter(self)
File "D:\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py", line 682, in init
prepare(preparation_data)
File "D:\Anaconda3\lib\multiprocessing\spawn.py", line 225, in prepare
w.start()_fixup_main_from_path(data['init_main_from_path'])

File "D:\Anaconda3\lib\multiprocessing\process.py", line 112, in start
File "D:\Anaconda3\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
self._popen = self._Popen(self)
run_name="mp_main") File "D:\Anaconda3\lib\multiprocessing\context.py", line 223, in _Popen

  File "D:\Anaconda3\lib\runpy.py", line 263, in run_path

return _default_context.get_context().Process._Popen(process_obj)
File "D:\Anaconda3\lib\multiprocessing\context.py", line 322, in _Popen
pkg_name=pkg_name, script_name=fname)
File "D:\Anaconda3\lib\runpy.py", line 96, in _run_module_code
return Popen(process_obj)
mod_name, mod_spec, pkg_name, script_name) File "D:\Anaconda3\lib\multiprocessing\popen_spawn_win32.py", line 89, in init

  File "D:\Anaconda3\lib\runpy.py", line 85, in _run_code

reduction.dump(process_obj, to_child)
exec(code, run_globals) File "D:\Anaconda3\lib\multiprocessing\reduction.py", line 60, in dump

  File "D:\PycharmProjects2\inpainting_gmcnn\pytorch\train.py", line 34, in <module>

ForkingPickler(file, protocol).dump(obj)
for i, data in enumerate(dataloader):BrokenPipeError
: File "D:\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py", line 278, in iter
[Errno 32] Broken pipe
return _MultiProcessingDataLoaderIter(self)
File "D:\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py", line 682, in init
w.start()
File "D:\Anaconda3\lib\multiprocessing\process.py", line 112, in start
self._popen = self._Popen(self)
File "D:\Anaconda3\lib\multiprocessing\context.py", line 223, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "D:\Anaconda3\lib\multiprocessing\context.py", line 322, in _Popen
return Popen(process_obj)
File "D:\Anaconda3\lib\multiprocessing\popen_spawn_win32.py", line 46, in init
prep_data = spawn.get_preparation_data(process_obj._name)
File "D:\Anaconda3\lib\multiprocessing\spawn.py", line 143, in get_preparation_data
_check_not_importing_main()
File "D:\Anaconda3\lib\multiprocessing\spawn.py", line 136, in _check_not_importing_main
is not going to be frozen to produce an executable.''')
RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.

    This probably means that you are not using fork to start your
    child processes and you have forgotten to use the proper idiom
    in the main module:

        if __name__ == '__main__':
            freeze_support()
            ...

    The "freeze_support()" line can be omitted if the program
    is not going to be frozen to produce an executable.

my own masl

Hey @shepnerd
How can I test your code on my own mask ?

Bad results for Places2 with rect mask

Hello, I find that the pre-trained model given "places2_512x680_freeform" works well with stroke mask, but the results are poor with rect mask. Can you provide the pre-trained model with rect mask? Thanks.

when will pre train converge

Hi, I used the pretrained imagenet weights with rect inpainting style to train my dataset (about 4000 images) . here are some hyperparameters: --batch_size 16 --mask_type rect --lr 1e-4 --train_spe 4000 --max_iters 72000
it seems the train is hard to converge. so what should i do next? continue to pretrain? how many iters should i train? thanks. see pics below:

I wonder how to implement spatial Variant reconstruction loss on irregular masks

Its easy to implement spatial Variant reconstruction loss on the rectangle masks.
Did you implement spatial Variant reconstruction loss on irregular masks?

The mask used for testing

Hello, my question is what does the mask used for testing on places2 , celebaHQ-256 looks like when you do quantitative experiment?