Tensorflow implementation of the paper Image Super-Resolution by Neural Texture Transfer accepted in CVPR 2019. This is a simplified version, where the reference images are used without augmentation, e.g., rotation and scaling.
- Pre-requisites
- Dataset
- Easy testing
- Custom testing
- Easy training
- Custom training
- Acknowledgement
- Contact
- Python 3.6
- TensorFlow 1.13.1
- requests 2.21.0
- pillow 5.4.1
- matplotlib 3.0.2
Tested on MacOS (Mojave).
This repo only provides a small training set of ten input-reference pairs for demo purpose.
The input images and reference images are stored in data/train/CUFED/input and data/train/CUFED/ref, respectively.
Corresponding input and refernece images are with the same file name.
To speed up the training process, patch matching and swapping are performed offline,
and the swapped feature maps will be saved to data/train/CUFED/map_321 (see offline_patchMatch_textureSwap.py for more details).
If you want to train your own model, please prepare your own training set or download either of the following demo training sets:
11,485 input-reference pairs (size 320x320) extracted from DIV2K.
Each pair is extracted from the same image without overlap but considering scaling and rotation.
$ python download_dataset.py --dataset_name DIV2K11,871 input-reference pairs (size 160x160) extracted from CUFED.
Each pair is extracted from the similar images, including five degrees of similarity.
$ python download_dataset.py --dataset_name CUFEDThis repo includes one grounp of samples from the CUFED5 dataset, where each input image corresponds to five reference images (different from the paper) with different degrees of similarity to the input image. Please download the full dataset by
$ python download_dataset.py --dataset_name CUFED5$ sh test.shThe results will be save to the folder demo_testing_srntt, including the following 6 images:
-
[1/6]
HR.png, the original image.
-
[2/6]
LR.png, the low-resolution (LR) image, downscaling factor 4x.
-
[3/6]
Bicubic.png, the upscaled image by bicubic interpolation, upscaling factor 4x.
-
[4/6]
Ref_XX.png, the reference images, indexed by XX.
-
[5/6]
Upscale.png, the upscaled image by a pre-trained SR network, upscaling factor 4x.
-
[6/6]
SRNTT.png, the SR result by SRNTT, upscaling factor 4x.
$ python main.py
--is_train False
--input_dir path/to/input/image/file
--ref_dir path/to/ref/image/file
--result_dir path/to/result/folder
--ref_scale default 1, expected_ref_scale divided by original_ref_scale
--is_original_image default True, whether input is original
--use_init_model_only default False, whether use init model, trained with reconstruction loss only
--use_weight_map defualt False, whether use weighted model, trained with the weight map.
--save_dir path/to/a/specified/model if it exists, otherwise ignor this parameterPlease note that this repo provides two types of pre-trained SRNTT models in SRNTT/models/SRNTT:
srntt.npzis trained by all losses, i.e., reconstruction loss, perceptual loss, texture loss, and adversarial loss.srntt_init.npzis trained by only the reconstruction loss, corresponding to SRNTT-l2 in the paper.
To switch between the demo models, please set --use_init_model_only to decide whether use srntt_init.npz.
$ sh train.shThe CUFED training set will be downloaded automatically.
To speed up the training process, patch matching and swapping are conducted to get the swapped feature maps in an offline manner.
The models will be saved to demo_training_srntt/model, and intermediate samples will be saved to demo_training_srntt/sample.
Parameter settings are save to demo_training_srntt/arguments.txt.
Please first prepare the input and reference images which are squared patches in the same size.
In addition, input and reference images should be stored in separated folders,
and the correspoinding input and reference images are with the same file name. Please refer to the data/train/CUFED folder for examples.
Then, use offline_patchMatch_textureSwap.py to generate the feature maps in ahead.
$ python main.py
--is_train True
--save_dir folder/to/save/models
--input_dir path/to/input/image/folder
--ref_dir path/to/ref/image/folder
--map_dir path/to/feature_map/folder
--batch_size default 9
--num_epochs default 100
--input_size default 40, the size of LR patch, i.e., 1/4 of the HR image, set to 80 for the DIV2K dataset
--use_weight_map defualt False, whether use the weight map that reduces negative effect
from the reference image but may also decrease the sharpness. Please refer to main.py for more parameter settings for training.
Test on the custom training model
$ python main.py
--is_train False
--input_dir path/to/input/image/file
--ref_dir path/to/ref/image/file
--result_dir path/to/result/folder
--ref_scale default 1, expected_ref_scale divided by original_ref_scale
--is_original_image default True, whether input is original
--save_dir the same as save_dir in trainingThanks to Tensorlayer for
facilitating the implementation of this demo code.
We have include the Tensorlayer 1.5.0 in SRNTT/tensorlayer.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent