ishanashastri / vsrnet_pytorch

PyTorch implementation of paper "Video super-resolution with convolutional neural networks"

VSRNet

A PyTorch implementation of VSRNet [Video super-resolution with convolutional neural networks] (https://ieeexplore.ieee.org/abstract/document/7444187/)

Requirements

• Anaconda
• PyTorch

conda install pytorch torchvision -c soumith
conda install pytorch torchvision cuda80 -c soumith # install it if you have installed cuda 8.0

• PyTorchNet

pip install git+https://github.com/pytorch/tnt.git@master

• tqdm

pip install tqdm

• opencv

conda install -c conda-forge opencv

• tensorboard_logger

pip install tensorboard_logger

• h5py

conda install h5py

• pyflow

Datasets

Train, Val, Test Video Dataset

The train and val datasets are sampled from CDVL dataset. We choose this dataset because we want to extend single frame based SRCNN to multi frame based VSRNet.

Train dataset (uf_X, X=2,3,4) is composed of multiple h5 files:

• Data_CDVL_LR_uf_4_ps_72_fn_5_tpn_225000.h5: patches sampled from LR frames, 225243x5x(72/X)x(72/X)
• Data_CDVL_HR_uf_4_ps_72_fn_5_tpn_225000.h5: patches sampled from HR frames, 225243x5x72x72
• Data_CDVL_LR_Bic_uf_4_ps_72_fn_5_tpn_225000.h5: patches sampled from Matlab bicubic interpolation upscaled frames, 225243x5x72x72
• Data_CDVL_LR_Bic_MC_uf_4_ps_72_fn_5_tpn_225000.h5: patches sampled from Matlab bicubic interpolation upscaled and optical flow motion compensated frames, 225243x5x72x72

Val dataset (uf_X, X=2,3,4) is composed of multiple h5 files:

• Data_CDVL_LR_uf_4_ps_72_fn_5_tpn_45000.h5: patches sampled from LR frames, 45159x5x(72/X)x(72/X)
• Data_CDVL_HR_uf_4_ps_72_fn_5_tpn_45000.h5: patches sampled from HR frames, 45159x5x72x72
• Data_CDVL_LR_Bic_uf_4_ps_72_fn_5_tpn_45000.h5: patches sampled from Matlab bicubic interpolation upscaled frames, 45159x5x72x72
• Data_CDVL_LR_Bic_MC_uf_4_ps_72_fn_5_tpn_45000.h5: patches sampled from Matlab bicubic interpolation upscaled and optical flow motion compensated frames, 45159x5x72x72

Test dataset (uf_X, X=2,3,4) is composed of multiple h5 folders:

• LR: LR frames of a scene, (1920/X)x(1080/X)x5xframe_number
• HR: HR frames of a scene, 1920x1080x1xframe_number
• LR_Bic: Matlab Bicubic upscaled LR frames of a scene, 1920x1080x5xframe_number
• LR_Bic_MC: Matlab Bicubic upscaled and Celiu optical flow motion compensated LR frames of a scene, 1920x1080x5xframe_number
• LR_MC: Celiu optical flow motion compensated LR frames of a scene, (1920/X)x(1080/X)x5xframe_number

Download the pre-processed h5 files from Dropbox here or Baidu Yun here with code: apkw , and then setup the path in the codes.

If you want to prepare your own train/val/test h5 files, please check the Matlab scripts in the matlab_pre_processing folder. Notice that you would need to install Celiu optical flow to run the Matlab code.

Test Image Dataset

The test image dataset are sampled from | Set 5 | Bevilacqua et al. BMVC 2012 | Set 14 | Zeyde et al. LNCS 2010 | BSD 100 | Martin et al. ICCV 2001 | Sun-Hays 80 | Sun and Hays ICCP 2012 | Urban 100 | Huang et al. CVPR 2015. Download the preprocessed H5 files from here, and then setup the path in test_image.py file.

Usage

Train SRCNN

SRCNN need to be trained to initialize VSRNet

python train_SRCNN.py

optional arguments:
--upscale_factor      super resolution upscale factor [default value is 4]
--num_epochs          super resolution epochs number [default value is 800]

Train VSRNet

python train_VSRNet.py

optional arguments:
--upscale_factor      super resolution upscale factor [default value is 4]
--num_epochs          super resolution epochs number [default value is 400]

Test Image

python test_image_SRCNN.py

optional arguments:
--upscale_factor      super resolution upscale factor [default value is 3]
--model_name          super resolution model name [default value is epochs_SRCNN/model_epoch_800.pth]

The output high resolution images are on results directory.

Test CDVL Video (preprocessed H5 files) using SRCNN

python test_video_CDVL_SRCNN.py

optional arguments:
--upscale_factor      super resolution upscale factor [default value is 3]
--model_name          SRCNN model name [default value is epochs_SRCNN/model_epoch_800.pth]
--is_real_time        whether to save results into video file or not [default value is False]
--delay_time          display delay time [default value is 1]

The output high resolution images are on results directory.

Test CDVL Video (preprocessed H5 files) using VSRNet

python test_video_CDVL_VSRNet.py

optional arguments:
--upscale_factor      super resolution upscale factor [default value is 3]
--vsrnet_model_name   VSRNet model name [default value is epochs_SRCNN/model_epoch_800.pth]
--srcnn_model_name    SRCNN model name [default value is epochs_SRCNN/model_epoch_800.pth]
--is_real_time        whether to save results into video file or not [default value is False]
--delay_time          display delay time [default value is 1]

The output high resolution images are on results directory.