ywu40 / fldr-vfi

Efficient Feature Extraction for High-resolution Video Frame Interpolation (BMVC 2022)

Efficient Feature Extraction for High-resolution Video Frame Interpolation

This is the official repository accompanying the BMVC 2022 paper:

Efficient Feature Extraction for High-resolution Video Frame Interpolation
Moritz Nottebaum, Stefan Roth and Simone Schaub-Mayer
BMVC 2022. [paper (open access)] [supplemental] [example results] [talk video] [preprint (arXiv)]

This repository contains the training and test code along with the trained weights to reproduce our results, and our test datasets Inter4K-S and Inter4K-L (subsets of Inter4K).

Installation

The following steps will set up a local copy of the repository.

1. Create conda environment:

conda create --name fldrnet
conda activate fldrnet

2. Install PyTorch:

conda install pytorch torchvision torchaudio cudatoolkit=11.6 -c pytorch -c conda-forge

3. Install the dependencies:

pip install -r requirements.txt

4. Handle CuPy problems (Optional):

pip uninstall cupy-cuda116
conda install -c conda-forge cupy

If CuPy throws importing errors during execution, a conda installation can fix it.

Training

The following command will train a model from scratch:

python train_it.py --x_train_data_path /to/xtrain/train  --toptim

The flag --x_train_data_path /to/xtrain/train contains the location of the training data. We used X-Train from Sim et al. The --toptim flag is optional and adds a post-training optimization of the temperature parameter as described in our paper in the occlusion estimation section.

Testing

1. Download the respective testset from the following links:

Dataset	Link
X-Test	XVFI repository
Xiph	Xiph benchmark
Inter4K	Our subset (Licence)

2. Use the file path for the test sets accordingly or update them in main.py:

--x_test_data_path /to/your/location
--xiph_data_path /to/your/location
--inter4k_data_path /to/your/location

3. The following line will evaluate the provided checkpoint "fLDRnet_X4K1000FPS_exp1_best_PSNR.pt" on all four testsets (X-Test, Xiph-4K, Inter4K-S, Inter4K-L):

python main.py --exp_num 1 --gpu 0 --papermodel --test5scales 

By adding the option --testsets you can choose on which data you want to evaluate (options are "Inter4K-S", "Inter4k-L", "X-Test","Xiph-4K"). The option --papermodel ensures all preferences are set according to the model of the paper. The option --test5scales adapts args.fractions, args.scales, args.phase, args.S_tst and args.moreTstSc to allow for additional scales for testing.

Run Model

In run_on_your_images.py you have the run_on_images(...) function, which simplifies to test the model on your own data. In the main() function you can see examplary code to test on X-Test.

Acknowledgements

We thank the contributors of the following repositories for using parts of their publicly available code and 4K datasets, which were necessary to adequately train and evaluate our method:

• XVFI
• Softmax-splatting
• Inter4K: Our Inter4K testset for video frame interpolation is a subset of the images occuring in the Inter4K dataset shared under CC BY-NC 4.0.

Citation

We hope you find our work useful. If you would like to acknowledge it in your project, please use the following citation:

@inproceedings{Nottebaum:2022:EFE,
  author    = {Nottebaum, Moritz and Roth, Stefan and Schaub-Meyer, Simone},
  title     = {Efficient Feature Extraction for High-resolution Video Frame Interpolation},
  booktitle = {British Machine Vision Conference  {BMVC}},
  year      = {2022}
}