jyang297 / dqbc

Official implementation of IJCAI 2023 paper "Video Frame Interpolation with Densely Queried Bilateral Correlation".

Video Frame Interpolation with Densely Queried Bilateral Correlation

Introduction

This repository is the official implementation of the IJCAI 2023 paper "Video Frame Interpolation with Densely Queried Bilateral Correlation". [paper]

Requirements

1. torch is necessary. The code has been developed with torch1.12.1.
2. Install other requirements as:

   pip install -r requirements.txt

Benchmarking

Download our pretrained model.

Download Vimeo90K dataset.

Download SNU_FILM dataset.

Download UCF101 dataset.

Download MiddleBury Other dataset.

Make your downloaded files structured like below:

.
├── configs
├── datas
├── datasets
│   ├── middlebury
│   │   ├── other-data
│   │   └── other-gt-interp
│   ├── snu_film
│   │   ├── test
│   │   ├── test-easy.txt
│   │   ├── test-extreme.txt
│   │   ├── test-hard.txt
│   │   └── test-medium.txt
│   ├── ucf101
│   │   ├── 1
│   │   ├── 1001
│   │   ...
│   │   ├── 981
│   │   └── 991
│   └── vimeo_triplet
│       ├── readme.txt
│       ├── sequences
│       ├── tri_testlist.txt
│       └── tri_trainlist.txt
├── pretrained
│   └── 510000.pth
├── experiments
├── losses
├── models
├── utils
├── validate
├── train.py
├── test.py
└── val.py

Run benchmarking by following commands:

python val.py --config configs/benchmarking/vimeo.yaml --gpu_id 0
python val.py --config configs/benchmarking/middlebury.yaml --gpu_id 0
python val.py --config configs/benchmarking/ucf101.yaml --gpu_id 0
python val.py --config configs/benchmarking/snu_film.yaml --gpu_id 0

To enable the augmented test ("Ours-Aug" in the paper), uncomment the val_aug: [T,R] line in the configuration files.

Training

The model was trained on the Vimeo90K-triplet training split.

Run the following command for training:

CUDA_VISIBLE_DEVICES=0,1,2,3 torchrun --nproc_per_node=4 --master_port 9999 train.py --config configs/train.yaml

Testing on a Custom Image Pair

First specify the path of the model weights in configs/test.yaml.

Then you can test the model on a customized image pair as:

python test.py --config configs/test.yaml --im0 <path to im0> --im1 <path to im1> --output_dir <path to output folder>

Cite

@inproceedings{ijcai2023p198,
  title     = {Video Frame Interpolation with Densely Queried Bilateral Correlation},
  author    = {Zhou, Chang and Liu, Jie and Tang, Jie and Wu, Gangshan},
  booktitle = {Proceedings of the Thirty-Second International Joint Conference on
               Artificial Intelligence, {IJCAI-23}},
  publisher = {International Joint Conferences on Artificial Intelligence Organization},
  editor    = {Edith Elkind},
  pages     = {1786--1794},
  year      = {2023},
  month     = {8},
  note      = {Main Track},
  doi       = {10.24963/ijcai.2023/198},
  url       = {https://doi.org/10.24963/ijcai.2023/198},
}