tnquang1416 / fi-dusgan

Implementation for Video Frame Interpolation Via Down-Up Scale Generative Adversarial Networks (2022).

FI-DUSGAN

This is the official implementation for Video Frame Interpolation Via Down-Up Scale Generative Adversarial Networks (2022).

Requirements

For using this implementation, we recommend PyTorch with version 1.5.0 or later.

Dataset

The target dataset of this implementation is the Vimeo-90k dataset. The sample data directory is organized following the frame interpolation subset of the Vimeo90k.

Please see the Vimeo-90k dataset documentation for more details.

Run the code

python train.py --path=cpt_folder

Pre-trained model

Please find it on Google Drive and then put in the pre-trained folder. Note that rename the file to "net_gen.pt" might be required.

Reference

@ARTICLE{9097443,
  author={Tran, Quang Nhat and Yang, Shih-Hsuan},
  journal={Computer Vision and Image Understanding}, 
  title={Video Frame Interpolation Via Down-Up Scale Generative Adversarial Networks}, 
  year={2022},
  volume={220},
  doi={https://doi.org/10.1016/j.cviu.2022.103434}}

fi-dusgan's Issues

Am I testing the model correctly?

Hello, I am training this model on an animated image dataset.

During training, train.py saves some images called train_x.png - see here https://github.com/tnquang1416/FI-DUSGAN/blob/master/train.py#L148. Based on inspection of the code and output images I believe that train_x.png show generated images, not training data. See an example here (the figure I've circled in white makes me thing that these are generated images):

These images generally show very high-quality images, which is great! However, when I then test my trained model with python test.py --path_gen=net_gen.pt, the resulting images are much lower in quality. I wonder if I am not testing the model correctly, or not saving the weights correctly after training.

Is there anything you recommend I try?

Error with valid and fake tensors

I am trying to run the code to apply to a new dataset, however when attempting to run the code to make sure it was working correctly (training on the vimeo 90k dataset), I had some errors.

When running the code I get an error with the valid and fake tensors, when attempting to train a model on a small subset of the data. Can you help? Furthermore, there were a few errors in the code that had to be fixed (missing imports etc.). Is there a more up to date version of the code that has not been pushed that does not include these errors?

Thanks in advance for your help

Traceback (most recent call last):  File "/home/FI-DUSGAN/train.py", line 277, in <module>    main()  
File "/home/FI-DUSGAN/train.py", line 270, in main    epoch, g_loss, d_loss = train(cur_gen_epoch, dataloader)  
File "/home/FI-DUSGAN/train.py", line 185, in train    gen_imgs, real_loss, fake_loss, g_loss, adv_loss = _train_interval(in_pres, in_lats, gt, 1)  
File "/home/FI-DUSGAN/train.py", line 108, in _train_interval    real_loss = adversarial_loss(gt_distingue, valid)  
File "/opt/conda/envs/mlp/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl    return forward_call(*input, **kwargs)  
File "/opt/conda/envs/mlp/lib/python3.9/site-packages/torch/nn/modules/loss.py", line 720, in forward    return F.binary_cross_entropy_with_logits(input, target,  
File "/opt/conda/envs/mlp/lib/python3.9/site-packages/torch/nn/functional.py", line 3160, in binary_cross_entropy_with_logits    

raise ValueError("Target size ({}) must be the same as input size ({})".format(target.size(), input.size()))ValueError: Target size (torch.Size([20, 1, 1, 1])) must be the same as input size (torch.Size([20, 1, 5, 11]))

Request for pretrain model

Would it be possible to make the model publicly available? I train dusgan since over 6 million iterations and I still don't have good looking images.