yepjin / avid

This respository contains the code for AVID: Any-Length Video Inpainting with Diffusion Model.

AVID
_{Any-Length Video Inpainting with Diffusion Model}

This respository contains the code for the paper AVID: Any-Length Video Inpainting with Diffusion Model. For more visualization results, please check our supplementary materils.

AVID: Any-Length Video Inpainting with Diffusion Model
Zhixing Zhang ¹, Bichen Wu ², Xiaoyan Wang ², Yaqiao Luo ², Luxin Zhang ², Yinan Zhao ², Peter Vajda ², Dimitris Metaxas ¹, and Licheng Yu ^2
1 Rutgers University ² Meta AI

AVID is a a video inpainting method versatile across a spectrum of video durations and tasks.

Recent advances in diffusion models have successfully enabled text-guided image inpainting. While it seems straightforward to extend such editing capability into video domain, there has been fewer works regarding text-guided video inpainting. Given a video, a masked region at its initial frame, and an editing prompt, it requires a model to do infilling at each frame following the editing guidance while keeping the out-of-mask region intact. There are three main challenges in text-guided video inpainting: ($i$) temporal consistency of the edited video, ($ii$) supporting different inpainting types at different structural fidelity level, and ($iii$) dealing with variable video length. To address these challenges, we introduce Any-Length Video Inpainting with Diffusion Model, dubbed as AVID. At its core, our model is equipped with effective motion modules and adjustable structure guidance, for fixed-length video inpainting. Building on top of that, we propose a novel Temporal MultiDiffusion sampling pipeline with an middle-frame attention guidance mechanism, facilitating the generation of videos with any desired duration. Our comprehensive experiments show our model can robustly deal with various inpainting types at different video duration range, with high quality.

Acknowledgement

This codebase is built on top of diffusers, AnimateDiff, and Tune-A-Video. We thank the authors for their great works.

Reference

If our work helps you, please consider to cite our paper. Thanks!

@article{zhang2023avid,
  title={AVID: Any-Length Video Inpainting with Diffusion Model},
  author={Zhang, Zhixing and Wu, Bichen and Wang, Xiaoyan and Luo, Yaqiao and Zhang, Luxin and Zhao, Yinan and Vajda, Peter and Metaxas, Dimitris and Yu, Licheng},
  journal={arXiv preprint arXiv:2312.03816},
  year={2023}
}