jialetao / mtia

[ECCV 2022] Motion Transformer for Unsupervised Image Animation

[ECCV 2022] Motion Transformer for Unsupervised Image Animation

This is the project page of the paper Motion Transformer for Unsupervised Image Animation (ECCV 2022).

2022.8.15: Codes are released.

Environments

The model is trained on 8 Tesla V100 cards, PyTorch version 1.6 and 1.8 with Python 3.6 are tested fine. Basic installations are given in requiremetns.txt.

pip install -r requirements.txt

Datasets

Following FOMM to download TaiChiHD,Voxceleb1, and MGIF. And for the TED384, following MRAA. After downloading and pre-processing, the dataset should be placed in the ./data folder or you can change the parameter root_dir in the yaml config file. Note that we save the video dataset in PNG frames format for better training IO performance. All train and test video frames are specified in txt files in the ./data folder. The tree structure of the dataset path is given in the following.

|-- data/taichi-png, data/TED384-png, data/voxceleb1-png, data/moving-gif-png
    |-- train
        |-- video1
            |-- 00000.png
            |-- 00001.png
            |-- ...
        |-- video2
            |-- 00000.png
            |-- 00001.png
            |-- ...
        |-- ...
    |-- test
        |-- video1
            |-- 00000.png
            |-- 00001.png
            |-- ...
        |-- video2
            |-- 00000.png
            |-- 00001.png
            |-- ...
        |-- ...

Checkpoints

Google drive Baiduyun passwd:qp6v

Training

We train the model on 8 cards and use pytorch DistributedDataPrallel.

CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 python -m torch.distributed.launch --nproc_per_node=8 run.py --config config/dataset.yaml

Evaluation

Evaluate video reconstruction with the following command, for more metrics, we recommend seeing FOMM-Pose-Evaluation.

CUDA_VISIBLE_DEVICES=0 python run.py --mode reconstruction --config path/to/config --checkpoint path/to/model.pth  

Demo

To make a demo animation, specify the driving video and source image, the result video will be saved to result.mp4.

python demo.py --config path/to/config --checkpoint path/to/model.pth --driving_video path/to/video.mp4 --source_image path/to/image.png --result_video path/to/result.mp4 --adapt_scale

Animation

Citation

@inproceedings{tao2022motion,
title={Motion Transformer for Unsupervised Image Animation},
author={Tao, Jiale and Wang, Biao and Ge, Tiezheng and Jiang, Yuning and Li, Wen and Duan, Lixin},
booktitle={European Conference on Computer Vision},
year={2022}
}

Acknowledgements

The implementation is partially borrowed from FOMM, TokenPose and TransPose, we thank the authors for their excellent works.

mtia's Issues

评估测试

i你好,非常感谢能开源这份代码,危\我现在在做一份测试评估关于在vox2测试集上的,我运行run.py的重建模式发生意外,请问,如何才能正确的使用重建模式?

can not download checkpoints

I have no idea how to download the pretrained checkpoints. Could you please provide a baidudisk link? Thanks a lot.

cannot download training data, anyone can provide preprocessed data link, baiduyun or google drive

i tried to download training data, but some links are broken. It's hard to obtain them again.
Does anyone have the preprocessed data link?

2it [13:53, 358.55s/it]Can not load video _QdPW8JrYzQ, broken link
36it [1:37:18, 228.99s/it]Can not load video onaiTOLPeUs, broken link
71it [3:06:09, 130.61s/it]Can not load video HCgv_HNoJrY, broken link
73it [3:08:57, 114.38s/it]Can not load video 95ovIJ3dsNk, broken link
117it [4:40:07, 124.83s/it]Can not load video mLzTo-y8Ef0, broken link
120it [4:47:27, 152.69s/it]Can not load video qH4GAXXH29s, broken link
130it [5:10:07, 149.93s/it]Can not load video RDIy58g9n2k, broken link
160it [6:13:53, 122.22s/it]Can not load video KzIp4IzDPG0, broken link
185it [7:07:17, 108.19s/it]Can not load video u08T3A7slkE, broken link
205it [8:02:42, 168.09s/it]Can not load video 5zWmdHmJMd0, broken link

could u provide checkpoint？

Thanks for your exciting work, for studying can u provide the checkpoint for FOMM on the TED-talk?
(in your paper's Qualitative comparisons)
Thanks a lot!

a issue about demo

You have done a great job! But when I tried to run the demo file, I encountered an error that was displayed as' load '_ A parameter is missing from checkpoint. Could you please tell me how to solve it? Thank you!