wywu / reenactgan

[ECCV 2018] ReenactGAN: Learning to Reenact Faces via Boundary Transfer

ReenactGAN: Learning to Reenact Faces via Boundary Transfer

Created by Wayne Wu, Yunxuan Zhang and Ren Li.

We present a novel learning-based framework for face reenactment. The proposed method, known as ReenactGAN, is capable of transferring facial movements and expressions from an arbitrary person’s monocular video input to a target person’s video. Instead of performing a direct transfer in the pixel space, which could result in structural artifacts, we first map the source face onto a boundary latent space. A transformer is subsequently used to adapt the source face’s boundary to the target’s boundary. Finally, a target-specific decoder is used to generate the reenacted target face. Thanks to the effective and reliable boundary-based transfer, our method can perform photo-realistic face reenactment. In addition, ReenactGAN is appealing in that the whole reenactment process is purely feed-forward, and thus the reenactment process can run in real-time.

Prerequisites

• Linux
• Python 3
• PyTorch 1.1.0 + CUDA 8.0 (Other versions may also work.)
• Dominate

Getting Started

Installing

Clone this repo:

git clone ...
cd ReenactGAN

Training

The bounday encoder is trained on WFLW and Helen dataset, and both of the boundary transformer and decoder are trained on CelebV Dataset. The training of the encoder requires a huge amount of time, so you can get the pretrained encoder at ./pretrained_models/v8_net_boundary_detection.pth.

To train the boundary transformer, run

sh script/train_Transformer.sh

You need to take care of the arguments --root_dir and --which_target. --root_dir refers to the directory of the dataset, and --which_target refers to which person to be the target

0: Emmanuel_Macron
1: Kathleen
2: Jack_Ma
3: Theresa_May
4: Donald_Trump

To train the decoder, run

sh script/train_Decoder.sh

Also, you need to take care of the value of --root_dir, which refers to the directory of the target person.

Testing

To test the model, run

sh script/move_models.sh ./checkpoints/Transformer_2019-xx-xx_xx-xx-xx/G_BA_xx.pth ./checkpoints/Decoder_2019-xx-xx_xx-xx-xx/xx_net_G.pth trump
sh script/test.sh

The images used for testing is at ./test_imgs/samples/image, and the text file, ./test_imgs/samples/images_list.txt, contains the list of these images. After the testing, you will get a floder named results, which contains the images of the real and reenacted faces, the boundaries and the transformed boundaries of the real faces. Here are some results.

You can get our trained models from Decoder and Transformer.

Citation

If you find ReenactGAN helpful for your research, please cite:

inproceedings{wayne2018reenactgan,
 author = {Wu, Wayne and Zhang, Yunxuan and Li, Cheng and Qian, Chen and Loy, Chen Change},
 title = {ReenactGAN: Learning to Reenact Faces via Boundary Transfer},
 booktitle = {ECCV},
 month = September,
 year = {2018}
} 

Questions

Please contact [email protected]

reenactgan's Issues

Training on other dataset

Thanks for the nice work
The result from the test script is great.

However, how could I train the network on my own dataset.
I could not find the code for training the PCA and Edge network.
I guess these two parts are highly related to the training data, hence could not use the pretrained model in the repository.

Thanks

98 points landmark detector

Where can I find the 98 points landmark detector? What are the description of the points?

Cannot find the model structure of Boundary Transformer

I cannot find the exact model structure of Boundary Transformer. Can you notify me of the location of the described code or some detail dimensions and the number of residual blocks you used for the Boundary transformer?

When will the code be released?

Hi ！ How to get the ground-true boundary maps

It seems that the training code of the encode network for transforming the face image to boundary maps does not include in this project, you have provided the pretrained model, can you provide me the code for producing the ground-true boundary maps? Thanks!!!

Which effect is better for swaping faces for video.

Hi,
Thanks for realease the code.
About the effect, there are some new papers this year, Compare to following papers with ours, which effect is better.
1、'Few-Shot Adversarial Learning of Realistic Neural Talking Head Models'
2、'One-shot Face Reenactment'

Any suggestions?
Many thanks.

How to train on another dataset

I would like to know how can we train on another dataset , and how to generate the all_98pt.txt file. I tried but it showed me an error.

sample_40000.txt not found

Is there any demo available?

There is no script for running a demo of pretrained model. If I want to run demo of pretrained model how should I run. @wywu @liren2515

is this one shot or few shot face reenactment?

thansk for ur works. i want to known is it one shot or few shot face reenactment?
i find ur target face is:
0: Emmanuel_Macron
1: Kathleen
2: Jack_Ma
3: Theresa_May
4: Donald_Trump
Is this model only good at this people?

where is encoder

Runtime question

Hi , thank you for releasing great project code.
I want to use boundary encoder that translate image to boundary map.
So I modify and reuse "test.sh" code.
But it's runtime is 2-3 seconds per image.
In the paper , it was written as " the reenactment process can run in real-time (30 FPS on one GTX 1080 GPU). "
I'm using the TITAN Xp GPU.
Is there anything I should be aware of or do wrong?
I'd appreciate it if you could give me advice.
thank you :)

gpu and CelebV/Donald_Trump

1，建议在read me中提醒大家--gpu_ids 0,1,2,3，这个参数在train_Transformer.sh中必须根据自己的设备情况设定。很多兄弟都习惯先拿colab或kaggle跑一跑，k80和p100只有一块gpu，只能--gpu_ids 0 ，写排序gpu，cuda就会报错。
2，在train_Decoder.sh中，你把CelebV/Donald_Trump写成了CelebV/Donald_trump小写。于是报错，找不到文件夹了。下载的数据集默认是大写。

祝顺利

Training transformer - Missing file

The script for training the transformer uses a 'sample_40000.txt' file, which is not present in the CelebV dataset. Can you please explain what the file contains and the data format in the file?