This repository contains the official PyTorch implementation of the paper:
Box-FaceS: A Bidirectional Method for Box-Guided Face Component Editing
Abstract: While the quality of face manipulation has been improved tremendously, the ability to control face components, e.g., eyebrows, is still limited. Although existing methods have realized component editing with user-provided geometry guidance, such as masks or sketches, their performance is largely dependent on the user's painting efforts. To address these issues, we propose Box-FaceS, a bidirectional method that can edit face components by simply translating and zooming the bounding boxes. This framework learns representations for every face component, independently, as well as a high-dimensional tensor capturing face outlines. To enable box-guided face editing, we develop a novel Box Adaptive Modulation (BAM) module for the generator, which first transforms component embeddings to style parameters and then modulates visual features inside a given box-like region on the face outlines. A cooperative learning scheme is proposed to impose independence between face outlines and component embeddings. As a result, it is flexible to determine the component style by its embedding, and to control its position and size by the provided bounding box. Box-FaceS also learns to transfer components between two faces while maintaining the consistency of image content. In particular, Box-FaceS can generate creative faces with reasonable exaggerations, requiring neither supervision nor complex spatial morphing operations. Through the comparisons with state-of-the-art methods, Box-FaceS shows its superiority in component editing, both qualitatively and quantitatively. To the best of our knowledge, Box-FaceS is the first approach that can freely edit the position and shape of the face components without editing the face masks or sketches.
Install the dependencies:
conda create -n boxfaces python=3.7
conda activate boxfaces
conda install pytorch==1.7.1 torchvision==0.8.2 torchaudio==0.7.2 cudatoolkit=11.0 -c pytorch
pip install -r requirements.txt
For docker users:
docker pull huangwenjingcs/ubuntu18-conda-cuda11-pytorch1.7
- To obtain the CelebA-HQ dataset, please refer to the Progressive GAN repository. The official way of generating CelebA-HQ can be challenging. You can get the pre-generated dataset from CelebA-HQ-dataset. Unzip the file and put the images them to "data/CelebA-HQ-img/".
- To obtain the Visual Genome Animals dataset, please refer to VGA. Download the images and put them to "data/VGA-img/". To obtain the full Visual Genome dataset, please refer to VisualGenome.
Once the datasets are set up, you can train the networks as follows:
- Edit
configs/<DATASET>.json
to specify the dataset, model and training configurations. - Run the training script with
python train.py -c configs/<DATASET>.json
. For example,
# train Box-FaceS with CelebA-HQ dataset, with a batch size of 16
python train.py -c configs/celebahq.json --bz 16
# train Box-FaceS with VGA dataset, with a batch size of 8 (paper setting on vga)
python train.py -c configs/vga.json --bz 8
The code will use all GPUS by default, please specify the devices you want to use by:
# train Box-FaceS in parallel, with a batch size of 16 (paper setting on celebahq)
CUDA_VISIBLE_DEVICES=0,1 python train.py -c configs/celebahq.json --bz 8
- The checkpoints are written to a newly created directory
saved/models/<DATASET>
Pre-trained models can be found on Google Drive:
Path | Description |
---|---|
checkpoint | Main folder. |
├ celeba_hq_256.pt | Box-FaceS trained with CelebA-HQ dataset at 256×256. |
├ vga_256.pt | Box-FaceS trained with Visual Genome Animals at 256×256. |
For editing the position and shape of component, run:
python move.py --cmd <INSTRUCTION> # editing results are saved to output/move/<INSTRUCTION>
For face component transfer, run:
python replace.py --index <COMPONENT_INDEX> # editing results are saved to output/replace/
For image reconstruction, run:
python reconstruction.py # editing results are saved to output/recon
To reproduce the quantitative results in the paper:
# to reproduce Table 1
python move.py --cmd <INSTRUCTION> --data_path data/test.txt --img_dir data/CelebA-HQ-img
# to reproduce Table 2
python comtrsf.py
# to reproduce Table 3
python reconstruction.py --data_path data/test.txt --img_dir data/CelebA-HQ-img
The results are saved to output/
.
- Reconstruction: MSE, LPIPS, SSIM, FID
- Component transfer: MSE$_{\text{irr}}$, IFG, FID
- Component editing: FID
If you want to see details, please follow metrics/README.md
.
If you find this work useful for your research, please cite our paper:
@inproceedings{huang2022box,
title={Box-FaceS: A Bidirectional Method for Box-Guided Face Component Editing},
author={Huang, Wenjing and Tu, Shikui and Xu, Lei},
booktitle={Proceedings of the 30th ACM International Conference on Multimedia},
pages={6061--6071},
year={2022}
}
This repository used some codes in pytorch-template and stylegan2-pytorch.