shizongyan / cmos-gan

Code for paper "TIP2023 - CMOS-GAN: Semi-supervised Generative Adversarial Model for Cross-Modality Face Image Synthesis"

CMOS-GAN: Semi-supervised Generative Adversarial Model for Cross-Modality Face Image Synthesis

We provide PyTorch implementations for CMOS-GAN (Paper). This PyTorch implementation produces results comparable to our original results reported in our paper. We will continue optimize the code to fix any issues reported.

If you use this code for your research, please cite:

S. Yu, H. Han, S. Shan and X. Chen, "CMOS-GAN: Semi-Supervised Generative Adversarial Model for Cross-Modality Face Image Synthesis," in IEEE Transactions on Image Processing, vol. 32, pp. 144-158, 2023, doi: 10.1109/TIP.2022.3226413.

Prerequisites

• Linux or macOS
• Python 3 (We used python 3.7/3.5)
• CPU or NVIDIA GPU + CUDA (We used cuda 11.1/10.2/10.0)

Getting Started

Installation

• Clone this repo:

git clone  https://github.com/skgyu/CMOS-GAN
cd CMOS-GAN
cd CMOS-GAN_code_refactor

• Install PyTorch 0.4+ (we use PyTorch 1.7.1/1.8.2/1.2.0) and other dependencies (e.g., torchvision and visdom).

Run CMOS-GAN

• Download the pre-trained RGB face recognition network.

  • We previously downloaded a pre-trained face recognition network from https://www.robots.ox.ac.uk/~vgg/data/vgg_face2/, but this model maybe no longer publicly available.
  • So we suggest use Our Pretrained models for PyTorch are converted from Caffe models authors of [1] provide.
  • Put the downloaded model (named 'ResNet50Backbone.pth') at 'CMOS-GAN_code_refactor/pretrained_model/ResNet50Backbone.pth'.

• sketch-to-photo synthesis

  • We download the CUFS [3] and CUFSF [3-4] dataset. We use CUFS as the paired data $S_p$, and use CUFSF as the unpaired data $S_u$.

  • We performed face alignment, cropping and augmentation. Our processed data can be downloaded here. Put the downloaded dataset folders (named 'AUG_3_9_AR', 'AUG_3_9_CUFSF', 'AUG_3_9_CUHK', and 'AUG_3_9_XM2VTS') at 'CMOS-GAN/dataset/Viewed/'. The complete relative paths are as follows.

    • 'CMOS-GAN/dataset/Viewed/AUG_3_9_AR'
    • 'CMOS-GAN/dataset/Viewed/AUG_3_9_CUFSF'
    • 'CMOS-GAN/dataset/Viewed/AUG_3_9_CUHK'
    • 'CMOS-GAN/dataset/Viewed/AUG_3_9_XM2VTS'

  • (Optional) If you want to perform face recognition using the 10,000 background images like the experiment in our paper, you need to download MORPH [2] to build the background set of the gallery, which contains 10000 RGB images. We aligned and cropped these images in the same way as for CUFS and CUFSF. Place the folder of gallery set (named '10000backgrounds') at 'CMOS-GAN/dataset/additional/10000backgrounds'.

  • Run the following script to start training and testing.

    bash script/S2P_CUFS_CUFSF/S2P_CUFS_CUFSF.sh
    

    You can modify the random seed in the yaml files in the 'script\S2P_CUFS_CUFSF' folder.

  • If you do not want to use the 10,000 background images when performing face recognition, you can run the following script to start training and testing.

    bash script/S2P_CUFS_CUFSF_nobackgrounds/S2P_CUFS_CUFSF_nobackgrounds.sh
    

  • If you do not want to perform face recognition, you can run the following scripts to start training and testing.

    bash script/S2P_CUFS_CUFSF/S2P_CUFS_CUFSF_no_recognition.sh
    

Similarly, NIR-to-VIS synthesis and RGB-to-depth synthesis can be performed using the following scripts.

• NIR-to-VIS synthesis

  • VIPLMumoFace3K

    bash script/RGBD_VIPLMumoFace3K/RGBD_VIPLMumoFace3K.sh
    

• RGB-to-depth synthesis

  • RealSenseII

    bash script/RGBD_RealSenseII/RGBD_RealSenseII.sh
    

  • BUAA

    bash script/RGBD_BUAA/RGBD_BUAA.sh
    

  • VIPLMumoFace3K

    bash script/RGBD_VIPLMumoFace3K/RGBD_VIPLMumoFace3K.sh
    

  • RealSenseII_VIPLMumoFace3K

    bash script/RGBD_RealSenseII_VIPLMumoFace3K/RGBD_RealSenseII_VIPLMumoFace3K.sh
    

VIPL-MumoFace-2K dataset

You can download the VIPL-MumoFace-2K Dataset from the following link:

https://vipl.ict.ac.cn/resources/databases/202309/t20230915_180574.html

References

[1] Q. Cao, L. Shen, W. Xie, O. M. Parkhi, A. Zisserman

VGGFace2: A dataset for recognising faces across pose and age

International Conference on Automatic Face and Gesture Recognition, 2018

[2] K. Ricanek and T. Tesafaye, “Morph: A longitudinal image database of normal adult age-progression,” in IEEE FG, pp. 341–345, 2006.

[3] X. Wang and X. Tang. Face Photo-Sketch Synthesis and Recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), vol. 31, no. 11, pages 1955-1967, 2009.

[4] W. Zhang, X. Wang and X. Tang. Coupled Information-Theoretic Encoding for Face Photo-Sketch Recognition. Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2011.