ruslanvasylev / sfsnet

SfSNet : Learning Shape, Reflectance and Illuminance of Faces in the Wild [Project Page]

Soumyadip Sengupta, Angjoo Kanazawa, Carlos D. Castillo, David W. Jacobs. In CVPR, 2018 (Spotlight).

Overview

• (0) Test script: test_SfSNet.m
• (1) Test images along with mask: Images_mask
• (2) Test images without mask: Images

Run 'test_SfSNet' on Matlab to run SfSNet on the supplied test images.

Dependencies

This code requires a working installation of Caffe and Matlab interface for Caffe. For guidelines and help with installation of Caffe, consult the installation guide and Caffe users group.

Please set the variable PATH_TO_CAFFE_MATLAB, in line 3 of test_SfSNet.m as $PATH_TO_CAFFE/matlab (path to matlab folder for the caffe installation)

Notes

We detect keypoints on the face using All-in-One Network and compute a mask from it. Unfortunately the code is not distributable. Ideally, you can use any keypoint detector and generate a mask based on the facial contour.

For the ease of use, we include a matlab function functions/create_mask_fiducial.m which computes a mask given keypoints. The keypoint definitions are shown in functions/facial_landmarks_68markup-768x619.jpg. Dlib C++ Library can be used to detect 68 keypoints based on this definition.

Training Code

We provided neccessary .prototxt training and solver files, along with python loss layers needed to train SfSNet. Please check SfSNet_train for more details.

Training Data

We provide 250k synthetic face images with Ground-Truth normal, albedo and lighting at 512x512 resolution generated with 3DMM. Caution the file size is ~120GB. This data is provided only for research purposes.

Note: Please fill out this form to request access to the synthetic data.

For real data, please download the aligned and cropped CelebA face dataset.

Citation

If you use this code for your research, please consider citing:

@InProceedings{sfsnetSengupta18,
  title={SfSNet: Learning Shape, Refectance and Illuminance of Faces in the Wild},
  author = {Soumyadip Sengupta and Angjoo Kanazawa and Carlos D. Castillo and David W. Jacobs},
  booktitle={Computer Vision and Pattern Regognition (CVPR)},
  year={2018}
}