zihangjiang / dr-learning-for-3d-face

Implementation for paper "Disentangled Representation Learning for 3D Face Shape" CVPR 2019

Disentangled Representation Learning for 3D Face Shape

This repository is the implementation of our CVPR 2019 paper "Disentangled Representation Learning for 3D Face Shape"

Authors: Zihang Jiang ,Qianyi Wu, Keyu Chen and Juyong Zhang .

Our Proposed Framework

Result Examples

We can manipulate 3D face shape in expression and identity code space.

Usage

Dataset

Please download FaceWareHouse dataset and extract it into the corresponding dir like data/FaceWarehouse_Data.

Requirements

1. Basic Environment

tensorflow-gpu = 1.9.0

Keras = 2.2.2

openmesh (both python version which can be installed through pip install openmesh, and compiled version are required)

1.5 quick start

For convenience, we included repos for get_dr and get_mesh in third party folder and provided scripts for easy install. See readme in third party folder for detail.

2. Requirements for Data Processing (About Deformation Representation Feature)

1. We provide a python interface for obtaining Deformation Representation (DR) Feature. Code are avaliable at Here to generate DR feature for each obj file by specific one reference mesh. Replace the get_dr.cpython-36m-x86_64-linux-gnu.so with compiled one and run python get_fwh_dr.py After that, you can change the data_path and data_format in src/data_utils.py.

2. To recover mesh from DR feature, you need to compile get_mesh, and replace the get_mesh.cpython-36m-x86_64-linux-gnu.so in src folder.

3. Also, python version of libigl is needed for mesh-IO and you need to replace the pyigl.so in src folder if the current one does not match your environment.

After all requirements are satisfied, you can use following command to train and test the model.

Training

Run following command to generate training and testing data for 3D face DR learning

python src/data_utils.py

We have provided DR feature of expression mesh on Meanface we used in this project on here.

Run this command to pretrain identity branch

python main.py -m gcn_vae_id -e 20

Run following command to pretrain expression branch

python main.py -m gcn_vae_exp -e 20

Run following command for end_to_end training the whole framework

python main.py -m fusion_dr -e 20

Testing

You can test on each branch and the whole framework like

python main.py -m fusion_dr -l -t

Our pretrained model can be found here.

Evaluation

The measurement.py and STED folder is for computation of numerical result mentioned in our paper, including two reconstruction metrics and two decompostion metrics.

Notes

1. if you train the model on your own dataset(for which topology is different from FaceWarehouse mesh), you have to recompute Mean_Face.obj and expression meshes on mean face as mentioned in our paper and regenerate the FWH_adj_matrix.npz in data/disentagle folder using src/igl_test.py.
2. We will release srcipts for data augmentation method metioned in our paper. You could put the augmented interpolated data in data/disentangle/Interpolated_results
3. Currently we have fully tested this package on Ubuntu 16.04 LTS environment with CUDA 9.0. Windows and MacOS are not ensured working.
4. Errors like Unknown CMake command "pybind11_add_module" you may encounter while building get_mesh and get_dr can be solved by

git submodule init
git submodule update --recursive

5. If you have comments or questions, please contact Zihang Jiang ([email protected]), Qianyi Wu ([email protected]), Keyu Chen ([email protected]), Juyong Zhang ([email protected]).

Citation

Please cite the following papers if it helps your research:

Disentangled Representation Learning for 3D Face Shape

@inproceedings{Jiang2019Disentangled,
      title={Disentangled Representation Learning for 3D Face Shape},
      author={Jiang, Zi-Hang and Wu, Qianyi and Chen, Keyu and Zhang, Juyong},
      booktitle={IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
      year={2019}
}

Acknowledgement

GCN part code was inspired by https://github.com/tkipf/keras-gcn.

License

Free for personal or research use, for commercial use please contact us via email.

dr-learning-for-3d-face's Issues

the folder ‘data/disentangle/real_data’ ？

Hello，What does the folder ‘data/disentangle/real_data’ in line 395 of file src/model.py refer to？

FileNotFoundError: [Errno 2] No such file or directory: 'data/disentangle/test_data/Feature141.npy'

error in V2M2

good job for me!
may be some error in V2M2 func, like follow

V2M2 generate bad mesh when run generate_mean_face, like follow

i guess new_v + V is need like V2M, V2M is work but also have some bad case, like follow

is it normal? can you give me some advice? thx~ a lot!

module 'src.pyigl' has no attribute 'eigen'?

pyigl version ?

can you share facewarehosue dataset?

hello, good job!
can you share facewarehouse dataset? i can not get it from teacher zhou. my email is [email protected]. thx~ very much!

the difference of "train"and "special_train"

hi, your work is excellent and i am interested in it. when reading your codes, i can't figure out the difference of the two functions "net.special_train" and "net.train" due to my weak mathematical foundation. could you please tell me why you use the two functions to train your model. thank you very much.

Can this method do expression transfer between raw image?

line 154 in '/src/data_utils.py' is wrong?

np.save('../data/disentangle/test_data',np.vstack(train_data)) should be changed to np.save('../data/disentangle/test_data',np.vstack(test_data)) ?

When will you release srcipts for data augmentation method metioned in our paper? Thanks!

When will you release srcipts for data augmentation method metioned in our paper? Thanks!

The topology of a new dataset

Hello! I want to apply your method on a new dataset. But a question that bothers me is how to make all 3d faces in the new dataset have the same vertex structure, for example, they could be averaged by direct addition. I would appreciate if you could give me some advice.

bad case generate by fusion net!

good job for me!
follow is result generate by fusion net

some bad case in result, plus result also have bad case, but id and exp is good, do you have any idea about it?
plus:

id:

exp:

and what is the meaning of
bias = 0.9 * (self.m_list + self.M_list) / (self.M_list - self.m_list)
i find 0.9-->1.0 can improve expression perform

def test_generate_one(self, id_net, exp_net):
        def I(z):
            z_mean, z_log_var, z = id_net.encoder(z)
            dr_id = id_net.decoder(z)
            return dr_id
        def E(z):
            z_mean, z_log_var, z = exp_net.encoder(z)
            dr_exp = exp_net.decoder(z)
            return dr_exp
        def F(y, x):
            return self.gcn_comp([y, x])
        def load_model():
            our_model.load_weights(('../model/our_model/our_model{}{}.h5').format(self.prefix, self.suffix))

        z = self.real

        our_model = Model(z, [I(z), E(z), F(I(z), E(z))])

        if self.load:
            load_model()
            print('!!!!!!!!!!!load fusion model success!!!!!!!!!!!')

        test_dr_feature = np.fromfile(os.path.join('dr_feature', 'Tester_106', 'dr_1.dat'))
        test_dr_feature = test_dr_feature[np.newaxis, :]
        
        test_dr_feature = normalize_fromfile(test_dr_feature, self.M_list, self.m_list)

        id_code = id_net.encoder.predict(test_dr_feature, batch_size=self.batch_size)[0]
        exp_code = exp_net.encoder.predict(test_dr_feature, batch_size=self.batch_size)[0]

        print('\n')
        print(id_code)
        print(exp_code)
        print('\n')

        bias = 0.9 * (self.m_list + self.M_list) / (self.M_list - self.m_list)

        norm_id = id_net.decoder.predict(id_code, batch_size=self.batch_size)
        norm_exp = exp_net.decoder.predict(exp_code, batch_size=self.batch_size)

        plus = norm_id+norm_exp+bias
        recon = self.gcn_comp.predict([norm_id, norm_exp], batch_size=self.batch_size) + bias

        id_feature = denormalize_fromfile(norm_id, self.M_list, self.m_list)
        exp_feature = denormalize_fromfile(norm_exp, self.M_list, self.m_list)
        plus_feature = denormalize_fromfile(plus, self.M_list, self.m_list)
        recon_feature = denormalize_fromfile(recon, self.M_list, self.m_list)

Hi， I am very interested in you project. How to deal with the situation in which two meshs have different topological structure(different vertex number...)?

How to deal with the situation in which two meshs have different topological structure(different vertex number...)?

the srcipts for data augmentation

Hi, I've read your paper but I couldn't generate interpolated data by myself :( . Can you release srcipts for data augmentation method metioned in your paper? Thanks very much.

Couldn't find face_{}.dat in FacewareHouse Dataset?

Sorry.
It seems that something named face_{}.dat is needed but I couldn't find it in FaceWareHouse dataset I downloaded.
Thanks for your help :)

Got extremely large face when performing data augmentation

I got extremely large face when I implemented the data augmentation method described in the paper. Here is my code

import config
import numpy as np
from lib_dr import get_dr, get_mesh
from tqdm import tqdm
import open3d as o3d
import openmesh as om


def generate_coefficients(m):
    r = np.random.uniform(0.5, 1.2)
    thetas = np.random.uniform(0, np.pi / 2, m - 1)
    c = np.repeat(r, m)
    for i in range(len(thetas)):
        c[m - i - 1] *= np.sin(thetas[i])
        c[:m - i - 1] *= np.cos(thetas[i])
    return c


n_aug = 500
m = 5

data_path = 'data/CoMA/data/FW_140/train.npy'
result_path = 'data/CoMA/data/FW_140/train_dr_{}.npy'.format(n_aug)

train_np = np.load(data_path)

everyone_template = 'data/FWH/Tester_{}/Blendshape/shape_0.obj'
mean_face_path = 'DR-Learning-for-3D-Face/data/disentangle/Mean_Face.obj'

features = []

for i in tqdm(range(1, 141)):
    feat = get_dr.get_dr(mean_face_path, everyone_template.format(i))
    features.append(feat)

features = np.array(features)

template_mesh = om.read_trimesh(mean_face_path)
mesh = o3d.geometry.TriangleMesh()
mesh.vertices = o3d.utility.Vector3dVector(template_mesh.points())
mesh.triangles = o3d.utility.Vector3iVector(template_mesh.face_vertex_indices())
mesh.compute_vertex_normals()

aug_res = []
for i in tqdm(range(n_aug)):
    c = generate_coefficients(m)
    ids = np.random.choice(features.shape[0], m, replace=False)
    samples = features[ids]
    tmp = get_mesh.get_mesh(mean_face_path, np.tensordot(samples, c, axes=[0, 0]))
    tmp = tmp.reshape(-1, 3)

    o3d.visualization.draw_geometries([mesh])
    aug_res.append(tmp)

aug_res = np.array(aug_res)
aug_res = np.concatenate([train_np, aug_res], axis=0)
print(aug_res.shape)
np.save(result_path, aug_res)

Is it a normal phenomenon?

how to get the front_part_v.txt and mean face in a new dataset?

what is 'delta' in data_utils.py?

First of all, thank you for publishing a great research.

I wonder what Delta means.

In data_utils.py line 13, delta = np.array([1,0,0,1,0,1,0,0,0])?

Thanks in advance :)

how to get "min_data.npy"and "max_data.npy" in "data/disentangle"?

How to determine the identity and expression of the interpolated face?

Thanks for your code!
I want to know how to determine the ID and expression of the augmented data.
Due to the new faces are interpolated by several identity samples, these samples are belong to different subjects, so what is identity of new data?

pretrained model?

Hi!

I think there was a pre-trained model(google drive link). Where did it go?

Thanks in advance :)

ps) Can you tell when an augmentation script will be released?