diegothomas / avatar-generation-3drw2019-

This is the source code of our 3DRW 2019 paper

Avatar-generation-3DRW2019-

This is the source code of our 3DRW 2019 paper

This code is made for academic researhc use only.

If you use part of this code for your research please cite the following paper:

@INPROCEEDINGS{, author = {Hayato Onizuka and Diego Thomas and Hideaki Uchiyama and {Rin-ichiro} Taniguchi}, title = {Landmark-guided deformation transfer of template facial expressions for automatic generation of avatar blendshapes}, booktitle = {3D Reconstruction in the Wild Workshop (3DRW), part of the 2019 International Conference on Computer Vision (ICCV)}, year = {2019}, month = {10}, }

avatar-generation-3drw2019-'s Issues

Blendshape semantics and 3D avatar resource

Hi Diego,

I working on the facial motion retargeting topic. Now I am trying to convert the blendshapes you shared here to BFM model as well as other avatars. Overall, it looks good on BFM after the deformation transfer. However, there're some flaws on details. For example, the eyelid is not fully closed for the eye close blendshape. Since I am not a professional modeling artist, I don't know how to finetune it.

I found some other blendshape resources of BFM model. I want to substitute some results by Blendshape Maker with those. For blendshapes of clear semantic meaning, it's easy to do, e.g., eye close BS. However, it's difficult for some other subtle BS. I am wondering do you have the description of the semantic meanings of the blendshapes, like this, so that I can know what each BS is.

BS generation from head template to whole body avatar model

Hello, I noticed that the avatar model in your paper and provided in this project only contain the head (and neck), I tried a new avatar model with only head, it worked well.

However, when I tried to generate BS from head templates to whole-body avatar model (still select the 51 facial landmarks), something got wrong. I guess error occurred in the step of deforming avatar neutral model towards source neutral model. Maybe I could constrain the avatar model to head part during non-rigid deformation? Not sure if it makes sense. Any sugguestions?

Thanks,
Jiawen

Corrupted model after conversion by Blendshape maker

Hi Diego,

I am using blendshape maker to transfer blendshapes with different source and avatar 3D model (Other than the models you shared in the repo). I found that it doesn't work with my target model. The transferred neutral model is corrupted somehow. At the beginning, I thought it's because the inaccurate landmark labelling. However, after I relabelled landmarks with greater care, the issue still exists. Do you have any suggestions? The issue is shown as the folloing pictures.

1640499948318-61e3b4b1-dc37-4141-aa46-0206aa937ca2

After refining some landmarks:

When I changed the avatar model to oldman model, it works fine. I am wondering is it because maybe blendshape maker just doesn't work for some 3D face model?

Some differences between paper and code

Hello, thanks for your amazing work.

In Section 4.3.2 of your paper, it mentioned " To measure the size of each facial part, we first use the rotation matrices Rk,s and Rk,t to align the facial part with the z axis."

In line 1481&1482，BMManager.py. In my understanding, Rot_S and Rot_A is the rotation matrix from z axis orientation to facial part normal vector orientation. So when we want to align the facial part with z axis, we should apply the inverse rotation matrix Rot_S.T and Rot_A.T to each faical part's landmarks.

However, when measuring facial parts size (line 1486 to 1502 and line 1528 to 1544), your code apply Rot_S and Rot_A to facial part landmarks instead of Rot_S.T and Rot_A.T.

It makes me confused, hope you can give me some explanation

Transfer blendshapes defined by quad polygons (1 face defined by 4 vertices)

Hi,

Thank you for this wonderful work and it's really handy!

I an trying to apply this to other target avatar and I found that many 3D cartoon face models are define by quad polygons, i.e., a face consists of four vertices. This work is designed for triangular mesh based 3D models. I am wondering if the transformation method can somehow be quickly adapted for quad mesh based 3D models. Any suggestions?

Br,
Aaron

BlendshapeMaker example data

Hello!

I've just created a github project (https://github.com/jerenchen/deformxfer) for my C++ implementation, which was largely based on your wonderful Python impl (thank you!) I'm also fetching your project repo for the example data when building my demo.

I wonder would you be able to take a look at the said project and let me know if you have any comments/concerns? I'd be happy to acknowledge any individuals for their contribution.

Thank you,
Jeren