jwyang / face-alignment

Face alignment in 3000 FPS

Face alignment in 3000 FPS

##Introduction

This project is aimed to reproducing (partially) the face alignment algorithm in the CVPR 2014 paper:

Face Alignment at 3000 FPS via Regressing Local Binary Features. Shaoqing Ren, Xudong Cao, Yichen Wei, Jian Sun; The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2014, pp. 1685-1692

##How to run the codes?

• First of all, we need prepare datasets, such as afw, lfpw, helen, ibug, etc. All these can be downloaded freely from https://ibug.doc.ic.ac.uk/resources/facial-point-annotations. Then get the filelist file Path_Images.txt for each dataset (please refer to the Q&A).

• For training, initialize variable dbnames as {'Dataset_a', 'Dataset_b', ..., }, then run train_model in command line window.

• For testing, run test_model in command line window after having obtained trained model. Please remember to initialize dbnames to be the names of dataset you would like to test on.

##Dependencies

• liblinear: http://www.csie.ntu.edu.tw/~cjlin/liblinear/.

##Learned Model

Off-the-shelf model can be downloaded here: http://pan.baidu.com/s/1i325Rbn, whose configure file can be found in folder "models". Its performance is analogy to the lbf_fast model evaluated in the original paper.

##Q&A

• How to get the file Path_Images.txt?

  It can be obtained by run bat file in the root folder of a dataset, the code is simply "dir /b/s/p/w *.jpg>Path_Images.txt".
  

• What is Ts_bbox.mat?

  This problem is solved in recent version. Ts_bbox is a transformation matrix to adapt bounding boxes obtained from face detector to the boxes suitable for the face alignment algorithm.
  

• How to define the variable dbnames in train_model and test_model functions?

  It is formed as a cell array {'dbname_1' 'dbname_2' ... 'dbname_N'}. For example, if we use the images in afw for trainig, we then define it as {'afw'}.
  

• Why does an error occur when initializing parallel computing?

  It may be caused by Matlab version. For Matlab 2014, it will be okay. For earlier version, please use matlabpool alternatively.
  

• Some function correspondences from Matlab 2014 to older version

  fitgeotrans -> cp2tform, transformPointsForward -> tformfwd
  

##Discussion

For those Tencent QQ users, you can join the group face hacker (180634020) to discuss more on facial algorithms. Along with the request, it is needed to provide your affiliation (university/company where you are studying/working).

face-alignment's Issues

train_model: loadLibrary Can't load

>> dbnames = {'ibug'};
>> train_model
Attention, if error occurs, plese ensure you used the correct version of parallel initialization.
Starting parallel pool (parpool) using the 'local' profile ... connected to 4 workers.
MATLAB:dispatcher:loadLibrary Can't load '/Applications/MATLAB_R2014b.app/bin/maci64/libmwocv.dylib': dlopen(/Applications/MATLAB_R2014b.app/bin/maci64/libmwocv.dylib, 10): Symbol not found: __ZN7CvBoost10calc_errorEP8CvMLDataiPSt6vectorIfSaIfEE
  Referenced from: /Applications/MATLAB_R2014b.app/bin/maci64/libmwocv.dylib
  Expected in: /usr/local/Cellar/opencv/2.4.11/lib/libopencv_ml.dylib
 in /Applications/MATLAB_R2014b.app/bin/maci64/libmwocv.dylib.
Caught MathWorks::System::FatalException

License

Thanks for the nice work, I am wondering what kind of license it takes?

Datasets site is offline.

http://ibug.doc.ic.ac.uk/resources/facial-point-annotations

Could you re-upload those files on other place?
Or just add to this repo.

train function in globalregression.m

Hi,

recently I tried to run the train_model code. But I 've got an error in the line 47 of gloablregression.m, saying 'could not find train function with input format'.

I've searched the source code but did not find any train function implementation. So I guess you may use train in NN toolbox, but the input format seems wrong.

Could you please tell me how I can fix the problem?

Also, could you please release one good model trained with this code? I want to use it for testing.

Thank you very much.

A problem about "img_gray"

if size(img_region, 3) == 1
Data{i}.img_gray = img_region;
else
Data{i}.img_gray = rgb2gray(img_region);
end
problem:You store all the sample data, the data is too large,which will cause be out of memory?

scale invariant "shapes_residual"

Thanks for your great work, help me through the paper. But there is still one problem confuse me.
In matlab code, i found you transform the intermediate's "shapes_residual" to the "meanshape_resize" coordinate system, while the latter's bounding box is exactly the same as intermediate's shape. Which means the "shapes_residual" would depends on the image resolution, e.g. same image, if i double its size, the "shapes_residual" would be doubled, and the binary feature still the same. Would this affect the global regression? Why not just transform to the unique param's "meanshape"?

the result in LFPW and Helen

first of all thank you for your code ! it works well !
but i notice that when i run it on LFPW, the err is much higher than the paper(5.17/3.35). but on Helen, the result is nearly the same . how do you think about this issue?
the second one i'm confusion is that for helen (194 landmakers), the paper use total 300 trees, averaged less than 2 for each maker. how it comes??? it seems like only decision tree instead random forest~~~~(>_<)~~~~

looking forward to your reply~~~
thanks again!

Training using Matlab face detector or voila jones algorithm.

Greeting....I have been trying to train the model using matlab's face detector.....but i have not been able to achieve anything. I see that in you loadsamples.m you have given an option for using cascade classifier but some how i cant understand how to use it. Also I coudnt understand the difference between data{}.bbox_gt and data{}.bbox_facedet.

Thank you in advance .

A problem about textread.m

Hi, I runned this code on matlab. ‘train_model.m’ works good, but running ‘test_model.m’ shows errors below:

>> test_model
Already initialized
错误使用 textread (line 165)
未找到文件(could not find files)。

出错(error) loadsamples (line 10)
imgpathlist = textread(imgpathlistfile, '%s', 'delimiter', '\n');

出错(error) test_model (line 36)
    te_data = loadsamples(imgpathlistfile, 1);

'textread.m' might be the problem,I can't find out. Can you help me?
Appreciate you.

About the speed of Liblinear SVM 'train'

Hi, thanks for your excellent codes. I got good performance when I use a set of 10000 pictures, with 5 landmarks. However, when I increase the landmarks to 9, the training of SVM is too slow -- after 'learn global regressors...', it can hardly go on.
So would you like to give me some advice on it?

A problem about "draw.m"?

In globalpredicition.m,"drawshapes(Te_Data{k}.img_gray, [Te_Data{k}.shape_gt Te_Data{k}.intermediate_shapes{1}(:,:, s) shape_newstage]);"
Te_Data{k}.shape_gt ,if it not exists,how to get the landmarks points?

Problem in split in node

Dear @jwyang
I downloaded the program and tested.
It works good.

But, I found a problem in the function splitnode() in train_randomfs.m

There's code to get image index for the left and right child nodes:

lcind = find(pdfeats(ind_colmax(ind_max), :) < thresh);
rcind = find(pdfeats(ind_colmax(ind_max), :) >= thresh);

As the node goes deep, lcind and rcind only refer the front index of training images.

I think it should be changed as

lcind = ind_samples(find(pdfeats(ind_colmax(ind_max), :) < thresh));
rcind = ind_samples(find(pdfeats(ind_colmax(ind_max), :) >= thresh));

Then, lcind and rcind come from original image index.

Please check it.

Undefined function 'train' corresponding to the input parameter of 'double' type.

The problem is"Undefined function 'train' corresponding to the input parameter of 'double' type".
The corresponding code is" model = train(deltashapes(:, o), sparse(binaryfeatures), param);"
my environment is 'matlab 2017a' . it seems that the function named 'train' is not the same one as the function named 'train' of the software.which is used to train BP network.

I am looking forward to you replay!
best wishes to you!

Requires bounding box to fit face

It looks like the points fitted based on bounding box size. The bounding box is require to cover perfectly on face. Since the codes used the groundtruth bounding box, it will be no problem for that. However, dataset with no groundtruth is the problem. Any idea for this? Or I missing something in the codes?

About performence

Hi,
Base on your matlab code, i have finally finished the c++ implement (with some modifications). But with the configuration define on your code, i.e. 68 landmarks, 6 tree depth, 10 trees per forest, 5 stages, i could hardly achieve 1000 FPS, actually 350~450 FPS only.
I notice that when generating the binary features, the binary tree traverse is quite time consuming, even longer than the global prediction (which is almost vector summation, can be optimized by avx instructions).
Btw, my laptop use cpu I7-3820QM 2.7GHz, all test is running on a single core. It would be grateful if you can share some advice.

about meanshape

hi. first of all, thanks for your code, it works well . now i want to do some change on it.
i want to change the meanshape to my own shape(each pic have their own shape ). but there are always some unknown problems .i'm thinking about if it is the problem of coordinate. so i want to ask you how your meanshape come from(what is the file used in calc_meanshape.m )? what else should i pay attention to if i want to do this change?

how to realise face alignment?

Recently,I have been read your code!The code has realise 68 feature landmarks alignment,but the whole face don't follow the 68 feature landmarks aligement to realise face alignment.How to realise it?And I found rotateshape function,And how to use the rotate function in the code? I found the globalprediction.m function don't have the rotateshape function,Whether to need the function?And how to use it?
I need some help!Thanks very much!

Radius & liblinear parameter C

Hi,

Thanks for your work ! Here are my questions :

• How did you determine "params.max_raio_radius" values? Did you do a cross-validation at some point ?
• How did you find that the optimal cost parameter C for liblinear regression is 1/(size(binaryfeatures, 1) ? Is it optimal for any dataset ?

Thanks

I hope to disscuss about "how to improve the efficient of the code?"

3000FPS is mentioned in the paper.But I run the code to test the efficient.The code deal with 224 images needing 5 minuters.I feel the efficient cann't meet our need.I feel training random forest need a long time.But I don't know how to improve it. What's more, the code run needs what conditions to the hardware? Meanwhile, I hope you can give me some directions. Thanks very much!

Binary feature dimension problem

I am trying to run the code using the trained model.
But I get N X 5429 binary features (N is the number of my images) after I run the first stage:
binfeatures = derivebinaryfeat(randf{min(s, params.max_numstage)}, Data, Param, min(s, params.max_numstage));

From the model i get Ws of the first stage is 10845X136. Apparently they do not match thus I could not get it through?

Could you please give me some hints on this issue?

A problem about "W_liblinear1(:, o) = model.w;"

"Length of label vector does not match # of instances.
Attempt to reference field of non-structure array.

Error in globalregression (line 58)
W_liblinear1(:, o) = model.w';"

I hope to get some suggestions!Thanks very much!

训练模型运行出错

您好，我想请教您两个问题：

1. 依赖库##Dependencies必须要下载吗？我运行train_model.m文件时会跳到globalregression.m函数中的model = train(deltashapes(:, o), sparse(binaryfeatures), param)这一行，出现错误“未定义与 'double' 类型的输入参数相对应的函数 'train'。”
   其中，我已经将 parpool(4)命令注释掉了
   2.该检测算法适合手掌关键点检测吗？

About the best model

Hi, may I ask what's your training set when training the model downloaded from http://pan.baidu.com/s/1i325Rbn and are the corresponding parameters setting in the file 'models/config_te_best.txt'?

i want to know where is the function ‘enlargingbbox’ defined in ‘src/loadsamples.m’？

i want to know where is the function ‘enlargingbbox’ defined in ‘src/loadsamples.m’？
THANKS

21 landmark dataset

Hi,
could you please tell me how to train a model in 21 landmark , download a 21 landmark dataset or just use a 68 landmark dataset and change the global parameter landmark number to 21?

The globalregression doesn't work

globalregression(binfeatures, Data, Param, s)
Error using globalregression (line 52)
An UndefinedFunction error was thrown on the workers for 'train'. This might be because the file containing 'train' is not accessible on the
workers. Use addAttachedFiles(pool, files) to specify the required files to be attached. See the documentation for
'parallel.Pool/addAttachedFiles' for more details.
Caused by:
Undefined function 'train' for input arguments of type 'double'.

And I checked the Error position in file globalregression.m:

model = train(deltashapes(:, o), sparse(binaryfeatures), param);

As far as I know, the function 'train' in the neural network toolbox should have the first param as a network, but here 'deltashapes(:, o)' is a vector with double values.

Please help me figure out this problem, and thank you very much!

best config for train and test

Hi
Thanks for you work.
I have tested your code and it works fine.
I also downloaded your the LBFRegModel_best.mat file.
It seem like the structure is quit different for the config_tr and config_te file.

And also I read from the original paper that the tree number in each forest is 1200 or 300(fast).
Even the LBFRegModel_best.mat I see there is only 10 trees.

Any hint?

Thanks in advance.

Face Tracking?

I tried your program for face tracking.

First, I assumed that one single image is given continuously like a video.
At the first frame, the alignment was successful because it starts with the mean shape.

But the problem happens from the second frame.
After the first frame, current shape is almost close to the ground truth.
The first stage of regressor is trained with the mean shape and augmented data.
Then, the result of regressor contains large displacement from the ground truth.
Therefore, At the second frame, the shape moves largely by first stage of regressor.
As the frame goes, shape diverges smoothly.

Somebody suggests to start with the mean shape at each frame.
But it sometimes fails if face is rotated largely.

Therefore, I think starting with the previous frame's shape is the best way.

Do you have any experience of idea about this problem of face tracking?

A problem about test our own face images

Now, the code is normal running.But I found your databases include the file" .pts".Because our databases don't have the file ".pts",Can I use your test_model.m to realise face-alignment? If poosible,how to realise ?
Thanks very much!

C++ implementation of your code

Greetings
I have implemented code for testing in C++ opencv. Results are as good as your matlab implementation. My concern is that it consumes around 0.5sec per image in C++. I have used parallel programming at any step possible(for me). Can you please suggest some optimizations to make it fast?

PS :- I have also removed concatination of feats, cnodes from real time. I have concatinated then before i load the image. This resulted in decrease in computation time in real time but not as much i was expecting.

Thank You.

A problem about "region radius"

params.radius = ([0:1/30:1]');
params.angles = 2_pi_[0:1/36:1]';

if we don't use polar coordinates but rectangular coordinate to solve the region radius,how we can do it

Parallel processing not working

Dear Yang;
First of all thank you for the code. I have been trying to get your code to work for some days now but all my attempts to start parpool have failed. I have googled and tried everything. As this the error encountered by a number of Matlab Users, would you recommend running the code without parallel processing?

The information about my System is.
Core m y510c CPU @ .8GHz 1GHz.
Win 8.1
MATLAB Version: 8.3.0.532 (R2014a)

Error is:

Error using parpool (line 111)
Failed to start a parallel pool. (For information in addition to the causing error, validate the profile 'LocalProfile1' in the
Cluster Profile Manager.)

Error in train_model (line 31)
parpool(4);

Caused by:
Error using parallel.internal.pool.InteractiveClient/start (line 330)
Failed to start pool.
Error using parallel.Job/submit (line 304)
All dimension arguments must be greater than zero

Could not LoadSamples

Hello @jwyang ,
I'm trying to run your matlab code on my machine now. But I met some issue when I tried to load the image in.
in the test_model file, after I ran the code "te_data = loadsamples(imgpathlistfile, 1);"
the data returned is always "Empty cell array: 0-by-1";
Could you please provide me some information on it?
Thank you

Training a new model

Hi,

Thank you for the code! It has been very useful to me. But I have run into a snag and I was hoping you can help.

I have been trying to train my own model with your code. I have a dataset of around 65000 images with 49 landmark points for each image.

While setting the parameters, I set "global_params.landmark_num" in InitializeGlobalParam() to 49. But the execution always crashes while loading the images at around the 16000 image mark or 30000 mark with the error - OpenCV Out of Memory exception.

However, if I leave the landmark_num to 68 the training completes successfully. But then if I try to Test the model or run the Demo mode, the program always crashes (which I think is what should happen since the global parameter claims there are 68 points but in reality there are only 49).

It would be really good if you could help me out a little.

A problem about "Test time consumption"

As I haven't done face alignment before,so I don't know where is the start-stop of test time consumption, could you mark it in the code of test_model with tic and toc?

运行自己的图片，缺少.pts

非常好的分享！感谢！
我成功地在你们提供的图片库中运行成功了，结果很好。
但是目前我想用自己的图片试试，而该图片缺少.pts文件，参考了之前的提问，我选择用matlab自带的工具箱来圈出脸部边界。至于landmarks(也就是shape_gt变量)，我是随机赋值的。结果发现shape_gt没有被更新，也就是没有程序没有找到landmarks，想请教一下这是什么原因呢，该怎么解决呀？
我的想法是shape_gt这个变量是标记点的坐标，应该会被模型重新打上标记，所以我随机初始化了，这是为了匹配上程序的输入数据的格式，不知道是不是我这样的想法有问题。还望赐教！

A problem about "Ts_bbox.mat"

load('Ts_bbox.mat').Is the file of "Ts_bbox.mat" lost? Or how to create "Ts_bbox.mat"?

About testing the face alignment without ground truth *.pts files

Hi, your code has been very helpful to me! Thank you for your contribution!
I have got your code running with afw dataset and the result is good.
However, I meet some problems when I try to do face alignment with my own images which do not have the ground truth .pts files.
I read through your code, and try to replace bbox_facedet with viola jones detector result. But the shape_gt variable seems confusing to me.
Could you tell me how can I modify your code to realise face alignment with my own images?