yinguobing / face-mesh-generator

Generate face mesh dataset using Google's FaceMesh model.

face-mesh-generator

Generate face mesh dataset using Google's FaceMesh model from annotated face datasets.

Watch this 30s video demo:

.

Features

There are built in features to help generating the dataset more efficiently.

• Automatically centralize the marked face.
• Rotate the image to align the face horizontally.
• Crop the face with custom scale range.
• Generate mark heatmaps.
• Write TensorFlow Record files, or export the processed image and marks.
• Support multiple public datasets. Check the full list here

Getting Started

These instructions will get you a copy of the project up and running on your local machine for development and testing purposes.

Prerequisites

Installing

First clone this repo.

# From your favorite development directory
git clone https://github.com/yinguobing/face-mesh-generator.git

Then download Google's FaceMesh tflite model and put it in the assets directory.

Model link: https://github.com/google/mediapipe/blob/master/mediapipe/modules/face_landmark/face_landmark.tflite

How to run

Take WFLW as an example. Download the dataset files from the official website. Extract all files to one directory.

First, Construct the dataset.

ds_wflw = fmd.wflw.WFLW("wflw")
ds_wflw.populate_dataset(wflw_dir)

wflw_dir is the directory for the extracted files.

Then, process the dataset.

process(ds_wflw)

There is a demo file generate_mesh_dataset.py demonstrating how to generate face mesh data and save them in a TFRecord file. Please refer to it for more details.

Authors

Yin Guobing (尹国冰) - yinguobing

License

Acknowledgments

All the authors who made their datasets and model public.

face-mesh-generator's Issues

碰到这样的错误

我是在facial-landmark-detection-hrnet项目中转过来的。我看facial-landmark-detection-hrnet的训练需要json文件，应该是通过本项目生成样本，但是我将wflw从官方下载解压后运行generate_mesh_dataset.py后，碰到如下问题：

D:\Anaconda3\envs\tf2.3\python.exe D:/Facial-Landmark-Detection/reference/facial-landmark-detection-hrnet-master/face-mesh-generator/generate_mesh_dataset.py
2020-12-28 19:23:22,786 - main - INFO - Starting to process dataset: wflw_train
Traceback (most recent call last):
File "D:/Facial-Landmark-Detection/reference/facial-landmark-detection-hrnet-master/face-mesh-generator/generate_mesh_dataset.py", line 341, in
process(ds_wflw)
File "D:/Facial-Landmark-Detection/reference/facial-landmark-detection-hrnet-master/face-mesh-generator/generate_mesh_dataset.py", line 82, in process
tf_writer = MeshRecordOperator(
File "D:\Facial-Landmark-Detection\reference\facial-landmark-detection-hrnet-master\face-mesh-generator\tfrecord_operator.py", line 44, in init
self._writer = tf.io.TFRecordWriter(filename)
File "D:\envs\tf2.3\lib\site-packages\tensorflow\python\lib\io\tf_record.py", line 298, in init
super(TFRecordWriter, self).init(
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd5 in position 70: invalid continuation byte
由此导致无法训练模型进行下一步工作，大佬可以指教一下吗

When I rungenerate_mesh_dataset.py A record file of 0 bytes is generated in an instant

2021-02-21 22:49:19,277 - __main__ - INFO - Starting to process dataset: 300vw
0it [00:00, ?it/s]
2021-02-21 22:49:19,290 - __main__ - INFO - Dataset done. Processed samples: 0, invalid samples: 0

When I rungenerate_mesh_dataset.py A record file of 0 bytes is generated in an instant

iris tracking using mediapipe

Hello,

I have recently find this library: https://blog.tensorflow.org/2020/11/iris-landmark-tracking-in-browser-with-MediaPipe-and-TensorFlowJS.html
and my concern is to use its updated version, which does the iris detection (in the webpage, the second GIF, the comparison of the older and newer version), I want to use the right side one because:

1. the model works more robust in this version
2. there is no shaking in face mesh
3. no facemesh disappearing during extreme head poses
4. does iris detection
5. it shows how eyes are closed or open via eye region landmarks, when eyes are closed the eye landmarks come very close to each other to show that eyelids are closed.

But this library is for JavaScript which I don't know that language and of course I can't install and use it in my python scripts. I am looking for its python version or a converter to convert it from JS to python.

Would you please help/guide me through this issue?
Does your model give me what I am looking for?

Thank you

Got these errors while working with WFLW datasets.

2020-11-04 01:07:11,584 - main - INFO - Starting to process dataset: wflw
0%| | 0/7500 [00:00<?, ?it/s]2020-11-04 01:07:11.703861: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN)to use the following CPU instructions in performance-critical operations: AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2020-11-04 01:07:11.753022: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x21557b03060 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-11-04 01:07:11.814474: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
0%| | 0/7500 [00:00<?, ?it/s]
2020-11-04 01:07:11,958 - main - ERROR - Error Traceback (most recent call last):
File "c:/Users/Admin/Desktop/New folder/face-mesh-generator/generate_mesh_dataset.py", line 142, in process
example = tf_writer.make_example(
File "c:\Users\Admin\Desktop\New folder\face-mesh-generator\mesh_record_operator.py", line 27, in make_example
'label/score': float_feature(score)
File "c:\Users\Admin\Desktop\New folder\face-mesh-generator\tfrecord_operator.py", line 26, in float_feature
return tf.train.Feature(float_list=tf.train.FloatList(value=[value]))
File "C:\Python38\lib\site-packages\google\protobuf\internal\python_message.py", line 542, in init
copy.extend(field_value)
File "C:\Python38\lib\site-packages\google\protobuf\internal\containers.py", line 282, in extend
new_values = [self._type_checker.CheckValue(elem) for elem in elem_seq_iter]
File "C:\Python38\lib\site-packages\google\protobuf\internal\containers.py", line 282, in
new_values = [self._type_checker.CheckValue(elem) for elem in elem_seq_iter]
File "C:\Python38\lib\site-packages\google\protobuf\internal\type_checkers.py", line 285, in CheckValue
raise TypeError(message)
TypeError: array([[[34.125233]]], dtype=float32) has type <class 'numpy.ndarray'>, but expected one of: numbers.Real
. sample index: 0
2020-11-04 01:07:12,142 - main - INFO - Dataset done. Processed samples: 1, invalid samples: 0

augmented dataset

hello author,

i have some queries regarding your implementation.

1. i see that it adds extra keypoints to the existing dataset. how can i use the original landmarks.
2. the tf record genearted does augmentation? how many augment samples?
3. how to generate train and test split tfrecords?
   thank you

A question

How Do I get a face mesh points as much as the video showing?

Error: tfrecord/helen.record; No such file or directory

I am getting below error when trying to process helen dataset.

tensorflow.python.framework.errors_impl.NotFoundError: tfrecord/helen.record; No such file or directory

Any idea how to resolve? Saw couple of others got similar issue. Is this a pending problem

Python 3.8
TF 2.7

Does the code use the google model？

I read the code and I do not find the model file. When I run the code, it return nothing.

NotFoundError: tfrecord/wflw.record

Hello there! While generating the training data with python3 generate_mesh_dataset.py getting the error above. Should not it itself generate that file?
compat.as_bytes(path), options._as_record_writer_options())
tensorflow.python.framework.errors_impl.NotFoundError: tfrecord/wflw.record; No such file or directory

No longer able to download the model

Error 404 page not found when trying to access
Model link: https://github.com/google/mediapipe/blob/master/mediapipe/models/face_landmark.tflite

tfrecord/ibug.record; No such file or directory

where or how to get ibug.record?

(venv) ➜  face-mesh-generator git:(master) ✗ python generate_mesh_dataset.py
2021-09-24 19:48:59,643 - __main__ - INFO - Starting to process dataset: ibug
Traceback (most recent call last):
  File "generate_mesh_dataset.py", line 335, in <module>
    process(ds_ibug)
  File "generate_mesh_dataset.py", line 81, in process
    tf_writer = MeshRecordOperator(
  File "/Users/users/Documents/_ai/face-mesh-generator/tfrecord_operator.py", line 44, in __init__
    self._writer = tf.io.TFRecordWriter(filename)
  File "/Users/users/Documents/_ai/face-mesh-generator/venv/lib/python3.8/site-packages/tensorflow/python/lib/io/tf_record.py", line 298, in __init__
    super(TFRecordWriter, self).__init__(
tensorflow.python.framework.errors_impl.NotFoundError: tfrecord/ibug.record; No such file or directory

ValueError: Could not open 'assets/face_landmark.tflite'.

(venv) ➜ face-mesh-generator git:(master) ✗ python generate_mesh_dataset.py
2021-09-24 19:46:31,683 - main - INFO - Starting to process dataset: ibug
Traceback (most recent call last):
File "generate_mesh_dataset.py", line 335, in
process(ds_ibug)
File "generate_mesh_dataset.py", line 78, in process
md = MeshDetector("assets/face_landmark.tflite")
File "/Users/users/Documents/_ai/face-mesh-generator/mesh_detector.py", line 16, in init
self.interpreter = tf.lite.Interpreter(model_path=model_path)
File "/Users/users/Documents/_ai/face-mesh-generator/venv/lib/python3.8/site-packages/tensorflow/lite/python/interpreter.py", line 365, in init
_interpreter_wrapper.CreateWrapperFromFile(
ValueError: Could not open 'assets/face_landmark.tflite'.

Unable to generate processed samples of WFLW dataset

Hi,
I need to generate processed samples from WFLW dataset. When I executed the code, it generates tfrecord/wflw.record file but with no contents. Success message is also generated as Dataset is done. Please find the snapshot of the log. Please help.

Uncompressed files are in the same folder as below:
├── WFLW_annotations
└── WFLW_images

tfrecord/wflw.record ; No such file or directory

I ran the generate_mesh_dataset.py and the path of the WFLW dataset was set to WFLW_images path. The directory is
wflw
--- WFLW_annotations
--- WFLW_images
But then also I am running into an error when I am running generate_mesh_dataset.py I am getting the error tfrecord/wflw.record ; No such file or directory. I have attached an image below of the error

Cannot generate dataset with python generate_mesh_dataset.py due to missing file

Traceback (most recent call last):
File "generate_mesh_dataset.py", line 327, in
process(ds_afw)
File "generate_mesh_dataset.py", line 78, in process
md = MeshDetector("assets/face_landmark.tflite")
File "/content/drive/MyDrive/Colab Notebooks/landmark/face-mesh-generator/mesh_detector.py", line 16, in init
self.interpreter = tf.lite.Interpreter(model_path=model_path)
File "/usr/local/lib/python3.7/dist-packages/tensorflow/lite/python/interpreter.py", line 458, in init
custom_op_registerers_by_func, experimental_preserve_all_tensors))
ValueError: Could not open 'assets/face_landmark.tflite'.

Where can I get 'assets/face_landmark.tflite'? Thanks!