susantabiswas / facerecog

Realtime Facial recognition system using Siamese neural network

Facial Recognition System

This face recognition library is built with ease and customization in mind. There are numerous control parameters to control how you want to use the features, be it face detection, face recognition on videos, or with a webcam.
At its core, the facial recognition system uses Siamese Neural network. Over the years there have been different architectures published and implemented. The library uses dlib's face recognition model, which is inspired from ResNet-34 network. The modified ResNet-34 has 29 Convolutional layers. The model achieved 99.38% accuracy on LFW dataset.

There are 4 different face detectors for usage. Wrappers for video and webcam processing are provided for convenience.

Table of Contents

• Facial Recognition System
  • Table of Contents
• Sample Output
  • Processed Video
  • Processed Images
• Architecture
• Setup
  • Clone the repo and install dependencies.
  • Docker Image
  • Dockerfile
• Project Structure
• Usage
  • Face Recognition
• References

Sample Output

Processed Video

Processed Images

Architecture

For face recognition, flow is:

media -> frame -> face detection -> Facial ROI -> Neural Network -> 128D facial encoding 

These are the major components:

1. Face Detection: There are 4 different face detectors with different cropping options.
2. Face Recognition: Responsible for handling facial recognition related functionalities like registering facial data etc.
3. Storage: The system provides abstract definitions of cache and persistent storage. For usage, a simple cache using python's native data structure is provided along side a persistent storage system with JSON. If needed the abstract classes can be extended to integrate better storage systems.
4. Utilities: Methods for handling image, video operations, validations, etc.

Setup

There are multiple ways to set this up.

Clone the repo and install dependencies.

git clone https://github.com/susantabiswas/FaceRecog.git
pip install -r requirements.txt

Docker Image

You can pull the docker image for this project and run the code there.
docker pull susantabiswas/face_recog:latest

Dockerfile

You can build the docker image from the docker file present in the repo.

docker build -t <name> .

Project Structure

FaceRecog/
├── Dockerfile
├── README.md
├── data/
├── docs/
├── face_recog/
│   ├── exceptions.py
│   ├── face_data_store.py
│   ├── face_detection_dlib.py
│   ├── face_detection_mtcnn.py
│   ├── face_detection_opencv.py
│   ├── face_detector.py
│   ├── face_recognition.py
│   ├── in_memory_cache.py
│   ├── json_persistent_storage.py
│   ├── logger.py
│   ├── media_utils.py
│   ├── persistent_storage.py
│   ├── simple_cache.py
│   └── validators.py
├── models/
│   ├── dlib_face_recognition_resnet_model_v1.dat
│   ├── mmod_human_face_detector.dat
│   ├── opencv_face_detector.pbtxt
│   ├── opencv_face_detector_uint8.pb
│   └── shape_predictor_5_face_landmarks.dat
├── requirements.txt
├── tests/
│   ├── conftest.py
│   ├── test_face_data_store.py
│   ├── test_face_detection_dlib.py
│   ├── test_face_detection_mtcnn.py
│   ├── test_face_detection_opencv.py
│   ├── test_face_recognition.py
│   ├── test_json_persistent_storage.py
│   ├── test_media_utils.py
│   └── test_simple_cache.py
└── video_main.py

Usage

Face Recognition

Depending on the use case, whether to aim for accuracy and stability or speed etc., you can pick the face detector. Also, there are customization options inside face detectors to decide the facial ROI.

To register a face using a webcam

# Inside project root
import video_main

# You can pick a face detector depending on Acc/speed requirements
face_recognizer = FaceRecognitionVideo(face_detector='dlib')
face_recognizer.register_face_webcam(name="Susanta")

To register a face using an image on disk

# Inside project root
import video_main

face_recognizer = FaceRecognitionVideo(face_detector='dlib')
face_recognizer.register_face_path(img_path='data/sample/conan.jpg', name="Conan")

To register a face using a loaded image

# Inside project root
from face_recog.media_utils import load_image_path
from face_recog.face_recognition import FaceRecognition

face_recognizer = FaceRecognition(
                    model_loc="models",
                    persistent_data_loc="data/facial_data.json",
                    face_detector="dlib",
                )
img = load_image_path("data/sample/1.jpg")
# Matches is a list containing information about the matches
# for each of the faces in the image
matches = face_recognizer.register_face(image=img, name=name)

Face recognition with a webcam feed

# Inside project root
import video_main

face_recognizer = FaceRecognitionVideo(face_detector='dlib')
face_recognizer.recognize_face_video(video_path=None, \
                                    detection_interval=2, save_output=True, \
                                    preview=True, resize_scale=0.25)

Face recognition on a video

# Inside project root
import video_main

face_recognizer = FaceRecognitionVideo(face_detector='dlib')
face_recognizer.recognize_face_video(video_path='data/trimmed.mp4', \
                                    detection_interval=2, save_output=True, \
                                    preview=True, resize_scale=0.25)

Face recognition using an image

# Inside project root
from face_recog.media_utils import load_image_path
from face_recog.face_recognition import FaceRecognition

face_recognizer = FaceRecognition(
                    model_loc="models",
                    persistent_data_loc="data/facial_data.json",
                    face_detector="dlib",
                )
img = load_image_path("data/sample/1.jpg")
# Matches is a list containing information about the matches
# for each of the faces in the image
matches = face_recognizer.recognize_faces(
                image=img, threshold=0.6
            )

There are 4 face detectors namely dlib (HOG, MMOD), MTCNN, OpenCV (CNN). All the face detectors are based on a common abstract class and have a common detection interface detect_faces(image).

# import the face detector you want, it follows absolute imports
from face_recog.media_utils import load_image_path
from face_recog.face_detection_dlib import FaceDetectorDlib

face_detector = FaceDetectorDlib(model_type="hog")
# Load the image in RGB format
image = load_image_path("data/sample/1.jpg")
# Returns a list of bounding box coordinates
bboxes = face_detector.detect_faces(image)

References

The awesome work Davis E. King has done: http://dlib.net/cnn_face_detector.py.html, https://github.com/davisking/dlib-models
You can find more about MTCNN from here: https://github.com/ipazc/mtcnn

facerecog's Issues

using images from disk rather then taking from webcam

Error when checking : expected input_1 to have shape (None, 3, 96, 96) but got array with shape (1, 3, 90, 70)
getting this error in add_user_image_path funtion
can u help me to fix it

Value Error

Hi, can you help me, should i resize my phoro before add for user or need scale 96x96?
It's message when just photo
ValueError: Error when checking input: expected input_1 to have shape (3, 96, 96) but got array with shape (3, 648, 464)

But when i resize the photo to 96x96, it added to base but doesn't do face recognition and also have got the error with resize (empty data).

thanks for answer

Error while restoring model

I am getting the following error while restoring the model.
The file for inception_blocks_v2 was also missing and could not be imported. I downloaded it from here and also its dependency fr_utils.py.

ValueError: Shape must be rank 1 but is rank 4 for 'bn1/cond/FusedBatchNorm' (op: 'FusedBatchNorm') with input shapes: [?,64,48,48], [1,64,1,1], [1,64,1,1], [1,64,1,1], [1,64,1,1].
Using Keras 2.2.0, Tensorflow 1.8.0

np.set_printoptions(threshold=np.nan)

ValueError: threshold must be numeric and non-NAN, try sys.maxsize for untruncated representation

No such file or directory: 'database/user_dict.pickle

FileNotFoundError Traceback (most recent call last)
in
1 # add a user
----> 2 add_user_img_path(user_db, FRmodel, "susanta", "images/2.jpg")

in add_user_img_path(user_db, FRmodel, name, img_path)
4 user_db[name] = img_to_encoding(img_path, FRmodel)
5 # save the database
----> 6 with open('database/user_dict.pickle', 'wb') as handle:
7 pickle.dump(user_db, handle, protocol=pickle.HIGHEST_PROTOCOL)
8 print('User ' + name + ' added successfully')

FileNotFoundError: [Errno 2] No such file or directory: 'database/user_dict.pickle'

@susantabiswas

How to handle it ?

Where is the Siamese neural network？

Where is the Siamese neural network？

ValueError while loading model.

ValueError: Shape must be rank 1 but is rank 4 for 'bn1/cond/FusedBatchNorm' (op: 'FusedBatchNorm') with input shapes: [?,64,48,48], [1,64,1,1], [1,64,1,1], [1,64,1,1], [1,64,1,1].
I am using Keras 2.2.0 and Tensorflow 1.8.0. I am not able to load the model and getting an error at the following line.
FRmodel = load_model('models/model.h5', custom_objects={'triplet_loss': triplet_loss}). Is there an updated model file?

LICENSE?

I see that the license was deleted. Can somebody help explain what the licensing or add the correct LICENSE file?