nasir6 / rppg

Robust Heart rate estimation from facial videos

This repo monitors real time cardiac activities of a person through remote photoplethysmography(rPPG) without any physical contact with sensor, by detecing blood volume pulse induced subtle color changes from video stream through webcam sensor or a video file.

Pre Processing

Skin pixels play significant role in extraction of rPPG signal therefore, we trained first ever deep learning model for semantic segmentation of skin and non skin pixels. This is novel technique for region of interst (ROI) selection and tracking. The model is robust to motion, multiple postures and segments skin pixels from non skin very accurately. rPPG signal exhibit different waveform when sampled from different rigions of skin, therefore, to consistently sample ROI from same part of skin we detect face as prerequisite step to semantic segmentation.

rPPG Signal Extraction

After detection and tracking ROI for signal extraction we compute the spatial red, green and blue channel mean of skin segmented pixels to minimise camera quantization error. Averaged values of RGB channel are temporally normalized and projected to plane orthogonal to skin-tone. The projected signal is alpha tuned to extract signal.

Post processing

We apply signal processing techniques, moving average filter of order 6 to remove outliers from signal. To estimate heart rate we compute power spectral density PSD applying fast fourier transformation (FFT) on rPPG signal. It is then band pass filtered to analyse only frequencies of interest. The maximum power spectrum represents the frequency of instant heart rate.

This code runs on cuda enabled device at 30 FPS and estimates heartbeat in one second intervel.

Pipeline

Requirements

• Python 3

• Numpy

• Pytorch

• OpenCv

• Matplotlib, Scipy, Pillow

• Git Lfs to track trained model parameters or alternatively download the model from google drive

• We have used deep learning for semantic segmentation of skin and non skin pixels from frames. The segmentation requires cuda enabled device

Clone this repository.

    git clone https://github.com/nasir6/rPPG.git

To run

    cd rPPG
    python3 run.py --source=0 --frame-rate=25

rppg's Issues

Open Source License

Can you please kindly add an Open Source MIT License to this project?

_pickle.UnpicklingError: invalid load key, 'v'. on Ubuntu 22.04

Downloading: "https://download.pytorch.org/models/resnet34-b627a593.pth" to /home/snowuyl/.cache/torch/hub/checkpoints/resnet34-b627a593.pth
100.0%
Traceback (most recent call last):
File "/home/snowuyl/samba/workspace_Python/rPPG/run.py", line 65, in
runPOS(source)
File "/home/snowuyl/samba/workspace_Python/rPPG/run.py", line 39, in call
capture = CaptureFrames(self.batch_size, source, show_mask=True)
File "/home/snowuyl/samba/workspace_Python/rPPG/capture_frames.py", line 20, in init
self.model.load_state_dict(torch.load('linknet.pth'))
File "/home/snowuyl/.local/lib/python3.10/site-packages/torch/serialization.py", line 713, in load
return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
File "/home/snowuyl/.local/lib/python3.10/site-packages/torch/serialization.py", line 920, in _legacy_load
magic_number = pickle_module.load(f, **pickle_load_args)
_pickle.UnpicklingError: invalid load key, 'v'.

Error

hi, there is an error when I run the code:

(py36) E:\github\rPPG-master\rPPG-master>python run.py --source=0 --frame-rate=25
init
Traceback (most recent call last):
File "run.py", line 66, in
runPOS(source)
File "run.py", line 39, in call
capture = CaptureFrames(self.batch_size, source, show_mask=True)
File "E:\github\rPPG-master\rPPG-master\capture_frames.py", line 20, in init
self.model.load_state_dict(torch.load('linknet.pth'))
File "D:\Anaconda\envs\py36\lib\site-packages\torch\serialization.py", line 585, in load
return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
File "D:\Anaconda\envs\py36\lib\site-packages\torch\serialization.py", line 755, in _legacy_load
magic_number = pickle_module.load(f, **pickle_load_args)
_pickle.UnpicklingError: invalid load key, 'v'.

References

Which paper did you refer to?

some questions

Thank you for the code，but I have some questions：
1.Because my computer does not take a camera, directly recorded a good video processing. Parameter “framefare” is the framerate of the video？？？
2.What is the principle of extracting heart rate? Do you have corresponding references

After run "python3 run.py --source=0 --frame-rate=25", The window flashed 0.1s

Extend data recording

Currently, only records 300 frames. How to extend to an infinite number of frames?

Questions about the pkl file

Kindly add the packages version in requirements.txt file

Can not compute HR with small face

Dear Sir, I am very interested in your work and many thanks for your work. Could I ask a question? It is fine when the face is large. But it can not compute HR when the face is not large. The HR is NaN. Thank you!

linknet.pth

I wonder if you would like to provide the training program for linknet.pth
I want to know how to train extraction, please!

Face anti-spoofing dataset - CASIA & MSU

Problem with HR measures

Hello,
thank you for your work! I'm very interessed on it. Can you help me, please? Everything seems all right, the code run, the mask is created and also the interface with graphs appears but the plot is always flat and the HR is not recorded (it is always NaN).

Face Segmentation Dataset

Hi Nasir,

Is that possible for you to share your face segmentation dataset.

Thank you very much,
Donghao

Do you have a paper for this program？

HRV from HR

Thanks for sharing your repo! I am running your codes and want to know how can I get HRV(heart rate variability) from HR. I think we need R-R intervals to calculate HRV, but I don't know how to get it. Do you have any ideas?

Segmentation fault (core dumped)

when runing this:
python3 run.py --source=0 --frame-rate=25

got this error:
Segmentation fault (core dumped)

any ideas?

It seems the code just gives a face seg result?

I test the code, and return the unstable face seg result?

Unable to download model files from the link

Whenever I use the google drive link to download the model, I get a 502 error, can you maybe email me the model files?
[email protected]

or check the link health?

How to bring HR value to display on the live frames

In Plot_cont.py file, we have update_data() to update the data and set HR value .. I was trying to pass this HR value into Capture_Frames class to display on to the frames.

As we have self.plotter = DynamicPlot(self.signal_size, self.batch_size) this line in run.py where i will get the object of DynamicPlot class. using this object was trying to call self.hr_texts in plot_cont.py file
hr_text = 'HR: ' + str(int(hr_fft))
self.hr_texts.set_text(hr_text)

Am new to this multiprocessing code, Please help me in this.

Some question about project

Hi dear,
1- You mention this works with skin pixels, But If one who has dark skin how this system is able to detect the Hear rate? This system is robust for white skins when color of skins be red this system recognition the one who has high heart rate, right?

2- Is it system robust to light conditions?

Error - "_pickle.UnpicklingError: invalid load key"

I got the following error message while running the program. Also, 'linknet.pth' is present the folder.
PS: I'm not using a CUDA enabled device and i tried to use map_location='cpu' argument in torch.load function.

Thanks!
Davi

Traceback (most recent call last):
  File "run.py", line 65, in <module>
    runPOS(source)
  File "run.py", line 39, in __call__
    capture = CaptureFrames(self.batch_size, source, show_mask=True)
  File "/home/davi/Dropbox/rPPG/nasir6_rPPG/capture_frames.py", line 20, in __init__
    self.model.load_state_dict(torch.load('linknet.pth'))
  File "/home/davi/.cv/OpenCV-3.4-py3/lib/python3.6/site-packages/torch/serialization.py", line 386, in load
    return _load(f, map_location, pickle_module, **pickle_load_args)
  File "/home/davi/.cv/OpenCV-3.4-py3/lib/python3.6/site-packages/torch/serialization.py", line 563, in _load
    magic_number = pickle_module.load(f, **pickle_load_args)
_pickle.UnpicklingError: invalid load key, 'v'.

can't able to download pth model using git-lfs

Git LFS: (0 of 0 files, 1 skipped) 0 B / 0 B, 83.27 MB skipped [f18ca579d42765d26bfdeb1a9833826a78ef42e61cb2974b6263c18d21fc8f84] Object does not exist on the server: [404] Object does not exist on the server
error: failed to fetch some objects from 'https://github.com/nasir6/rPPG.git/info/lfs'

TypeError: can't multiply sequence by non-int of type 'float'

The problem is that the framerate was seen as string.
I have added the cast to float and it works.

Model performance metrics

Thanks for the open-sourced repository. From previous GitHub issues I understand that there is no paper published but I would like to find out if there are performance metrics for the rPPG model for us users to learn about its robustness.

Could you please send me the pretrain model 'linknet.pth'?

Sorry for asking this request. but I could't download the pretrain model 'linknet.pth' from your repository. If it's quite all right, please send this file to [email protected]. Thanks very much.

Don't Read torch.load( ) with CPU

My environment is python3.8 with MacBookAir(M1) .

I download the source code and run on terminal ' python run.py --source=0 --frame-rate25 '

I got this error message.

init
Traceback (most recent call last):
File "run.py", line 65, in
runPOS(source)
File "run.py", line 39, in call
capture = CaptureFrames(self.batch_size, source, show_mask=True)
File "/Users/jaehyuni/Desktop/rPPG-master/rPPG/capture_frames.py", line 21, in init
self.model.load_state_dict(torch.load('linknet.pth'), map_location='cpu')
File "/opt/homebrew/Caskroom/miniforge/base/envs/yolo/lib/python3.8/site-packages/torch/serialization.py", line 608, in load
return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
File "/opt/homebrew/Caskroom/miniforge/base/envs/yolo/lib/python3.8/site-packages/torch/serialization.py", line 787, in _legacy_load
result = unpickler.load()
File "/opt/homebrew/Caskroom/miniforge/base/envs/yolo/lib/python3.8/site-packages/torch/serialization.py", line 743, in persistent_load
deserialized_objects[root_key] = restore_location(obj, location)
File "/opt/homebrew/Caskroom/miniforge/base/envs/yolo/lib/python3.8/site-packages/torch/serialization.py", line 175, in default_restore_location
result = fn(storage, location)
File "/opt/homebrew/Caskroom/miniforge/base/envs/yolo/lib/python3.8/site-packages/torch/serialization.py", line 151, in _cuda_deserialize
device = validate_cuda_device(location)
File "/opt/homebrew/Caskroom/miniforge/base/envs/yolo/lib/python3.8/site-packages/torch/serialization.py", line 135, in validate_cuda_device
raise RuntimeError('Attempting to deserialize object on a CUDA '
RuntimeError: Attempting to deserialize object on a CUDA device but torch.cuda.is_available() is False. If you are running on a CPU-only machine, please use torch.load with map_location=torch.device('cpu') to map your storages to the CPU.

So, I entered capture_frames.py .

edit
self.model.load_state_dict(torch.load('linknet.pth'), map_location='cpu')

Still the same error occurs.

Help me plz!

After run "python3 run.py --source=0 --frame-rate=25"

Thank you very much for exposing the code.
When I run "python3 run.py --source=0 --frame-rate=25", I have changed the source to int, but it still appears:"init
time 3.2095234394073486“.

What paper is this based on?

Hi, thank you for your code. What paper is your work based on?