prouast / heartbeat Goto Github PK

I somehow managed to compile the code through mingw. Now This is what i am getting after executing the exe file. I have updated .net framework and visual c++ but still cant get rid of it. Is there something missing in the project code?
Is it because the code was written for unix but i am running it on windows?

Hi!
When I try to compile, I get bunch of issues related to opencv2. such as:

fatal error: opencv2/core/core.hpp: No such file or directory
#include <opencv2/core/core.hpp>

Any thoughts?

Hi @prouast,

First all of all, thanks for the great work. I wanted to ask two things specifically:

1. Is this code base still relevant with respect to the current SOTA in rPPG estimation? If not, which are the improvements that you think should be tackled with the highest priority at its current state?
2. How robust to motion is your implementation?
3. How challenging would it be to port it all to Python? I gave a quick glance at the repository, and was debating which path would be the easiest between re-implementing your whole logic in Python or wrapping your C++ code with Python binders.

As you may have already assumed, I would need this code to talk to a larger Python module.

First of all, thank you for the amazing research done @prouast. I was wondering whether the FPS affects the accuracy of the pulse count. I am getting ~6.5 FPS on my PC. I saw on your youtube video that you were getting constant 30FPS. Also my FPS count was variable and not locked. If there is a way to lock FPS let me know.

Hello. I am trying to understand the rescanning portion of your code here:

How is this condition satisfied and what is its purpose? I am assuming time is incremented by 1, so how is time-lastScanTime' never != 1? Also, it looks like timeBase is set to .001, so how can (time - lastScanTime) * timeBase` ever be >= 1/reScanFrequency? From my understanding, this results in .001 >= 1, which is never true. Please let me know what I am missing.

Thanks!

Hi,
I can compile and run the Heartbeat demo now. But I see the bpm value is varying all the time. I think the reason cause this issue is the difference between my test environment and your develop environment. Can you provide the environment requirements for this project (like inside or outside, how much light should project on face etc.).

thank you

Hello. At first, thank you for your amazing work!
I tried to get high heartrate using your code, but I can't achieve it. When I measure heartrate at 30 fps, I could detect high bpm easily , but below 15 fps I couldn't detect high bpm.
I tried to figure out why, so I analyzed raw signal and figured out there is so much fluctuation after workout. Maybe this is the reason why I can't get high bpm at low fps.
However, I watched your youtube video about rppg, and I shocked because you can get high bpm at low fps.
So, I wonder why this happens and how can I get high bpm at low fps. Please let me know.

Thank you for your great work again!

In your current code:

• you define a rectangle on the forehead.
• Then use the 4 corners and do openCV optical flow keypoint tracking to get smoother tracking
• use estimateAffine2D to rectify the tracked area.
• Based on this rectangle you create a mask and analyze only the RGB image based on the mask.

What do you think how accurate the PPG analysis would be if i defined 1000 optical flow keypoints and got the mean RGB value from those 1000 pixel points?
The keypoint tracking ensures that we are reading the same skin pixels throughout consecutive frame. But i don't know if 1000 points is too low or if we need a continuous 2D space.

Thanks for your advice.

Thank you again for posting this code.
I am able to make it run and I find it works great. I was even able to integrate it in to a openframeworks.cc application.

I was hoping you could point me to the right part of the code where I might be able to extract the moment of a peak. My goal is to make the heart beat readings audible by playing a sound every time we get a peak. The hope if that it will align more or less with the person's actual pulse.

Thank you for any advice.

Hi thanks for sharing this project. when I compile and running the heartbeat file showing the below error:

Using algorithm 0.
Face classifier xml not found!

any way to fix ?

thanks

g++ -std=c++11 -LC:\Users\Anon\Desktop\ffmpeg-20171225-613f789-win32-shared\bin -LC:\Anaconda\pkgs\opencv3-3.1.0-py35_0\Library\lib -lopencv_core310 -lopencv_highgui310 -lopencv_imgcodecs310 -lopencv_imgproc310 -lopencv_objdetect310 -lopencv_video310 -lopencv_videoio310 -lavcodec-58 -lavformat-58 -lavutil-56 -lswscale-5 ?

Thank you so much for sharing this create example project.
It works very well as far as I can tell.

You know what would be super nice ;), if your code would integrate OSC (open sound control) as an output option. This would make it so easy for other to grab that data and use it in their own projects. Also it would make it Independent of c++.

Just a thought.

All the best.

• Environment
  MacOS Mojave 10.14.2

• Compiling using:
  g++ -std=c++11 -I/usr/local/Cellar/opencv/4.0.1/include/opencv4 -lopencv_core -lopencv_dnn -lopencv_highgui -lopencv_imgcodecs -lopencv_imgproc -lopencv_objdetect -lopencv_video -lopencv_videoio Heartbeat.cpp opencv.cpp RPPG.cpp Baseline.cpp -o Heartbeat

• Getting following warnings

        Mat transform = estimateRigidTransform(corners_0v, corners_1v, false);
                        ^
/usr/local/Cellar/opencv/4.0.1/include/opencv4/opencv2/video/tracking.hpp:258:1: note: 'estimateRigidTransform' has been explicitly marked deprecated here
CV_DEPRECATED CV_EXPORTS Mat estimateRigidTransform( InputArray src, InputArray dst, bool fullAffine );
^
/usr/local/Cellar/opencv/4.0.1/include/opencv4/opencv2/core/cvdef.h:346:43: note: expanded from macro 'CV_DEPRECATED'
#    define CV_DEPRECATED __attribute__ ((deprecated))

• Running Heartbeat getting following error

Using rPPG algorithm 0.
Using face detection algorithm 0.
Face classifier xml not found!

[Process completed]

• Even though 'haarcascade_frontalface_alt.xml' file is located in the same directory. Any guidance is appreciated.

Hi, I think your work is very interesting, and I wonder if you ever published articles about this? And how can I find your articles? Thanks!

Do you think your method would work on an camera feed with irPass filter, like discussed here:
https://arxiv.org/pdf/1902.05194.pdf ?

In your code you get the mean RGB value. So maybe using the mean IR as they mention "we compute the mean IR value on each region" could work too?

Hello, great work!

Is there a way to verify if the person is placing a photo or image instead of a real face ?

Thank you.

I cant figure out whats wrong with it? I am using windows 8 , compiler TDM-GCC mingw , opencv 3.4.1

this one seems to work: https://www.rouast.com/pdf/rouast2016remote_b.pdf
I mean "Remote Photoplethysmography: Evaluation of Contactless Heart Rate Measurement in an Information Systems Setting" link

你好，请问你在浦源创建的“rPPG非接触式心率测量”的应用的源码是来源于这个仓库吗

Thanks your code ,but when i compile the source code , the error happend

undefined reference to symbol 'avcodec_open2@@LIBAVCODEC_FFMPEG_56'
/usr/lib/gcc/x86_64-linux-gnu/5/../../../x86_64-linux-gnu/libavcodec.so: error adding symbols: DSO missing from command line

I need help .Thankyou.

Could you pls elaborate more about your detrend algorithm?
Thanks

Thanks for your great code. I am having trouble compiling on my Mac. If you already compiled on Mac, can you share the executable? Thanks!

Hi,
I have a question when I run the code. Hope that you could help me to get out.

when I debug the code with the pdb tools, I found the faceValid is true when I first get into the rppg.processFrame() function, which leads to not call the detectFace function before call the traceFace function. I don't know where makes the faceValid true at the beginning of the code. Do I understand wrong with the program ?

Thank you.

have you got an android studio version of this code?

I have recorded some video footage using a Nikon D750 DSLR camera and have kept auto-focus and auto for all other settings. The room is well lit, the subjects face fills the frame and there is little movement.
The results are poor when compared to the ground-truth, which has been measured using a Polar H10 chest-strap sensor.
When checking the metadata for the videos I have noticed that the ISO varies significantly (280 - 1800) across different subjects. In particular, it's high for those with dark skin tone and performs poorly with those. The best performing subject doesn't have the lowest ISO though.

@prouast Is the ISO a significant factor in determining the quality of the results or is there anything else I should consider with the data?

Hi

I took a video from one single picture for 20 seconds,then using it to test.however,I can also get the heart rate just like human's. Do you have any idea about this?

Regards
MLQUE

I can't figure out what the getFps function is doing. I see it is called in in RPPG.cpp to create the fps variable, but I can't figure out what the intention is. Could you clarify and break down what the goal of this function and the fps variable is? Forgive me if this is already written in a paper/blog, but I've searched extensively and can find no information.

The function in opencv.cpp:

        double getFps(Mat &t, const double timeBase) {

        double result;

        if (t.empty()) {
            result = 1.0;
        } else if (t.rows == 1) {
            result = std::numeric_limits<double>::max();
        } else {
            double diff = (t.at<int>(t.rows-1, 0) - t.at<int>(0, 0)) * timeBase;
            result = diff == 0 ? std::numeric_limits<double>::max() : t.rows/diff;
        }

        return result;
    }

The code in RPPG.cpp:

        if (faceValid) {

        // Update fps
        fps = getFps(t, timeBase);
        cout << fps << endl;
        // Remove old values from raw signal buffer
        while (s.rows > fps * maxSignalSize) {
            push(s);
            push(t);
            push(re);
        }
        cout << "s: " << s.rows << " t: " << t.size() <<endl;

        assert(s.rows == t.rows && s.rows == re.rows);

        // New values
        Scalar means = mean(frameRGB, mask);
        // Add new values to raw signal buffer
        double values[] = {means(0), means(1), means(2)};
        s.push_back(Mat(1, 3, CV_64F, values));
        t.push_back(time);

        // Save rescan flag
        re.push_back(rescanFlag);

        // Update fps
        fps = getFps(t, timeBase);

I ask because if I change the ROI size, the s and t variables change which drastically effects (in a bad way) the signal quality/bpm prediction. Why is this?

I'm trying to load baseline physiological data from a csv file. I'm running into segmentation faults. It'd be great if you could include some documentation on how to use the baseline argument and the format of the baseline csv file

How to run on windows10？

I have found no effective steps to build and run.

could you share a heart python

Hi
Actually heart rate changes rapidly for example from 80 to 50 then 70 ... .
So i think the max change should be for instance 2 BPM per second. Do you have any idea about this?

My second question is about the time in csv file. Could you please explain a little bit about it? (eg. 9394881, 9395903, 9396906)

Also could please explain a little bit about max time window ?

And finally, because i think your app is not accurate when there are a lot of movements and also the system could not detect face at a distance more than 0.5 m. Do you think using dlib will improve your app?

Regards
Javad

I cant figure out the problem. I am using windows 8, opencv 3.4.1, mingw32 minimalist compiler.

I want to use this repo with Visual Studios and make an exe file. Can this be done? which is the main file that I should run?

Hi
After compiling the source-code, and trying to run the program I got this error.
Do you have nay idea for fixing this issue?

centan@centan-Vostro-3800:~/Desktop/Heartbeat$ ./Heartbeat
Using algorithm 0.
SIZE: 640x480
FPS: 30
MSEC: 0
Tracking face
OpenCV Error: Assertion failed ((npoints = prevPtsMat.checkVector(2, CV_32F, true)) >= 0) in calcOpticalFlowPyrLK, file /home/centan/Downloads/opencv-3.1.0/modules/video/src/lkpyramid.cpp, line 1114
terminate called after throwing an instance of 'cv::Exception'
what(): /home/centan/Downloads/opencv-3.1.0/modules/video/src/lkpyramid.cpp:1114: error: (-215) (npoints = prevPtsMat.checkVector(2, CV_32F, true)) >= 0 in function calcOpticalFlowPyrLK

could you update the repo to include pos ,fav and chrome algorithms

Hi, iam working on rppg for about 4 months and I take this project as a source to study from I changed some stuff like using ncnn implantation and more my question is there is respiratory rate implantation to use in cpp

edit: i read a lot on hypered can and mtts can but this need lots of efforts to be generated for android device any advice?

https://openexr.com/en/latest/install.html#install
"On macOS, we do not recommend installation via HomeBrew because the distribution is outdated."

I test on video with -i arguments, but the log prints Found no face, why is that?

I tried the algorithm on paper mask or just print photo, it can still get the heart rate, is there anyway to improve the algorithm? Thx.

Hello,
I try to compile your code on Debian 9.8 i386 with opencv & opencv_contrib 3.1.0. unfortunately, I got these error message:

could you help me solve this problem

Thanks so much!