brycedrennan / eulerian-magnification

A tool to discover hidden variation in video.

Eulerian Video Magnification

Amplify tiny movements in video.

Based on the amazing research done at MIT: http://people.csail.mit.edu/mrub/vidmag/

Warning from the author: This code is not in a working state. I'm slowly adding testing to identify whats wrong, but this project is very low priority. Pull-requests welcome!

Installation

• Install OpenCV
• pip install eulerian-magnification

or

docker build -t eulerian .
docker run -it eulerian /bin/bash

Requirements

• Python 3.5, untested on Python 2.7.
• OpenCV 3+.
• numpy, scipy, matplotlib

On windows you can download the needed python dependencies here. Make sure you install the MKL version of numpy as the scipy binary depends on it.

Usage

This technique works best with videos that have very little motion. Pre-processing a video through a stabilization algorithm may help. Some excellent videos sources can be found here: http://people.csail.mit.edu/mrub/vidmag/

Once you've downloaded the video simply run::

import eulerian_magnification as em

vid, fps = em.load_video_float(source_path)
em.eulerian_magnification(vid, fps, 
        freq_min=50.0 / 60.0,
        freq_max=1.0,
        amplification=50,
        pyramid_levels=3
)

freq_min and freq_max specify the frequency in hertz that will be amplified. amplification specifies how much that signal will be amplified.

It can take a while to find the best parameters for a specific video. To help with that there is the show_frequencies function::

import eulerian_magnification as em

vid, fps = em.load_video_float(source_path)
em.show_frequencies(vid, fps)

This will show a graph of the average value of the video as well as a graph of the signal strength at various frequencies.

Todo

• Butterworth and IIR filters
• Optimized memory usage to allow processing of larger files

Troubleshooting

When I process the video it looks all weird - alternating from bright to dark - what am I doing wrong?

Most likely the video you're trying to process just has too much movement. Try running it through a video stabilizer. Even with stabilization, it can be hard to find the correct frequency and amplification parameters that isolate the hidden motion you're trying to display.

Additionally, some videos are better suited to motion amplification using a laplacian pyramid.

Windows: IndexError: tuple index out of range

On windows with OpenCv2 it may be necessary to add C:\OpenCV2.3\build\x86\vc10\bin to the system path for videos to load properly. Make sure you adjust the path to the actual location of your opencv library.

Push to Pypi

git tag 0.22
git push --tags
python setup.py sdist
twine upload dist/*
rm dist -r

Author

Bryce Drennan [email protected]

eulerian-magnification's Issues

Gaussian pyramid

When I was reviewing the code for the gaussian_video function:

def gaussian_video(video, shrink_multiple):
    """Create a gaussian representation of a video"""
    vid_data = None
    for x in range(0, video.shape[0]):
        frame = video[x]
        gauss_copy = numpy.ndarray(shape=frame.shape, dtype="float")
        gauss_copy[:] = frame
        for i in range(shrink_multiple):
            gauss_copy = cv2.pyrDown(gauss_copy)

        if x == 0:
            vid_data = numpy.zeros((video.shape[0], gauss_copy.shape[0], gauss_copy.shape[1], 3))
        vid_data[x] = gauss_copy
    return vid_data

It seems like it's just downsampling and overwriting gauss_copy 4 times (lines 106-107). I thought the algorithm saved all levels of the pyramid instead of just the lowest one?

why is 50.0/60.0

import eulerian_magnification as em

vid, fps = load_video_float(source_path)
em.eulerian_magnification(vid, fps,
freq_min=50.0 / 60.0,
freq_max=1.0,
amplification=50,
pyramid_levels=3
)

Bad output

I tried running your example.py file with the face.mp4 file from http://people.csail.mit.edu/mrub/vidmag/. The show_frequencies appears to work, but eulerian_magnification produced a bad video file. VLC won't play it, and it is only 414 bytes.

Here is the log:

Loading media/face.mp4
/Users/Andreas/Sites/lies/eulerian_magnify.py:87: FutureWarning: comparison to `None` will result in an elementwise object comparison in the future.
  if frame == None or x >= frame_count:
29
/usr/local/lib/python2.7/site-packages/numpy/core/numeric.py:462: ComplexWarning: Casting complex values to real discards the imaginary part
  return array(a, dtype, copy=False, order=order)
Loading /Users/Andreas/Sites/lies/media/face.mp4
Applying bandpass between 0.833333333333 and 1.0 Hz
Amplifying signal by factor of 50
/Users/Andreas/Sites/lies/eulerian_magnify.py:147: ComplexWarning: Casting complex values to real discards the imaginary part
  img[:] = g_video[x]

Does not work (list index out of range)

I cant get it to work:

Loading eulerian_source_videos/baby.mp4
Traceback (most recent call last):
  File "replicate_study.py", line 84, in <module>
    replicate_study()
  File "replicate_study.py", line 75, in replicate_study
    pyramid_levels=pyramid_levels
  File "/home/calle/eulerian-magnification/eulerian_magnification/base.py", line 27, in eulerian_magnification
    vid_data = collapse_laplacian_video_pyramid(vid_pyramid)
  File "/home/calle/eulerian-magnification/eulerian_magnification/pyramid.py", line 68, in collapse_laplacian_video_pyramid
    return pyramid[0]
IndexError: list index out of range

Here is the exact way i tried to use it from a fresh ubuntu 1604 vm.

    6  sudo apt install python3-pip
    7  sudo apt install git
   12  git clone https://github.com/brycedrennan/eulerian-magnification
   13  cd eulerian-magnification/
   16  pip3 install -r requirements.txt 
   19  pip3 install opencv
   22  pip3 install matplotlib
   25  sudo apt-get install python3-tk
   28  sudo python3 setup.py install
   31  pip3 install numpy scipy matplotlib
   32  python3 replicate_study.py 
   33  history

Empty File

The AVI written out is 8 kb and appears empty. The code works with the test videos from the MIT site. Those videos are:

960 x 624, H.264, and color profile SD (6-1-6) (from Get Info on the file on OS X).

My videos are

800 x 532, H.264 AAC, and color profile HD (1-1-1).

Using the 'file' command, both report "SO Media, MPEG v4 system, version 2".

I'm not sure what else might be causing the problem. Do you have any ideas why the output might be empty, or what else to check to see how the videos are different?

Illumination Problem

Hi

Just to explain, I'm very new to Python, and somewhat new to Ubuntu and OpenCV.
I've coded before, so I caught on quickly how the code works and began to implement it, but even when I run the example code, I seem to be getting a flash of bright light that complete blinds the image, followed by complete darkness, and so on.

I'm not too sure what the problem is, as I roughly understand the code and the setup. I thought it may have been the setup and that I don't have the correct versions installed or something, but figured that I would have gotten an error by now.

Also, with regards to the FFT plot displayed by the "show_frequencies" function --- What is this scale related to? I'm using the breathing baby video, and have my FFT spiking at 9? is that 9/30 Hz? I notice you're using a sampling time related to the fps (frames per second?), which is output in the Python Shell when running the "show_frequencies" function.

Any help would be appreciated.
Thank you very much.
Regards.

Support phase-based pipeline

Implement improvements described at http://people.csail.mit.edu/nwadhwa/phase-video/

Little spelling

Still haven't had this working but I've noticed that some of the code is written eulErian-magnification while other parts say eulArian-Magnification. I'm new to python and coding so this might be a silly question: Do I need to CD to /media/face.mp4 to run the script? I keep getting a gaussian image error

eulerian_magnification() got an unexpected keyword argument 'image_processing'

module 'eulerian_magnification' has no attribute 'load_video_float'

Hello,
When using the module downloaded from pip, I get this error when trying out the sample code from the readme.
My video is an .mp4 file taken from the paper.

ReadMe show_frequencies example

I was trying to run show_frequencies(vid_data) as described but had two issues:

1. The function requires a second fps argument. I imagine that the readme could be updated to say that.
2. after adding the fps argument, kept on getting an type error with vid_data (my video filename) being used as a string. I believe the best method is to load the video with cv2, then use that in show_frequencies. It wasn't made clear to me that this was necessary from the readme, but the actual code seems to suggest that the argument 'vid_data' needs to already be something loaded in, not just a filename.

What is the output of the algorithm?

Hi,
I really don't understand what is the output of the algorithm? It seems to that the original video is not modified after the run and there is no new video created in my work directory. Maybe i need to do some other steps in order to receive the output?
Thanks!

Getting 'tuple index out of range' error in OS X

Installed opencv by tapping hombrew/science and doing brew install opencv, then did pip install matplotlib scipy numpy.

After opening up the file by going

python -i eulerian_magnify.py

I entered

eulerian_magnification('movie.mov', image_processing='gaussian', freq_min=50.0 / 60.0, freq_max=1.0, amplification=50)

and got

Loading will.mov
WARNING: Couldn't read movie file will.mov
Applying bandpass between 0.833333333333 and 1.0 Hz
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "eulerian_magnify.py", line 19, in eulerian_magnification
    vid_data = temporal_bandpass_filter(vid_data, fps, freq_min=freq_min, freq_max=freq_max)
  File "eulerian_magnify.py", line 61, in temporal_bandpass_filter
    fft = scipy.fftpack.fft(data, axis=axis)
  File "/usr/local/lib/python2.7/site-packages/scipy/fftpack/basic.py", line 230, in fft
    n = tmp.shape[axis]
IndexError: tuple index out of range

It seems like opencv's binaries are already in the path.

Memory error

With a 35" video (58MB), I get a frame_count of 1179, a height of 1080 and width of 1920.

Since you're creating a 4d matrix of uint8 zeros, this means:

In[1] > print "{} TB required!".format(1179 * 1080 * 1920 * 3 * 256.0 / 1024**4)
1.70765519142 TB required!

I would need to degrade my video to 320*180 in order to have a feasible (48.5GB) size, and I assume the result will be disastrous.

Do you have an idea for a better way to handle this magnification, which doesn't require 30872 times more memory than the input video size ?

Thanks for your suggestions

package spelling issue

I received the following report via email:
"Thanks for all your work you have done on this code and permitting the public to use and help out with its development. I wanted to let you know that I found a small glitch that is causing problems implementing the function. It may be quite a simple fix. I noticed the pip package manager wouldn't recognize the 'eulerian_magnification' as currently spelled, but only as 'eularian_magnification'. I used the tarball to install the package on my linux machine, but when I import the function in python using the as-installed misspelled version everything goes well except when I use anything related to 'em.function_name' as the code appears to expect the correct spelling."

No License

This repository has no license. Am I allowed to relicense it to some FOSS license GPL/BSD (I don't really care) to fork it and base some other functionality on it?

Issue about invoking parameter

When I try to call the top function just like the usage following:
em.eulerian_magnification('media/face.mp4', image_processing='gaussian', pyramid_levels=3, freq_min=50.0 / 60.0, freq_max=1.0, amplification=50);
I find that the first parameter doesn't make sense because it's a string, not a video object. And in the eulerian_magnification function, there is no preprocess about the first parameter which should obtain an object like a video. So I get a error when the program comes to 'for frame_number, frame in enumerate(video)' in the pyramid.py(line 37) because the first parameter isn't an iterable object. What should I do solve this question?

Empty output video

Output files all seem to have zero size. I get this error when running the code:

eulerian_magnify.py:144: ComplexWarning: Casting complex values to real discards the imaginary part
  img[:] = g_video[x]

System: OS X, Enthought Canopy Python, Python 2.7.3, 64-bit, OpenCV 2.4.2

Video (*_magnified.avi) size is 28 KB.

Real Time Implementation of Eulerian-Magnification

Hi, @brycedrennan
As a part of an academic project, we want to use this algorithm to detect respiration of babies and identifying the pattern of respiration in real-time with a help of simple mobile camera.This can be useful to various hospitals to replace the sensors used on infant body.
For now, the problem we are facing is the processing time this algorithm requires when a real-time video stream is passed into this.
I hope you can help us in figuring out the best approach that can give us real-time results.
Thanks :)

Spelling Error : Readme.md

In Readme.md
The spelling of eularian is wrong.
Correct version :

import eularian_magnification as em
em.eularian_magnification('videos/baby.mp4', image_processing='gaussian', pyramid_levels=3, freq_min=50.0 / 60.0, freq_max=1.0, amplification=50)

implementing live feed use

Hi is it possible to use this on a real time video stream?