sudheerachary / mesh-flow-video-stabilization

Online video stabilization using a novel MeshFlow motion model

Mesh-Flow-Video-Stabilization

The MeshFlow is a spatial smooth sparse motion field with motion vectors only at the mesh vertexes. The MeshFlow is produced by assigning each vertex an unique motion vector via two median filters. The path smoothing is conducted on the vertex profiles, which are motion vectors collected at the same vertex location in the MeshFlow over time. The profiles are smoothed adaptively by a novel smoothing technique, namely the Predicted Adaptive Path Smoothing (PAPS), which only uses motions from the past.

Getting Started

• To stabilize a video execute the script src/Stabilization.py

  • python Stabilization.py <path_to_video>
    

• To run experiments ipython-notebook is present in src/mesh_flow.ipynb

• The stable output video is saved to home directory of cloned repository

Prerequisites

• Required Packages:

  • opencv: pip install --user opencv-python

  • numpy: pip install --user numpy

  • scipy: pip install --user scipy

  • tqdm: pip install --user tqdm

• Optional Packages:

  • cvxpy: pip install --user cvxpy

Motion Propagation

Mesh Flow only operate on a sparse regular grid of vertex profiles, such that the expensive optical flow can be replaced with cheap feature matches. For one thing, they are similar because they both encode strong spatial smoothness. For another, they are different as one is dense and the other is sparse. Moreover, the motion estimation methods are totally different. Next, we show estimatation of spacial coherent motions at mesh vertexes.

Initial motion vectors

• initial motion meshes are dumped into results/old_motion_vectors

Final motion vectors

• final motion meshes are dumped into results/new_motion_vectors

Predicted Adaptive Path Smoothing (PAPS)

A vertex profile represents the motion of its neighboring image regions. MeshFlow can smooth all the vertex profiles for the smoothed motions. It begin by describing an offline filter, and then extend it for online smoothing.

Path smoothening

• Green is the optimized result
• Blue is the motion vector accumulation
• vertex profile paths are dumped into results/paths

Acknowledgments

• MeshFlow: Minimum Latency Online Video Stabilization
• MeshFlow Video Denoising
• SteadyFlow: Spatially Smooth Optical Flow for Video Stabilization

mesh-flow-video-stabilization's Issues

SyntaxError: invalid syntax

Tried to run: python Stabilization.py shaky-5.avi

with the shaky-5.avi in the same src directory, and got this:
File "Stabilization.py", line 23
print method.name+' has taken: '+str(end_time-start_time)+' sec'
^
SyntaxError: invalid syntax

error

x_paths = np.zeros((old_frame.shape[0]/PIXELS, old_frame.shape[1]/PIXELS, 1))

TypeError: 'float' object cannot be interpreted as an integer

while running this code i got error

Please Help!

Error reading the video

Hello, when I run:
python Stabilization.py ../data/shaky-5.avi
or
python Stabilization.py ../data/small-shaky-5.avi
I get the following error:

OpenCV Error: Assertion failed (scn == 3 || scn == 4) in ipp_cvtColor, file /home/travis/miniconda/conda-bld/conda_1486587069159/work/opencv-3.1.0/modules/imgproc/src/color.cpp, line 7456
Traceback (most recent call last):
File "Stabilization.py", line 252, in
x_motion_meshes, y_motion_meshes, x_paths, y_paths = read_video(cap)
File "Stabilization.py", line 21, in timed
result = method(*args, **kwargs)
File "Stabilization.py", line 52, in read_video
old_gray = cv2.cvtColor(old_frame, cv2.COLOR_BGR2GRAY)
cv2.error: /home/travis/miniconda/conda-bld/conda_1486587069159/work/opencv-3.1.0/modules/imgproc/src/color.cpp:7456: error: (-215) scn == 3 || scn == 4 in function ipp_cvtColor

How can I solve this? Should I use another version of opencv? Thank you!

Real-time stabilization

Can this code be implemented to work on real-time incoming frames? Thanks.

empty H matrix

Hi,
when I run './data/shaky-5.avi', i get the following error:
File "/home/ubuntu/Video_Stabilization/Mesh-Flow-Video-Stabilization/src/MeshFlow.py", line 21, in point_transform
a = H[0,0]*pt[0] + H[0,1]*pt[1] + H[0,2]
TypeError: 'NoneType' object is not subscriptable
it seems that openCV doesn't always manage to find a homography matrix, due to a very small number of features found. this, of course, can be fixed by changing the 'feature_params', but I was wondering if anyone encountered this with the original code.
I run with Python 3.

thanks, Offir

How I fixed some bugs and ran the code successfully

I faced some issues when running the code with newer versions of python and OpenCV. This is how I fixed them and managed to run the code successfully:
Make the following adjustment in Stabilization.py:

line  23: print(method.__name__+' has taken: '+str(end_time-start_time)+' sec')
line 265: print('Time elapsed: ', str(time.time()-start_time))
line  60: VERTICAL_BORDER = int((HORIZONTAL_BORDER*old_gray.shape[1])/old_gray.shape[0])
line  66: x_paths = np.zeros((int(old_frame.shape[0]/PIXELS), int(old_frame.shape[1]/PIXELS), 1))
line  67: y_paths = np.zeros((int(old_frame.shape[0]/PIXELS), int(old_frame.shape[1]/PIXELS), 1))

Make the following adjustment in MeshFlow.py:

line  43: cols, rows = int(old_frame.shape[1]/PIXELS), int(old_frame.shape[0]/PIXELS)
line  80: x_motion_mesh[key] = x_motion[key]+temp_x_motion[key][int(len(temp_x_motion[key])/2)]
line  85: y_motion_mesh[key] = y_motion[key]+temp_y_motion[key][int(len(temp_y_motion[key])/2)]

numpy.core.multiarray failed to import

Camera path

Thank you for your code.
In your README you demonstrate plot of path's original video and stabilized. How do you get this paths?
I thought that it's path of frame's center - x. So x' = H*x, where H is matrix homography and (x'[0] / x'[2], x'[1] / x'[2]) is new coordinates (x, y) of center next frame. But I get very strange plot.
Could you explain me my mistake, please?

cvxpy version

Which is the exact version of cvxpy?
I get a ModuleNotFoundError when importing 'canonInterface' in cvxpy/utilities/quadratic.py.

Code is full of bugs

in the Stabilization.py file:
line 23 - print statement missing parenthesis
line 66 - integer division produces float which is invalid as a key
line 67 - integer division produces float which is invalid as a key
line 265 - print statement missing parenthesis

in the MeshFlow.py file:
line 81 - integer division produces float which is invalid as a key
line 86 - integer division produces float which is invalid as a key

I've also noticed this stabilization method is insanely slow, a 10 minute video is taking hours to stabilize, I was expecting better performance so these results are quite disappointing.

still some bug when frame size width < height

I have modify Stabilization.py and MeshFlow.py according to the issues before and test data/*.avi is OK.

when test my video, the width < height, arised array bounds overflow. here is the error log.
frame_num 0 size (240, 134, 3)
Traceback (most recent call last):
File "src/Stabilization.py", line 306, in
generate_stabilized_video(cap, x_motion_meshes, y_motion_meshes, new_x_motion_meshes, new_y_motion_meshes, write_name)
File "src/Stabilization.py", line 23, in timed
result = method(args, **kwargs)
File "src/Stabilization.py", line 245, in generate_stabilized_video
new_frame = mesh_warp_frame(frame, new_x_motion_mesh, new_y_motion_mesh)
File "/data/video_stabilization/Mesh-Flow-Video-Stabilization/src/MeshFlow.py", line 175, in mesh_warp_frame
map_x[:, j] = map_x[:, PIXELSx_motion_mesh.shape[0]-1]
IndexError: index 239 is out of bounds for axis 1 with size 134

so I modify MeshFlow.py

# # repeat motion vectors for remaining frame in y-direction
# for i in range(PIXELS*x_motion_mesh.shape[0], map_x.shape[0]):
#         map_x[i, :] = map_x[PIXELS*x_motion_mesh.shape[0]-1, :]
#         map_y[i, :] = map_y[PIXELS*x_motion_mesh.shape[0]-1, :]

# # repeat motion vectors for remaining frame in x-direction
# for j in range(PIXELS*x_motion_mesh.shape[1], map_x.shape[1]):
#         map_x[:, j] = map_x[:, PIXELS*x_motion_mesh.shape[0]-1]
#         map_y[:, j] = map_y[:, PIXELS*x_motion_mesh.shape[0]-1]

# video weight < height
# repeat motion vectors for remaining frame in y-direction
for i in range(PIXELS*x_motion_mesh.shape[0], map_x.shape[0]):
        map_x[:, i] = map_x[:, PIXELS*x_motion_mesh.shape[0]-1]
        map_y[:, i] = map_y[:, PIXELS*x_motion_mesh.shape[0]-1]

# repeat motion vectors for remaining frame in x-direction
for j in range(PIXELS*x_motion_mesh.shape[1], map_x.shape[1]):
        map_x[j, :] = map_x[PIXELS*x_motion_mesh.shape[0]-1, :]
        map_y[j, :] = map_y[PIXELS*x_motion_mesh.shape[0]-1, :]

the video output is drawing, but the stab video has a white edge in the middle of the height and in the right of the width, and I have not figure out.

list index out of range

hi there, when I am trying to run the code after installing all the prerequisites, it shows me this error. please help me out.!!

/home/darkratio/.virtualenvs/cv/lib/python3.5/importlib/_bootstrap.py:222: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
return f(*args, **kwds)
/home/darkratio/.virtualenvs/cv/lib/python3.5/importlib/_bootstrap.py:222: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
return f(*args, **kwds)
Traceback (most recent call last):
File "Stabilization.py", line 248, in
file_name = sys.argv[1]

Is there any version written in PyTorch?

Does the code use a offline mode other than online mode ?

stable_video_change

Thanks for your sharing,why are the characters in the stable video obtained larger than in the original video?

Is the parameter lamada_t in your implementation different from the one in the paper?

I find that in your implementation parameter lamada_t is fixed.But in the papaer MeshFlow: Minimum Latency Online Video Stabilization, lamada_t is an adaptive parameter. Why?

It seems that this work doesn't support real time video stabilization ?

When I am trying to run the Mesh flow stabilization for real time video stabilization I am getting the below error

Does this work support real-time video stabilizaion?

Does this work support real-time video stabilizaion?
Thanks

Is the stabilized video cropped?

When I run this program and get the error in stabilization.py

file_name =(sys.argv[1])
IndexError: list index out of range
How to solve this Index error .Sir

Originally posted by @LeoSeinThanKyaw in 1cb9994#commitcomment-30972156

an error or imperfection when the frame don't have any feature point

In mesh_flow.ipynb , an error occur when one frame has no feature point:
# calculate optical flow
p1, st, err = cv2.calcOpticalFlowPyrLK(old_gray, frame_gray, p0, None, **lk_params)

This problem occur definitely when a video has a blank or black frame, can you provide a more rebust code for this solution.

Thank you very much!