mrinaljain17 / mydia

A simple and efficient wrapper for reading videos as NumPy tensors

Mydia

Reading videos as NumPy arrays was never more simple. This library provides an entire range of additional functionalities such as custom frame selection, frame resizing, pixel normalization, grayscale conversion and much more.

READ THE DOCUMENTATION

Getting started

1. Read a video, given its path

# Import
from mydia import Videos

# Initialize video path
video_path = r".docs/examples/sample_video/bigbuckbunny.mp4"

# Create a reader object
reader = Videos()

# Call the 'read()' function to get the video tensor
# which will be of shape (1, 132, 720, 1280, 3)
video = reader.read(video_path)

The tensor can be interpreted as:

• 1 video
• Having 132 frames,
• Dimension (width x height) of each frame: 1280x720 pixels
• 3 denotes that the video is in RGB format

2. You can even use multiple workers for reading the videos in parallel

from mydia import Videos

video_paths = [
    "path/to/video_1", 
    "path/to/video_2", 
    "path/to/video_3",
    ...,
]

reader = Videos()
video = reader.read(video_path, workers=4)

3. View detailed examples here

Requirements

• Python 3.x (preferably from the Anaconda Distribution)

• FFmpeg: The backend for reading and processing the videos.

  The recommended (and probably the easiest) way of installing FFmpeg is via the conda package manager.

      conda install -c mrinaljain17 ffmpeg

  However, if you are not using conda, then

  For Linux users -

      $ sudo apt-get update
      $ sudo apt-get install ffmpeg

  For Windows or OSX users -

  Download the required binaries from here. Extract the zip file and add the location of binaries to the PATH variable.

Installation

1. Using the conda package manager (recommended):

       conda install -c mrinaljain17 mydia

2. Using pip:

       pip install mydia

The following python packages that mydia depends on, will also be installed, along with their dependencies.

• ffmpeg-python
• Numpy
• tqdm - Required for displaying the progress bar.

License

Copyright 2018 Mrinal Jain.

Released under the MIT License.

mydia's Issues

Feature: Provide other methods for normalization of pixels

Currently, each pixel is divided by 255 to be in the range (0, 1), inclusive.

Other methods like min-max normalization, z-score normalization should be provided.

mkv video files do not have "nb_frames" field in video header

I tried reading a .mkv file and it crashed.

    video = mydia.Videos().read(video_path)
  File ".../python3.6/site-packages/mydia/mydia.py", line 234, in read
    video_tensor = np.vstack(map(self._read_video, paths_iterator))
  File "<__array_function__ internals>", line 6, in vstack
  File ".../python3.6/site-packages/numpy/core/shape_base.py", line 280, in vstack
    arrs = atleast_2d(*tup)
  File ".../python3.6/site-packages/mydia/mydia.py", line 286, in _read_video
    fps, total_frames = self._probe(path)
  File ".../python3.6/site-packages/mydia/mydia.py", line 365, in _probe
    total_frames = int(video_stream["nb_frames"])
KeyError: 'nb_frames'

To reproduce:

1. Locate a video file which works
2. Convert it to .mkv as follows: ffmpeg -i working.mp4 -vc copy test.mkv
3. Try to read mkv file

I printed the video_stream data in _probe(), here was the result:

{'index': 0, 'codec_name': 'h264', 'codec_long_name': 'H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10', 'profile': 'High', 'codec_type': 'video', 'codec_time_base': '1/50', 'codec_tag_string': '[0][0][0][0]', 'codec_tag': '0x0000', 'width': 700, 'height': 500, 'coded_width': 700, 'coded_height': 500, 'has_b_frames': 2, 'sample_aspect_ratio': '0:1', 'display_aspect_ratio': '0:1', 'pix_fmt': 'yuv420p', 'level': 30, 'chroma_location': 'left', 'field_order': 'progressive', 'refs': 1, 'is_avc': 'true', 'nal_length_size': '4', 'r_frame_rate': '25/1', 'avg_frame_rate': '25/1', 'time_base': '1/1000', 'start_pts': 0, 'start_time': '0.000000', 'bits_per_raw_sample': '8', 'disposition': {'default': 1, 'dub': 0, 'original': 0, 'comment': 0, 'lyrics': 0, 'karaoke': 0, 'forced': 0, 'hearing_impaired': 0, 'visual_impaired': 0, 'clean_effects': 0, 'attached_pic': 0, 'timed_thumbnails': 0}, 'tags': {'HANDLER_NAME': 'VideoHandler', 'ENCODER': 'Lavc57.107.100 libx264', 'DURATION': '00:00:05.120000000'}}

Do .mp4 files work reliably? If so, converting input to .mp4 should work as a workaround.

Eliminate usage of scikit-video at the backend, because of it's development being stagnant

scikit-video should not be used for reading the videos as the development/maintenance of the library seems to be quite stagnant (based on the repository's activity)

Instead, after considering a few alternatives, ImageIO seems to be suitable for usage, since most of the functionality provided by scikit-video is available in ImageIO (not directly though). This would help in making the transition simple and more efficient (hopefully).

Further, the reason for this change will also be justified by relevant tests and performance comparisons.

Any suggestions/improvements are welcome 😄

Feature: Support for the user to give custom indices for frame selection in addition to the default modes available.

Internally, the _read_video() function gets an array to indices of frames to be read from a function called _get_frame_indices().

In addition, the user should be able to:

1. Pass an array/list of indices indicating the frames to be extracted.
2. Pass a custom function that will return an array of indices - the input(s) made available to the function will be the same as that of _get_frame_indices()

[Optimization] Using multiple workers

Figure out techniques to optimize the video reading and processing time.

Feature: Support for pytorch/theano like tensors - with channels first

PyTorch and theano (and possible other frameworks) have the channels in the first dimension (the zeroth dimension being the batch size).

As of now, the channels are in the last dimension (for frameworks like Tensorflow).

The channels here, refer to the RGB or gray channel.

Feature: Creating a generator for accessing the processed tensor.

Since this kind of data can get extremely large (as we are dealing with multiple videos), a generator should be returned to the user, that will yield the videos one-by-one.

Optimization when reading frames.

mydia/mydia/mydia.py

Line 159 in 5ff2384

if idx in indices:

Terminate the loop if the required frames have been already read.

Optimization in reading videos

Figure out techniques to optimize and reduce the video reading and processing time.

Feature: Option to save the processed video(s) tensor.

Support to save the video tensor. Two different functionalities could be provided for this feature:

1. Saving the tensor in .npz(or .npy) format.
2. Writing each video back to the disk - low priority