okankop / vidaug

Effective Video Augmentation Techniques for Training Convolutional Neural Networks

Video Augmentation Techniques for Deep Learning

This python library helps you with augmenting videos for your deep learning architectures. It converts input videos into a new, much larger set of slightly altered videos.

Original Video

Requirements and installation

Required packages:

• numpy
• PIL
• scipy
• skimage
• OpenCV (i.e. cv2)

For installation, simply use sudo pip install git+https://github.com/okankop/vidaug. Alternatively, the repository can be download via git clone https://github.com/okankop/vidaug and installed by using python setup.py sdist && pip install dist/vidaug-0.1.tar.gz.

Examples

A classical video classification with CNN using augmentations on videos. Train on batches of images and augment each batch via random crop, random crop and horizontal flip:

from vidaug import augmentors as va

sometimes = lambda aug: va.Sometimes(0.5, aug) # Used to apply augmentor with 50% probability
seq = va.Sequential([
    va.RandomCrop(size=(240, 180)), # randomly crop video with a size of (240 x 180)
    va.RandomRotate(degrees=10), # randomly rotates the video with a degree randomly choosen from [-10, 10]  
    sometimes(va.HorizontalFlip()) # horizontally flip the video with 50% probability
])

for batch_idx in range(1000):
    # 'video' should be either a list of images from type of numpy array or PIL images
    video = load_batch(batch_idx)
    video_aug = seq(video)
    train_on_video(video)

The videos below show examples for most augmentation techniques:

Augmentation Type	Augmented Video
Piecewise Affine Transform
Superpixel
Gausian Blur
Invert Color
Rondom Rotate
Random Resize
Translate
Center Crop
Horizontal Flip
Vertical Flip
Add
Multiply
Downsample
Upsample
Elastic Transformation
Salt
Pepper
Shear

vidaug's Issues

Is it for one video (sequence of images) or a batch of videos?

Thank you for providing your code.
While checking the example code

for batch_idx in range(1000):
    # 'video' should be either a list of images from type of numpy array or PIL images
    video = load_batch(batch_idx)
    video_aug = seq(video)
    train_on_video(video)

the loop is running on batch_idx (i suppose batch of videos) but in comment it says video is list of images. Can I give a batch of videos or I will have to give single video at a time?

RandomRotate does not preserve range

Hey,

when using RandomRotate with numpy arrays, the images are normalized. For most use cases it might be preferable to prevent the range of input imgs. Skikit.transform.rotate has the flag preserver_range (default False).

https://scikit-image.org/docs/stable/api/skimage.transform.html#skimage.transform.rotate

It could be a good idea to forward it to the user or set it to True in the va.RandomRotate class.

Cheers,
Max.

How to input the videos

Hi,
Thanks for the code. Just wanted to know how to pass multiple images to the code for Video Augmentation.

Thanks,
Mritula

There are no functions like: video = load_batch(batch_idx) and train_on_video(video) in repo: Give the example to run this repo with original function

for batch_idx in range(1000):
# 'video' should be either a list of images from type of numpy array or PIL images
video = load_batch(batch_idx)
video_aug = seq.augment_images(video)
train_on_video(video)

How can I load batch images through load_batch() function

for batch_idx in range(1000):
# 'video' should be either a list of images from type of numpy array or PIL images
video = load_batch(batch_idx)
video_aug = seq(video)
train_on_video(video)

Sequence exists with only message "Killed"

Hi,
I am applying the pepper or salt augmenters but the process keeps getting killed and I am not sure why. Only message is "Killed". I tried to look for where this might be coming from but could not identify the source.

Not working for .mp4 format?

Hello
Can you please guide me that why this code is not running for .mp4 format?
Thank you

camera vibration augmentation

Hello
How are you?
Thanks for contributing to this project.
I have a question.
Does this library contain a camera motion augmentation?
Here the camera motion means a camera's vibration(shaking).

Please check the above sequence frames which are shaking by camera vibration.
Could u implement this augmentation in your repo?
Thanks

RandomResize does not work with ndarray

\lib\site-packages\vidaug\augmentors\affine.py in __call__(self, clip)
     90         new_size = (new_h, new_w)
     91         if isinstance(clip[0], np.ndarray):
---> 92             return [scipy.misc.imresize(img, size=(new_h, new_w),interp=self.interpolation) for img in clip]
     93         elif isinstance(clip[0], PIL.Image.Image):
     94             return [img.resize(size=(new_w, new_h), resample=self._get_PIL_interp(self.interpolation)) for img in clip]

\site-packages\vidaug\augmentors\affine.py in <listcomp>(.0)
     90         new_size = (new_h, new_w)
     91         if isinstance(clip[0], np.ndarray):
---> 92             return [scipy.misc.imresize(img, size=(new_h, new_w),interp=self.interpolation) for img in clip]
     93         elif isinstance(clip[0], PIL.Image.Image):
     94             return [img.resize(size=(new_w, new_h), resample=self._get_PIL_interp(self.interpolation)) for img in clip]

AttributeError: module 'scipy.misc' has no attribute 'imresize'

When trying to use RandomResize it gives the above error. Seems to be caused by deprecated scipy API.

imresize is deprecated! imresize is deprecated in SciPy 1.0.0, and will be removed in 1.3.0. Use Pillow instead: numpy.array(Image.fromarray(arr).resize()).

Error deo

Testing the horizontal_flipping.ipynb with a video.mp4

Hello, I'm testing your example, horizontal_flipping.ipynb, with a video.mp4 instead salt.gif, it doesn't work, do you know why?
Thanks!

How to run this application?

What I see from your ReadMe file is, you are training in batches.
I wanted to know, How to generate Augmented Videos as you shown in some output gifs??
Could you please mention clearly about it!!??

How about using ImageDataGenerator in Keras for videos?

Hello,
First, I would like to thank you for this tremendous contribution to video data augmentation. I am looking for a way to use video data augmentation while training models on video classification/labelling, I know that for images ImageDataGenerator from Keras exists as a data augmentor. However, it is not specifically targetted for videos.
However, I saw a kind of augmentation used in this website article for video classification. I don't know if you came across ImageDataGenerator, but I would like to know if it is possible to use ImageDataGenerator rather than your functions to apply data augmentation for videos ?

AttributeError: module 'numpy' has no attribute 'float'

Hi!

First of all: thanks for this awesome package!
But I'm receiving this numpy error:

AttributeError: module 'numpy' has no attribute 'float'.
`np.float` was a deprecated alias for the builtin `float`. To avoid this error in existing code, use `float` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.float64` here.
The aliases was originally deprecated in NumPy 1.20; for more details and guidance see the original release note at:
    https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations

The error occurs in augmentors/intesity.py#L162

On the SuperPixel Augmentation

I can not get the right effect for superpixel augmentation with the original video on the homepage. Would you please tell me what is the proper parameter configuration for the superpixel augmentation?

Naming error

I found that the directory name is augmentors, while in the document it's augmenters
Also, va.RandomRotate(angle=10) should be va.RandomRotate(degree=10)
Are these functions tested?

Document for the project

Hello okankop!

This is a really impressive repo and it definitely saved me from writing those data augmentation functions. Thanks greatly for the great work!

But I do think it will be even better if you can provide a document, however easy it is. Maybe just add something to explain the input parameters at the bottom of the form in README.md.

Thanks again!

'Sequential' object has no attribute 'augment_images'

hi，thank you for your code. When I use your example an error occurred ‘Sequential' object has no attribute 'augment_images’. Can you tell me how to slove it?

from vidaug import augmentors as va

sometimes = lambda aug: va.Sometimes(0.5, aug) # Used to apply augmentor with 50% probability
seq = va.Sequential([
va.RandomCrop(size=(240, 180)), # randomly crop video with a size of (240 x 180)
va.RandomRotate(degrees=10), # randomly rotates the video with a degree randomly choosen from [-10, 10]
sometimes(va.HorizontalFlip()) # horizontally flip the video with 50% probability
])

for batch_idx in range(1000):
# 'video' should be either a list of images from type of numpy array or PIL images
video = load_batch(batch_idx)
video_aug = seq.augment_images(video)
train_on_video(video)

Thank you!

Downsample augmentation is not working because of the floating value of the nb_return_frame variable.

nb_return_frame = np.floor(self.ratio * len(clip))
numpy floor return a value in float64 and linspace function doesn't support float to integer conversion.

nb_return_frame = int(np.floor(self.ratio * len(clip)))
I tried to run it like this and it works!

Note : It is the same for Upsample augmentation too!