techlowd / speech-driven-animation

Speech-Driven Animation

This library implements the end-to-end facial synthesis model described in this paper.

Prerequisites

The models provided are checked out using git LFS. You can install git LFS by following these instructions.

Downloading the models

The models were hosted on git LFS. However the demand was so high that I reached the quota for free gitLFS storage. I have moved the models to GoogleDrive. Models can be found here.

Installing

To install the library do:

$ pip install .

Running the example

To create the animations you will need to instantiate the VideoAnimator class. Then you provide an image and audio clip (or the paths to the files) and a video will be produced.

Choosing the model

The model has been trained on the GRID, TCD-TIMIT, CREMA-D and LRW datasets. The default model is GRID. To load another pretrained model simply instantiate the VideoAnimator with the following arguments:

import sda
va = sda.VideoAnimator(gpu=0, model_path="crema")# Instantiate the animator

The models that are currently uploaded are:

• grid
• timit
• crema
• lrw

Example with Image and Audio Paths

import sda
va = sda.VideoAnimator(gpu=0)# Instantiate the animator
vid, aud = va("example/image.bmp", "example/audio.wav")

Example with Numpy Arrays

import sda
import scipy.io.wavfile as wav
from PIL import Image

va = sda.VideoAnimator(gpu=0)# Instantiate the animator
fs, audio_clip = wav.read("example/audio.wav")
still_frame = Image.open("example/image.bmp")
vid, aud = va(frame, audio_clip, fs=fs)

Saving video with audio

va.save_video(vid, aud, "generated.mp4")

Using the encodings

The encoders for audio and video are made available so that they can be used to produce features for classification tasks.

Audio Encoder

The Audio Encoder (which is made of Audio-Frame encoder and RNN) is provided along with a dictionary which has information such as the feature length (in seconds) required by the Audio Frame encoder and the overlap between audio frames.

import sda
encoder, info = sda.get_audio_feature_extractor(gpu=0)