vimalmanohar / pychain-old

PyTorch implementation of LF-MMI for End-to-end ASR

PyTorch implementation of LF-MMI for End-to-end ASR

End-to-end version of lattice-free MMI is mainly based on these two papers:

1. "End-to-end speech recognition using lattice-free MMI", Hossein Hadian, Hossein Sameti, Daniel Povey, Sanjeev Khudanpur, Interspeech 2018 (pdf)
2. "Purely sequence-trained neural networks for ASR based on lattice-free MMI", Daniel Povey, Vijayaditya Peddinti, Daniel Galvez, Pegah Ghahrmani, Vimal Manohar, Xingyu Na, Yiming Wang and Sanjeev Khudanpur, Interspeech 2016, (pdf) (slides,pptx)

The code is a modification of the version in the kaldi repository with no dependency on the kaldi base.

Installation

Install PyTorch

Install PyTorch

Setup OpenFst

Download and install OpenFST

OPENFST_PATH shold be set to the root of the OpenFst installation. i.e. $OPENFST_PATH/include and $OPENFST_PATH/openfst should contain the required headers and libraries.

./configure --prefix=`pwd` --enable-static --enable-shared --enable-ngram-fsts CXX="g++" LIBS="-ldl" CPPFLAGS="-D_GLIBCXX_USE_CXX11_ABI=0" CXXFLAGS="-D_GLIBCXX_USE_CXX11_ABI=0"
make
make install

Note that the option -D_GLIBCXX_USE_CXX11_ABI=0 must be compatible with the option used when compiling PyTorch. Details here.

Install the bindings

cd pytorch_binding
python setup.py install

Structure of the directory

• / contains the instructions and makefiles etc.
  • src contains C++/CUDA code agnostic of PyTorch
  • pytorch_bindings contains the pytorch-based module of the LF-MMI objf function
    • src contains C++ level implementation of the LF-MMI objf function
    • pychain_pytorch contains python level implementation of the LF-MMI objf function. This includes the actual loss-function module, which can be bound the C++ level code.
      • __init__.py the actual implementation of the python module
  • openfst_bindings contains the openfst-based module to interact with FSTs

pychain-old's Issues

Implement DataLoader that can read MFCCs and FSTs

DataLoader class to load MFCC features and e2e-format FSTs from disk into a pair of features Tensor and FST and create minibatches.

We can follow the way Deepspeech does this i.e. to use an SCP-like file that has the filepaths so that the features and FSTs can be read at the time of creating the minibatch. This should be implemented such that it can return a single pair (features, fst) read from disk when used with python iterator.

• This probably needs as input an SCP-like file that has all the paths for features as well as FSTs including additional auxiliary information such as the duration of the utterance, which will be required by the Sampler.
• We assume that the SCP-like file is sorted by length so we can create minibatches of same chunk size. A Sampler will use this information to create minibatches. #3

Implement denominator computation in python

Create a new class (maybe derived from Function) with the arguments:

[in] chain_training_options

A struct containing various options relevant to LF-MMI training:

• leaky_hmm_coefficient [0.1]

[in] transition_matrix

SparseTensor of size N x N x D, where
N = num states in denominator graph
D = num pdfs
T = chunk size
B = minibatch size
A sparse tensor representation of the transition probabilities of the arcs in the denominator graph. Depending on the implementation, we might require this to be stored as real numbers or in log format.

[in] initial_probs

Tensor of size N x 1
A tensor containing the initial probabilities for the states in the denominator graph. This is also needed for leaky-hmm computation.

[in] nnet_outputs

Tensor of size D x T x B
A tensor of neural network activations.

[out] nnet_gradients

Tensor of size D x T x B
Gradients computed by the network. In a python implementation, this will be computed using autograd backward() function.

Implement denominator computation in C++ and CUDA

Implement the denominator computation in C++ instead of python as in #1.
This will involve writing CUDA kernels such as the one already in kaldi (https://github.com/kaldi-asr/kaldi/blob/master/src/chain/chain-denominator.cc and https://github.com/kaldi-asr/kaldi/blob/master/src/chain/chain-kernels.cu). However this might not actually be required if the python version is already fast.

Implement Sampler to create minibatches

Sampler to create minibatches based on the length of the utterances so that similar length utterances are together.
This needs to be moved to a separate module outside the loss function module.

It has access to a list of utterances sorted by length so that same-size batches can be created easily.

• Create batches of appropriate size.
• Return randomly sampled batches.
• Random sampling within a batch.
• Different length utterances should have different minibatch sizes (This might be different from the usual approach which uses same minibatch size. We have to use the kaldi approach for this).

[Note from Dan: we could consider padding with silence or speed-perturbing (like Hossein does) to make sure the utterance lengths are not all distinct. Also, I’d like to be able to support chunks of utterances, but this is not a hurry right now. You may have to have a mechanism to use different minibatch sizes for different utterance lengths, if you have very variable utterance lengths.]