terry-yip / speech-to-text

Speaker diarization and speech to text

Speech To Text

Project to combine VAD, Speaker Diarization, Speech Recognition together.

Getting Started

As DeepSpeech pre-trained English model is too big to commit to git. You could download from (https://github.com/mozilla/DeepSpeech/releases/download/v0.6.1/deepspeech-0.6.1-models.tar.gz)

Please put the downloaded model files into folder speech-to-text/deepspeech/models

# Create and activate a virtual environment
python3 -m venv speech-to-text/env
source speech-to-text/env/bin/activate

# Install prerequisites
pip3 install -r requirements.txt

# Speech To Text
python3 speech_to_text.py --audio=wavs/test2.wav

It will output the txt file with speakers and speech text, side by side the wav file

Overview

It's just to combine speaker diarization and speech recognization together.

Only support 16k sample rate PCM wav file. You can use ffmpeg to convert sound file format. i.e.

ffmpeg -i input.mp3 -acodec pcm_s16le -ar 16000 output.wav

Main flows

1. Filter out silence frames and break down to segments with webrtcvad.

2. Generate utterances spec with librosa

3. Get utterances features with ghostvlad

4. Classify features with uisrnn model

5. Recognize speeches segment by segment with deepspeech

It might takes long period(tens minutes) if the wav is too big.(seems uisrnn part takes the longest) The test wavs in the wavs folder are from movie sound clips. The speech accuracy is not perfect, it might relative to the pretrained deepspeech model and the background noise

Prerequisites

• pytorch
• keras
• tensorflow
• pyaudio
• librosa
• webrtcvad
• deepspeech

References

• DeepSpeech
• Speaker-Diarization
• uis-rnn
• py-webrtcvad
• librosa
• pyAudioAnalysis
• kaldi
• awesome-diarization

Tip

Following are the libs version installed in my env, just for your reference.

• absl-py 0.7.1
• astor 0.8.0
• astroid 2.4.2
• audioread 2.1.8
• cffi 1.12.3
• decorator 4.4.0
• deepspeech 0.5.1
• gast 0.2.2
• google-pasta 0.1.7
• grpcio 1.22.0
• h5py 2.9.0
• isort 5.6.4
• joblib 0.13.2
• Keras 2.2.4
• Keras-Applications 1.0.8
• Keras-Preprocessing 1.1.0
• lazy-object-proxy 1.4.3
• librosa 0.7.0
• llvmlite 0.29.0
• Markdown 3.1.1
• mccabe 0.6.1
• numba 0.45.0
• numpy 1.16.4
• pip 19.2
• protobuf 3.9.0
• PyAudio 0.2.11
• pycparser 2.19
• pylint 2.6.0
• PyYAML 5.1.1
• resampy 0.2.1
• scikit-learn 0.21.2
• scipy 1.3.0
• setuptools 41.0.1
• six 1.12.0
• SoundFile 0.10.2
• tensorboard 1.14.0
• tensorflow 1.14.0
• tensorflow-estimator 1.14.0
• termcolor 1.1.0
• toml 0.10.1
• torch 1.1.0.post2
• typed-ast 1.4.1
• webrtcvad 2.0.10
• Werkzeug 0.15.5
• wheel 0.33.4
• wrapt 1.11.2

speech-to-text's Issues

class VladPooling(keras.engine.Layer): AttributeError: module 'keras.engine' has no attribute 'Layer

Pls Help me for solving this issue

Version of libs

Hi,

I'm trying to run your code, but are having some trouble with incompatible libraries.
Which version did you use for..

pytorch
keras
tensorflow
pyaudio
librosa
webrtcvad
deepspeech

?

Thanks :)

wavTranscriber - IndexError: list index out of range

Hi @terry-yip

I have one question if you dont mind.

When I run your app with

python3 speech_to_text.py --audio=wavs/test2.wav

i get error:

DEBUG:__main__:Processing speech recognition
Traceback (most recent call last):
  File "speech_to_text.py", line 69, in <module>
    main(sys.argv[1:])
  File "speech_to_text.py", line 35, in main
    output_graph, alphabet, lm, trie = wavTranscriber.resolve_models(model_path)
  File "/home/ivan/PycharmProjects/speech-to-text/wavTranscriber.py", line 77, in resolve_models
    pb = glob.glob(dirName + "/*.pb")[0]
IndexError: list index out of range

Can you please tell me if you can point where is the problem in this case?

I run this code on Ubuntu.

nice work btw!

EDIT: I have solved this step.
Can you please provide what versions of deepspeech, tensorflow etc. you used?

wavTranscriber.py IndexError: list index out of range

Hi!
I'm getting the same error.
pb = glob.glob(dirName + "/*.pb")[0]
IndexError: list index out of range

Can yu pls tell how this issue was solved.
Thanks

alphabet = glob.glob(dirName + "/alphabet.txt")[0] IndexError: list index out of range

DEBUG:speaker_diarization:[Speaker diarization] Clustering utterance features
DEBUG:speaker_diarization:[Speaker diarization] Tagging segments speakers
DEBUG:main:Processing speech recognition
Traceback (most recent call last):
File "/content/speech-to-text/speech_to_text.py", line 69, in
main(sys.argv[1:])
File "/content/speech-to-text/speech_to_text.py", line 35, in main
output_graph, alphabet, lm, trie = wavTranscriber.resolve_models(model_path)
File "/content/speech-to-text/wavTranscriber.py", line 80, in resolve_models
alphabet = glob.glob(dirName + "/alphabet.txt")[0]
IndexError: list index out of range

Please resolve this issue.

Which Python version is used here!