codeforequity-at / botium-speech-processing

Botium Speech Processing

Botium Speech Processing

Botium Speech Processing is a unified, developer-friendly API to the best available free and Open-Source Speech-To-Text and Text-To-Speech services.

What is it ?

Botium Speech Processing is a get-shit-done-style Open-Source software stack, the configuration options are rudimentary: it is highly opinionated about the included tools, just get the shit done.

• With Kaldi a reasonable speech recogniction performance is available with freely available data sources.
• MaryTTS is currently the best freely available speech synthesis software
• SoX is the swiss army-knife for audio file processing

While the included tools in most cases cannot compete with the big cloud-based products, for lots of applications the trade-off between price and quality is at least reasonable.

Read about the project history here

Possible Applications

Some examples what you can do with this:

• Synthesize audio tracks for Youtube tutorials
• Build voice-enabled chatbot services (for example, IVR systems)
  • see the Rasa Custom Voice Channel
• Classification of audio file transcriptions
• Automated Testing of Voice services with Botium

Installation

Software and Hardware Requirements

• 8GB of RAM (accessible for Docker) and 40GB free HD space (for full installation)
• Internet connectivity
• docker
• docker-compose

Note: memory usage can be reduced if only one language for Kaldi is required - default configuration comes with two languages.

Full Installation (Prebuilt Docker Images)

Clone or download this repository and start with docker-compose:

> docker-compose up -d

This will download the latest released prebuilt images from Dockerhub. To download the latest developer images from Dockerhub:

> docker-compose --env-file .env.develop up

Point your browser to http://127.0.0.1 to open the Swagger UI and browse/use the API definition.

Slim Cloud-Specific Installation (Prebuilt Docker Images)

For the major cloud providers there are additional docker-compose files. If using those, the installation is more slim, as there is only the frontend-service required. For instance, add your Azure subscription key and Azure region key to the file docker-compose-azure.yml and start the services:

> docker-compose -f docker-compose-azure.yml up -d

Optional: Build Docker Images

You can optionally built your own docker images (if you made any changes in this repository, for instance to download the latest version of a model). Clone or download this repository and run docker-compose:

> docker-compose -f docker-compose-dev.yml up -d

This will take some time to build.

Configuration

This repository includes a reasonable default configuration:

• Use MaryTTS for TTS
• Use Kaldi for STT
• Use SoX for audio file conversion
• Languages included:
  • German
  • English

Configuration changes with environment variables. See comments in this file.

Recommendation: Do not change the .env file but create a .env.local file to overwrite the default settings. This will prevent troubles on future git pull

Request-Specific Configuration

If there is a JSON-formatted request body, or a multipart request body, certain sections are considered:

• credentials to override the server default credentials for cloud services
• config to override the server default settings for the cloud API calls

See samples below

Securing the API

The environment variable BOTIUM_API_TOKENS contains a list of valid API Tokens accepted by the server (separated by whitespace or comma). The HTTP Header BOTIUM_API_TOKEN is validated on each call to the API.

Caching

For performance improvements, the result of the speech-to-text and text-to-speech calls are cached (by MD5-hash of audio or input text). To enforce reprocessing empty the cache directories:

• frontent/resources/.cache/stt
• frontent/resources/.cache/tts

Testing

Point your browser to http://127.0.0.1/ to open Swagger UI to try out the API.

Point your browser to http://127.0.0.1/dictate/ to open a rudimentary dictate.js-interface for testing speech recognition (for Kaldi only)

Attention: in Google Chrome this only works with services published as HTTPS, you will have to take of this yourself. For example, you could publish it via ngrok tunnel.

Point your browser to http://127.0.0.1/tts to open a MaryTTS interface for testing speech synthesis.

Real Time API

It is possible to stream audio from real-time audio decoding: Call the /api/sttstream/{language} endpoint to open a websocket stream, it will return three urls:

• wsUri - the Websocket uri to stream your audio to. By default, it accepts wav-formatted audio-chunks
• statusUri - check if the stream is still open
• endUri - end audio streaming and close websocket

File System Watcher

Place audio files in these folders to receive the transript in the folder watcher/stt_output:

• watcher/stt_input_de
• watcher/stt_input_en

Place text files in these folders to receive the synthesized speech in the folder watcher/tss_output:

• watcher/tts_input_de
• watcher/tts_input_en

API Definition

See swagger.json:

• HTTP POST to /api/stt/{language} for Speech-To-Text

  curl -X POST "http://127.0.0.1/api/stt/en" -H "Content-Type: audio/wav" -T sample.wav

• HTTP POST to /api/stt/{language} for Speech-To-Text with Google, including credentials

  curl -X POST "http://127.0.0.1/api/stt/en-US?stt=google" -F "google={"credentials": {"private_key": "xxx", "client_email": "xxx"}}" -F [email protected]

• HTTP POST to /api/stt/{language} for Speech-To-Text with Google, including switch to MP3 encoding

  curl -X POST "http://127.0.0.1/api/stt/en-US?stt=google" -F "google={"config": {"encoding": "MP3"}}" -F [email protected]

• HTTP POST to /api/stt/{language} for Speech-To-Text with IBM, including credentials

  curl -X POST "http://127.0.0.1/api/stt/en-US?stt=ibm" -F "google={"credentials": {"apikey": "xxx", "serviceUrl": "xxx"}}" -F [email protected]

• HTTP GET to /api/tts/{language}?text=... for Text-To-Speech

  curl -X GET "http://127.0.0.1/api/tts/en?text=hello%20world" -o tts.wav

• HTTP POST to /api/convert/{profile} for audio file conversion

  curl -X POST "http://127.0.0.1/api/convert/mp3tomonowav" -H "Content-Type: audio/mp3" -T sample.mp3 -o sample.wav

Contributing

To be done: contribution guidelines.

We are open to any kind of contributions and are happy to discuss, review and merge pull requests.

Big Thanks

This project is standing on the shoulders of giants.

• Kaldi GStreamer server and Docker images
• MaryTTS
• SVOX Pico Text-to-Speech
• Kaldi
• Kaldi Tuda Recipe
• Zamia Speech
• Deepspeech and Deepspeech German
• SoX
• dictate.js

Changelog

2022-03-06

• Voice effects to consider audio file length

2022-02-28

• Applied Security Best Practices (not run as root user)

2022-01-12

• Added support for Azure Speech Services

2021-12-07

• Added endpoints for streaming audio and responses

2021-12-01

• Added option to hand over cloud credentials in request body

2021-01-26

• Added several profiles for adding noise or other audio artifacts to your files
• Added custom channel for usage with Rasa

2020-12-18

• Adding support for Google Text-To-Speech
• Adding support for listing and using available TTS voices
• Added sample docker-compose configurations for PicoTTS and Google

2020-03-05

• Optional start/end parameters for audio file conversion to trim an audio file by time codes formatted as mm:ss (01:32)

2020-02-22

• Additional endpoint to calculate the Word Error Rate (Levenshtein Distance) between two texts
• When giving the hint-parameter with the expected text to the STT-endpoint, the Word Error Rate will be calculated and returned
• When multiple STT- or TTS-engines are configured, select the one to use with the stt or tts parameter (in combination with the Word Error Rate calculation useful for comparing performance of two engines)

2020-01-31

• Using pre-trained models from Zamia Speech for speech recognition
• Using latest Kaldi build
• Added file system watcher to transcribe and synthesize audio files

botium-speech-processing's Issues

BUILD FAILED in 48s 1 actionable task: 1 executed ERROR: Service 'tts' failed to build: The command '/bin/sh -c cat components.json | jq .[].artifact | xargs ./marytts insta

Getting this error:

BUILD FAILED in 48s
1 actionable task: 1 executed
ERROR: Service 'tts' failed to build: The command '/bin/sh -c cat components.json | jq .[].artifact | xargs ./marytts insta

Cycling redirects

Hi!
I'm trying to start botium in docker under nginx reverse proxy.

location / {
    	proxy_pass http://127.0.0.1:8008/;
        proxy_http_version 1.1;
		proxy_redirect off;
    }

But i have sycling redirects in Network tab of Chrome Dev Tools on any page

Same issue on https://speech.botiumbox.com/ , https://speech.botiumbox.com/dictate, https://speech.botiumbox.com/tts

Also at page https://speech.botiumbox.com/tts
Try to click on any button
In Chrome console

tts:132 Uncaught ReferenceError: requestSynthesis is not defined
    at HTMLInputElement.onclick (tts:132)

botium speech processing frontend always restart and nginx always 502 Bad Gateway

Describe the bug
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
0bd4cc8c1b2d botium/botium-speech-kaldi-en:latest "/usr/bin/supervisor…" 22 minutes ago Up 21 minutes 56180/tcp botiumspeechprocessing_stt-en_1
c1b3c5309212 botium/botium-speech-watcher:latest "/usr/bin/supervisor…" 22 minutes ago Up 21 minutes botiumspeechprocessing_watcher_1
eea24e613233 botium/botium-speech-kaldi-de:latest "/usr/bin/supervisor…" 22 minutes ago Up 21 minutes 56180/tcp botiumspeechprocessing_stt-de_1
794598aa6dd7 nginx "/docker-entrypoint.…" 22 minutes ago Up 21 minutes 0.0.0.0:80->80/tcp, :::80->80/tcp botiumspeechprocessing_nginx_1
cd8a36723aa8 botium/botium-speech-dictate:latest "docker-entrypoint.s…" 22 minutes ago Up 21 minutes 56100/tcp botiumspeechprocessing_dictate_1
06df15d556b7 botium/botium-speech-frontend:latest "/bin/sh -c 'npm run…" 22 minutes ago Restarting (1) 32 seconds ago botiumspeechprocessing_frontend_1
1b5e2047d9f6 botium/botium-speech-marytts:latest "/bin/sh -c ./marytts" 22 minutes ago Up 21 minutes 59125/tcp botiumspeechprocessing_tts_1

docker logs botiumspeechprocessing_frontend_1

[email protected] start-dist /app
node -r dotenv-flow/config ./src/server.js

dotenv-flow: "BOTIUM_SPEECH_PROVIDER_TTS" is already defined in process.env and will not be overwritten
dotenv-flow: "BOTIUM_SPEECH_PROVIDER_STT" is already defined in process.env and will not be overwritten
/app/node_modules/mkdirp/lib/mkdirp-native.js:35
throw er
^

Error: EACCES: permission denied, mkdir '/app/resources/.cache/stt'
at Object.mkdirSync (fs.js:1014:3)
at mkdirpNativeSync (/app/node_modules/mkdirp/lib/mkdirp-native.js:29:10)
at Function.mkdirpSync [as sync] (/app/node_modules/mkdirp/index.js:21:7)
at Object. (/app/src/routes.js:31:26)
at Module._compile (internal/modules/cjs/loader.js:1085:14)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:1114:10)
at Module.load (internal/modules/cjs/loader.js:950:32)
at Function.Module._load (internal/modules/cjs/loader.js:790:12)
at Module.require (internal/modules/cjs/loader.js:974:19)
at require (internal/modules/cjs/helpers.js:101:18) {
errno: -13,
syscall: 'mkdir',
code: 'EACCES',
path: '/app/resources/.cache/stt'
}
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] start-dist: node -r dotenv-flow/config ./src/server.js
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] start-dist script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR! /home/node/.npm/_logs/2024-02-06T16_35_40_834Z-debug.log

and nginx 502 Bad Gateway
so i guess if you can fix the issus frontend always restart so everything will be better.

Audio File Size Limit

Hi, I am running it through docker and am able to successfully convert speech to text using http file upload

The problem is, when I try uploading a large audio file (188mb), I get Content-Length too long Error in the master.log file.

I have already increased and resolved the nginx limit, and express limit, The request goes through these and reaches tornado but tornado doesn't let it through due to its own limit.

503: Service Unavailable

I ran:

curl -X POST "http://127.0.0.1/api/stt/en" -H "Content-Type: audio/wav" -T test1.wav

response:

{"status":"error","message":"Calling url http://stt_en:80/client/dynamic/recognize failed with code 503: Service Unavailable"}

I have tried a test through the Swagger API page and get the same response.

Am I required to do further configuration?

HTTP 200 error code on /dictate API endpoint

Hi,

Firstly, thanks for your efforts!

I seem to have all the requisite containers running:

$ docker ps
CONTAINER ID        IMAGE                             COMMAND                  CREATED             STATUS              PORTS                NAMES
71c1ef071742        botiumspeechprocessing_frontend   "/bin/sh -c 'npm sta…"   37 minutes ago      Up 8 seconds        56000/tcp            botiumspeechprocessing_frontend_1
ac5b093ba5a8        botiumspeechprocessing_tts        "/bin/sh -c ./marytts"   37 minutes ago      Up 2 seconds        59125/tcp            botiumspeechprocessing_tts_1
3e43c78a63fc        botiumspeechprocessing_dictate    "docker-entrypoint.s…"   37 minutes ago      Up 6 seconds        56100/tcp            botiumspeechprocessing_dictate_1
93b06574ac09        botiumspeechprocessing_stt_de     "/usr/bin/supervisord"   37 minutes ago      Up 5 seconds        80/tcp               botiumspeechprocessing_stt_de_1
15199910a926        botiumspeechprocessing_watcher    "/usr/bin/supervisord"   37 minutes ago      Up 6 seconds                             botiumspeechprocessing_watcher_1
26689b50571c        botiumspeechprocessing_stt_en     "/usr/bin/supervisord"   37 minutes ago      Up 9 seconds        80/tcp               botiumspeechprocessing_stt_en_1
7256a8ac9e27        nginx                             "nginx -g 'daemon of…"   37 minutes ago      Up 3 seconds        0.0.0.0:80->80/tcp   botiumspeechprocessing_nginx_1

However, http://127.0.0.1/dictate URL autocompletes to http://127.0.0.1/dictate/demos/demos/demos/demos/demos/demos/demos/demos/demos/demos/demos/demos/demos/demos/demos/demos/demos/demos/mob.html while giving HTTP 200.

Please note, /api/stt/en seems to work fine with a wav file.

Thanks!

ERROR: Service 'frontend' failed to build: The command '/bin/sh -c npm install --no-optional --production' returned a non-zero code: 127

Hi, I'm getting this error. Any idea how I could solve this?