mgonzs13 / whisper_ros

silero-vad + whisper.cpp (speech-to-text) for ROS 2

whisper_ros

This repository provides a set of ROS 2 packages to integrate whisper.cpp into ROS 2 using audio_common. Besides, silero-vad is used to perform VAD (Voice Activity Detection).

Installation

$ cd ~/ros2_ws/src
$ git clone https://github.com/mgonzs13/audio_common.git
$ git clone --recurse-submodules https://github.com/mgonzs13/whisper_ros.git
$ sudo apt install portaudio19-dev
$ pip3 install -r audio_common/requirements.txt
$ pip3 install -r whisper_ros/requirements.txt
$ cd ~/ros2_ws
$ colcon build

CUDA

To run llama_ros with CUDA, you have to install the CUDA Toolkit and the following line in the CMakeLists.txt must be uncommented:

option(WHISPER_CUBLAS "whisper: support for cuBLAS" ON)

Usage

Run Silero for VAD and Whisper for STT:

$ ros2 launch whisper_bringup whisper.launch.py

Send a goal action to listen:

$ ros2 action send_goal /whisper/listen whisper_msgs/action/STT "{}"

Or try the example of a whisper client:

$ ros2 run whisper_ros whisper_client_node

whisper_ros's Issues

How to transcribe in spanish

Hello,
I change the param "language": LaunchConfiguration("language", default="es") and the param "translate": LaunchConfiguration("translate", default=False).
The issue is that the programm doesn't transcribe well in spanish. I speak in spanish but the programm always try to translate to english.
The model that i load is the distil-large-v3 with this params:
[whisper_node-1] [INFO] Processing, 4 threads, lang = es, task = transcribe, timestamps = 1
Thank you