modal-labs / quillman

A chat app that transcribes audio in real-time, streams back a response from a language model, and synthesizes this response as natural-sounding speech.

QuiLLMan: Voice Chat with LLMs

A complete chat app that transcribes audio in real-time, streams back a response from a language model, and synthesizes this response as natural-sounding speech.

This repo is meant to serve as a starting point for your own language model-based apps, as well as a playground for experimentation. Contributions are welcome and encouraged!

The language model used is Zephyr. OpenAI Whisper is used for transcription, and Metavoice Tortoise TTS is used for text-to-speech. The entire app, including the frontend, is made to be deployed serverlessly on Modal.

You can find the demo live here.

[Note: this code is provided for illustration only; please remember to check the license before using any model for commercial purposes.]

File structure

1. React frontend (src/frontend/)
2. FastAPI server (src/app.py)
3. Whisper transcription module (src/transcriber.py)
4. Tortoise text-to-speech module (src/tts.py)
5. Zephyr language model module (src/llm_zephyr.py)

Read the accompanying docs for a detailed look at each of these components.

Developing locally

Requirements

• modal installed in your current Python virtual environment (pip install modal)
• A Modal account
• A Modal token set up in your environment (modal token new)

Develop on Modal

To serve the app on Modal, run this command from the root directory of this repo:

modal serve src.app

In the terminal output, you'll find a URL that you can visit to use your app. While the modal serve process is running, changes to any of the project files will be automatically applied. Ctrl+C will stop the app.

Deploy to Modal

Once you're happy with your changes, deploy your app:

modal deploy src.app

[Note that leaving the app deployed on Modal doesn't cost you anything! Modal apps are serverless and scale to 0 when not in use.]

quillman's Issues

Cuda image `THIS IMAGE IS DEPRECATED and is scheduled for DELETION`

Loading the example image provided in llm_vicuna.py throws a huge warning:

==========
== CUDA ==
==========

CUDA Version 11.7.0

Container image Copyright (c) 2016-2022, NVIDIA CORPORATION & AFFILIATES. All rights reserved.

This container image and its contents are governed by the NVIDIA Deep Learning Container License.
By pulling and using the container, you accept the terms and conditions of this license:
https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license

A copy of this license is made available in this container at /NGC-DL-CONTAINER-LICENSE for your convenience.
*************************
** DEPRECATION NOTICE! **
*************************
THIS IMAGE IS DEPRECATED and is scheduled for DELETION.
    https://gitlab.com/nvidia/container-images/cuda/blob/master/doc/support-policy.md

I've tried updating it to later version and I found myself in the unfortunate rabbit hole of "pytorch has been compiled with the Cuda 11.7 version".

I raise this issue hoping someone at their discretion could update providing an example of how it should be done. Thank you. :)

Could not find the quantized model in .pt or .safetensors format

modal serve src.app

The cache for model files in Transformers v4.22.0 has been updated. Migrating your old cache. This is a one-time only operation. You can interrupt this and resume the migration later on by calling `transformers.utils.move_cache()`.
0it [00:00, ?it/s]0it [00:00, ?it/s]
Downloading (…)okenizer_config.json:   0%|          | 0.00/727 [00:00<?, ?B/s]Downloading (…)okenizer_config.json: 100%|██████████| 727/727 [00:00<00:00, 196kB/s]
Downloading tokenizer.model:   0%|          | 0.00/500k [00:00<?, ?B/s]Downloading tokenizer.model: 100%|██████████| 500k/500k [00:00<00:00, 94.7MB/s]
Downloading (…)cial_tokens_map.json:   0%|          | 0.00/411 [00:00<?, ?B/s]Downloading (…)cial_tokens_map.json: 100%|██████████| 411/411 [00:00<00:00, 511kB/s]
Loading GPTQ quantized model...
Could not find the quantized model in .pt or .safetensors format, exiting...
Traceback (most recent call last):
  File "/pkg/modal/_container_entrypoint.py", line 301, in handle_user_exception
    yield
  File "/pkg/modal/_container_entrypoint.py", line 443, in call_function_async
    obj.__enter__()
  File "/root/src/llm_vicuna.py", line 79, in __enter__
    model = load_quantized(MODEL_NAME)
  File "/FastChat/fastchat/serve/load_gptq_model.py", line 60, in load_quantized
    exit()
  File "/usr/lib/python3.8/_sitebuiltins.py", line 26, in __call__
    raise SystemExit(code)
SystemExit: None
Runner failed with exception: SystemExit(None)
2023-05-30T15:45:37+0000 User exception caught, exiting

Error running Quillman code with `modal serve src.app`

Hi there,

I am experiencing an error running the code with modal serve src.app. Here is the error message:

Building image im-uapgkrskiwyzvdznvrsjdwytfles

=> Step 0: FROM base

=> Step 1: RUN cd /FastChat && python download-model.py anon8231489123/vicuna-13b-GPTQ-4bit-128g
Traceback (most recent call last):
  File "download-model.py", line 196, in <module>
    links, sha256, is_lora = get_download_links_from_huggingface(model, branch)
  File "download-model.py", line 130, in get_download_links_from_huggingface
    fname = dict[i]['path']
KeyError: 0
Terminating task due to error: command "cd /FastChat && python download-model.py anon8231489123/vicuna-13b-GPTQ-4bit-128g" had exit status: 1

Could you please help me identify the root cause of this error and provide guidance on how to fix it?

Thank you

Getting an error when I run "modal serve src.app"

Hello,
I get the error "task exited with failure, status = exit status: 1" when I try to run modal serve src.app.
What could be the reason for this?

Demo not working

This sounds like a cool app, but unfortunately the demo doesn't seem to be working: https://modal-labs--quillman-web.modal.run/

Thanks!

Feature request: add an option to type the query instead of speak it

For those who don't have a microphone.

Why not use Whisper for text to speech itself?

And would that give any cost savings with Modal?

https://towardsdatascience.com/speech-to-text-with-openais-whisper-53d5cea9005e

modal : The term 'modal' is not recognized as the name of a cmdlet, function, script file, or operable program.

downloaded zip, ran pip install modal-client, then modal serve src.app, and got the following error:

modal : The term 'modal' is not recognized as the name of a cmdlet, function, script file, or operable program.
Check the spelling of the name, or if a path was included, verify that the path is correct and try again.
At line:1 char:1
+ modal serve src.app
+ ~~~~~
+ CategoryInfo : ObjectNotFound: (modal:String) [], CommandNotFoundException
+ FullyQualifiedErrorId : CommandNotFoundException