pipecat-ai / pipecat Goto Github PK
View Code? Open in Web Editor NEWOpen Source framework for voice and multimodal conversational AI
License: BSD 2-Clause "Simplified" License
Open Source framework for voice and multimodal conversational AI
License: BSD 2-Clause "Simplified" License
The conflict is caused by:
pipecat-ai[daily,openai,silero] 0.0.36 depends on daily-python~=0.10.1; extra == "daily"
pipecat-ai[daily,openai,silero] 0.0.35 depends on daily-python~=0.10.1; extra == "daily"
pipecat-ai[daily,openai,silero] 0.0.34 depends on daily-python~=0.10.1; extra == "daily"
pipecat-ai[daily,openai,silero] 0.0.33 depends on daily-python~=0.10.1; extra == "daily"
pipecat-ai[daily,openai,silero] 0.0.32 depends on daily-python~=0.10.0; extra == "daily"
pipecat-ai[daily,openai,silero] 0.0.31 depends on daily-python~=0.9.0; extra == "daily"
pipecat-ai[daily,openai,silero] 0.0.30 depends on daily-python~=0.9.0; extra == "daily"
pipecat-ai[daily,openai,silero] 0.0.29 depends on daily-python~=0.9.0; extra == "daily"
pipecat-ai[daily,openai,silero] 0.0.28 depends on daily-python~=0.9.0; extra == "daily"
pipecat-ai[daily,openai,silero] 0.0.27 depends on daily-python~=0.9.0; extra == "daily"
pipecat-ai[daily,openai,silero] 0.0.26 depends on daily-python~=0.9.0; extra == "daily"
pipecat-ai[daily,openai,silero] 0.0.25 depends on daily-python~=0.9.0; extra == "daily"
pipecat-ai[daily,openai,silero] 0.0.24 depends on daily-python~=0.9.0; extra == "daily"
pipecat-ai[daily,openai,silero] 0.0.23 depends on daily-python~=0.8.0; extra == "daily"
pipecat-ai[daily,openai,silero] 0.0.22 depends on daily-python~=0.8.0; extra == "daily"
pipecat-ai[daily,openai,silero] 0.0.21 depends on daily-python~=0.7.4; extra == "daily"
pipecat-ai[daily,openai,silero] 0.0.20 depends on daily-python~=0.7.4; extra == "daily"
pipecat-ai[daily,openai,silero] 0.0.19 depends on daily-python~=0.7.4; extra == "daily"
pipecat-ai[daily,openai,silero] 0.0.18 depends on daily-python~=0.7.4; extra == "daily"
pipecat-ai[daily,openai,silero] 0.0.17 depends on daily-python~=0.7.4; extra == "daily"
pipecat-ai[daily,openai,silero] 0.0.16 depends on daily-python~=0.7.4; extra == "daily"
pipecat-ai[daily,openai,silero] 0.0.15 depends on daily-python~=0.7.4; extra == "daily"
pipecat-ai[daily,openai,silero] 0.0.14 depends on daily-python~=0.7.4; extra == "daily"
pipecat-ai[daily,openai,silero] 0.0.13 depends on daily-python~=0.7.4; extra == "daily"
pipecat-ai[daily,openai,silero] 0.0.12 depends on daily-python~=0.7.4; extra == "daily"
pipecat-ai[daily,openai,silero] 0.0.11 depends on daily-python~=0.7.4; extra == "daily"
pipecat-ai[daily,openai,silero] 0.0.10 depends on daily-python~=0.7.4; extra == "daily"
pipecat-ai[daily,openai,silero] 0.0.9 depends on daily-python~=0.7.4; extra == "daily"
Description:
To enhance the development of consumer-facing applications, it would be extremely beneficial to have an out-of-the-box feature that provides a real-time counter for the usage of tokens/characters for various services such as LLMs (Language Models), TTS (Text-to-Speech), STT (Speech-to-Text), and others. This feature would enable developers to set and monitor usage limits effectively.
Benefits:
Use Case:
As a developer building a consumer-facing application, I want to set a usage limit on the number of tokens/characters for services like LLMs, TTS, and STT. Having a real-time counter integrated into the system would allow me to track usage efficiently and ensure that my application stays within the predefined limits.
Suggested Implementation:
Conclusion:
Implementing a real-time usage counter for service providers would greatly improve the ability of developers to manage and optimize the use of various services, leading to more efficient and cost-effective applications.
The OpenAI Text-To-Speech sounds very natural in different languages. Is this something pipecat wants to support?
Hello, I'd like to use Italian language but I cannot find where to change it
I am trying to implement a FastAPI server that can accept both Twilio calls & websocket calls but found that the FastAPIWebSocket transport did not drop in nicely.
I was using the examples/websocket-server
client as a means to check my work and was confused why the WebSocketServerTransport worked whereas the FastAPI one did not. I noticed playback from the voice assistant was working but no inputs were getting thru.
I did some digging at the code for the transport input and found that the following monkey patched fixed it for this particular usecase:
transport = FastAPIWebsocketTransport(
websocket=websocket_client,
params=FastAPIWebsocketParams(
audio_out_enabled=True,
add_wav_header=True,
vad_enabled=True,
vad_analyzer=SileroVADAnalyzer(),
vad_audio_passthrough=True,
serializer=ProtobufFrameSerializer()
)
)
async def _patched_receive_messages(self):
async for message in self._websocket.iter_bytes():
frame = self._params.serializer.deserialize(message)
if not frame:
continue
if isinstance(frame, AudioRawFrame):
await self.push_audio_frame(frame)
else:
await self._internal_push_frame(frame)
await self._callbacks.on_client_disconnected(self._websocket)
transport._input._receive_messages = partial(_patched_receive_messages, transport._input)
The two changes were:
self._websocket.iter_bytes()
instead of self._websocket.iter_text()
to tease out the protobuf data_internal_push_frame
function that isnt used by the current implementationThis makes sense when working with the Twilio WS api but would be nice if it could be generalized to accommodate this workflow
I modified the llm service code to local ollama. After the program started, everything was fine at first. The robot asked me what kind of story I wanted to hear. I answered it, and then it told me the first paragraph. When it finished telling this When I was asked about my next step needs, the microphone on the page turned gray, and there was no error message in the log. The page was obviously waiting for my response, but because the microphone turned gray, I couldn't answer it. I observed that the Chrome tab shows that the microphone is running normally, I posted the log below, hoping someone can help me.
import random # Generate a random number between A and B (inclusive) A = 1 B = 100 random_number = random.randint(A, B) print(random_number)
In this code, randint(A, B)
generates a random integer N
such that `A In the heart of this enchanted forest, a curious adventurer stumbled upon a hidden path <the adventurer: a young girl with wild curly hair, wearing a flowing white dress, holding a lantern> She had been searching for the legendary Moonflower, said to bloom only once a year under the gentle light of the crescent moon How would you like the story to continue?]I've found an AttributeError: 'AzureLLMService' object has no attribute '_endpoint'
error when attempting to use the AzureLLMService
instead of the BaseOpenAILLMService
in one of the provided example projects.
The error appears to be an instance of using an uninitialised attribute. The AzureLLMService
constructor calls the parent BaseOpenAILLMService
constructor before initialising its own AttributeError
attribute, resulting in an AttributeError
when the parent constructor tries to access the non-existent _endpoint
I'm using an M3 Pro MacBook Pro running MacOS Sonoma 14.1 (23B2073) and Python 3.11.7 using a virtual environment.
I've been trying to run the simple-chatbot
example project but using the Azure OpenAI Service by replacing the provided BaseOpenAILLMService
with AzureLLMService
and updating the appropriate environment variables where required.
The resulting invocation (bot.py
line 108) looks like:
llm = AzureLLMService(
api_key=os.getenv("AZURE_OAI_KEY"),
endpoint=os.getenv("AZURE_OAI_ENDPOINT"),
model=os.getenv("AZURE_OAI_MODEL"),
)
Granted that all other required environment variables are provided and correct (they are), running the server.py
as instructed by the README.md
successfully runs the FastAPI server.
The actual exception is raised when accessing the /start
endpoint via web browser, which launches the Daily UI but prevents the bot from joining the room.
The example project should work just fine by replacing the BaseOpenAILLMService
with the AzureLLMService
.
An exception is being raised:
File "[REDACTED]/pipecat/examples/simple-chatbot/bot.py", line 108, in main
llm = AzureLLMService(
^^^^^^^^^^^^^^^^
File "[REDACTED]/pipecat/src/pipecat/services/azure.py", line 81, in __init__
super().__init__(api_key=api_key, model=model)
File "[REDACTED]/pipecat/src/pipecat/services/openai.py", line 63, in __init__
self.create_client(api_key=api_key, base_url=base_url)
File "[REDACTED]/pipecat/src/pipecat/services/azure.py", line 90, in create_client
azure_endpoint=self._endpoint,
^^^^^^^^^^^^^^
AttributeError: 'AzureLLMService' object has no attribute '_endpoint'
The current implementation for the AzureLLMService
is:
class AzureLLMService(BaseOpenAILLMService):
def __init__(
self,
*,
api_key,
endpoint,
api_version="2023-12-01-preview",
model
):
super().__init__(api_key=api_key, model=model)
self._endpoint = endpoint
self._api_version = api_version
self._model: str = model
Its parent class is the BaseOpenAILLMService
(abbreviated):
class BaseOpenAILLMService(LLMService):
def __init__(self, model: str, api_key=None, base_url=None):
super().__init__()
self._model: str = model
self.create_client(api_key=api_key, base_url=base_url)
The AttributeError
observed when using the AzureLLMService
class can be attributed to the order of attribute initialisation and method invocation in the inheritance hierarchy.
When an instance of AzureLLMService
is created, the constructor (__init__
method) of AzureLLMService
is invoked. The first line of this constructor calls the constructor of its parent class, BaseOpenAILLMService
, using super().__init__(api_key=api_key, model=model)
. This invokes the parent constructor before the _endpoint
attribute is initialised in the AzureLLMService
constructor.
The BaseOpenAILLMService
constructor, in turn, calls the create_client
method (self.create_client(api_key=api_key, base_url=base_url)
). Due to the principles of inheritance and polymorphism, the create_client
method of the AzureLLMService
class is invoked, as it overrides the create_client
method of the parent class.
However, at this point, the _endpoint
attribute has not yet been initialised in the AzureLLMService
constructor, as the self._endpoint = endpoint
line has not been executed. Therefore, when the create_client
method of AzureLLMService
tries to access self._endpoint
, it raises an AttributeError
because the _endpoint
attribute does not exist.
This sequence of events violates the expected initialisation order and results in attempting to access an attribute before it has been properly initialised, leading to the AttributeError.
Given the current implementation, I suggest to simply move the super().__init__(api_key=api_key, model=model)
call to the very end of the AzureLLMService
constructor so that all required attributes can be defined BEFORE they are required.
The following implementation works perfectly and passes the Azure LLM Integration Tests.
class AzureLLMService(BaseOpenAILLMService):
def __init__(
self,
*,
api_key,
endpoint,
api_version="2023-12-01-preview",
model
):
self._endpoint = endpoint
self._api_version = api_version
self._model: str = model
super().__init__(api_key=api_key, model=model)
I'll be happy to open a PR for this, just wanted to be sure first. Please let me know if you have any questions.
Hi Pipecat Team,
I'm exploring the integration of Pipecat (Pipeline: Speech-to-text -> LLM -> Text-to-Speech) into my React Native app using Expo. I have a couple of questions on the best approach for this integration:
Daily Integration: Should Daily be integrated directly within my app's client, or would it be more efficient to deploy Pipecat to AWS (perhaps using a Lambda architecture with state management) and have the Expo app communicate with the cloud service and all the pipecat code living in aws?
Best Practices: Are there any specific best practices or recommendations you have for integrating Pipecat with a React Native app, especially considering the constraints and capabilities of Expo?
Any guidance or examples you could provide would be greatly appreciated!
Thank you for your help!
I've tried running the example locally and within docker.
It always fails in the same way for me:
`
ERROR: Exception in ASGI application
Traceback (most recent call last):
File "...uvicorn/protocols/websockets/websockets_impl.py", line 244, in run_asgi
result = await self.app(self.scope, self.asgi_receive, self.asgi_send) # type: ignore[func-returns-value]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "...uvicorn/middleware/proxy_headers.py", line 70, in call
return await self.app(scope, receive, send)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
...
File ".../pipecat/examples/twilio-chatbot/bot.py", line 39, in run_bot
serializer=TwilioFrameSerializer(stream_sid)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: TwilioFrameSerializer.init() takes 1 positional argument but 2 were given
`
How to change the speech recognition interface? I want to use iFlytek's or ourself's speech recognition.
I want to have a conversation in Chinese.
I am experiencing issues with my audio processing pipeline. When I comment out transport.output() in the pipeline, I can see the interaction logs without hearing the audio from ElevenLabs. However, if I uncomment that line, I neither hear the audio output nor see the transcription logs from Deepgram.
Here is the relevant part of my code:
import asyncio
import os
import sys
import aiohttp
from loguru import logger
from pipecat.frames.frames import Frame, TranscriptionFrame
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineParams, PipelineTask
from pipecat.processors.aggregators.llm_response import (
LLMAssistantResponseAggregator,
LLMUserResponseAggregator,
)
from pipecat.processors.frame_processor import FrameDirection, FrameProcessor
from pipecat.services.deepgram import DeepgramSTTService
from pipecat.services.elevenlabs import ElevenLabsTTSService
from pipecat.services.openai import OpenAILLMService
from pipecat.transports.base_transport import TransportParams
from pipecat.transports.local.audio import LocalAudioTransport
from pipecat.vad.silero import SileroVADAnalyzer
from pipecat.vad.vad_analyzer import VADParams
logger.remove(0)
logger.add(sys.stderr, level="DEBUG")
class TranscriptionLogger(FrameProcessor):
async def process_frame(self, frame: Frame, direction: FrameDirection):
await super().process_frame(frame, direction)
if isinstance(frame, TranscriptionFrame):
print(f"Transcription: {frame.text}")
async def main():
async with aiohttp.ClientSession() as session:
transport = LocalAudioTransport(
TransportParams(
audio_in_enabled=True,
audio_out_enabled=True,
transcription_enabled=True,
vad_enabled=True,
vad_analyzer=SileroVADAnalyzer(params=VADParams(min_volume=0.6)),
vad_audio_passthrough=True,
)
)
stt = DeepgramSTTService(os.environ["DEEPGRAM_API_KEY"])
llm = OpenAILLMService(
api_key=os.environ["OPENAI_API_KEY"],
model="gpt-3.5-turbo-0125",
)
messages = [
{
"role": "system",
"content": "Say hello.",
},
]
tma_in = LLMUserResponseAggregator(messages)
tma_out = LLMAssistantResponseAggregator(messages)
tts = ElevenLabsTTSService(
aiohttp_session=session,
api_key=os.environ["ELEVENLABS_API_KEY"],
voice_id=os.environ["ELEVENLABS_VOICE_ID"],
)
pipeline = Pipeline(
[
transport.input(),
stt,
tma_in,
llm,
tts,
transport.output(),
tma_out,
]
)
task = PipelineTask(pipeline, PipelineParams(allow_interruptions=True))
runner = PipelineRunner()
await runner.run(task)
if __name__ == "__main__":
asyncio.run(main())
Additionally, when I add an introductory message from the LLM, I can hear the initial message, but the interaction with the LLM stops. Here is the code snippet for that:
async def say_something():
messages.append(
{
"role": "system",
"content": "Please briefly introduce yourself to the user.",
}
)
await task.queue_frames([LLMMessagesFrame(messages)])
# await runner.run(task)
await asyncio.gather(runner.run(task), say_something())
I am using a Mac and pipecat-ai==0.0.31. Do you see any obvious issues in my code, or could the problem be elsewhere?
This error just started appearing this afternoon, on the latest version of the code, didn't update any libraries, same daily room. Don't know what could have caused this
2024-06-30 05:51:24.814 | ERROR | pipecat.vad.silero:voice_confidence:74 - Error analyzing audio with Silero VAD: The following operation failed in the TorchScript interpreter.
Traceback of TorchScript, serialized code (most recent call last):
File "code/__torch__/vad/model/vad_annotator.py", line 26, in forward
if _2:
_3 = torch.format(_0, (torch.size(x0))[-1])
ops.prim.RaiseException(_3, "builtins.ValueError")
~~~~~~~~~~~~~~~~~~~~~~~ <--- HERE
else:
pass
Traceback of TorchScript, original code (most recent call last):
File "/home/keras/notebook/nvme1/adamnsandle/silero-models-research/vad/model/vad_annotator.py", line 484, in forward
num_samples = 512 if sr == 16000 else 256
if x.shape[-1] != num_samples:
raise ValueError(f"Provided number of samples is {x.shape[-1]} (Supported values: 256 for 8000 sample rate, 512 for 16000)")
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ <--- HERE
batch_size = x.shape[0]
builtins.ValueError: Provided number of samples is 640 (Supported values: 256 for 8000 sample rate, 512 for 16000)
Hi there, I'm getting this error when using the VAD module:
Provided number of samples is 320 (Supported values: 256 for 8000 sample rate, 512 for 16000)
here is my little demo code. The sample rate is 8000 with 1 channel
wav = wave.open(r'demo-instruct.wav', "rb")
total_frames = wav.getnframes()
sample_rate = wav.getframerate()
num_channels = wav.getnchannels()
print(sample_rate)
print(num_channels)
vad_analyzer = SileroVADAnalyzer(
sample_rate=sample_rate, params=VADParams())
sent_frames = 0
while sent_frames < total_frames:
# Read 100ms worth of audio frames.
frames = wav.readframes(128)
audio_data = np.frombuffer(frames, dtype=np.int16)
# Convert back to bytes
frames_to_send = audio_data.tobytes()
if len(frames_to_send) > 0:
sent_frames += len(frames_to_send)
new_vad_state = vad_analyzer.analyze_audio(frames_to_send)
print(new_vad_state)
Any ideas on this? The program itself doesn't raise an error...it is something inside silero itself that is printing the error but not actually raising:
Traceback of TorchScript, original code (most recent call last):
File "/home/keras/notebook/nvme1/adamnsandle/silero-models-research/vad/model/vad_annotator.py", line 484, in forward
num_samples = 512 if sr == 16000 else 256
if x.shape[-1] != num_samples:
raise ValueError(f"Provided number of samples is {x.shape[-1]} (Supported values: 256 for 8000 sample rate, 512 for 16000)")
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ <--- HERE
batch_size = x.shape[0]
builtins.ValueError: Provided number of samples is 320 (Supported values: 256 for 8000 sample rate, 512 for 16000)
EndPipeFrame
class does not exist, but it is referenced from src/pipecat/pipeline/merge_pipeline.py
and examples/foundational/04-utterance-and-speech.py
.
Trying to run the chatbot.py, and encountered the following errors. I think there are two errors:
I have populated the env variables. And here is the shape of my env:
OPENAI_API_KEY=sk-xxxx
ELEVENLABS_API_KEY=xxx
ELEVENLABS_VOICE_ID=CnV6BQOHeZCIv4McSXDH
DAILY_SAMPLE_ROOM_URL=https://xx.daily.co/xx-x
DAILY_API_KEY=xxxx
Detailed error
(smol) sasha@iSashair daily-ai-sdk % python src/examples/starter-apps/chatbot.py
Using cache found in /Users/sasha/.cache/torch/hub/snakers4_silero-vad_master
Eleven Labs API Key: xxx
Eleven Labs Voice ID: xxx
20 2024-03-16 14:30:07,548 🎬 Starting frame consumer thread
{"timestamp":"2024-03-16T21:30:07.577665Z","level":"ERROR","fields":{"message":"startTranscription (request 3) encountered an error: Transcription(Properties(UnknownCallClientError))"},"target":"daily_core::native::ffi::call_client"}
40 2024-03-16 14:30:07,578 on_error: Error handling startTranscription: Transcription(Properties(UnknownCallClientError))
!!! in here, pipeline.source is <Queue maxsize=0 _getters[1]>
10 2024-03-16 14:30:17,075 Generating chat via openai: [{"content": "You are Chatbot, a friendly, helpful robot. Your goal is to demonstrate your capabilities in a succinct way. Your output will be converted to audio. Respond to what the user said in a creative and helpful way, but keep your responses brief. Start by introducing yourself.", "role": "system", "name": "system"}]
[W NNPACK.cpp:64] Could not initialize NNPACK! Reason: Unsupported hardware.
20 2024-03-16 14:30:18,795 === OpenAI LLM TTFB: 1.7094330787658691
40 2024-03-16 14:30:19,559 audio fetch status code: 401, error: <bound method ClientResponse.text of <ClientResponse(https://api.elevenlabs.io/v1/text-to-speech/CnV6BQOHeZCIv4McSXDH/stream?output_format=pcm_16000&optimize_streaming_latency=2) [401 Unauthorized]>
<CIMultiDictProxy('Date': 'Sat, 16 Mar 2024 21:30:18 GMT', 'Server': 'uvicorn', 'Content-Length': '476', 'Content-Type': 'application/json', 'Access-Control-Allow-Origin': '*', 'Access-Control-Allow-Headers': '*', 'Access-Control-Allow-Methods': 'POST, OPTIONS, DELETE, GET, PUT', 'Access-Control-Max-Age': '600', 'strict-transport-security': 'max-age=31536000; includeSubDomains', 'Via': '1.1 google', 'Alt-Svc': 'h3=":443"; ma=2592000,h3-29=":443"; ma=2592000')>
>
I was trying to implement your interruptible langchain example by moulding it to my usecase.As we can call the tools in OpenAI,similarly I wanted to implement the tools with langchain.Is there a way to use the tools of Langchain in Pipecat?
Storytelling Chatbot client sometime reports the following error on opening http://0.0.0.0:7860:
Application error: a client-side exception has occurred (see the browser console for more information).
Here is the browser console log:
TypeError: Cannot read properties of undefined (reading 'ondevicechange')
at nx (d50e61c5-729bf43d5cce1859.js:19:31033)
at Z.startListeningForDeviceChanges (d50e61c5-729bf43d5cce1859.js:19:42326)
at new Z (d50e61c5-729bf43d5cce1859.js:19:49895)
at Function.value (d50e61c5-729bf43d5cce1859.js:19:118289)
at 647-07448eb149f02d56.js:1:32531
at aI (fd9d1056-15205bcf7b0e6812.js:1:72882)
at a3 (fd9d1056-15205bcf7b0e6812.js:1:84323)
at a5 (fd9d1056-15205bcf7b0e6812.js:1:84961)
at a8 (fd9d1056-15205bcf7b0e6812.js:1:84845)
at a5 (fd9d1056-15205bcf7b0e6812.js:1:84941)
In the cases I don't get the above error, I get the following message:
This demo is currently at capacity. Please try again later.
To summarize, this demo doesn't work for me, or I don't know how to run it.
We had it in a previous version, but we refactored some internals and it went away. We'll have it back soon.
I'm working on a Google Gemini LLM service for Pipecat and interested in any feedback people have about the LLMMessagesFrame class.
All the other LLMs with a chat (multi-turn) fine-tuning that I've worked with have adopted OpenAI's messages array format. Google's format is a bit different.
role
can only be user
or model
. Contrast with user
, assistant
, or system
for OpenAI.parts: [<string>, ...]
instead of just content: <string>
.https://ai.google.dev/api/python/google/ai/generativelanguage/Content
https://ai.google.dev/gemini-api/docs/get-started/tutorial?lang=python#encode_messages
We could do at least three different things.
I lean towards (1). I think it will be fairly straightforward and we can always do (2) later if we need to.
But I haven't yet gotten to the context management code here, and that may complicate things. Note: we can't use the Google library's convenience functions for multi-turn chat context management, because pipelines need to be interruptible. One important part of interruptibility is making sure that the LLM context includes only sentences that the LLM has "said" out loud to the user.
Any other thoughts here are welcome!
Also, Discord thread is here if people want to hash things out ephemerally before etching pixels into the stone tablet of an issue comment: https://discord.com/channels/1239284677165056021/1239284677823565826/1240682255584854027
Google text to speech seems to have good support in a lot of different languages. Is adding this to pipecat a good idea?
Description:
To enhance the capabilities and interactivity of our application, it would be highly beneficial to integrate function calling with the Google Gemini Chatbot. This feature would enable the chatbot to invoke predefined functions based on user inputs and context, allowing for more dynamic and functional conversations According to the Gorilla leaderboard, this would provide performance much better than GPT-3.5 and at a lower cost than GPT-4.
I attempted this integration myself but was unsuccessful:
#
# Copyright (c) 2024, Daily
#
# SPDX-License-Identifier: BSD 2-Clause License
#
import asyncio
import json
from typing import Callable, List
from loguru import logger
from pipecat.frames.frames import (
Frame,
LLMFullResponseEndFrame,
LLMFullResponseStartFrame,
LLMMessagesFrame,
LLMResponseEndFrame,
LLMResponseStartFrame,
TextFrame,
VisionImageRawFrame,
)
from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext, OpenAILLMContextFrame
from pipecat.processors.frame_processor import FrameDirection
from pipecat.services.ai_services import LLMService
try:
import google.ai.generativelanguage as glm
import google.generativeai as gai
except ModuleNotFoundError as e:
logger.error(f"Exception: {e}")
logger.error(
"In order to use Google AI, you need to `pip install pipecat-ai[google]`. Also, set `GOOGLE_API_KEY` environment variable."
)
raise Exception(f"Missing module: {e}")
class GoogleLLMService(LLMService):
"""This class implements inference with Google's AI models
This service translates internally from OpenAILLMContext to the messages format
expected by the Google AI model. We are using the OpenAILLMContext as a lingua
franca for all LLM services, so that it is easy to switch between different LLMs.
"""
def __init__(self, api_key: str, model: str = "gemini-1.5-flash-latest", tools: List[Callable] = None, **kwargs):
super().__init__(**kwargs)
gai.configure(api_key=api_key)
self._tools = tools or []
self._client = gai.GenerativeModel(model, tools=self._tools)
def can_generate_metrics(self) -> bool:
return True
def _get_messages_from_openai_context(self, context: OpenAILLMContext) -> List[glm.Content]:
openai_messages = context.get_messages()
google_messages = []
for message in openai_messages:
role = message["role"]
content = message["content"]
if role == "system":
role = "user"
elif role == "assistant":
role = "model"
parts = [glm.Part(text=content)]
if "mime_type" in message:
parts.append(
glm.Part(inline_data=glm.Blob(mime_type=message["mime_type"], data=message["data"].getvalue()))
)
google_messages.append({"role": role, "parts": parts})
return google_messages
async def _async_generator_wrapper(self, sync_generator):
for item in sync_generator:
yield item
await asyncio.sleep(0)
async def _process_context(self, context: OpenAILLMContext):
await self.push_frame(LLMFullResponseStartFrame())
try:
logger.debug(f"Generating chat: {context.get_messages_json()}")
messages = self._get_messages_from_openai_context(context)
await self.start_ttfb_metrics()
response = self._client.generate_content(
messages, generation_config=gai.GenerationConfig(temperature=0), tools=context.tools, stream=True
)
await self.stop_ttfb_metrics()
async for chunk in self._async_generator_wrapper(response):
try:
for candidate in chunk.candidates:
if candidate.content:
if candidate.content.parts:
for part in candidate.content.parts:
if part.text:
await self.push_frame(LLMResponseStartFrame())
await self.push_frame(TextFrame(part.text))
await self.push_frame(LLMResponseEndFrame())
if candidate.function_call:
for function_call in candidate.function_call:
function_name = function_call.name
arguments = function_call.args
result = await self.call_function(function_name, arguments)
# Add function call and result to context
context.add_message({"role": "assistant", "content": f"Function call: {function_name}"})
context.add_message(
{"role": "function", "name": function_name, "content": json.dumps(result)}
)
# Send function response back to the model
function_response = glm.Content(
parts=[
glm.FunctionResponse(
name=function_name, response={"content": json.dumps(result)}
)
]
)
# Re-process context with function result
context.add_message(function_response)
await self._process_context(context)
return
except Exception as e:
# Google LLMs seem to flag safety issues a lot!
if chunk.candidates[0].finish_reason == 3:
logger.debug(f"LLM refused to generate content for safety reasons - {messages}.")
else:
logger.error(f"{self} error: {e}")
except Exception as e:
logger.error(f"{self} exception: {e}")
finally:
await self.push_frame(LLMFullResponseEndFrame())
async def process_frame(self, frame: Frame, direction: FrameDirection):
await super().process_frame(frame, direction)
context = None
if isinstance(frame, OpenAILLMContextFrame):
context: OpenAILLMContext = frame.context
elif isinstance(frame, LLMMessagesFrame):
context = OpenAILLMContext.from_messages(frame.messages)
elif isinstance(frame, VisionImageRawFrame):
context = OpenAILLMContext.from_image_frame(frame)
else:
await self.push_frame(frame, direction)
if context:
await self._process_context(context)
when i run example simple-bot on cloud machine, i got these error
2024-07-11 10:54:32.151 | ERROR | pipecat.transports.services.daily:join:280 - Time out joining https://autoark.daily.co/1r4by424NvuhJS8hxxxx
{"timestamp":"2024-07-11T02:54:44.715005Z","level":"ERROR","fields":{"message":"Failed to fetch room information: GET failed: Transport(Transport { kind: ConnectionFailed, message: Some("Connect error"), url: Some(Url { scheme: "https", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("gs.daily.co")), port: None, path: "/rooms/check/autoark/1r4by424NvuhJS8hOpRF", query: None, fragment: None }), source: Some(Error { kind: TimedOut, message: "connection timed out" }) })"},"target":"daily_core::call_client"}
{"timestamp":"2024-07-11T02:54:44.715073Z","level":"ERROR","fields":{"message":"Failed to fetch room information: GET failed: Transport(Transport { kind: ConnectionFailed, message: Some("Connect error"), url: Some(Url { scheme: "https", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("gs.daily.co")), port: None, path: "/rooms/check/autoark/1r4by424NvuhJS8hOpRF", query: None, fragment: None }), source: Some(Error { kind: TimedOut, message: "connection timed out" }) })"},"target":"daily_core::event"}
{"timestamp":"2024-07-11T02:54:44.734809Z","level":"ERROR","fields":{"message":"join (request 2) encountered an error: Connection(Api(RoomLookup(RoomInfoFetchFailed(Get(Transport(Transport { kind: ConnectionFailed, message: Some("Connect error"), url: Some(Url { scheme: "https", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("gs.daily.co")), port: None, path: "/rooms/check/autoark/1r4by424NvuhJS8hxxxx", query: None, fragment: None }), source: Some(Error { kind: TimedOut, message: "connection timed out" }) }))))))"},"target":"daily_core::native::ffi::call_client"}
{"timestamp":"2024-07-11T02:54:44.734919Z","level":"ERROR","fields":{"message":"sendAppMessage (request 3) encountered an error: AppMessage(Signalling(ChannelNotOpen))"},"target":"daily_core::native::ffi::call_client"}
using the Simple-chatbot example and when I turn on speaker, the bot heard itself and interrupted. is there a way to avoid this?
2024-07-11 14:13:05.423 | ERROR | pipecat.pipeline.task:_process_up_queue:135 - Error running app: Error getting audio (status: 422, error: {"detail":[{"type":"string_type","loc":["body","stream_chunk_size"],"msg":"Input should be a valid string","input":20}]})
Any plans to add Deepgram Speech to Text, performs better than whisper in many cases.
Hi There, I came across an error after all deployments were in place and it seems that I need a full Twilio account instead of a trial one.
Is that correct to fully use this chatbot powered by LLMs?
Thanks.
As mentioned in Discord, Pipecat bots don't automatically maintain an LLM context object for you; instead, you have to use a bunch of aggregators and other tools to manage that yourself.
We originally built that because there were plenty of use cases where you didn't necessarily want to automatically store everything the user or bot said in the context, but we should consider adding an 'easy mode' that does this for you.
Here are the docs for the new Whisper models available in Groq Cloud. I'd like to replace my Deepgram STT provider with the Groq transcription service. It would be great if you could add this integration. 🙏
Websocket transport is not working right now. It was before 0.0.9. We should re-add it again.
We want to build out some code and examples to make it easier to deploy Pipecat bots. Right now, we're looking at the following services:
Right now, the basic structure looks like a Dockerfile running a FastAPI server that serves a static HTML frontend UI. That server also spins up bots on demand in threads. We're still figuring out how to handle Cloudflare Workers.
Where else do you want to be able to run Pipecat bots? Is there another approach that makes more sense to you?
I am trying to install pipecat with this in the requirements.txt pipecat-ai[deepgram,openai]
But it returns the error:
WARNING: pipecat-ai 0.0.29 does not provide the extra 'deepgram'
ctranslate2==4.3.0
https://pypi.org/project/ctranslate2/#history
I am assuming this is a typo?
First up, brilliant project! Still trying to wrap my head around the concept, but making some baby steps.
My question is: Why do we have to toggle recording when at the same time the "interruptible" feature is demoed in other examples?
if (e.data?.cue === "user_turn") {
// Delay enabling local mic input to avoid feedback from LLM
setTimeout(() => daily.setLocalAudio(true), 500);
setStoryState("user");
} else {
daily.setLocalAudio(false);
setStoryState("assistant");
}
Is the "always listening" feature a unique one of the Daily default video call screen which does not apply to custom apps? Would it record the audio of the TTS output (which is a typical problem of any speech-to-speech systems I have written myself)?
Thanks for clarification, looking forward to v0.0.24 :)
in the file services.openai.py, why is line 147 function_name += tool_call.function.name
and not function_name = tool_call.function.name
?
In particular, i am getting the following error:
Uncaught exception in LLMUserContextAggregator#0: The LLM tried to call a function named 'collect_namecollect_agecollect_payment_method', but there isn't a callback registered for that function.
it happens whenever i execute a function, for example, giving my full name, then the flow goes on to collecting my age, and i try to correct my full name saying "oh, i am sorry, my name is not {previous}. it is {new_name}". I have noticed that in the patient-intake example, every function call modifies the context and just allows one function tool. Can i keep a context with many function tools (so as to call them more than once during the flow of events).??
Re-enable function calling in GPTs, and add a foundational example showing how it works.
______________________________________ ERROR collecting src/pipecat/pipeline/merge_pipeline.py ______________________________________
ImportError while importing test module '/mnt/c/Users/DELL/Documents/pipchatbot/pipecat/src/pipecat/pipeline/merge_pipeline.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/home/loop/miniconda3/lib/python3.11/importlib/init.py:126: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
src/pipecat/pipeline/merge_pipeline.py:2: in
from pipecat.pipeline.frames import EndFrame, EndPipeFrame
E ModuleNotFoundError: No module named 'pipecat.pipeline.frames'
_____________________________ ERROR collecting src/pipecat/processors/aggregators/openai_llm_context.py _____________________________
ImportError while importing test module '/mnt/c/Users/DELL/Documents/pipchatbot/pipecat/src/pipecat/processors/aggregators/openai_llm_context.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/home/loop/miniconda3/lib/python3.11/importlib/init.py:126: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
src/pipecat/processors/aggregators/openai_llm_context.py:13: in
from openai._types import NOT_GIVEN, NotGiven
E ModuleNotFoundError: No module named 'openai'
_______________________________ ERROR collecting src/pipecat/serializers/abstract_frame_serializer.py _______________________________
ImportError while importing test module '/mnt/c/Users/DELL/Documents/pipchatbot/pipecat/src/pipecat/serializers/abstract_frame_serializer.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/home/loop/miniconda3/lib/python3.11/importlib/init.py:126: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
src/pipecat/serializers/abstract_frame_serializer.py:3: in
from pipecat.pipeline.frames import Frame
E ModuleNotFoundError: No module named 'pipecat.pipeline.frames'
__________________________________ ERROR collecting src/pipecat/serializers/protobuf_serializer.py __________________________________
ImportError while importing test module '/mnt/c/Users/DELL/Documents/pipchatbot/pipecat/src/pipecat/serializers/protobuf_serializer.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/home/loop/miniconda3/lib/python3.11/importlib/init.py:126: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
src/pipecat/serializers/protobuf_serializer.py:3: in
from pipecat.pipeline.frames import AudioFrame, Frame, TextFrame, TranscriptionFrame
E ModuleNotFoundError: No module named 'pipecat.pipeline.frames'
__________________________________________ ERROR collecting src/pipecat/services/azure.py ___________________________________________
ImportError while importing test module '/mnt/c/Users/DELL/Documents/pipchatbot/pipecat/src/pipecat/services/azure.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/home/loop/miniconda3/lib/python3.11/importlib/init.py:126: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
src/pipecat/services/azure.py:14: in
from openai import AsyncAzureOpenAI
E ModuleNotFoundError: No module named 'openai'
________________________________________ ERROR collecting src/pipecat/services/fireworks.py _________________________________________
ImportError while importing test module '/mnt/c/Users/DELL/Documents/pipchatbot/pipecat/src/pipecat/services/fireworks.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/home/loop/miniconda3/lib/python3.11/importlib/init.py:126: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
src/pipecat/services/fireworks.py:7: in
from pipecat.services.openai import BaseOpenAILLMService
src/pipecat/services/openai.py:25: in
from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext, OpenAILLMContextFrame
src/pipecat/processors/aggregators/openai_llm_context.py:13: in
from openai._types import NOT_GIVEN, NotGiven
E ModuleNotFoundError: No module named 'openai'
__________________________________________ ERROR collecting src/pipecat/services/ollama.py __________________________________________
ImportError while importing test module '/mnt/c/Users/DELL/Documents/pipchatbot/pipecat/src/pipecat/services/ollama.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/home/loop/miniconda3/lib/python3.11/importlib/init.py:126: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
src/pipecat/services/ollama.py:7: in
from pipecat.services.openai import BaseOpenAILLMService
src/pipecat/services/openai.py:25: in
from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext, OpenAILLMContextFrame
src/pipecat/processors/aggregators/openai_llm_context.py:13: in
from openai._types import NOT_GIVEN, NotGiven
E ModuleNotFoundError: No module named 'openai'
__________________________________________ ERROR collecting src/pipecat/services/openai.py __________________________________________
ImportError while importing test module '/mnt/c/Users/DELL/Documents/pipchatbot/pipecat/src/pipecat/services/openai.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/home/loop/miniconda3/lib/python3.11/importlib/init.py:126: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
src/pipecat/services/openai.py:25: in
from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext, OpenAILLMContextFrame
src/pipecat/processors/aggregators/openai_llm_context.py:13: in
from openai._types import NOT_GIVEN, NotGiven
E ModuleNotFoundError: No module named 'openai'
_____________________________________ ERROR collecting src/pipecat/transports/services/daily.py _____________________________________
ImportError while importing test module '/mnt/c/Users/DELL/Documents/pipchatbot/pipecat/src/pipecat/transports/services/daily.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/home/loop/miniconda3/lib/python3.11/importlib/init.py:126: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
src/pipecat/transports/services/daily.py:18: in
from daily import (
E ImportError: cannot import name 'CallClient' from partially initialized module 'daily' (most likely due to a circular import) (/mnt/c/Users/DELL/Documents/pipchatbot/pipecat/src/pipecat/transports/services/daily.py)
____________________________________ ERROR collecting tests/integration/integration_azure_llm.py ____________________________________
ImportError while importing test module '/mnt/c/Users/DELL/Documents/pipchatbot/pipecat/tests/integration/integration_azure_llm.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/home/loop/miniconda3/lib/python3.11/importlib/init.py:126: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
tests/integration/integration_azure_llm.py:3: in
from pipecat.pipeline.openai_frames import OpenAILLMContextFrame
E ModuleNotFoundError: No module named 'pipecat.pipeline.openai_frames'
___________________________________ ERROR collecting tests/integration/integration_ollama_llm.py ____________________________________
ImportError while importing test module '/mnt/c/Users/DELL/Documents/pipchatbot/pipecat/tests/integration/integration_ollama_llm.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/home/loop/miniconda3/lib/python3.11/importlib/init.py:126: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
tests/integration/integration_ollama_llm.py:2: in
from pipecat.pipeline.openai_frames import OpenAILLMContextFrame
E ModuleNotFoundError: No module named 'pipecat.pipeline.openai_frames'
___________________________________ ERROR collecting tests/integration/integration_openai_llm.py ____________________________________
ImportError while importing test module '/mnt/c/Users/DELL/Documents/pipchatbot/pipecat/tests/integration/integration_openai_llm.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/home/loop/miniconda3/lib/python3.11/importlib/init.py:126: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
tests/integration/integration_openai_llm.py:3: in
from pipecat.pipeline.openai_frames import OpenAILLMContextFrame
E ModuleNotFoundError: No module named 'pipecat.pipeline.openai_frames'
____________________________________________ ERROR collecting tests/test_aggregators.py _____________________________________________
ImportError while importing test module '/mnt/c/Users/DELL/Documents/pipchatbot/pipecat/tests/test_aggregators.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/home/loop/miniconda3/lib/python3.11/importlib/init.py:126: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
tests/test_aggregators.py:6: in
from pipecat.pipeline.aggregators import (
E ModuleNotFoundError: No module named 'pipecat.pipeline.aggregators'
____________________________________________ ERROR collecting tests/test_aggregators.py _____________________________________________
ImportError while importing test module '/mnt/c/Users/DELL/Documents/pipchatbot/pipecat/tests/test_aggregators.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/home/loop/miniconda3/lib/python3.11/importlib/init.py:126: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
tests/test_aggregators.py:6: in
from pipecat.pipeline.aggregators import (
E ModuleNotFoundError: No module named 'pipecat.pipeline.aggregators'
____________________________________________ ERROR collecting tests/test_ai_services.py _____________________________________________
ImportError while importing test module '/mnt/c/Users/DELL/Documents/pipchatbot/pipecat/tests/test_ai_services.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/home/loop/miniconda3/lib/python3.11/importlib/init.py:126: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
tests/test_ai_services.py:6: in
from pipecat.pipeline.frames import EndFrame, Frame, TextFrame
E ModuleNotFoundError: No module named 'pipecat.pipeline.frames'
____________________________________________ ERROR collecting tests/test_ai_services.py _____________________________________________
ImportError while importing test module '/mnt/c/Users/DELL/Documents/pipchatbot/pipecat/tests/test_ai_services.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/home/loop/miniconda3/lib/python3.11/importlib/init.py:126: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
tests/test_ai_services.py:6: in
from pipecat.pipeline.frames import EndFrame, Frame, TextFrame
E ModuleNotFoundError: No module named 'pipecat.pipeline.frames'
______________________________________________ ERROR collecting tests/test_pipeline.py ______________________________________________
ImportError while importing test module '/mnt/c/Users/DELL/Documents/pipchatbot/pipecat/tests/test_pipeline.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/home/loop/miniconda3/lib/python3.11/importlib/init.py:126: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
tests/test_pipeline.py:5: in
from pipecat.pipeline.aggregators import SentenceAggregator, StatelessTextTransformer
E ModuleNotFoundError: No module named 'pipecat.pipeline.aggregators'
______________________________________________ ERROR collecting tests/test_pipeline.py ______________________________________________
ImportError while importing test module '/mnt/c/Users/DELL/Documents/pipchatbot/pipecat/tests/test_pipeline.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/home/loop/miniconda3/lib/python3.11/importlib/init.py:126: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
tests/test_pipeline.py:5: in
from pipecat.pipeline.aggregators import SentenceAggregator, StatelessTextTransformer
E ModuleNotFoundError: No module named 'pipecat.pipeline.aggregators'
________________________________________ ERROR collecting tests/test_protobuf_serializer.py _________________________________________
ImportError while importing test module '/mnt/c/Users/DELL/Documents/pipchatbot/pipecat/tests/test_protobuf_serializer.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/home/loop/miniconda3/lib/python3.11/importlib/init.py:126: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
tests/test_protobuf_serializer.py:3: in
from pipecat.pipeline.frames import AudioFrame, TextFrame, TranscriptionFrame
E ModuleNotFoundError: No module named 'pipecat.pipeline.frames'
________________________________________ ERROR collecting tests/test_protobuf_serializer.py _________________________________________
ImportError while importing test module '/mnt/c/Users/DELL/Documents/pipchatbot/pipecat/tests/test_protobuf_serializer.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/home/loop/miniconda3/lib/python3.11/importlib/init.py:126: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
tests/test_protobuf_serializer.py:3: in
from pipecat.pipeline.frames import AudioFrame, TextFrame, TranscriptionFrame
E ModuleNotFoundError: No module named 'pipecat.pipeline.frames'
________________________________________ ERROR collecting tests/test_websocket_transport.py _________________________________________
ImportError while importing test module '/mnt/c/Users/DELL/Documents/pipchatbot/pipecat/tests/test_websocket_transport.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/home/loop/miniconda3/lib/python3.11/importlib/init.py:126: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
tests/test_websocket_transport.py:7: in
from pipecat.transports.websocket_transport import WebSocketFrameProcessor, WebsocketTransport
E ModuleNotFoundError: No module named 'pipecat.transports.websocket_transport'
________________________________________ ERROR collecting tests/test_websocket_transport.py _________________________________________
ImportError while importing test module '/mnt/c/Users/DELL/Documents/pipchatbot/pipecat/tests/test_websocket_transport.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/home/loop/miniconda3/lib/python3.11/importlib/init.py:126: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
tests/test_websocket_transport.py:7: in
from pipecat.transports.websocket_transport import WebSocketFrameProcessor, WebsocketTransport
E ModuleNotFoundError: No module named 'pipecat.transports.websocket_transport'
====================================================== short test summary info ======================================================
ERROR src/pipecat/pipeline/merge_pipeline.py
ERROR src/pipecat/processors/aggregators/openai_llm_context.py
ERROR src/pipecat/serializers/abstract_frame_serializer.py
ERROR src/pipecat/serializers/protobuf_serializer.py
ERROR src/pipecat/services/azure.py
ERROR src/pipecat/services/fireworks.py
ERROR src/pipecat/services/ollama.py
ERROR src/pipecat/services/openai.py
ERROR src/pipecat/transports/services/daily.py
ERROR tests/integration/integration_azure_llm.py
ERROR tests/integration/integration_ollama_llm.py
ERROR tests/integration/integration_openai_llm.py
ERROR tests/test_aggregators.py
ERROR tests/test_aggregators.py
ERROR tests/test_ai_services.py
ERROR tests/test_ai_services.py
ERROR tests/test_pipeline.py
ERROR tests/test_pipeline.py
ERROR tests/test_protobuf_serializer.py
ERROR tests/test_protobuf_serializer.py
ERROR tests/test_websocket_transport.py
ERROR tests/test_websocket_transport.py
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 22 errors during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Most unittests are broken due to bad imports from pipecat.pipeline
.
So if they are still considered useful, I can assist there.
Hello.
A question has been revolving in my mind.I wanted to display the contents that are getting collected by the bot(The information it is gathering while talking to the user).Is there any way to show the file through screen sharing or chat messages?
Screen share is preferable as we want to display images or pdf.But we can also adjust to the chat part where we can share a simple json.
I'm running 13-whisper-transcription.py.
I made one modification to it, adding:
import os
os.environ['KMP_DUPLICATE_LIB_OK']='True'
To overcome this error:
OMP: Error #15: Initializing libiomp5.dylib, but found libiomp5.dylib already initialized.
OMP: Hint This means that multiple copies of the OpenMP runtime have been linked into the program. That is dangerous, since it can degrade performance or cause incorrect results. The best thing to do is to ensure that only a single OpenMP runtime is linked into the process, e.g. by avoiding static linking of the OpenMP runtime in any library. As an unsafe, unsupported, undocumented workaround you can set the environment variable KMP_DUPLICATE_LIB_OK=TRUE to allow the program to continue to execute, but that may cause crashes or silently produce incorrect results. For more information, please see http://www.intel.com/software/products/support/.
Now, I see this cryptic error:
20 2024-03-19 00:23:29,580 🎬 Starting frame consumer thread
TRANSCRIPTION
20 2024-03-19 00:23:29,581 Call_joined: {'participants': ..... [DETAILS OF THE CALL]
zsh: abort python 13-whisper-transcription.py -u -k
/Users/ddd/opt/anaconda3/envs/py311/lib/python3.11/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '
when trying v0.0.18, this error is given on loading the story page
Traceback (most recent call last):
File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/home/jwee/projects/pipecat/examples/storytelling-chatbot/src/bot.py", line 18, in <module>
from processors import StoryProcessor, StoryImageProcessor
File "/home/jwee/projects/pipecat/examples/storytelling-chatbot/src/processors.py", line 5, in <module>
from pipecat.frames.frames import (
ImportError: cannot import name 'LLMFullResponseEndFrame' from 'pipecat.frames.frames' (/home/jwee/envs/alfabert/lib/python3.10/site-packages/pipecat/frames/frames.py)
After following the readme ode the storytelling chatbot example, I see the narrator is starting the story. But the fragment if the user input is not started. The microphone does not become active.
I see in the readme, Deepgram mentioned. But I can not find it being used in the code. I want to help improving the example, but I am stuck. Is daily.co supposed to stream/transcribe user input?
Someone knows how to take next step?
I'm running 01-say-one-thing.py - having made no modifications to it, just passing it a URL and token. I see this cryptic error:
File "/Users/ddd/daily-ai-sdk/src/examples/foundational/01-say-one-thing.py", line 51, in
asyncio.run(main(url))
File "/Users/ddd/opt/anaconda3/envs/py311/lib/python3.11/asyncio/runners.py", line 190, in run
return runner.run(main)
^^^^^^^^^^^^^^^^
File "/Users/ddd/opt/anaconda3/envs/py311/lib/python3.11/asyncio/runners.py", line 118, in run
return self._loop.run_until_complete(task)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/ddd/opt/anaconda3/envs/py311/lib/python3.11/asyncio/base_events.py", line 653, in run_until_complete
return future.result()
^^^^^^^^^^^^^^^
File "/Users/ddd/daily-ai-sdk/src/examples/foundational/01-say-one-thing.py", line 45, in main
await transport.run(pipeline)
^^^^^^^^^^^^^^^^^^^^^^^
TypeError: BaseTransportService.run() takes 1 positional argument but 2 were given
Most of the examples, for instance 06a-image-sync.py, need dotenv imported to grab the variables from the .env file
from dotenv import load_dotenv
load_dotenv()
without it, you get the following non-str key serialization error:
foundational$ python 06a-image-sync.py -u "[MY ROOM URL]" -k "[MY KEY]"
Using cache found in /Users/ddd/.cache/torch/hub/snakers4_silero-vad_master
20 2024-03-19 00:37:03,467 🎬 Starting frame consumer thread
--- Logging error ---
Traceback (most recent call last):
File "/Users/ddd/opt/anaconda3/envs/py311/lib/python3.11/site-packages/dailyai/services/ai_services.py", line 54, in run
async for output_frame in self.process_frame(frame):
File "/Users/ddd/opt/anaconda3/envs/py311/lib/python3.11/site-packages/dailyai/services/ai_services.py", line 115, in process_frame
async for audio_chunk in self.run_tts(text):
File "/Users/ddd/opt/anaconda3/envs/py311/lib/python3.11/site-packages/dailyai/services/elevenlabs_ai_service.py", line 34, in run_tts
async with self._aiohttp_session.post(
File "/Users/ddd/opt/anaconda3/envs/py311/lib/python3.11/site-packages/aiohttp/client.py", line 1187, in aenter
self._resp = await self._coro
^^^^^^^^^^^^^^^^
File "/Users/ddd/opt/anaconda3/envs/py311/lib/python3.11/site-packages/aiohttp/client.py", line 599, in _request
resp = await req.send(conn)
^^^^^^^^^^^^^^^^^^^^
File "/Users/ddd/opt/anaconda3/envs/py311/lib/python3.11/site-packages/aiohttp/client_reqrep.py", line 712, in send
await writer.write_headers(status_line, self.headers)
File "/Users/ddd/opt/anaconda3/envs/py311/lib/python3.11/site-packages/aiohttp/http_writer.py", line 129, in write_headers
buf = _serialize_headers(status_line, headers)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "aiohttp/_http_writer.pyx", line 132, in aiohttp._http_writer._serialize_headers
File "aiohttp/_http_writer.pyx", line 109, in aiohttp._http_writer.to_str
TypeError: Cannot serialize non-str key None
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/ddd/opt/anaconda3/envs/py311/lib/python3.11/logging/init.py", line 1110, in emit
msg = self.format(record)
^^^^^^^^^^^^^^^^^^^
File "/Users/ddd/opt/anaconda3/envs/py311/lib/python3.11/logging/init.py", line 953, in format
return fmt.format(record)
^^^^^^^^^^^^^^^^^^
File "/Users/ddd/opt/anaconda3/envs/py311/lib/python3.11/logging/init.py", line 687, in format
record.message = record.getMessage()
^^^^^^^^^^^^^^^^^^^
File "/Users/ddd/opt/anaconda3/envs/py311/lib/python3.11/logging/init.py", line 377, in getMessage
msg = msg % self.args
~~~~^~~~~~~~~~~
TypeError: not all arguments converted during string formatting
I run the storybot example ,but it does not work.So I try on the line at https://storytelling-chatbot.fly.dev/ ,aslo.
no more info from log :
2024-07-03 11:11:32.787 | INFO | pipecat.transports.services.daily:join:268 - Enabling transcription with settings language='en' tier='nova' model='2-conversationalai' profanity_filter=True redact=False endpointing=True punctuate=True includeRawResponse=True extra={'interim_results': True}
2024-07-03 11:11:33.110 | DEBUG | pipecat.transports.services.daily:on_transcription_started:482 - Transcription started: {'model': '2-conversationalai', 'transcriptId': 'aae9d1ec-c552-4d2c-a542-056674890de4', 'startedBy': '9f6a5f95-d0d4-4c96-b3c7-9f1006fe0d52', 'tier': 'nova', 'language': 'en'}
2024-07-03 11:11:33.243 | INFO | pipecat.transports.services.daily:on_participant_joined:455 - Participant joined 6edd95bd-d4f4-4272-ab5c-f482247d2337
2024-07-03 11:11:33.244 | DEBUG | __main__:on_first_participant_joined:111 - Participant joined, storytime commence!
2024-07-03 11:11:33.244 | DEBUG | pipecat.services.openai:_stream_chat_completions:96 - Generating chat: [{"content": "You are a creative storyteller who loves to tell whimsical, fantastical stories. Your goal is to craft an engaging and fun story. Start by asking the user what kind of story they'd like to hear. Don't provide any examples. Keep your response to only a few sentences.", "role": "system", "name": "system"}]
/{path_name:path}
INFO: 10.16.39.2:42872 - "GET /alpha-mask.gif HTTP/1.1" 200 OK
2024-07-03 11:11:34.361 | DEBUG | pipecat.services.openai:run_tts:313 - Generating TTS: [, I am ready to begin. What type of story would you like to hear?]
2024-07-03 11:11:36.942 | DEBUG | pipecat.pipeline.runner:run:32 - Runner PipelineRunner#0 finished running PipelineTask#0
2024-07-03 11:11:36.942 | DEBUG | pipecat.transports.services.daily:__init__:70 - Loaded native WebRTC VAD
2024-07-03 11:11:36.942 | DEBUG | pipecat.processors.frame_processor:link:132 - Linking PipelineSource#1 -> DailyInputTransport#0
2024-07-03 11:11:36.942 | DEBUG | pipecat.processors.frame_processor:link:132 - Linking DailyInputTransport#0 -> LLMUserResponseAggregator#0
2024-07-03 11:11:36.942 | DEBUG | pipecat.processors.frame_processor:link:132 - Linking LLMUserResponseAggregator#0 -> OpenAILLMService#0
2024-07-03 11:11:36.942 | DEBUG | pipecat.processors.frame_processor:link:132 - Linking OpenAILLMService#0 -> StoryProcessor#0
2024-07-03 11:11:36.942 | DEBUG | pipecat.processors.frame_processor:link:132 - Linking StoryProcessor#0 -> StoryImageProcessor#0
2024-07-03 11:11:36.942 | DEBUG | pipecat.processors.frame_processor:link:132 - Linking StoryImageProcessor#0 -> OpenAITTSService#0
2024-07-03 11:11:36.942 | DEBUG | pipecat.processors.frame_processor:link:132 - Linking OpenAITTSService#0 -> DailyOutputTransport#0
2024-07-03 11:11:36.943 | DEBUG | pipecat.processors.frame_processor:link:132 - Linking DailyOutputTransport#0 -> LLMAssistantResponseAggregator#0
2024-07-03 11:11:36.943 | DEBUG | pipecat.processors.frame_processor:link:132 - Linking LLMAssistantResponseAggregator#0 -> PipelineSink#1
2024-07-03 11:11:36.943 | DEBUG | pipecat.processors.frame_processor:link:132 - Linking Source#1 -> Pipeline#1
2024-07-03 11:11:36.943 | DEBUG | pipecat.pipeline.runner:run:28 - Runner PipelineRunner#0 started running PipelineTask#1
2024-07-03 11:11:45.274 | DEBUG | pipecat.transports.services.daily:_on_transcription_message:879 - Transcription (from: 6edd95bd-d4f4-4272-ab5c-f482247d2337): [Fanny
It would be great if you could provide examples using this with local LLM tooling (e.g. Ollama / llama.cpp, whisper/piper etc...)
Neat project!
hi guys, is there a tool to create the images for the chatbot base on one image? could you help recommend, thanks..
Hi 👋
I am having trouble running a local example that integrates LocalAudioTransport, WhisperSTTService, ElevenLabsTTSService, and OpenAILLMService.
I have successfully managed to run Whisper locally for transcription and another script that uses Eleven Labs and OpenAI for TTS and LLM services, respectively. However, I am struggling to combine these components to create a fully functional local conversation system.
To illustrate, here are the two examples I have working independently:
Example 1: Passing an LLM message to the TTS provider:
import asyncio
import os
import sys
import aiohttp
from loguru import logger
from pipecat.frames.frames import EndFrame, LLMMessagesFrame
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineTask
from pipecat.services.elevenlabs import ElevenLabsTTSService
from pipecat.services.openai import OpenAILLMService
from pipecat.transports.base_transport import TransportParams
from pipecat.transports.local.audio import LocalAudioTransport
logger.remove(0)
logger.add(sys.stderr, level="DEBUG")
async def main():
async with aiohttp.ClientSession() as session:
transport = LocalAudioTransport(TransportParams(audio_out_enabled=True))
tts = ElevenLabsTTSService(
aiohttp_session=session,
api_key=os.getenv("ELEVENLABS_API_KEY"),
voice_id=os.getenv("ELEVENLABS_VOICE_ID"),
)
llm = OpenAILLMService(
api_key=os.getenv("OPENAI_API_KEY"),
model="gpt-3.5-turbo-0125",
)
messages = [
{
"role": "system",
"content": "You are a helpful LLM in a WebRTC call. Your goal is to demonstrate your capabilities in a succinct way. Your output will be converted to audio so don't include special characters in your answers. Respond to what the user said in a creative and helpful way.",
},
]
pipeline = Pipeline([llm, tts, transport.output()])
task = PipelineTask(pipeline)
async def say_something():
await asyncio.sleep(1)
await task.queue_frames([LLMMessagesFrame(messages), EndFrame()])
runner = PipelineRunner()
await asyncio.gather(runner.run(task), say_something())
if __name__ == "__main__":
asyncio.run(main())
Example 2: Using Whisper locally:
import asyncio
import sys
from loguru import logger
from pipecat.frames.frames import Frame, TranscriptionFrame
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineTask
from pipecat.processors.frame_processor import FrameDirection, FrameProcessor
from pipecat.services.whisper import Model, WhisperSTTService
from pipecat.transports.base_transport import TransportParams
from pipecat.transports.local.audio import LocalAudioTransport
logger.remove(0)
logger.add(sys.stderr, level="DEBUG")
class TranscriptionLogger(FrameProcessor):
async def process_frame(self, frame: Frame, direction: FrameDirection):
if isinstance(frame, TranscriptionFrame):
print(f"Transcription: {frame.text}")
async def main():
transport = LocalAudioTransport(TransportParams(audio_in_enabled=True))
stt = WhisperSTTService()
tl = TranscriptionLogger()
pipeline = Pipeline([transport.input(), stt, tl])
task = PipelineTask(pipeline)
runner = PipelineRunner()
await runner.run(task)
if __name__ == "__main__":
asyncio.run(main())
Despite these individual successes, I'm unable to connect the transcriptions with the LLM and have a continuous conversation. Could you provide or add an example of a fully working local setup that demonstrates how to achieve this?
Thank you!
dot-env.template lists:
OPENAI_API_KEY
However, right now, many of the examples look for:
OPENAI_CHATGPT_API_KEY
These should be consistent with each other
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.