yacinemtb / talk Goto Github PK
View Code? Open in Web Editor NEWLet's make sand talk
Let's make sand talk
Would be easier to help if there are more issues, otherwise the best I can really do is nits or orthogonal feature prs as is OSS tradition :)
Maybe track the fixMe's somewhere, would make a PR but don't want to repeat work if you are already on it
Basically to allow more varity in the response and also change the prompts based on the model's supported format e.g. alpaca vs vicuna
Since the microphone is always listening, the AI will create a feedback loop with its own outputs if headphones aren't used. We could potentially differentiate between voices.
Current CUBLAS is asked via prompt in the build.sh script, but the command should instead grep lspci
output, or check if its a aarch64 vs x86 and automatically enable the flag only if a CUDA capable device is detected
I've got (maybe too many) 3070s plugged in, but none of them are being utilized. Is there anywhere cuda needs to be explicitly specified as the backend?
Right now, I'm planning to initiate the response with a "vim pedal", aka a hotkey, because knowing when to respond is difficult. https://github.com/yacineMTB/talk/blob/master/index.ts#L108-L135
When humans speak to each other, we use intonation and other signals to let the other human know when the floor is open, and we also use it to let the other human know that we want the floor.
Right now, we just need some naive event firing when the speaker stops speaking.
Is this something that we can get out of whisper.cpp's embeddings? Possibly a classifier trained on top of the embeddings?
Also I wouldn't shy away from running a python sidecar that takes requests from the main node proc.
Figuring out how to either get whisper.cpp, or some sidecar, that takes a byte stream and outputs a continous "activation function" based on likelihood to respond
Okay so im just having a look over the github, I know nothing else about this project.
So, it takes live microphone input, converts it to text, and even seems smart enough to wait for a lul in the speech to actually finalize the current input. Right, thats pretty flash, I like that a lot.
Now that I have the speech as text, what exactly do I do with it? How do I make use of this? I really want other apps to use this data. Does this have any kind of API system or something? Basically, how do I actually get the data out of this bad boy?
Much appreciated!
For those, such as myself (MacBook Pro, Intel + AMD GPU), without powerful GPUs, it would be great to self host on a serverless GPU service (e.g. Runpod).
Also having Docker build would make setup much easier. We could distribute pre-build Docker images for quick installation.
Here are some related examples I found:
Sometimes a whisper timeout will cause the application to crash. Happens intermittently with both short and long conversations.
Need help with steps to reproduce
After ~4 minutes, the audio process (record_audio.sh) stops sending chunks.
For some reason, adding a listener on stderr fixes the issue:
audioProcess.stderr.on('data', (data) => {
console.log(data);
});
Not sure why this works, but the application works fine for 45+ minutes with this snippet.
This thing needs to respond back to us on some event.
Right now, the strategy to reduce latency is to generate precanned responses constantly. Maybe we can also follow the same strategy with some TTS system?
Ideally this would
For now we can just save it as a wav file. The scope of this task is figuring out what reasonable candidates we have for TTS, with one of the goals being low latency.
Non-zero status code returned while running Gather node. Name:'/enc_p/emb/Gather' Status Message: indices element out of data bounds, idx=144 must be within the inclusive range [-130,129]
Aborted (core dumped)
The provided error indicates that there's an issue with the Piper ONNX model, which is used for generating synthetic speech. The model appears to have trouble with the 'Gather' operation—an operation that pulls out specific indices from a tensor—at a certain index in the input data.
Specifically, the error message 'indices element out of data bounds, idx=144 must be within the inclusive range [-130,129]'
suggests the index at '144' is not within the limit set to run the Gather operation.
Without knowing the specifics of this STT model, it's hard to give a specific solution. However, you can try the following:
Thanks in advance
Now, our model can speak!
Here's what the time looks like, as observed by the cpp
[kache@whitebox ~]$ echo "I need to renable the response reflex. Though, I think to start with, we can just use a simple keypress." | piper --model ~/models/piper/en-gb-southern_english_female-low.onnx
Load time: 0.157878 sec
WARNING: Piper was not compiled with pcaudiolib. Output audio will be written to the current directory.
Output directory: "/home/kache/."
/home/kache/./1686592663635066288.wav
Real-time factor: 0.0332105 (infer=0.172695 sec, audio=5.2 sec)
This gives us a really good lowerbound on time
However, most of the time being observed from the node proc (refer to video) is from resource acquisition.
We should fix this!
This issue tracks fixing this by building a custom c binding around piper, or coming up with a side car server/mechanism
/c/Program Files/nodejs/npm: line 37: 1993 Segmentation fault "$NODE_EXE" "$NPM_CLI_JS" "$@"
The user may want to interrupt the LLM while it's speaking, so that it doesn't ramble.
I think that a simple "detect user talking" solution might be good enough, since in normal human conversation, a person could interrupt with any sentence. If the user speaks above a certain word threshold while the LLM is speaking, the LLM should stop and yield.
I am working on this right now, leaving this issue as a placeholder.
I am trying to install and when doing
npx cmake-js compile --CDWHISPER_CUBLAS="ON" -T whisper-addon -B Release
it gives
npx: installed 70 in 2.291s
Unexpected token .
Would love to have this be able to talk to me about a research paper I load into a vector database or similar functionality. I have some custom made txt based research assistant but a voice one would be extremely useful
Sometimes you will want to whisper 'stop'
Sometimes you may want to say 'wait wait wait, this is not what I meant'
Sometimes you will scream 'Damn it, forget what I said, let's start from the beginning'
This is not possible currently since we pass history has well in the conversation.
Some ways to achieve this -
So I have gotten it to run the script succesfully
strau@James MINGW64 ~/Downloads/talk-master/talk-master (master)
$ ./build.sh
Starting Script...
Installing npm dependencies in the current directory...
npm WARN [email protected] No repository field.
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: [email protected] (node_modules\fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for [email protected]: wanted {"os":"darwin","arch":"any"} (current: {"os":"win32","arch":"x64"})
added 19 packages from 17 contributors, updated 1 package and audited 349 packages in 7.688s
34 packages are looking for funding
run npm fund
for details
found 0 vulnerabilities
Do you want to turn CUBLAS ON? [y/n] n
Installing npm dependencies for whisper.cpp examples...
npm WARN [email protected] No repository field.
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: @esbuild/[email protected] (node_modules@esbuild\freebsd-arm64):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for @esbuild/[email protected]: wanted {"os":"freebsd","arch":"arm64"} (current: {"os":"win32","arch":"x64"})
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: @esbuild/[email protected] (node_modules@esbuild\darwin-x64):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for @esbuild/[email protected]: wanted {"os":"darwin","arch":"x64"} (current: {"os":"win32","arch":"x64"})
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: @esbuild/[email protected] (node_modules@esbuild\android-arm):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for @esbuild/[email protected]: wanted {"os":"android","arch":"arm"} (current: {"os":"win32","arch":"x64"})
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: @esbuild/[email protected] (node_modules@esbuild\android-x64):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for @esbuild/[email protected]: wanted {"os":"android","arch":"x64"} (current: {"os":"win32","arch":"x64"})
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: @esbuild/[email protected] (node_modules@esbuild\freebsd-x64):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for @esbuild/[email protected]: wanted {"os":"freebsd","arch":"x64"} (current: {"os":"win32","arch":"x64"})
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: @esbuild/[email protected] (node_modules@esbuild\android-arm64):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for @esbuild/[email protected]: wanted {"os":"android","arch":"arm64"} (current: {"os":"win32","arch":"x64"})
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: @esbuild/[email protected] (node_modules@esbuild\linux-arm):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for @esbuild/[email protected]: wanted {"os":"linux","arch":"arm"} (current: {"os":"win32","arch":"x64"})
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: @esbuild/[email protected] (node_modules@esbuild\linux-arm64):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for @esbuild/[email protected]: wanted {"os":"linux","arch":"arm64"} (current: {"os":"win32","arch":"x64"})
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: @esbuild/[email protected] (node_modules@esbuild\linux-ia32):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for @esbuild/[email protected]: wanted {"os":"linux","arch":"ia32"} (current: {"os":"win32","arch":"x64"})
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: @esbuild/[email protected] (node_modules@esbuild\linux-loong64):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for @esbuild/[email protected]: wanted {"os":"linux","arch":"loong64"} (current: {"os":"win32","arch":"x64"})
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: @esbuild/[email protected] (node_modules@esbuild\darwin-arm64):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for @esbuild/[email protected]: wanted {"os":"darwin","arch":"arm64"} (current: {"os":"win32","arch":"x64"})
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: @esbuild/[email protected] (node_modules@esbuild\netbsd-x64):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for @esbuild/[email protected]: wanted {"os":"netbsd","arch":"x64"} (current: {"os":"win32","arch":"x64"})
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: @esbuild/[email protected] (node_modules@esbuild\linux-s390x):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for @esbuild/[email protected]: wanted {"os":"linux","arch":"s390x"} (current: {"os":"win32","arch":"x64"})
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: @esbuild/[email protected] (node_modules@esbuild\openbsd-x64):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for @esbuild/[email protected]: wanted {"os":"openbsd","arch":"x64"} (current: {"os":"win32","arch":"x64"})
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: @esbuild/[email protected] (node_modules@esbuild\linux-x64):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for @esbuild/[email protected]: wanted {"os":"linux","arch":"x64"} (current: {"os":"win32","arch":"x64"})
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: @esbuild/[email protected] (node_modules@esbuild\linux-ppc64):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for @esbuild/[email protected]: wanted {"os":"linux","arch":"ppc64"} (current: {"os":"win32","arch":"x64"})
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: @esbuild/[email protected] (node_modules@esbuild\sunos-x64):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for @esbuild/[email protected]: wanted {"os":"sunos","arch":"x64"} (current: {"os":"win32","arch":"x64"})
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: @esbuild/[email protected] (node_modules@esbuild\win32-arm64):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for @esbuild/[email protected]: wanted {"os":"win32","arch":"arm64"} (current: {"os":"win32","arch":"x64"})
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: @esbuild/[email protected] (node_modules@esbuild\linux-riscv64):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for @esbuild/[email protected]: wanted {"os":"linux","arch":"riscv64"} (current: {"os":"win32","arch":"x64"})
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: @esbuild/[email protected] (node_modules@esbuild\linux-mips64el):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for @esbuild/[email protected]: wanted {"os":"linux","arch":"mips64el"} (current: {"os":"win32","arch":"x64"})
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: [email protected] (node_modules\fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for [email protected]: wanted {"os":"darwin","arch":"any"} (current: {"os":"win32","arch":"x64"})
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: @esbuild/[email protected] (node_modules@esbuild\win32-ia32):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for @esbuild/[email protected]: wanted {"os":"win32","arch":"ia32"} (current: {"os":"win32","arch":"x64"})
audited 402 packages in 4.311s
41 packages are looking for funding
run npm fund
for details
found 0 vulnerabilities
Compiling whisper.cpp examples...
info find VS using VS2019 (16.11.33214.272) found at:
info find VS "C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools"
info find VS run with --verbose for detailed information
info TOOL Using Visual Studio 16 2019 generator.
info CMD BUILD
info RUN [
info RUN 'cmake',
info RUN '--build',
info RUN 'C:\Users\strau\Downloads\talk-master\talk-master\whisper.cpp\build',
info RUN '--config',
info RUN 'Release',
info RUN '--target',
info RUN 'whisper-addon'
info RUN ]
Microsoft (R) Build Engine version 16.11.2+f32259642 for .NET Framework
Copyright (C) Microsoft Corporation. All rights reserved.
whisper.vcxproj -> C:\Users\strau\Downloads\talk-master\talk-master\whisper.cpp\build\bin\Release\whisper.dll
common.vcxproj -> C:\Users\strau\Downloads\talk-master\talk-master\whisper.cpp\build\examples\Release\common.lib
whisper-addon.vcxproj -> C:\Users\strau\Downloads\talk-master\talk-master\whisper.cpp\build\bin\Release\whisper-addon.node
Moving compiled whisper.cpp code to bindings directory...
Installing npm dependencies for llama.cpp examples...
npm WARN [email protected] No repository field.
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: [email protected] (node_modules\fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for [email protected]: wanted {"os":"darwin","arch":"any"} (current: {"os":"win32","arch":"x64"})
audited 329 packages in 1.648s
34 packages are looking for funding
run npm fund
for details
found 0 vulnerabilities
Compiling llama.cpp examples...
info find VS using VS2019 (16.11.33214.272) found at:
info find VS "C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools"
info find VS run with --verbose for detailed information
info TOOL Using Visual Studio 16 2019 generator.
info CMD BUILD
info RUN [
info RUN 'cmake',
info RUN '--build',
info RUN 'C:\Users\strau\Downloads\talk-master\talk-master\llama.cpp\build',
info RUN '--config',
info RUN 'Release',
info RUN '--target',
info RUN 'llama-addon'
info RUN ]
Microsoft (R) Build Engine version 16.11.2+f32259642 for .NET Framework
Copyright (C) Microsoft Corporation. All rights reserved.
ggml.vcxproj -> C:\Users\strau\Downloads\talk-master\talk-master\llama.cpp\build\ggml.dir\Release\ggml.lib
Auto build dll exports
llama.vcxproj -> C:\Users\strau\Downloads\talk-master\talk-master\llama.cpp\build\bin\Release\llama.dll
common.vcxproj -> C:\Users\strau\Downloads\talk-master\talk-master\llama.cpp\build\examples\common.dir\Release\common.lib
Auto build dll exports
llama-addon.vcxproj -> C:\Users\strau\Downloads\talk-master\talk-master\llama.cpp\build\bin\Release\llama-addon.node
Moving compiled llama.cpp code to bindings directory...
Do you want to download models? [y/n] n
Skipping model download...
Script completed successfully!
but when i go to npm run start I Get this error
strau@James MINGW64 ~/Downloads/talk-master/talk-master (master)
$ npm run start
[email protected] start C:\Users\strau\Downloads\talk-master\talk-master
npx ts-node ./index.ts
Error: Cannot find module './bindings/whisper/whisper-addon'
Require stack:
npx ts-node ./index.ts
npm ERR! A complete log of this run can be found in:
npm ERR! C:\Users\strau\AppData\Roaming\npm-cache_logs\2023-06-13T09_43_32_959Z-debug.log
2023-06-13T09_43_32_959Z-debug.log
a question on the Piper repo from @ishan0102 rhasspy/piper#109 (comment)
Also quick question, does this mean piper can only be called by spinning up a docker image and running these commands? Is there a way to get piper into my path so that I can call it from anywhere? I'm trying to get this repo working on my M1 mac: https://github.com/yacineMTB/talk
using the docker commands here: rhasspy/piper#109 (comment)
it's possible that it could work if the command that's run here is updated to use docker exec piper ...
: https://github.com/yacineMTB/talk/blob/master/src/depedenciesLibrary/voice.ts#L25
An issue to capture the state of the architecture.
I'll edit this with the state, and open it up for discussion
does this work for windows or only for linux if so can anyone help me i been trying to run this for the past two days had no luck. If anything ill pay top dollar to get this project running really intrested in running this repo. Please get back to me if your willing to help.
Ik Ik you worked at stripe, but should also add a crypto address for donations (pref Ethereum)
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.