Real-time transcription about swiftwhisper

[cerupcat]

I'd also like to see an SFSpeechRecognizer-like API for easy replacement of SFSpeechRecognizer.

[libratiger]

+1 for this feature

[fakerybakery]

Yes, this would be a great feature.

[fakerybakery]

Right now, I'm setting a timer to start + stop the transcription every 2 seconds, however it's not that accurate because if a word is cut off, then whisper tries to improvise, and the text often has hallucinations.

[jacobjiangwei]

how would whiper officially support real time? The cut off issue is same for official library, correct? @fakerybakery

[cerupcat]

how would whiper officially support real time? The cut off issue is same for official library, correct? @fakerybakery

The Whisper cpp repo has examples of how to implement realtime.

[fakerybakery]

I think that the whisper.cpp library stores some of the previous recording history and uses that to fix the cutoff issue, but I'm not sure.

[jacobjiangwei]

how would whiper officially support real time? The cut off issue is same for official library, correct? @fakerybakery

The Whisper cpp repo has examples of how to implement realtime.

thanks for point that out. Just curious, why it's in Obj-C but not in swift version...

[fakerybakery]

I don't know why, but if someone could port the example to Swift, I would really appreciate that (I'm really bad at Obj-C).

[brytonsf]

I think that the whisper.cpp library stores some of the previous recording history and uses that to fix the cutoff issue, but I'm not sure.

Yep, I believe it does too – see this line (and line 245)

[barkb]

Don't have a great understanding, but to me it looks like whisper.objc is storing the contents of a buffer when it fills up, then calling it's transcribe function against what it just stored, while clearing the buffer and re-enqueuing it. I don't know a ton about AVFAudio, but does anyone know if you could use AVAudioEngine and AVAudioPCMBuffer to create similar functionality? I'm thinking you could call Whisper.transcribe here with the buffer data if you can get that buffer data back from AVAudioEngine. Does anyone know if that would work?

[ldenoue]

@barkb have you ever found a solution to this real-time idea?