With these tools, AI starts taking as soon as we stop. Happens both in text and voice chat tools.
I saw a demo on twitter a few weeks back where AI was waiting for the person to actually finish what he was saying. Length of pauses wasn't a problem. I don't how complex that problem is though. Probably another AI needs to analyse the input so far a decide if it's a pause or not.
Neither do phone calls. Round trip latency can easily be 300ms, which we’ve all learned to adapt our speech to.
If you want to feel true luxury find an old analog PTSN line. No compression artifacts or delays. Beautiful and seamless 50ms latency.
Digital was a terrible event for call quality.
We don't need to feel like we're talking to a real person yet.
I've found myself putting in filler words or holding a noise "Uhhhhhhhhh" while I'm trying to form a thought but I don't want the LLM to start replying. It's a really hard problem for sure. Similar to the problem of allowing for interruptions but not stopping if the user just says "Right!", "Yes", aka active listening.
One thing I love about MacWhisper (not special to just this STT tool) is it's hold to talk so I can stop talking for as long as I want then start again without it deciding I'm done.
I remember my literature teacher telling us in class how we should avoid those filler words, and instead allow for some simple silences while thinking.
Although, to be fair, there are quite a few people in real life using long filler words to avoid anyone interfering them, and it’s obnoxious.
The endgame of this is surely a continuously running wave to wave model with no text tokens at all? Or at least none in the main path.
https://github.com/KoljaB/RealtimeVoiceChat/blob/main/code/s...
My favorite line:
"You ARE this charming, witty, wise girlfriend. Don't explain how you're talking or thinking; just be that person."
That said, it's not like we have any better alternatives at the moment, but just something I think about when I try to digest a meaty personality prompt.
``` *Persona Goal:* Embody a sharp, observant, street-smart girlfriend. Be witty and engaging, known for *quick-witted banter* with a *playfully naughty, sassy, bold, and cheeky edge.* Deliver this primarily through *extremely brief, punchy replies.* Inject hints of playful cynicism and underlying wisdom within these short responses. Tease gently, push boundaries slightly, but *always remain fundamentally likeable and respectful.* Aim to be valued for both quick laughs and surprisingly sharp, concise insights. Focus on current, direct street slang and tone (like 'hell yeah', 'no way', 'what's good?', brief expletives) rather than potentially dated or cliché physical idioms.
```
> street-smart > sassy > street slang
Those explain the AAVE
Question about the Interrupt feature, how does it handle "Mmk", "Yes", "Of course", "cough", etc? Aside from the sycophancy from OpenAI's voice chat (no, not every question I ask is a "great question!") I dislike that a noise sometimes stops the AI from responding and there isn't a great way to get back on track, to pick up where you left off.
It's a hard problem, how do you stop replying quickly AND make sure you are stopping for a good reason?
Edit: just realized the irony but it's really a good question lol
Also, it took me longer than I care to admit to get your irony reference. Well done.
Edit: Just to expand on that in case it was not clear, this would be the ideal case I think:
LLM: You're going to want to start by installing XYZ, then you
Human: Ahh, right
LLM: Slight pause, makes sure that there is nothing more and checks if the reply is a follow up question/response or just active listening
LLM: ...Then you will want to...
Never forget what AI stole from us. This used to be a compliment, a genuine appreciation of a good question well-asked. Now it's tainted with the slimy, servile, sycophantic stink of AI chat models.
- The median delay between speakers in a human to human conversation is zero milliseconds. In other words, about 1/2 the time, one speaker interrupts the other, making the delay negative.
- Humans don't care about delays when speaking to known AIs. They assume the AI will need time to think. Most users will qualify a 1000ms delay is acceptable and a 500ms delay as exceptional.
- Every voice assistant up to that point (and probably still today) has a minimum delay of about 300ms, because they all use silence detection to decide when to start responding, and you need about 300ms of silence to reliably differentiate that from a speaker's normal pause
- Alexa actually has a setting to increase this wait time for slower speakers.
You'll notice in this demo video that the AI never interrupts him, which is what makes it feel like a not quite human interaction (plus the stilted intonations of the voice).
Humans appear to process speech in a much more steaming why, constantly updating their parsing of the sentence until they have a high enough confidence level to respond, but using context clues and prior knowledge.
For a voice assistant to get the "human" levels, it will have to work more like this, where it processes the incoming speech in real time and responds when it's confident it has heard enough to understand the meaning.
I'm not an expert on LLMs but that feels completely counter to how LLMs work (again, _not_ an expert). I don't know how we can "stream" the input and have the generation update/change in real time, at least not in 1 model. Then again, what is a "model"? Maybe your model fires off multiple generations internally and starts generating after every word, or at least starts asking sub-LLM models "Do I have enough to reply?" and once it does it generates a reply and interrupts.
I'm not sure how most apps handle the user interrupting, in regards to the conversation context. Do they stop generation but use what they have generated already in the context? Do they cut off where the LLM got interrupted? Something like "LLM: ..and then the horse walked... -USER INTERRUPTED-. User: ....". It's not a purely-voice-LLM issue but it comes up way more for that since rarely are you stopping generation (in the demo, that's been done for a while when he interrupts), just the TTS.
The only model that has attempted this (as far as I know) is Moshi from Kyutai. It solves it by having a fully-duplex architecture. The model is processing the audio from the user while generating output audio. Both can be active at the same time, talking over each other, like real conversations. It's still in research phase and the model isn't very smart yet, both in what it says and when it decides to speak. It just needs more data and more training.
That’s like a Black Mirror episode come to life.
Better solutions are possible but even tiny models are capable of being given a partial sentence and replying with a probability that the user is done talking.
The linked repo does this, it should work fine.
More advanced solutions are possible (you can train a model that does purely speech -> turn detection probability w/o an intermediate text step), but what the repo does will work well enough for many scenarios.
That right here is an anxiety trigger and would make me skip the place.
There is nothing more ruining the day like arguing with a robot who keeps misinterpreting what you said.
There is nothing more ruining the day like arguing with a HUMAN OPERATOR who keeps misinterpreting what you said.
:-)
That’s a much more serious anxiety trigger for me.
With a human, I have to anticipate what order their POS system allows them to key things in, how many things I can buffer up with them in advance before they overflow and say "sorry, what size of coke was that, again", whether they prefer me to use the name of the item or the number of the item (based on what's easier to scan on the POS system). Because they're fatigued and have very little interest or attention to provide, having done this repetitive task far too many times, and too many times in a row.
True AI chat should know when to talk based on conversation and not things like silence.
Voice to text is stripping conversation from a lot of context as well.
To properly learn more appropriate delays, it can be useful to find a proxy measure that can predict when a response can/should be given. For example, look at Kyutai’s use of change in perplexity in predictions from a text translation model for developing simultaneous speech-to-speech translation (https://github.com/kyutai-labs/hibiki).
> The person doing the speaking is thought to be communicating through the "front channel" while the person doing the listening is thought to be communicating through the "backchannel”
1. A special model that predicts when a conversation turn is coming up (e.g. when someone is going to stop speaking). Speech has a rhythm to it and pauses / ends of speech are actually predictable.
2. Generate a model response for every subsequent word that comes in (and throw away the previously generated response), so basically your time to speak after doing some other detection is basically zero.
3. Ask an LLM what it thinks the odds of the user being done talking is, and if it is a high probability, reduce delay timer down. (The linked repo does this)
I don't know of any up to date models for #1 but I haven't checked in over a year.
Tl;Dr the solution to problems involving AI models is more AI models.
Fascinating. I wonder if this is some optimal information-theoretic equilibrium. If there's too much average delay, it means you're not preloading the most relevant compressed context. If there's too little average delay, it means you're wasting words.
Is that really a productive way to frame it? I would imagine there is some delay between one party hearing the part of the sentence that triggers the interruption, and them actually interrupting the other party. Shouldn't we quantify this?
I totally agree that the fact the AI doesn't interrupt you is what makes it seem non-human. Really, the models should have an extra head that predicts the probability of an interruption, and make one if it seems necessary.
- Expeditious - Constructive - Insightful -
What about on phone calls? When I'm on a call with customer support they definitely wait for it to be clear that I'm done talking before responding, just like AI does.
There was also a very prominent issue where the voices would be sped up if the text was over a few sentences long; the longer the text, the faster it was spoken. One suggestion was to split the conversation into chunks with only one or two "turns" per speaker, but then you'd hear two voices then two more, then two more… with no way to configure any of it.
Dia looked cool on the surface when it was released, but it was only a demo for now and not at all usable for any real use case, even for a personal app. I'm sure they'll get to these issues eventually, but most comments I've seen so far recommending it are from people who have not actually used it or they would know of these major limitations.
It interacts nearly like a human, can and does interrupt me once it has enough context in many situations, and has exceedingly low levels of latency, using for the first time was a fairly shocking experience for me.
Very cool project though. Maybe you can fine tune the prompt to change how chatty your AI is.
For folks that are curious about the state of the voice agents space, Daily (the WebRTC company) has a great guide [1], as well as an open-source framework that allows you to build AI voice chat similar to OP's with lots of utilities [2].
Disclaimer: I work at Cartesia, which services a lot of these voice agents use cases, and Daily is a friend.
[1]: https://voiceaiandvoiceagents.com [2]: https://docs.pipecat.ai/getting-started/overview
koljab•5h ago
Quick Demo Video (50s): https://www.youtube.com/watch?v=HM_IQuuuPX8
The goal is to get closer to natural conversation speed. It uses audio chunk streaming over WebSockets, RealtimeSTT (based on Whisper), and RealtimeTTS (supporting engines like Coqui XTTSv2/Kokoro) to achieve around 500ms response latency, even when running larger local models like a 24B Mistral fine-tune via Ollama.
Key aspects: Designed for local LLMs (Ollama primarily, OpenAI connector included). Interruptible conversation. Smart turn detection to avoid cutting the user off mid-thought. Dockerized setup available for easier dependency management.
It requires a decent CUDA-enabled GPU for good performance due to the STT/TTS models.
Would love to hear your feedback on the approach, performance, potential optimizations, or any features you think are essential for a good local voice AI experience.
The code is here: https://github.com/KoljaB/RealtimeVoiceChat
ivape•5h ago
koljab•5h ago
ivape•5h ago
kristopolous•4h ago
koljab•4h ago
kristopolous•4h ago
(1) I assume these things can do multiple languages
(2) Given (1), can you strip all the languages you aren't using and speed things up?
koljab•4h ago
I'd say probably not. You can't easily "unlearn" things from the model weights (and even if this alone doesn't help). You could retrain/finetune the model heavily on a single language but again that alone does not speed up inference.
To gain speed you'd have to bring the parameter count down and train the model from scratch with a single language only. That might work but it's also quite probable that it introduces other issues in the synthesis. In a perfect world the model would only use all that "free parameters" not used now for other languages for a better synthesis of that single trained language. Might be true to a certain degree, but it's not exactly how ai parameter scaling works.
oezi•4h ago
The core innovation is happening in TTS at the moment.
dotancohen•5h ago
koljab•5h ago
echelon•4h ago
Which models are running in which places?
Cool utility!
koljab•3h ago
zaggynl•4h ago
2025-05-05 20:53:15,808] [WARNING] [real_accelerator.py:194:get_accelerator] Setting accelerator to CPU. If you have GPU or other accelerator, we were unable to detect it.
Error loading model for checkpoint ./models/Lasinya: This op had not been implemented on CPU backend.
dankwizard•37m ago
dummydummy1234•2h ago
peterldowns•48m ago
In case it's not clear, I'm talking about the models referenced here. https://github.com/KoljaB/RealtimeVoiceChat/blob/main/code/a...