Voice UX: Latency, Turn-Taking, and Barge-In
Voice agent latency is the delay between the moment a user stops speaking and the moment the agent's audio reaches their ear. Users judge that delay against the rhythm of human conversation, where the median gap between speakers is roughly 200 ms across 10 languages (Stivers et al., 2009, PNAS). Meeting that bar is not just a model problem; it is a budgeting problem. This chapter breaks the response gap into a per-stage latency budget, then covers the three behaviors that decide whether an agent feels conversational: end-of-turn detection, barge-in handling, and acknowledgment during slow reasoning.
Why 200 ms Is the Bar?
Human turn-taking runs on a strict clock. Stivers et al. measured question-answer transitions across 10 languages and found the same pattern everywhere: speakers aim to minimize the gap between turns, and the median transition lands near 200 ms. Gaps well beyond that read as hesitation, confusion, or disengagement, and listeners assign meaning to them whether the speaker intended it or not.
A voice agent inherits this judgment. A response that arrives in two seconds is not perceived as "a fast API call"; it is perceived as an awkward silence. The user cannot see the pipeline, so every millisecond of transport, transcription, reasoning, and synthesis is billed to a single account: the pause. Voice UX engineering is the discipline of managing that account deliberately.
Two consequences follow:
- Latency is a budget, not a metric. The end-to-end gap is the sum of every stage. Improving one stage while ignoring the others moves the total by that stage's share and no more.
- Perceived latency can be managed separately from measured latency. Overlapping stages, speaking early, and acknowledging delays all change how fast the agent feels without changing how fast any component runs.
What Is a Voice Agent Latency Budget?
A latency budget assigns every stage of the conversational loop a share of the response gap, so regressions are caught per stage instead of discovered as a slow product. The stages below cover the path from the user's last word to the agent's first audible sound.
| Stage | What adds delay | How to shrink it |
|---|---|---|
| Audio capture and transport | Frame buffering; network hops to wherever processing runs | Stream small frames; process on the device so audio travels zero network distance |
| End-of-turn detection | The wait for confidence that the user is done | Tune endpointing per domain; act on early signals, confirm on final ones |
| Transcription finalization | Finalizing the last words of the utterance | Streaming speech-to-text that emits partials mid-utterance, so the final text arrives nearly complete |
| Reasoning | LLM time to first token; any function or retrieval calls | Start reasoning on partial transcripts; keep retrieval snippets small; acknowledge past a threshold |
| Speech synthesis | Time from first response token to first audio byte | Streaming synthesis that speaks from the first tokens instead of the full response |
| Playback start | Output buffering on the client | Keep output buffers as small as stable playback allows |
Two structural facts dominate this table. First, a cloud-cascaded agent pays a network round-trip at every remote stage, and those round-trips are the one line item that engineering effort inside the models cannot reduce. Second, the stages can overlap: reasoning can start before the turn ends, and synthesis can start before reasoning finishes. An agent that executes the loop sequentially pays the sum of all stages; an agent that overlaps them pays closer to the longest one.
Orca Streaming Text-to-Speech reaches 128 ms first-token-to-speech, 2.6x faster than ElevenLabs Streaming at 335 ms (TTS latency benchmark). Because Orca runs on-device, that 128 ms is the whole synthesis line in the budget: there is no network round-trip stacked on top. The same logic applies to every stage moved on-device, which is why inference location is a latency decision before it is anything else. Chapter 6 treats it as one.
How Does End-of-Turn Detection Work?
End-of-turn detection, also called endpointing, is the decision that the user has finished speaking and the agent may respond. It is the highest-leverage timing decision in the system: respond too early and the agent talks over the user; wait too long and the agent adds dead air to a budget that is already tight.
The naive implementation is a fixed silence timeout: no speech for N milliseconds means the turn is over. It fails in both directions. People pause mid-sentence to think, recall a number, or read from a card, and a fixed timeout cuts them off. Set the timeout long enough to tolerate those pauses and every turn ends with the full timeout of silence appended to the response gap.
Production endpointing layers three signals:
- Acoustic: voice activity detection classifies each audio frame as speech or non-speech, providing the raw signal that a pause has started and how long it has lasted.
- Linguistic: the streaming transcript indicates whether the utterance is complete. "I want to pay my" predicts continuation; "I want to pay my bill" predicts completion.
- Behavioral: a tiered response. On medium confidence, start reasoning speculatively but do not speak. On high confidence, commit. If the user resumes, discard the speculative work silently.
The speculative tier is what lets an agent be both patient and fast: it tolerates a long pause without cutting in, yet responds almost instantly when the turn genuinely ends, because the response was already being prepared.
The acoustic layer only helps if it is accurate under noise, because every false speech detection delays endpointing and every miss truncates a turn. On Picovoice's open-source VAD benchmark, Cobra Voice Activity Detection reaches 98.9% detection accuracy at 0 dB SNR with 5 false activations per 100 non-speech frames, against 87.7% for Silero VAD and 50% for WebRTC VAD under the same conditions, while using 3.7% CPU on a Raspberry Pi Zero versus Silero's 45%. Cobra runs on-device, so the endpointing signal carries no network delay of its own.
How Should a Voice Agent Handle Barge-In?
Barge-in is the user speaking while the agent is speaking. It is not an edge case; it is how people correct errors, skip information they already have, and redirect the conversation. An agent that finishes its sentence over the user's interruption signals that the user is not in control, and that impression does not wash off.
Correct barge-in handling spans two layers, and both must fire:
- Media layer: the instant inbound speech is detected during playback, stop or mute outbound audio and flush the playback buffer. Detection here is the VAD again, now running against the caller's channel while the agent speaks, which is why echo handling matters: the agent must not barge in on itself.
- Logic layer: cancel or invalidate everything in flight. A pending LLM generation, a queued synthesis job, or an unexecuted function call that survives the interruption will resurface as a response to a question the user already abandoned. Stale responses are the signature failure of half-implemented barge-in.
After cancellation, the system re-enters listening with conversational state intact. The user's interruption is the start of a new turn, and whatever partial response the agent produced belongs in the dialog history as an interrupted turn, not erased, so the reasoning layer knows what the user did and did not hear. Chapter 4 covers the orchestration patterns that make cancellation clean.
A useful production metric is barge-in latency: time from the user's first interrupting syllable to silence from the agent. It is measurable from event logs and correlates directly with whether users describe the agent as "listening."
Acknowledgment Strategies for Slow Reasoning
Some turns can be legitimately slow. A function call into a booking system, a retrieval query, or a long reasoning chain can push the response past any acceptable silent gap. The fix is not to eliminate the delay but to stop it from being silent.
Humans do this constantly: "let me check that," "one moment." These cues reframe the delay as progress. A voice agent should implement the same behavior as an explicit orchestration policy rather than a scripted personality trait:
- Set a threshold. If the response is not ready within a fixed interval, emit a short acknowledgment. The threshold is a policy decision the orchestrator enforces, not something the LLM decides per turn.
- Keep cues short and low-commitment. "I'm checking that now" buys seconds. A long filler paragraph costs more time than it covers and invites barge-in.
- Make the cue truthful. Tie acknowledgments to real system state, such as a function call in flight. An agent that says "looking that up" while doing nothing trains users to distrust its signals.
- Never acknowledge over the user. Cues respect turn ownership; they fill the agent's silence, not the user's speech.
- Resume instantly. The moment the result lands, speak the answer. Streaming synthesis matters here: the answer should start playing from its first tokens, not after full generation.
Acknowledgment is a patch for a budget hole, and the durable fix is shrinking the hole. Every stage that runs on-device removes a network round-trip from the gap the acknowledgment has to cover, and a first response that starts in 128 ms via Orca needs no filler at all for in-domain turns. The honest split, consistent with the hybrid pattern Chapter 1 established: reserve acknowledgment cues for the turns that genuinely leave the device, such as cloud reasoning or third-party API calls, and make every other turn fast enough not to need them.
What's Next?
Endpointing decides when the agent may think; barge-in decides when it must stop. What happens in between is the reasoning and orchestration layer: intent handling, dialog state, function calling, and retrieval. Chapter 4 covers it.