How Voice Agents Work
A voice agent is a real-time system that listens to speech, extracts meaning, decides on a response, and speaks back, all while the user is still allowed to interrupt. Every voice agent, regardless of vendor or framework, is built from the same five functions: audio capture, speech understanding, reasoning, speech synthesis, and an orchestration runtime that keeps them synchronized. This chapter explains each part, why coordinating them in real time is the hard problem, and where each component can run: in the cloud or on the device.
What Is a Voice Agent?
A voice agent is an autonomous conversational system that interacts with users understanding spoken language in real time. Users speak to it over a phone line, a device microphone, or an embedded application, and it responds with synthesized speech.
Three properties separate voice agents from the systems that came before them:
- Not an IVR. Interactive voice response systems play prerecorded prompts and force callers through fixed menus ("press 2 for billing"). A voice agent understands open-ended speech and decides its next action from context.
- Not a chatbot with a microphone. Text chatbots can take seconds to respond without breaking the interaction. Spoken conversation cannot. In a study of turn-taking across 10 languages, the median gap between speakers was roughly 200 ms (Stivers et al., 2009, PNAS). A voice agent competes with that human baseline on every turn.
- Not a sequential pipeline. Production agents overlap their stages: reasoning starts before the user finishes speaking, and audio playback starts before the full response is generated.
The definition above says nothing about where the models run. Treating inference location (cloud, on-device, or hybrid) as an explicit architecture decision, rather than defaulting to cloud APIs, is the single highest-leverage choice in voice agent design. It sets your latency floor, your privacy posture, and your cost curve at scale. The platform vendors have already made this choice: Apple Intelligence runs a roughly 3-billion-parameter foundation model on-device and escalates to Private Cloud Compute only for requests that need a larger model (Apple Machine Learning Research), and Google ships Gemini Nano for on-device inference on Android (Android developer docs). Chapter 6 covers this decision in depth.
The Conversational Loop
Every voice agent runs a continuous loop with five stages:
| Stage | Function | What it produces |
|---|---|---|
| Listen | Capture streaming audio from the user | Raw audio frames |
| Understand | Detect speech, transcribe it, find the end of the turn | Text plus timing signals |
| Reason | Interpret intent and decide the response or action | Response text or a function call |
| Respond | Convert the response into speech | Streaming audio |
| Speak | Play audio back while continuing to listen | Interruptible playback |
The loop is not turn-by-turn. In a production agent the stages run concurrently: the speech-to-text engine emits partial transcripts while the user is mid-sentence, the language model starts reasoning on those partials, and the text-to-speech engine begins synthesizing audio from the first generated tokens. This streaming, event-driven overlap is what makes an agent feel conversational instead of transactional.
Orca Streaming Text-to-Speech is built for the "Respond" stage of this loop. It synthesizes speech from LLM tokens as they arrive instead of waiting for the full response, reaching 128 ms first-token-to-speech, 2.6x faster than ElevenLabs Streaming at 335 ms (TTS latency benchmark). Because synthesis runs on the device, no network round-trip is added on top.
Why Real-Time Voice Is Hard?
Building the loop is not the hard part. Running it under real-world conditions is. Four constraints turn a voice agent from an API integration into a distributed systems problem:
Latency compounds across every stage
Each stage adds delay: audio transport, endpoint detection, transcription, reasoning, synthesis, playback. None of these is slow in isolation; their sum is what the user hears as a pause. A cloud-cascaded agent also pays a network round-trip at every remote stage. Budgeting latency per stage, and removing round-trips where possible, is the core engineering discipline of voice UX. Chapter 3 gives per-stage budgets.
Turn-taking is a prediction problem
The agent must decide, from audio alone, whether the user is done speaking or just pausing. Cut in too early and the agent interrupts. Wait too long and the silence feels broken. This is why voice activity detection and end-of-turn detection are first-class components rather than preprocessing details. Picovoice's VAD benchmark compares Cobra, Silero, and WebRTC VAD on accuracy and compute.
Users interrupt
People barge in over the agent's speech, correct themselves mid-utterance, and change intent while the agent is responding. The system must detect the interruption, stop playback, and re-enter listening without losing conversational state.
Components do not share a clock
Speech recognition, reasoning, and synthesis are independent processes emitting asynchronous events. Without an orchestration layer that owns timing and state, the failure modes are predictable: premature cutoffs, double-talk, and responses to questions the user already retracted.
The consequence: a voice agent must be designed as an event-driven, interrupt-aware system that treats timing signals and partial results as first-class inputs, not as edge cases.
The Voice Agent Stack
Voice agent architectures separate into two layers.
The core conversational layer answers: can the agent hold a conversation? It contains the components below. Remove any one of them and you no longer have a conversational agent.
| Component | Role in the system | On-device engine (Picovoice) |
|---|---|---|
| Audio I/O and transport | Stream audio in and out with low latency | Audio Processor |
| Wake word (embedded agents) | Activate the agent hands-free without streaming audio anywhere | Porcupine Wake Word (benchmark) |
| Voice activity detection | Detect speech vs silence; feed turn-taking logic | Cobra Voice Activity Detection (benchmark) |
| Speech-to-text | Transcribe streaming speech with word timings | Cheetah Streaming Speech-to-Text (benchmark) |
| Reasoning | Interpret intent; decide the response or action | picoLLM for open-ended dialog, Rhino Speech-to-Intent for command-and-control |
| Text-to-speech | Synthesize streaming audio from the response | Orca Streaming Text-to-Speech (benchmark) |
| Orchestration runtime | Own timing, state, and interruption across all components | Your application loop; Chapter 4 covers patterns |
To see these components assembled end to end, the Picovoice cookbook walks through a complete LLM-powered voice agent and a fully embedded voice assistant.
The operational layer answers: can that conversation run at scale, under load, within compliance constraints? It adds memory and context, observability, integrations, and security. These components do not change how the agent converses; they decide whether it survives production. Chapter 7 and Chapter 8 cover them.
For constrained domains, the "Understand" and "Reason" stages do not need to be separate components. Rhino Speech-to-Intent maps speech directly to structured intents in a single step, skipping the transcribe-then-parse round-trip. On Picovoice's NLU benchmark, this approach produced 6x fewer errors than the Big Tech average (Amazon Lex, Google Dialogflow, IBM Watson, Microsoft LUIS). For open-ended conversation, a full STT-to-LLM path remains the right tool.
Cascade vs Speech-to-Speech Architectures
The stack above describes a cascade architecture: speech becomes text, text is reasoned over, text becomes speech. Cascades are the dominant production pattern because every boundary is inspectable: you can log the transcript, audit the reasoning input, and debug each stage independently.
Speech-to-speech (S2S) models process audio and generate audio without an intermediate text representation. They promise lower latency and prosody awareness, and they trade away the inspectability that production teams rely on for debugging, compliance, and guardrails. Chapter 10 examines when that trade is worth making.
Where the Stack Runs: Cloud, On-Device, or Hybrid
Each component in the core layer can execute in the cloud or on the device. The choice shapes four properties of the finished agent:
- Latency: on-device inference removes the network round-trip from every stage that runs locally, and keeps the agent responsive when connectivity degrades.
- Privacy and compliance: audio that never leaves the device creates no data-residency, retention, or transmission obligations to mitigate. Chapter 8 covers the compliance consequences, and the medical voice agent tutorial shows the pattern applied to HIPAA-sensitive audio.
- Reliability: an on-device agent works offline; a cloud agent's uptime is bounded by its network path. In-car voice control and factory voice agents are built on this property.
- Cost: cloud voice pipelines bill by usage, so costs grow with conversation volume and stay unbounded. On-device processing is cost-effective at scale because inference runs on hardware you already ship.
Hybrid is not a compromise position; it is how the largest deployed voice AI systems are built. Apple Intelligence answers on-device by default and escalates to Private Cloud Compute only when a request needs a larger model (Apple Machine Learning Research). The same split applies to voice agents: wake word, voice activity detection, and speech-to-text on the device; heavyweight reasoning in the cloud only when the domain demands it. Every stage that moves on-device removes a round-trip, a data transfer, and a usage-metered bill. Chapter 6 gives a decision framework, a full on-device reference build, and the hybrid patterns in between.
What's Next?
Chapter 2 turns this anatomy into a build decision: managed voice-agent APIs, orchestration frameworks, or component-level builds, and how to choose based on control, latency, and cost at scale.