Voice Agent Reference Architectures
A voice agent reference architecture is a proven component topology for composing audio capture, speech understanding, reasoning, and speech synthesis into a production system. Four tiers cover the deployment spectrum: a single-agent assistant inside a web or mobile app, specialized and localized agents, integrated distributed systems such as AI call centers, and embedded implementations that run entirely on the device. This chapter gives the topology, the decision criteria, and a worked example for each tier.
Two axes define every architecture in this chapter. The first is system scope: how many components, integrations, and concurrent sessions the design must coordinate. The second is inference location: whether each component runs in the cloud, on the device, or in a hybrid split. The tiers below progress along the first axis. The second axis is a choice you make inside every tier, and it sets your latency floor, privacy posture, and cost curve, as Chapter 1 and Chapter 6 establish.
Tier 1: Single-Agent Web or App Assistant
The baseline production pattern is one agent, one user, one continuous conversational loop, embedded in a browser or application. The client captures microphone audio, the pipeline detects speech, transcribes it, reasons over the transcript, and streams synthesized speech back while the user can still interrupt.
| Component | Role | Runs where |
|---|---|---|
| Client audio I/O | Capture microphone frames, play synthesized audio | Device |
| Voice activity detection | Gate the pipeline; feed endpointing | Device or cloud |
| Streaming speech-to-text | Emit partial and final transcripts | Device or cloud |
| Reasoning (LLM) | Decide the response; emit function calls for actions | Device or cloud |
| Function handlers | Execute lookups, transactions, side effects | Backend service |
| Streaming text-to-speech | Synthesize audio from the first tokens | Device or cloud |
| Orchestration loop | Own timing, state, and barge-in | With the pipeline |
A function-calling extension turns this assistant from a talker into an actor: the reasoning engine emits structured calls (order status, scheduling, account lookup) that a backend service executes and returns for verbalization. The conversational loop is unchanged; only the reasoning stage gains tools. Chapter 4 covers the patterns.
When to use it. First production deployments, product assistants, internal tools, and any single-task conversational feature where one agent serves one user per session and deep system integration is not yet required.
The LLM voice agent recipe builds this exact tier with Cobra Voice Activity Detection, Cheetah Streaming Speech-to-Text, picoLLM, and Orca Streaming Text-to-Speech. Because every stage runs in-process on the user's machine, the topology loses its WebSocket: there is no session server to authenticate, scale, or fail over, and no per-turn network round-trip in the latency budget.
Tier 2: Specialized and Localized Agents
Tier 2 keeps the Tier 1 loop intact and changes what runs inside it. Specialization enters through model choice and configuration, not through new orchestration: a domain-tuned vocabulary for the speech-to-text stage, a constrained intent grammar for the reasoning stage, or multilingual recognition with per-language synthesis voices.
Two specialization patterns dominate:
- Domain-specific agents. A clinical, financial, or legal agent needs accurate transcription of specialized terminology and structured output into a system of record (an EMR, a CRM, a case file). The speech loop is unchanged; the speech model, the prompt, and the downstream integration carry the domain. The medical voice agent tutorial walks through this pattern with the added constraint that patient audio never leaves the device, which removes the data-transmission problem that Chapter 8 analyzes.
- Localized and multilingual agents. A single session must recognize more than one language, keep context across a language switch, and answer in a voice that matches. Multilingual voicebots and the speech-to-speech translation recipe show the pattern, with on-device translation as the bridge between recognition and synthesis.
How Do You Build a Multilingual Voice Agent?
Multilingual support is a Tier 2 specialization because it changes the models inside the loop, not the loop itself. A production multilingual agent has to answer four questions, each of which maps to a component decision:
| Requirement | Component decision |
|---|---|
| Which language is the user speaking? | Detect it explicitly with spoken language identification, or fix it per deployment when the market is known |
| Transcribe that language accurately | A speech-to-text model trained for each target language, evaluated on that language's audio, not assumed from English results |
| Reason across a language switch | Keep dialog state language-agnostic in code (Chapter 4); the transcript language changes, the task state does not |
| Answer in a matching voice | Per-language synthesis voice selection, or a translation bridge when the agent must cross languages within a turn |
Two architecture patterns cover most products. A per-language pipeline fixes the language at session start (a caller picks a language, or the market determines it) and loads the matching STT and TTS models; it is the simpler build and fits IVR-replacement and single-market apps. A language-switching pipeline detects the language per utterance with spoken language identification, then routes to the right recognizer and voice, holding one shared task state across the switch; it is what a genuinely multilingual assistant or a translation agent needs.
The accuracy point is the one teams underestimate: a speech-to-text engine's English score does not transfer. Picovoice publishes multilingual word error rates so the per-language decision is measured rather than assumed. Cheetah Streaming Speech-to-Text and Leopard Speech-to-Text report per-language WER for English, French, German, Spanish, Italian, and Portuguese on the STT benchmark, against Amazon, Google, and Azure on the same audio. Evaluate on the languages you will actually ship before committing an engine.
The on-device constraint holds across languages. Spoken language identification, per-language transcription, on-device translation, and per-language synthesis all run locally, so a multilingual agent keeps the same offline operation, privacy posture, and cost curve as a single-language one. The speech-to-speech translation recipe assembles the language-switching pattern end to end, and multilingual voicebots is the use-case view.
When to use it. Regulated or terminology-heavy domains, and products serving more than one language or market, where the Tier 1 loop is proven and the differentiation lives in accuracy and output structure.
For constrained specialist domains, Rhino Speech-to-Intent replaces the transcribe-then-parse path with direct speech-to-intent inference on a custom domain grammar. On the NLU benchmark, this approach produced 6x fewer errors than the Big Tech average (Amazon Lex, Google Dialogflow, IBM Watson, Microsoft LUIS). A specialized agent that only ever needs to fill a known set of intents does not need an LLM in the loop at all.
Tier 3: Integrated Distributed Systems
Tier 3 is where a voice agent stops being an application feature and becomes infrastructure: telephony ingress, concurrent sessions, integrations with business systems, and more than one specialized agent behind a single conversation.
What does an AI call center architecture look like?
An AI call center connects the conversational loop to the phone network and to the systems of record that a human agent would use. The topology has four layers:
| Layer | Function | Notes |
|---|---|---|
| Telephony ingress | Terminate PSTN/SIP calls; carry 8 kHz G.711 call audio | Chapter 5 covers codecs and gateways |
| Media gateway | Bridge call audio into a bidirectional stream per session | Twilio-style media streams or self-hosted SBC |
| Orchestrator | Own per-call state, barge-in, DTMF, handoff, and teardown | One instance per concurrent call |
| Agents and integrations | STT, reasoning, TTS, plus CRM, ticketing, scheduling APIs | Escalation path to human agents |
At this tier, multi-agent orchestration earns its place. Long calls decompose into phases (qualify, resolve, close), each handled by a narrowly prompted agent with a limited toolset. The orchestrator summarizes each phase and injects the summary as context for the next agent, which keeps prompts small, isolates failures, and makes each phase testable on its own. The caller hears one continuous conversation.
When to use it. Inbound and outbound phone automation, IVR replacement, support deflection, agent assist, and any workflow where call volume, compliance, and integration count make the agent a distributed system with an SLA. Chapter 7 covers what it takes to operate this tier.
The honest position is that Tier 3 orchestration is application territory, and the speech engines sit behind the gateway. Because the engines run on infrastructure you control, the entire recognition and synthesis path can stay on-premises: call audio reaches your servers through the gateway and is processed there, with no third-party speech API in the data path, which is the property regulated call centers ask for first. The call assist recipe shows real-time transcription and suggestion behind a live call, and on-device call screening applies the same loop at the endpoint instead of the data center. Cloud speech APIs win this tier when you have no server footprint at all and accept usage-based pricing that scales with call volume; self-hosted engines are cost-effective at scale and keep the audio inside your boundary.
Tier 4: Embedded and Edge Implementations
Tier 4 removes the network from the architecture. Every component of the conversational loop runs on the product's own hardware: a vehicle head unit, a factory terminal, a kiosk, a medical device, a mobile app that must work offline. There is no gateway, no session server, and no cloud dependency; the topology is a single process pipeline over local audio I/O.
The reference pipeline, engine by engine:
| Stage | Engine | Why it is on the device |
|---|---|---|
| Activation | Porcupine Wake Word | Hands-free activation without streaming audio anywhere |
| Gating | Cobra Voice Activity Detection | Feed endpointing; skip compute on silence |
| Understanding | Cheetah Streaming Speech-to-Text | Partial transcripts with no round-trip |
| Reasoning | Rhino Speech-to-Intent or picoLLM | Structured intents for control domains; on-device LLM for dialog |
| Response | Orca Streaming Text-to-Speech | 128 ms first-token-to-speech (TTS benchmark) |
When to use it. Environments where connectivity is unreliable or absent, where audio must not leave the device for privacy or regulatory reasons, where the latency budget cannot afford a network round-trip, or where per-interaction cloud pricing breaks the unit economics of a shipped product. This tier is Chapter 6's subject in full.
This tier is where the on-device stack is not one option among several but the defining constraint, and it is the tier Picovoice was built for. The embedded voice assistant recipe assembles the full pipeline above on embedded hardware. In-car voice control applies it where the vehicle cannot assume coverage, and the factory voice agent tutorial applies it where plant-floor audio stays on the plant floor. Voice-directed warehouse work follows the same pattern in the voice picking recipe.
How to Choose: The Architecture Continuum
| Tier | Scope | Decision trigger | Inference location |
|---|---|---|---|
| 1. Single agent | One app, one loop | First production voice feature | Any; on-device removes the session server |
| 2. Specialized | Domain or language depth | Accuracy and structured output drive value | Any; on-device removes data transmission |
| 3. Distributed | Telephony, integrations, concurrency | Call volume and business-system reach | Cloud or self-hosted; on-prem engines keep audio inside |
| 4. Embedded | The device is the product | Offline, latency, privacy, or unit economics | On-device by definition |
Deployments move along this continuum rather than picking a final tier on day one: a Tier 1 assistant grows domain models into Tier 2, gains telephony into Tier 3, or ships into hardware as Tier 4.
The tiers describe scope, not inference location, and conflating the two is the design error to avoid. A Tier 3 call center can run its speech stack on-premises; a Tier 1 web assistant can run fully in the browser. The hybrid pattern applies at every tier: run wake word, VAD, speech-to-text, and synthesis locally, and reserve the cloud for reasoning that exceeds local hardware. That is the split Apple chose for Apple Intelligence, a roughly 3-billion-parameter on-device model with Private Cloud Compute for heavier requests (Apple Machine Learning Research), and Google ships Gemini Nano for on-device inference on Android (Android developer docs). Cloud-only remains the right call when you need frontier-scale reasoning on every turn or have no deployable hardware at all.
What's Next?
Chapter 10 looks past the cascade: speech-to-speech models, what they promise and what they trade away, and why model efficiency keeps pushing inference toward the device.