On-Device Voice Agents
An on-device voice agent runs its inference, wake word, voice activity detection, speech-to-text, reasoning, and text-to-speech, on the hardware the user is holding, driving, or standing in front of, instead of streaming audio to a cloud API. Where each of those components executes is not an implementation detail. It is an architecture decision that fixes four properties of the finished product before a line of application code is written: the latency floor, the privacy posture, the behavior without a network, and the shape of the cost curve as usage grows. This chapter examines each consequence, the precedent set by Apple and Google, hybrid patterns that split the stack, a complete on-device reference build, and the cases where cloud inference is the right call.
Why Is Inference Location an Architecture Decision?
Every component in the voice agent stack from Chapter 1 can run in the cloud or on the device, and the choice is made per component, not per agent. A cloud default feels convenient because it reduces integration to API calls. The cost of that convenience is structural: every remote stage adds a network round-trip to every conversational turn, sends user audio off the device, fails when the network does, and bills by usage forever.
These properties cannot be patched in later: a team that discovers a latency, compliance, or unit-economics problem after launch has to re-architect, not tune. Inference location belongs in the first architecture review, alongside build approach (Chapter 2) and transport (Chapter 5).
The question to ask per component is not "can the cloud do this?" but "does this component need the cloud?" Wake word detection, voice activity detection, streaming transcription, intent inference, and speech synthesis all run on commodity hardware today, down to Raspberry Pi and, for wake word and intent, Arm Cortex-M microcontrollers (Porcupine Wake Word, Rhino Speech-to-Intent). The component that genuinely needs a datacenter, frontier-scale reasoning, is one stage out of five.
The Four Consequences of Inference Location
| Property | Cloud inference | On-device inference |
|---|---|---|
| Latency | Network round-trip added to every remote stage, every turn | No round-trip; bounded by local compute |
| Privacy and compliance | Audio and transcripts leave the device; residency, retention, and transit obligations follow | Audio never leaves the device; those obligations never arise |
| Offline reliability | Agent availability is bounded by the network path | Agent works with no connectivity |
| Cost | Usage-based pricing; spend grows unbounded with conversation volume | Cost-effective at scale; inference runs on hardware you already ship |
Latency: removing round-trips instead of optimizing them
Chapter 3 established the bar: median human turn gaps are roughly 200 ms across 10 languages (Stivers et al., 2009, PNAS). A cloud cascade pays transport twice per remote stage, audio up and results down, before any model computes. On-device inference deletes those legs entirely. The remaining latency is a function of local compute, which the team controls and benchmarks, not of cell coverage, which nobody controls.
Orca Streaming Text-to-Speech reaches 128 ms first-token-to-speech, 2.6x faster than ElevenLabs Streaming at 335 ms, and it is the only engine in the open-source TTS latency benchmark below the 200 ms threshold. It gets there without a network dependency: synthesis starts from LLM tokens as they arrive, on the device.
Privacy and compliance: obligations that never arise
Voice is biometric, identifying, and content-rich. The moment audio leaves the device, an organization inherits questions about where it is processed, how long it is retained, who can access it, and which jurisdiction's rules apply. On-device processing answers those questions by construction: audio that is never transmitted has no residency to manage, no retention schedule to enforce, and no transit to encrypt. Chapter 8 treats this in depth, and the medical voice agent tutorial applies it to HIPAA-sensitive audio.
Offline reliability: the agent that works in a tunnel
A cloud agent's uptime is the product of its own uptime and the network's. For a car entering a parking garage, a warehouse picker between access points, or a kiosk on a congested link, that product drops below what a voice interface can tolerate. On-device agents make connectivity irrelevant to the core loop. This is why in-car voice control, voice picking, and factory voice agents are built on-device: the environments where voice is most valuable are the environments where connectivity is least dependable.
Cost: the shape of the curve, not the size of the bill
Cloud voice APIs meter by usage: seconds of audio, characters synthesized, tokens generated. That pricing is proportional to conversation volume, so the bill grows unbounded as the product succeeds; the unit economics are set by the vendor's meter, not the builder's engineering. On-device inference is cost-effective at scale because the compute is the CPU already inside the phone, car, or appliance being shipped. Success adds devices, and each device brings its own compute with it.
The resource budgets required for Picovoice products are small. Porcupine Wake Word runs in under 1 MB with keyword files under 25 KB, Cobra Voice Activity Detection is a 450 KB library. Cheetah Streaming Speech-to-Text ships a 34 MB model, and Orca Streaming Text-to-Speech synthesizes within 29 MB of peak memory from a 7 MB model. A full listen-to-speak pipeline fits inside the memory budget of a mid-range phone app.
What Did Big Tech Choose? Apple Intelligence and Gemini Nano
The strongest evidence that on-device-first is a sound production architecture is that the two companies with the largest deployed device fleets chose it. Apple Intelligence runs a roughly 3-billion-parameter foundation model on the device and escalates to Private Cloud Compute only for requests that exceed its capability (Apple Machine Learning Research). Google ships Gemini Nano for on-device inference on Android (Android developer docs).
Both vendors had unconstrained access to datacenter capacity, and both still placed the default inference path on the device, reserving the cloud for escalation. Latency, privacy, and the economics of serving a billion-device fleet all point the same direction, for a single product line as much as for a platform vendor.
Hybrid Voice Agents: On-Device by Default, Cloud on Escalation
Hybrid is not a compromise between two pure architectures; it is the architecture Apple shipped. The design rule is simple: every stage runs on-device unless a specific request needs capability the device cannot provide, and only that request escalates.
| Pattern | On-device | Cloud | Fits |
|---|---|---|---|
| Fully on-device | Entire loop: wake word, VAD, STT, intent or LLM, TTS | Nothing | Vehicles, industrial, medical, offline-first products |
| Escalation hybrid | Entire loop by default | Larger LLM for turns that exceed the local model | Assistants with open-ended long-tail queries |
| Split cascade | Wake word, VAD, STT, TTS | Reasoning every turn | Agents tied to cloud-resident business systems |
| Cloud cascade | Audio capture only | Everything else | Telephony agents with no deployable hardware |
Every stage moved on-device removes a round-trip from the latency budget, a data flow from the compliance review, and a metered line item from the cloud bill. The split cascade already keeps raw audio on the device, sending only text upward; the escalation hybrid goes further and touches the cloud only for the turns that need it.
The on-device RAG voice document QA recipe shows retrieval-grounded question answering running fully locally, and the AI call assist recipe shows on-device speech components operating alongside cloud systems in a live-call setting. The same engines serve both ends of the table, so moving between patterns is a configuration change, not a rewrite.
A Full On-Device Stack, Component by Component
The pipeline below is a complete conversational loop with zero cloud dependencies. The embedded voice assistant recipe and the LLM voice agent recipe assemble it end to end.
| Stage | Engine | Footprint and measured performance |
|---|---|---|
| Wake | Porcupine Wake Word | Under 1 MB; 97.3% detection accuracy at 1 false alarm per 10 hours in 10 dB SNR noise; 0.6% CPU on Raspberry Pi 5 (benchmark) |
| Detect speech | Cobra Voice Activity Detection | 450 KB library; 98.9% true positive rate at 5% false positive rate in 0 dB SNR noise, versus 87.7% for Silero and 50% for WebRTC VAD; 3.7% CPU on Raspberry Pi Zero (benchmark) |
| Transcribe | Cheetah Streaming Speech-to-Text | 34 MB model; 10.1% English word error rate versus 11.9% for Google Streaming; 590 ms word emission latency versus 830 ms Google and 920 ms Amazon (benchmark) |
| Understand and decide | Rhino Speech-to-Intent or picoLLM | Rhino: under 2.5 MB; 97.3% command acceptance versus 84.3% Amazon Lex and 77.3% Google Dialogflow, averaged over 7 noise conditions from 6 to 24 dB SNR (benchmark). picoLLM: 2-bit compressed Llama-3-8b retains 95% of float16 MMLU performance (benchmark) |
| Speak | Orca Streaming Text-to-Speech | 7 MB model, 29 MB peak memory; 128 ms first-token-to-speech, 2.6x faster than ElevenLabs Streaming at 335 ms (benchmark) |
Walking a turn through the pipeline: Porcupine listens continuously and activates the agent without any audio leaving the device, which is the property that makes always-on listening acceptable in the first place. Cobra separates speech from silence and feeds the turn-taking logic from Chapter 3. Cheetah emits partial transcripts while the user is still speaking.
At the reasoning stage the stack forks. For command-and-control domains, car controls, appliance settings, warehouse workflows, Rhino maps speech directly to a structured intent in one step, with 6x fewer errors than the Big Tech average (Amazon Lex, Google Dialogflow, IBM Watson, Microsoft LUIS) on the NLU benchmark. For open-ended dialog, picoLLM runs compressed language models locally; its compression retains 99.9% of Llama-3-8b's MMLU score at 3-bit quantization, versus 83.1% for GPTQ (LLM compression benchmark). Orca then speaks the response, synthesizing from tokens as the model generates them.
Every engine in the table runs on Android, iOS, Linux, macOS, Windows, Raspberry Pi, and web browsers, so one architecture spans every deployment target. Noise-heavy environments add Koala Noise Suppression in front of the pipeline; multi-speaker products add Eagle Speaker Recognition.
When Does the Cloud Win?
An honest architecture review names the cases where on-device is the wrong default:
- Frontier-scale reasoning. A compressed on-device model handles constrained and mid-complexity dialog. Open-domain reasoning at the frontier, long multi-document synthesis, complex code generation, still requires models that do not fit on a device. The escalation hybrid answers it: keep the loop local and escalate the turn that needs the datacenter, as Apple Intelligence does with Private Cloud Compute.
- No deployable hardware. A telephony agent answering PSTN calls has no user device to run on; the "device" is a phone line. Inference runs server-side by necessity, and the architecture choices shift to the transport and scaling questions of Chapter 5 and Chapter 7.
- Server-side batch workloads. Transcribing recorded archives has no conversational latency constraint and no per-device footprint constraint. Accuracy-per-dollar dominates, and engines built for that job, such as Leopard Speech-to-Text, can run wherever the audio already lives.
What's Next?
Chapter 7 covers what happens after the architecture ships: testing and evaluation harnesses for conversational systems, telemetry, graceful degradation, and capacity planning, including how the scaling model changes when inference lives on the devices you ship.