The Future of Voice Agents: Speech-to-Speech and the Edge
Two architectural shifts are competing to define the next generation of voice agents. Speech-to-speech (S2S) models replace the transcribe-reason-synthesize cascade with a single model that listens and responds directly in audio. At the same time, model efficiency gains keep moving inference from the data center to the device. This chapter analyzes both directions: what S2S promises, what it trades away, which components survive either way, and why the platform vendors have already voted on inference location. It is analysis, not prediction; where the future is uncertain, we say so.
What Is a Speech-to-Speech Model?
A speech-to-speech model processes input audio and generates output audio without an intermediate text representation. Where a cascade converts speech to text, reasons over the text, and converts the response back to speech, an S2S model carries continuous audio representations through the reasoning step itself.
The motivating observation is real: text is a lossy interface. "I guess so" spoken flatly, sarcastically, or enthusiastically produces the same transcript, and a cascade reasons over that transcript with the prosody already discarded. An architecture that preserves acoustic information through reasoning can, in principle, respond to how something was said, not only to what was said.
The claimed advantages follow from removing stage boundaries:
- Lower latency ceiling. Fewer handoffs between models means fewer serialization points, and the model can begin forming a response while still listening.
- Prosody-aware responses. Tone, hesitation, and emphasis reach the reasoning step instead of being flattened into tokens.
- Learned conversational behavior. Turn-taking, backchannels, and interruption handling can emerge from training rather than being engineered as explicit rules.
What S2S Trades Away?
The cascade did not become the dominant production architecture by accident. Its defining property is that every stage boundary is an inspection point, and production teams build on those boundaries. S2S removes them, and everything built on them has to be reinvented:
| Property | Cascade | Speech-to-speech |
|---|---|---|
| Debugging | Log the transcript, replay each stage in isolation | Failures surface as audio; internal states need new tooling |
| Guardrails | Filter and constrain at the text boundary before synthesis | No text checkpoint; control must condition the model itself |
| Compliance | Transcripts feed redaction, retention, and audit pipelines | The auditable artifact is derived after the fact, not native |
| Evaluation | Word error rate, intent accuracy, per-stage benchmarks | Perceptual and conversational metrics, still unstandardized |
| Component choice | Swap the STT, LLM, or TTS engine independently | One model owns the loop; swapping means retraining |
The guardrail point deserves emphasis. In a cascade, the text between reasoning and synthesis is where policy lives: profanity filters, claim checkers, PII redaction, and scripted disclosures all operate on text before it becomes audio. An S2S model that goes from audio to audio has no such checkpoint, so steering it means conditioning the generation itself, a weaker and less verifiable control. For regulated deployments, the transcript is not a debugging convenience; it is the compliance record. Chapter 8 covers why that record is load-bearing.
None of this makes S2S a dead end. It makes S2S a research direction whose production story is unfinished. Published S2S work to date has concentrated on interaction quality; the inspectability, steerability, and audit problems are acknowledged open areas across the field. Teams should watch the direction and hold production adoption to the same bar the cascade already clears.
What Survives the Shift?
If S2S matures, it replaces the middle of the pipeline, not the system around it. Four things keep their jobs in either architecture:
- Voice activity detection. Something must decide which audio is worth processing at all. Feeding silence to a large model wastes compute in any architecture, and endpoint decisions still gate the loop. Cobra Voice Activity Detection plays the same gating role in front of an S2S model as in front of a cascade (VAD benchmark).
- Wake word. An always-on product cannot stream always-on audio to a large model, local or remote. Activation stays a small, dedicated, on-device function; Porcupine Wake Word is that function regardless of what wakes up behind it.
- Orchestration. Function calls, business-system integrations, escalation to humans, session lifecycle, and state across calls are application logic, not model behavior. An S2S model changes what the orchestrator wraps, not whether one exists. Chapter 4's patterns carry over.
- Evaluation and observability. Whatever the architecture, production teams need to know whether the agent answered correctly, complied with policy, and stayed within latency budget. S2S makes this harder, not optional, and the harness investments from Chapter 7 transfer.
For constrained domains, there is a shorter path to "audio in, action out" that is shipping today: Rhino Speech-to-Intent maps speech directly to structured intents without an intermediate transcript, on-device, with results that beat the transcribe-then-parse cascade by 6x on error rate against the Big Tech average (NLU benchmark). It is not a general S2S model, and it does not try to be: it trades open-ended generation for a verifiable, auditable output space, which is the opposite of the S2S trade.
Why Model Efficiency Pushes Inference Toward the Device?
The second shift is quieter and further along. The question "is the model good enough?" is being replaced by "how small can a good-enough model be?", and every improvement in that answer moves inference toward the hardware the user already owns.
The platform vendors have already built for this direction. Apple Intelligence runs a roughly 3-billion-parameter foundation model on the device and escalates to Private Cloud Compute only for requests beyond it (Apple Machine Learning Research). Google ships Gemini Nano for on-device inference on Android (Android developer docs). These are not experiments; they are the default AI architecture of the two dominant mobile platforms, and they encode a judgment: route the request locally first, pay for the cloud only when the task demands it.
Compression research is what makes that judgment hold for voice agents. picoLLM Compression learns a non-uniform bit allocation per weight instead of quantizing every weight to the same depth. On Llama-3-8b at 2-bit compression, picoLLM retains 95% of the float16 model's MMLU score (61.3 vs 64.9), where GPTQ at the same size collapses to 25.1, chance-level for a four-choice benchmark (LLM compression benchmark). The practical reading: reasoning quality that required a cloud GPU now fits in the memory budget of hardware you ship, and the gap between "cloud model" and "device model" is a compression problem, not a law of nature.
For voice agents specifically, three forces compound in the same direction:
- Latency. Every stage moved on-device deletes a network round-trip from a budget measured against the roughly 200 ms human turn gap (Stivers et al., 2009). Chapter 3 does the arithmetic.
- Privacy. Audio that never leaves the device has no residency, retention, or transit problem to mitigate, which matters more, not less, as agents handle richer conversations.
- Cost structure. Cloud pipelines are usage-metered, so cost grows with conversation volume. On-device inference is cost-effective at scale because the compute ships with the product.
Note what this argument does not claim: it does not claim the cloud disappears. Frontier-scale reasoning, tasks with no deployable hardware, and workloads that fit a data center stay in the cloud. The direction of travel is the hybrid split, on-device by default with cloud escalation, and it is the split the platform vendors chose first. And if S2S models mature, the same efficiency pressure applies to them: a compute-intensive audio-native model is exactly the kind of workload that starts in the cloud and migrates down as compression catches up.
The safest way to build for both futures is to keep the survivable components modular and on-device now. A stack of wake word, VAD, streaming speech-to-text, picoLLM, and Orca Streaming Text-to-Speech keeps every boundary inspectable today and lets you swap the middle when a production-grade alternative earns its way in. Betting the whole loop on one opaque model is a decision you cannot partially reverse.
What's Next?
Chapter 11 turns the guide into a build plan: the decision framework in one page, a first-week milestone list, and where to start with the Free Trial.