Reasoning and Orchestration for Voice Agents
The reasoning and orchestration layer sits between what the user said and what the agent says back. It interprets intent, tracks dialog state, calls functions against real systems, grounds answers in retrieved data, and does all of it under the timing and interruption constraints that Chapter 3 established. The design principle that holds the layer together: models decide what should happen, deterministic code decides how it happens, and an orchestration runtime owns when.
What Does the Reasoning Layer Actually Decide?
Strip away vendor terminology and the reasoning layer runs a two-step loop on every completed user turn:
- Think: interpret the transcript in the context of the conversation and decide whether the response is plain speech or an action.
- Act: if an action is needed, execute it deterministically, feed the result back into context, and generate the spoken response.
The separation is the point. Interpretation, ambiguity resolution, and language generation belong in a model, because they are open-ended. Validation, permissions, business rules, and side effects belong in code, because they must be exact, auditable, and testable. A voice agent that lets the model execute side effects directly has no boundary to log, no boundary to test, and no boundary to secure.
Voice sharpens this discipline beyond what text chat requires. Sessions are long, turns overlap, input is noisy, and every extra reasoning step is audible as silence. Determinism matters more than in chat: lower the decoding temperature for transactional flows, track state explicitly in code rather than trusting the model's memory, and log every decision boundary so a failed call can be replayed and audited.
Intent Handling: LLM, Speech-to-Intent, or Both?
Every user turn must resolve to an intent before anything else can happen. There are two production architectures for getting there, and the choice is a domain question, not a fashion question.
| Approach | Pipeline | Fits | Failure mode |
|---|---|---|---|
| Transcribe then reason | Speech-to-text, then LLM interprets the transcript | Open-ended dialog, unbounded topics | Errors compound across two stages; LLM can hallucinate intents or arguments |
| Speech-to-intent | One model maps audio directly to intent plus slot values | Closed domains with a defined command set | Out-of-domain requests are rejected rather than answered |
For open-ended conversation, the transcribe-then-reason path is the right tool: only an LLM can handle "actually, before that, can you explain the difference between the two plans?" For closed domains, the two-stage pipeline pays twice for nothing. Speech-to-text errors feed the NLU as ground truth, the NLU adds its own errors, and both stages add latency. Fusing them into a single acoustic-to-intent model removes the error handoff and the intermediate step, and its output is deterministic: a request either maps to a defined intent with typed slots or is rejected, with no hallucinated middle ground.
Rhino Speech-to-Intent is that fused architecture, running on-device. On Picovoice's open-source NLU benchmark, Rhino produced 6x fewer errors than the Big Tech average (Amazon Lex, Google Dialogflow, IBM Watson, Microsoft LUIS). Head to head across noise levels from 6 to 24 dB SNR, Rhino averaged 97.3% command acceptance against 84.3% for Amazon Lex and 77.3% for Google Dialogflow, and the gap widens in high noise: 94% versus 76% and 67% at 6 dB SNR. The two approaches also compose: route the defined command set through Rhino for deterministic sub-second handling, and fall through to the LLM path for everything open-ended.
How Do Voice Agents Manage Dialog State?
Dialog state is everything the agent must remember for the conversation to stay coherent: what the user asked, what the agent answered, which function calls ran and what they returned, and which task is in progress. Voice adds a distinctive requirement: state must survive interruption, because barge-in truncates turns mid-flight and the agent needs to know what the user actually heard.
Production practice separates three tiers:
- Working context: recent turns and function results, injected into each reasoning step. As sessions grow, summarize or prune older turns so the context window holds intent without holding every word.
- Task state: the current workflow (a booking half-completed, a slot still unfilled) tracked explicitly in code as a state machine, not inferred from the transcript. When the user barges in and changes direction, code-level state is what lets the agent cancel cleanly and resume correctly.
- Long-term memory: preferences and identifiers persisted externally, re-injected selectively at session start. Persist what matters, not everything said; voice sessions generate far more tokens than their decisions require.
The rule that keeps all three honest: the transcript is evidence, not the database. Anything the business depends on lives in structured state that code wrote and code can read back.
How Does Function Calling Work in a Voice Agent?
Function calling is how the agent acts on the world. Instead of generating free-form instructions, the model emits a structured request, a function name with typed arguments, and the orchestrator executes it against the booking system, the CRM, the device API. The model chooses the action; the orchestrator decides whether, when, and how it runs.
Four rules keep function calling production-safe in voice:
- The model never executes side effects directly. Every action passes through code that validates arguments, checks permissions, and enforces business rules.
- Every call has a clear request and response boundary. That boundary is where you log, test, and audit.
- Results re-enter context before speech. The agent speaks about what the function returned, not what the model hoped it would return.
- Every in-flight call is cancelable or safely ignorable. If the user barges in mid-execution, the result must not surface as a stale response, and a side effect that already committed must be reconciled, not pretended away.
Function calling also carries a latency tax the orchestrator must manage: each call requires two reasoning passes, one to emit the request and one to incorporate the result. That gap is exactly where the acknowledgment policy from Chapter 3 fires: past a threshold, say "I'm checking that now," and resume the instant the result lands.
For a defined command domain, speech-to-intent is function calling with the middleman removed. Rhino's output is already the structured call: intent name plus typed slot values, produced in one on-device inference with no transcript to parse and no arguments to hallucinate. Reserve LLM function calling for the requests that genuinely need open-ended interpretation, and the latency tax shrinks to the turns that earn it.
Retrieval Grounding: RAG for Voice Agents
Retrieval-augmented generation grounds the agent's answers in verified data instead of model weights: policy documents, account records, product manuals. In voice, RAG has one constraint text chat never faces: the retrieval happens inside a pause the user is listening to.
That constraint dictates the architecture. Retrieval belongs in the orchestration layer as a structured function call, not buried in the model prompt, so the runtime can time it, cache it, and cancel it like any other action. The working rules:
- Fetch small, high-relevance snippets, not documents; every retrieved token is context the reasoning step must chew through before speaking.
- Cache frequent queries; the questions callers ask cluster hard.
- Inject results into the next reasoning step rather than re-architecting the prompt mid-turn.
- If retrieval crosses the acknowledgment threshold, say so, then deliver.
RAG does not require a cloud. The on-device RAG cookbook recipe builds voice document Q&A where retrieval, reasoning, and speech all run locally: embeddings and search over the document, picoLLM for generation, Orca Streaming Text-to-Speech for the answer. picoLLM makes the reasoning step fit on real hardware: its compression retains 95% of Llama-3-8b's float16 MMLU score at 2-bit quantization (61.3 versus 64.9), where GPTQ collapses to 25.1 (picoLLM benchmark). For knowledge bases that are sensitive, static, or needed offline, on-device RAG removes the round-trip and the data exposure in one move; for corpora that outgrow the device, hybrid retrieval with local reasoning is the same pattern Apple Intelligence normalized, on-device by default with cloud escalation for the heavy cases (Apple Machine Learning Research).
The Orchestration Runtime's Job: Timing, Interruption, State
Everything above assumes a component that owns the clock. The orchestration runtime is that component. It is not intelligent; it is authoritative. Its responsibilities:
- Timing: react to turn events from endpointing, start reasoning on speculative end-of-turn signals, enforce the acknowledgment threshold, and start synthesis from the first response tokens.
- Interruption: treat user speech as the highest-priority signal in the system. On barge-in: stop playback, cancel pending reasoning and function calls, mark the interrupted turn in state, return to listening.
- State: hold the single source of truth for conversation state and emit explicit lifecycle events (listening, thinking, speaking) that UIs, analytics, and monitoring consume, so no component infers state from timing heuristics.
The runtime is where a voice agent's quality is decided, because it is the only component that sees every signal. A mediocre model behind a disciplined runtime interrupts cleanly and never speaks stale answers. A frontier model behind a sloppy runtime talks over users and answers retracted questions. To see the loop assembled in code, the LLM voice agent cookbook recipe walks through a complete runtime with on-device components.
What's Next?
The reasoning layer operates inside whatever delivery environment the deployment imposes, and for phone-based agents that environment brings its own physics: 8 kHz audio, SIP session semantics, and media gateways. Chapter 5 covers telephony and real-time transport.