How to Build a Voice Agent: API, Framework, or Components
There are three ways to build a voice agent: call a managed voice-agent API that runs the whole conversational loop for you, assemble the loop yourself with an orchestration framework such as Pipecat or LiveKit Agents, or build at the component level, choosing and integrating each engine directly. The approaches trade control, latency, and cost at scale against speed to a first demo. This chapter compares them and gives a decision framework, including the cases where the managed path is the right answer.
The Three Build Approaches
Chapter 1 established that every voice agent contains the same functions: audio I/O, speech understanding, reasoning, synthesis, and an orchestration runtime. The build approaches differ in how much of that stack you own.
Managed voice-agent APIs
A managed voice-agent API exposes the entire conversational loop behind a single real-time endpoint. You stream audio in, configure a prompt and function calls, and receive synthesized speech back. The vendor operates the speech-to-text, the LLM integration, the text-to-speech, and the turn-taking logic. OpenAI's Realtime API is the reference example; hosted agent platforms with visual builders and bundled telephony sit one abstraction higher on the same path. Enterprise conversational suites, where voice is one channel inside a larger CX product, are the far end of it.
You trade for that convenience three ways: every conversation transits the vendor's cloud, every stage of the pipeline is the vendor's choice, and every minute of audio is metered under usage-based pricing.
Orchestration frameworks
Open-source frameworks such as Pipecat and LiveKit Agents give you the conversational loop as code: pipeline abstractions, turn-taking state machines, interruption handling, and transport integration. You choose the STT, LLM, and TTS engines and plug them in as pipeline stages. The framework owns coordination; you own component selection and deployment.
This is the middle position on control. You can swap any engine, self-host the pipeline, and inspect every boundary. The latency and cost profile depends entirely on which engines you plug in: a framework pipeline built from cloud APIs inherits cloud round-trips and cloud meters, while the same pipeline built from on-device engines does not.
Picovoice engines slot into framework pipelines as local stages. Cheetah Streaming Speech-to-Text and Orca Streaming Text-to-Speech run in-process on the machine hosting the pipeline, so the STT and TTS stages execute without a network hop, and per-stage latency stops depending on network conditions. The LLM voice agent recipe shows the full loop in Python.
Component-level builds
At the component level you integrate engines directly and write the orchestration yourself, or keep it inside your existing application loop. This is the maximum-control path: you decide every model, every buffer size, every timing threshold, and every deployment target, from a Linux server to a mobile app to an embedded board.
It is also the only path that reaches targets the other two cannot express. A managed API cannot run offline in a vehicle. A cloud-composed framework pipeline cannot promise that audio never leaves the device. Component-level builds can, because inference location is chosen per component.
Picovoice is built for the component path. The engines cover the full core stack (Porcupine Wake Word, Cobra Voice Activity Detection, Cheetah Streaming Speech-to-Text, Rhino Speech-to-Intent, picoLLM, Orca Streaming Text-to-Speech), run on-device across mobile, web, embedded, and server targets, and publish open-source benchmarks (STT, TTS, NLU) so component selection is a measurement, not a guess. The embedded voice assistant recipe is a complete component-level build.
Comparing the Approaches
| Dimension | Managed voice-agent API | Orchestration framework | Component-level build |
|---|---|---|---|
| Control | Prompt, function calls, and configuration only; pipeline internals are up to the vendor | Full choice of engines; loop logic is the framework's, extensible in code | Total: every engine, threshold, and deployment target is yours |
| Latency floor | Bounded below by a network round-trip on every turn, plus the vendor's internal pipeline | Set by the engines you choose; cloud stages add round-trips, local stages do not | Set by hardware and models; fully on-device removes the network from the budget (Chapter 3) |
| Cost model at scale | Usage-based pricing; cost grows unbounded with volume | Framework is free; cost profile is the sum of the engines you plug in | On-device engines are cost-effective at scale |
| Lock-in | High: conversation logic, voices, and telemetry live inside one vendor's runtime | Low to moderate: engines swap behind stage interfaces; the pipeline code is yours | Lowest: each component replaceable behind your own interfaces |
| Time to first demo | Hours: stream audio, get a talking agent | Days to weeks - depending on the complexity - wire chosen engines into a pipeline template | Days to weeks, even years depending on components used and project complexity |
Two dimensions deserve expansion:
- Latency floor is structural, not tunable. A managed API can optimize its internals, and a well-built one overlaps its stages aggressively, but it cannot remove the round-trip between the user and the cloud. Component builds that run stages on-device delete that term from the budget instead of shrinking it.
- Cost at scale follows the meter. Managed APIs price the bundle per minute or per session, and the bill scales linearly with usage forever. Component builds separate the costs: engines that run on-device do not add a per-conversation cloud line item.
How to Choose
Answer three questions:
1. Does your product have a hard constraint the cloud cannot meet? Offline operation, audio that must stay on the device for privacy or compliance (Chapter 8), or deployment to hardware without reliable connectivity (in-car, factory floor) all force the component path. No amount of managed-API convenience satisfies a constraint the architecture cannot express.
2. What does your cost curve look like at target volume? Model the bill at production scale, not at pilot scale. Usage-based pricing that is negligible in a demo compounds across every conversation, every month, indefinitely. If projected volume makes the cloud meter the dominant cost, move stages on-device before the bill arrives, not after.
3. How much engineering do you want to own? Component builds and framework pipelines need engineers who can reason about streaming audio and real-time state. Managed APIs need an integration. Be honest about which team you have, and note that cookbook recipes and framework templates have compressed the component path's startup cost: the coordination patterns are published, not invented per project.
A hybrid answer is legitimate and is how the platform vendors themselves deploy. Apple Intelligence runs a roughly 3-billion-parameter model on-device and escalates to Private Cloud Compute only for requests that need more (Apple Machine Learning Research); Gemini Nano brings the same on-device-first pattern to Android (Android developer docs). For a voice agent, the equivalent split keeps wake word, VAD, STT, and TTS on-device and reserves the cloud for reasoning that genuinely needs a frontier model. Chapter 6 develops the hybrid patterns.
Teams migrate along this spectrum in one direction. Prototyping on a managed API and re-platforming to components later means rebuilding the conversational loop after your product depends on it, under vendor-specific behaviors you now have to reproduce. Starting at the component level with on-device engines costs days more up front and removes the migration entirely: the stack you demo is the stack you ship.
When a Managed Cloud API Wins?
The component path is not the universal answer. A managed voice-agent API is the better choice when:
- The domain needs frontier-scale reasoning on every turn. If the agent's value is open-ended reasoning at the level of the largest hosted models, and no smaller model clears the quality bar, the cloud is where that model lives. On-device LLMs like picoLLM cover bounded domains well; they do not replace a frontier model for unbounded ones.
- There is no deployable hardware. A pure telephony agent has no user device to run on. The pipeline lives in a server rack either way, and a managed runtime can be the fastest way to operate it (Chapter 5 covers the telephony path).
- You are validating demand, not shipping. For a proof of concept whose purpose is to test whether users want the product at all, hours-to-demo beats architectural purity. Treat the managed build as disposable and price the migration into the plan.
- Voice is a minor channel with low volume. If conversation volume is small and projected to stay small, the usage-based bill stays small with it, and the meter is a fair trade for zero operations.
Being clear-eyed about these cases is the point of the exercise: the decision framework above exists so the choice is deliberate, not a default.
What's Next?
Chapter 3 gets quantitative about the property every build approach competes on: latency budgets per stage, end-of-turn detection, and barge-in handling.