Telephony and Real-Time Transport for Voice Agents
Voice agent telephony is the plumbing that connects a real-time agent to a phone call: the PSTN or SIP leg that carries the call, the media gateway that converts it into packetized audio, and the bidirectional stream that feeds the agent runtime. The phone network imposes constraints no browser deployment has: 8 kHz narrowband audio, strict call lifecycle semantics, and callers who interrupt without ceremony. This chapter maps the transport paths, the codecs, the gateway architecture, and the session lifecycle that a production phone agent must handle.
How Does a Voice Agent Connect to a Phone Call?
A phone call and a voice agent speak different protocols, so a bridge sits between them. The standard inbound flow:
- A caller dials a number managed by a telephony provider (Twilio, Telnyx, Vonage, or a carrier SIP trunk into self-hosted infrastructure such as Asterisk or FreeSWITCH).
- The provider terminates the PSTN or SIP leg and opens a bidirectional media stream, in practice a WebSocket or SIP/RTP session, between the live call and the voice agent runtime.
- From then on the call is a continuous streaming session: caller audio flows to the agent frame by frame, and synthesized speech flows back into the call.
Every session carries a unique call or stream identifier, and that identifier must travel with every inbound and outbound frame. At any real concurrency, audio without a strictly enforced session ID ends up in the wrong call, and cross-call contamination is both a product failure and a privacy incident.
The agent runtime itself, the conversational loop of speech-to-text, reasoning, and text-to-speech with its orchestration layer, sits behind this bridge and is transport-agnostic by design: the same runtime should serve a phone call, a browser session, and an embedded device, with only the media edge changing.
PSTN, SIP, or WebRTC: Which Transport Path?
| Path | What it is | Audio quality | Where it fits |
|---|---|---|---|
| PSTN | The public switched telephone network; every dialable number | 8 kHz narrowband | Reaching anyone with a phone number; the default for customer-facing agents |
| SIP | Session Initiation Protocol; signaling for VoIP calls and trunks | Codec-negotiated, narrowband to wideband | Enterprise PBXs, contact center platforms, carrier trunks into your own infrastructure |
| WebRTC | Browser-native real-time media stack | Wideband Opus, 48 kHz capable | Click-to-call in web and mobile apps; agent-assist consoles; no phone number involved |
The paths are not exclusive. A contact center deployment routes PSTN callers through a SIP trunk while supervisors monitor over WebRTC, and the agent runtime behind the gateway serves all three. The architectural consequence of choosing PSTN reach: you inherit the phone network's audio floor, which the next section quantifies.
Which Codecs Do Phone Agents Deal With?
Telephony audio is standardized around ITU-T G.711: 8 kHz sampling, mono, 64 kbit/s, with two companding variants, mu-law in North America and Japan and A-law in the rest of the world (ITU-T Recommendation G.711). Sampling at 8 kHz caps the audio bandwidth below 4 kHz, which is why phone audio sounds the way it does and why phone speech recognition is harder than broadband: consonant energy above that ceiling is simply gone before any model hears the signal.
WebRTC paths use Opus, the IETF-standardized codec (RFC 6716) that scales from narrowband up to 48 kHz fullband; WebRTC implementations are required to support both Opus and G.711 (RFC 7874).
The engineering rules that follow:
- Align formats end to end. If the gateway emits mu-law, feed the speech-to-text engine mu-law or convert once at ingress; emit synthesis in the format the gateway consumes. Every transcoding hop adds latency and a failure mode while adding zero quality on a narrowband call.
- Pick models for the band you have. A speech-to-text engine evaluated on clean 16 kHz recordings will not reproduce those results on G.711 phone audio. Evaluate on call audio before committing.
- Do not upsample your way out. Converting 8 kHz audio to 16 kHz recovers nothing; the information was never captured.
Media Gateways and the Session Lifecycle
The media gateway is the component that translates between the call and the stream: terminating PSTN or SIP media, packetizing caller audio into frames, and injecting synthesized audio back into the call. Everything conversational lives behind it. A production phone agent must handle the full session lifecycle explicitly:
- Setup: answer the call, open the media stream, bind the session ID, load any caller context.
- Streaming: send and receive audio concurrently with non-blocking, event-driven loops. A pipeline that blocks on synthesis while inbound audio queues produces dead air on one side and missed barge-in on the other.
- Barge-in: enforced at two levels, exactly as Chapter 3 prescribes. Media level: cut outbound audio the moment inbound speech is detected. Logic level: cancel pending reasoning and synthesis so no stale response surfaces. Phone callers interrupt without visual cues to stop them, and a phone agent that talks through interruptions reads as a broken IVR.
- DTMF: keypad tones still arrive, and they must be detected explicitly and routed into call-control logic outside the speech pipeline so they never pollute transcripts. "Press 0 for an operator" remains a legitimate deterministic escape hatch.
- Transfer and escalation: the clean handoff pattern is stop the agent's stream, announce the transfer verbally, mute during the transition, redirect the call leg. Agent-assist scenarios, where the AI stays on the line advising a human, require strict separation of inbound and outbound streams so the agent never injects audio into the caller's channel unless explicitly authorized.
- Teardown: on hangup or redirect, close the media stream immediately, flush buffers, and persist whatever the session owes to records and analytics.
Scaling this architecture is horizontal: each call is an independent, long-lived streaming session, so capacity planning is about WebSocket concurrency, orchestrator throughput, and downstream model rate limits rather than a single hot path. Chapter 7 covers capacity and monitoring; the telephony-specific metrics worth instrumenting from day one are end-of-turn-to-first-audio latency, barge-in success rate, and caller abandonment during silence.
What Is an AI Phone Agent?
An AI phone agent is a voice agent deployed on a telephone number: it answers or places calls over PSTN or SIP, converses in open-ended speech, executes actions through function calls, and hands off to humans when the conversation exceeds its mandate. It is the successor to the IVR tree, replacing menu navigation with conversation, and the architecture is precisely this chapter plus the two before it: a media gateway bridging the call to a streaming session, an agent runtime running the conversational loop, and an orchestration layer enforcing timing and interruption over a channel where callers expect immediate turn-taking.
What separates a production AI phone agent from a demo is rarely the model. It is the transport discipline: format alignment, two-level barge-in, explicit DTMF and transfer handling, and session hygiene at concurrency.
Where Does Speech Processing Sit Behind the Gateway?
An honest placement: on a phone call, the media gateway and the network path to the caller are fixed infrastructure, and no vendor's inference removes the PSTN leg. What an architect controls is everything behind the gateway, and that is where speech processing choices show up in the caller's experience.
The engines slot in behind the gateway as the agent runtime's components. Cheetah Streaming Speech-to-Text transcribes the inbound stream incrementally, Orca Streaming Text-to-Speech starts the outbound response in 128 ms first-token-to-speech (TTS benchmark), and Koala Noise Suppression cleans caller audio frame by frame before recognition, 4.3x more effective than RNNoise averaged across noise levels and 17.3x at 0 dB SNR, at effectively the same compute cost (noise suppression page). Because the engines run inside the runtime process rather than as third-party cloud calls, the telephony hop is the only network leg in the loop, and per-call audio is not fanned out to additional external services, which simplifies both the latency budget and the compliance story (Chapter 8). Two cookbook recipes show the pattern in working code: AI call assist for live in-call transcription and assistance, and on-device AI call screening for an agent that answers, converses, and screens entirely on the phone itself.
Call screening is the instructive edge case. When the "phone agent" runs on the caller's or callee's own device rather than in a data center, the gateway disappears from the architecture entirely, and with it the per-minute media infrastructure. That does not generalize to inbound contact center traffic, which needs the gateway and will keep it. The takeaway is narrower and durable: treat the telephony edge as fixed cost, and spend your optimization budget behind the gateway, where engine choice and inference location are yours to decide.
What's Next?
Everything behind the gateway can also run without one. Chapter 6 makes the case study explicit: the full on-device voice agent stack, the latency, privacy, reliability, and cost consequences, and the hybrid patterns between all-cloud and all-device.