Inside the Streaming Cascade Powering Voice AI

When you talk to a voice agent and it answers in under a second, that’s not one fast model. It’s three models running in parallel with streaming connections between them, each overlapping its work with the others. The architecture behind that experience is a cascaded streaming pipeline: speech-to-text feeds an LLM feeds text-to-speech, and every stage starts before the previous one finishes.

I spent a lot of 2025 and early 2026 working with developers building voice agents. The single question I heard most often was some version of “how do they get it that fast?” The answer is never a single magic model. It’s a set of architectural decisions about streaming, buffering, and parallelism that together make the sub-second experience possible.

How the streaming cascade works

A voice agent has three mandatory stages. Audio comes in from a microphone or phone line. An STT model turns it into text. An LLM processes that text and generates a response. A TTS model turns the response back into audio. Simple enough on a whiteboard.

What makes it real-time is that none of these stages waits for the one before it to finish. The STT streams partial transcripts as the user is still talking. The LLM starts generating the moment it has a complete utterance, without waiting for the speaker to stop. The TTS begins synthesising audio from the first sentence fragment, sometimes before the LLM has finished its full response.

This overlap is the streaming cascade, and it is the single most important architectural pattern in production voice AI today. Without it, response times sit at two to three seconds. With it, the same pipeline hits seven to nine hundred milliseconds.

What is the latency budget for a real-time voice agent?

The total time between a user finishing a sentence and hearing the agent’s first word is called the end-to-end latency budget. Industry benchmarks from production deployments put it between 700 milliseconds and 1.2 seconds for good conversational flow. Beyond 1.5 seconds, users start noticing the delay. Beyond two seconds, the conversation feels broken.

That budget breaks down across the pipeline stages:

• Network transit from the user’s device to the server: 20-60ms
• Audio buffering before STT can process: 20-40ms
• STT producing a final transcript: 100-350ms
• Endpointing, or detecting the user finished speaking: 200-500ms
• LLM generating its first response token (TTFT): 200-400ms
• TTS synthesising the first audio chunk (TTFB): 40-150ms
• Network back to the user: 20-60ms
• Client audio buffer before playback: 20-60ms

The biggest variable is endpointing. Traditional systems use silence detection: wait for N milliseconds of quiet, then assume the speaker is done. The problem is the tradeoff. A short timeout interrupts the user. A long timeout adds a pause before every response. Deepgram’s Flux model and others using model-based turn detection cut this by analysing the linguistic content of partial transcripts to predict when a thought is complete, before the silence threshold fires. This shaves 200-600ms off the budget compared to pure silence detection, while reducing false interruptions by roughly thirty percent.

How do STT, LLM, and TTS overlap in a streaming pipeline?

The key to the cascade is that each component streams its output to the next, and components execute concurrently. The end-to-end latency is not the sum of all stages. It is the time for the first stage to produce useful output, plus the overlap time for the remaining stages to begin.

Here is the sequence in practice:

The user speaks. Audio chunks arrive at the server in 20-millisecond packets at 16kHz sample rate, 640 bytes of PCM int16 each. The STT model processes these incrementally, producing partial transcripts with every chunk and a final transcript when it detects the user has stopped.

The moment the STT emits a final transcript, the LLM receives it and starts generating tokens. It does not wait for the STT to finish processing the next utterance. The LLM streams tokens back one at a time over server-sent events or WebSocket.

A sentence buffer accumulates those tokens until a sentence boundary is detected. Each complete sentence is forwarded to the TTS immediately, while the LLM continues generating the rest of its response. The TTS model synthesises audio from that sentence and streams audio chunks back to the client as they are generated.

The measured result from production benchmarks on a cascaded pipeline using streaming STT, a cloud LLM API, and streaming TTS: best-case time-to-first-audio of 729 milliseconds, with a P50 of 947 milliseconds. The sequential estimate would be over two seconds. Streaming overlap saves the difference.

What role does voice activity detection play in the pipeline?

Voice Activity Detection, or VAD, is the component that decides whether the audio coming in contains speech or silence. It runs on every audio chunk before STT processing and serves three distinct purposes in the pipeline.

First, it prevents the STT from wasting compute on silence. Audio processing costs money per second. Running VAD on the incoming stream and only forwarding speech frames to the STT is the cheapest latency fix in the entire pipeline.

Second, it drives turn-taking. The VAD state machine transitions through four states: silence, speaking, stopping, and stopped. The transition from stopping to stopped is where endpointing happens. The VAD timeout value is one of the most tuned parameters in any production voice system. Set it too short and the agent interrupts the user. Set it too long and every response has a dead pause before it.

Third, it enables barge-in handling. When the user starts speaking while the agent is talking, the system has to stop TTS playback, flush the audio buffer, cancel the in-flight LLM generation, and redirect user speech to the STT pipeline, all within about 200 milliseconds. This is a state machine coordinating four systems simultaneously, and it breaks if VAD is slow or unreliable.

The standard for production deployments is Silero VAD, a two-megabyte model that processes 32-millisecond audio chunks in under a millisecond on CPU. It runs on every chunk, all the time, and its output is the clock that every other component follows.

What are the main transport options for streaming voice?

The transport layer between the user and the server determines the latency floor of the entire system. Two options dominate: WebSocket over TCP and WebRTC over UDP.

WebSocket is simpler. One connection, full-duplex, no NAT traversal complexity. The tradeoff is that TCP guarantees delivery, which means when a packet drops, the connection stalls while it retransmits. A single dropped 20-millisecond audio frame is barely perceptible. A 200-millisecond stall while TCP retransmits a lost packet creates an audible glitch. This head-of-line blocking pushes WebSocket P99 latency to 200-500 milliseconds or higher.

WebRTC uses UDP, which drops late packets instead of stalling the stream. The codec’s error concealment fills the gap. P99 latency on WebRTC sits at 80-150 milliseconds. It also comes with built-in acoustic echo cancellation, which WebSocket users have to implement themselves, a non-trivial signal-processing problem.

The emerging production pattern uses both. WebRTC carries audio between the user and the media server, handling the unpredictable client network with UDP’s resilience. WebSocket carries data between the media server and the AI providers, where TCP’s simplicity is fine on the controlled server-to-server path.

How do orchestration frameworks simplify the pipeline?

Building the cascade from scratch means managing streaming connections to three separate providers, running a VAD state machine, implementing a sentence buffer between the LLM and TTS, handling barge-in with four-way coordination, and debugging timing issues across all of it. Orchestration frameworks exist to wrap this complexity.

Pipecat treats everything as a stream of frames flowing through processors. Define the pipeline from transport through STT, LLM, and TTS back to transport, and Pipecat manages the streaming between stages. It handles VAD, turn detection, interruptions, and sentence buffering out of the box. By April 2026 it had reached version 1.0 after two years of development, with contributions from every major foundation lab and cloud provider.

LiveKit Agents takes a session-based approach with WebRTC transport. Your agent joins a room as a headless participant, subscribes to the user’s microphone, processes audio through the pipeline, and publishes responses. It handles turn-taking and barge-in natively, and comes with first-class function calling support.

The orchestration layer is where most of the practical complexity lives. The models themselves are straightforward. The streaming between them is where the bugs hide.

How do speech-to-speech models compare to the cascaded approach?

The cascaded pipeline is not the only architecture. Speech-to-speech models like OpenAI’s GPT-4o-realtime and Qwen2.5-Omni process audio tokens directly, skipping text entirely. Single model, audio in, audio out.

The tradeoffs are significant. The cascaded approach lets you swap any component independently. Want a cheaper STT for one language and a more accurate one for another? Swap it. Want a different TTS voice for different use cases? Swap it. Need to log, analyse, or score intermediate text for debugging? It is right there in the transcript. The text layer is an observability goldmine that speech-to-speech models hide from you.

On latency, as of early 2026, native speech-to-speech models still trail the cascaded approach. Qwen2.5-Omni measured around 13 seconds time-to-first-audio in independent benchmarks. The cascaded pipeline hits sub-second. OpenAI’s real-time model was closer, around 400 milliseconds voice-to-voice, but at significantly higher cost and with provider lock-in.

For production deployments in 2026, the cascaded architecture remained the standard. Speech-to-speech was compelling for demos and low-volume use cases, but the flexibility, observability, and cost profile of the cascade kept it dominant.

FAQ

What is the most common bottleneck in a streaming voice pipeline?

Endpointing latency is the single biggest variable. VAD timeout values, model-based turn detection, and the tradeoff between false interrupts and dead air dominate the latency budget. Most teams spend their optimisation effort here before anywhere else.

Can I build a voice agent without an orchestration framework?

Yes, but you will end up writing your own state machine for turn-taking, barge-in handling, and sentence buffering. The arXiv tutorial published in March 2026 showed a complete implementation with around 300 lines of Python per component. It is tractable. It is also where most bugs in production voice systems live.

How much does a streaming voice pipeline cost per minute?

Cost breaks roughly into three equal parts: STT at around $0.0077 per minute for production-grade streaming models, LLM inference varying widely by model choice, and TTS at roughly similar rates to STT. The total ranges from $0.02 to $0.05 per minute depending on the LLM tier. Streaming costs less than batch because you pay only for the audio actually processed, not the full call duration.

Why does WebRTC beat WebSocket for voice latency?

WebRTC uses UDP, which drops late packets rather than stalling to retransmit them. A dropped 20-millisecond audio frame is inaudible. A 200-millisecond TCP retransmission stall creates a perceptible gap. At the P99 tail, this difference is the entire latency budget.

Can I use the same architecture for speech-to-speech translation?

Yes, the cascaded architecture extends naturally. Insert a machine translation stage between STT and TTS, or more commonly between the LLM and TTS depending on where you want the translation to happen. The streaming overlap pattern is the same. The latency budget extends by the MT stage, typically adding 100-300 milliseconds.