Inside the Voice Agent Pipeline: STT, LLM, and Streaming TTS

Google shipped Gemini 3.5 Live Translate today, a single model that takes speech in one language and returns speech in another. It is the end-to-end speech model that cascaded pipelines are supposed to make obsolete. Yet almost every production voice agent still runs speech-to-text, then an LLM, then text-to-speech. Why?

A voice agent is three models chained together in real time. Speech-to-text transcribes what the user said. A large language model figures out what to say back. Text-to-speech turns that response into audio the user hears. The magic is not any single model. It is how fast they hand off to each other and overlap while doing it.

This is a deep look at how that pipeline works, where the latency hides, and why the industry standard is still a three-stage cascade. I have spent the last few years building SDKs and demos around these systems at Deepgram, working with teams who ship voice agents in production. Here is what I have learned.

Why Voice Agents Need Three Models Instead of One

The obvious question is why not just train one model that takes audio in and returns audio out. End-to-end speech models exist. OpenAI’s Realtime API, Gemini Live, and Qwen2.5-Omni all accept raw audio and return raw audio through a single model call. They sound natural, they preserve tone and emotion, and they skip the serialization overhead of converting speech to text and back.

So why does almost every production voice agent still use the three-stage cascade?

The short answer is control. A cascaded pipeline lets you swap any component independently. Want Deepgram for STT because it transcribes fast in noisy environments, Claude for the LLM because its tool calling is reliable, and ElevenLabs for TTS because its voices sound the way your brand needs them to? In a cascade, that is configuration. In a speech-to-speech model, you get what the vendor chose, and if the TTS voice does not work for your use case, you cannot replace it without replacing the whole model.

There is a deeper reason too. Speech-to-speech models struggle with function calling. When a voice agent needs to look up a customer record, update a database, or transfer a call, the LLM needs to emit structured tool calls that downstream systems can parse reliably. Text-based LLMs have mature tool-calling semantics. Speech-to-speech models, in mid-2026, still vary wildly in how they handle this. Some block silently while waiting for a tool result. Others drop the thread entirely.

The cascade also makes debugging tractable. When a voice agent says something wrong, you need to know whether the STT misheard, the LLM misreasoned, or the TTS mispronounced. Text sits between every stage. You can log exactly what was transcribed, what the LLM produced, and what was synthesized. With a speech-to-speech model, the audio goes in and audio comes out, and figuring out where the failure happened means re-running the whole thing with different trace settings. In production, that is a liability.

The Three Stages

Stage 1: Streaming Speech-to-Text

The pipeline starts with audio arriving from the user’s microphone, phone line, or WebRTC session. On Deepgram’s platform, that audio arrives in 20-millisecond chunks over a WebSocket connection. Each chunk is 640 bytes of PCM int16 audio at 16 kHz.

The STT model does not wait for the user to finish speaking. It streams partial transcripts back as audio arrives. Every 100 to 300 milliseconds, the model emits an update. “I want to” becomes “I want to book” becomes “I want to book a flight”. These are partial results with is_final: false. They are useful for showing the user what the system is hearing, but they are not stable enough to send to the LLM yet.

When the model detects a natural pause, it emits a final transcript with is_final: true and speech_final: true. That is the signal that the user has stopped speaking and the utterance is complete. This is the transcript you send to the LLM.

The latency from the last word spoken to the final transcript arriving is usually 200 to 500 milliseconds on a well-optimized streaming STT model. Deepgram Nova-3, for example, consistently lands around 300 to 400 milliseconds P50 on moderately clean audio. Noisy environments push that higher, but the streaming architecture means the pipeline does not have to wait for the full transcript before starting to work.

Stage 2: The LLM as Reasoning Layer

Once the final transcript arrives, it gets sent to the LLM along with the conversation history and system prompt. The LLM is the brain of the operation. It decides what to say next, whether to call a tool, and how to phrase the response.

For voice agents, the LLM needs to be fast and concise. A model that generates paragraph-length responses adds hundreds of milliseconds of latency before TTS can start. The best voice agent LLMs are configured with tight max_tokens limits, around 50 to 150 tokens per response, and system prompts that encourage short, conversational replies. “Sure, let me check that for you” is a better response than “I would be happy to assist you with checking the status of your account. Let me look that up right now.”

The LLM should stream its output token by token using SSE or a WebSocket. The first few tokens typically arrive within 200 to 400 milliseconds. That is often a complete sentence opener like “I can help with that” or “Your order is on its way”. Those first tokens can start feeding the TTS engine while the LLM finishes generating the rest.

Streaming overlap is where the pipeline starts to feel real time. The LLM does not need to finish before TTS begins. The system just needs a sentence boundary.

Stage 3: Streaming Text-to-Speech

The TTS engine receives text and returns audio chunks. In a streaming pipeline, it does not wait for the full LLM response. It starts synthesizing the first sentence as soon as a period or question mark tells the system the sentence is complete.

The latency from receiving text to emitting the first audio chunk is called time to first byte. For modern streaming TTS models like Deepgram Aura-2, that is typically under 200 milliseconds. The audio starts streaming back to the user while the LLM is still generating the second sentence. By the time the LLM finishes, the user has already heard the first sentence of the response and is processing it. The pipeline has bought itself 500 to 800 milliseconds of perceived head start.

Voice quality matters for trust. Clipped or robotic TTS makes users distrust the agent, even if the reasoning is correct. The TTS models in production in 2026, Aura-2, ElevenLabs Turbo, Cartesia Sonic, produce speech with natural prosody, correct emphasis, and domain-specific pronunciation. The gap between TTS quality and human speech has narrowed enough that users stop noticing the voice after the first interaction, which is the goal.

The Sentence Buffer: The Critical Primitive You Have Never Heard Of

The sentence buffer is the orchestration primitive that makes streaming pipelines work. It sits between the LLM and TTS, accumulating tokens until it detects a natural break, then flushes the complete sentence to the TTS engine.

Without a sentence buffer, sending individual tokens to TTS produces choppy, unnatural speech. “I” gets pronounced in isolation. “Can” follows as its own word. The rhythm breaks.

The buffer has to handle a few edge cases. Abbreviations like “Dr.” and “Mr.” should not trigger sentence breaks. Decimal numbers like “3.14” should not either. A minimum character count, usually around 10, prevents single-word sentences from being sent to TTS. And when the LLM stream ends, the buffer flushes any remaining text even if no sentence boundary was detected.

Most voice agent frameworks have their own version of this primitive. Pipecat calls it SentenceAggregator. LiveKit has a similar text processing pipeline. Deepgram’s Voice Agent API handles it inside the runtime. The implementation details differ but the concept is the same: buffer until you have a complete thought, then speak it.

Where the Latency Actually Goes

The total time from the user stopping speaking to them hearing the first word of the response is the sum of three components, overlapped as much as the pipeline allows.

STT finalization takes the last chunk of audio and produces the confirmed transcript. On Deepgram Nova-3, this is typically 300 to 400 milliseconds. The LLM time to first token is another 200 to 400 milliseconds for a fast model like Claude Haiku or GPT-4o-mini. TTS time to first byte adds 150 to 250 milliseconds.

In a sequential pipeline where each stage waits for the previous one to finish, those numbers add up to roughly 700 to 1050 milliseconds. That is borderline for natural conversation. Users start to notice gaps above 700 milliseconds.

In a streaming pipeline where the stages overlap, the perceived latency is lower. The LLM starts processing while STT is still emitting partial transcripts (on a speculative basis). TTS starts speaking while the LLM is still generating. The measured end-to-end time from speech final to first audio can drop below 500 milliseconds with well-optimized components, which feels instantaneous to most users.

Network latency adds another 50 to 150 milliseconds per hop. That is why co-locating the STT, LLM, and TTS infrastructure matters, and why regional endpoints exist. Google shipped Gemini 3.5 Live Translate today for 70+ languages, a single model that does speech-to-speech translation in one region. A cascaded STT-LLM-TTS pipeline does not have that luxury. When an agent in Sydney talks to a pipeline running in Oregon, the round trip adds latency that no amount of model optimization can fix.

The Hybrid Path

Some teams are experimenting with a hybrid approach. Use a speech-to-speech model for the input side, capturing tone and emotion from the user’s voice, but output text instead of audio. Then route that text through a dedicated TTS engine for full control over the voice.

This gives you the prosodic awareness of speech-to-speech on the input side and the voice quality, cloning support, and latency control of a dedicated TTS on the output. The trade-off is that you still have to manage two model interfaces, and the input-side model might not output text in a format that your downstream systems can parse cleanly. But for applications where emotional nuance on the input side is critical, healthcare triage, therapy bots, conflict resolution, it is worth the engineering overhead.

What Production Voice Agents Actually Use Today

If you look at the voice agents shipping in production in mid-2026, the overwhelming majority use a streaming three-stage cascade. Vapi, Retell, Bland.ai, and most of the platforms you have heard of run STT to LLM to TTS with streaming overlap at every stage. The frameworks that support them, Pipecat, LiveKit Agents, the HuggingFace speech-to-speech pipeline, all implement the same pattern.

The reasons are consistent across every team I have talked to. Modularity means you can pick the best STT for your language and noise profile, the best LLM for your tool-calling needs, and the best TTS for your brand voice. Debuggability means you can log the transcript between every stage and trace exactly where something went wrong. Cost control means you can use a cheap STT for high-volume calls and a premium TTS only for the first impression.

The speech-to-speech models will keep improving. Function calling will get more reliable. Latency will come down. But for production voice agents that need to be reliable, debuggable, and cost predictable, the three-stage cascade is going to be the standard for a while yet. The pipeline is not a compromise. It is the architecture that makes voice agents shippable.

FAQ

What is a voice agent pipeline?

A voice agent pipeline chains three models together: speech-to-text (STT) converts audio to text, a large language model (LLM) generates a response, and text-to-speech (TTS) converts that response back to audio. In a streaming pipeline, these stages overlap to reduce perceived latency. It is the most common architecture for production voice agents in 2026.

Why use three models instead of one speech-to-speech model?

A three-stage cascade gives you modularity. You can swap any component independently, debug failures by reading the text between stages, and use different providers for different parts of the pipeline. Speech-to-speech models are simpler to integrate but limit your control over individual components and make debugging harder.

What is a sentence buffer in a voice agent?

A sentence buffer sits between the LLM and TTS, accumulating tokens until a natural break is detected (period, question mark, or a minimum token count). It prevents individual words from being sent to TTS one at a time, which would produce choppy speech. Most frameworks implement their own version of this primitive.

How fast does a voice agent need to be?

Research shows that users notice response delays above 500 to 700 milliseconds. A well-optimized streaming pipeline can achieve 400 to 800 milliseconds from the end of the user’s speech to the first audio of the response. Each component, STT, LLM, and TTS, contributes 150 to 400 milliseconds, and streaming overlap makes the total feel faster than the sum.

Can I build a voice agent pipeline myself?

Yes. The components are available as APIs from multiple providers. You need a streaming STT provider (Deepgram, AssemblyAI, Google), a fast LLM (Claude Haiku, GPT-4o-mini), and a streaming TTS provider (Deepgram Aura-2, ElevenLabs, Cartesia). You wire them together with a sentence buffer and a WebSocket for audio transport. Frameworks like Pipecat and LiveKit handle most of the orchestration for you.