Six tools that power production voice agents

I spent the last few months building real-time voice agent demos and SDK examples at Deepgram. The part that surprised me most was not the models. It was how many supporting tools you need to make a voice agent reliable in production. An API key gets you a transcript. It does not get you a product.

Here are six tools that shaped how I build voice agents, ordered from most foundational to most surprising.

1. Deepgram Nova-3

Nova-3 is the streaming STT model that runs beneath most of the voice agents I worked on. Sub-300ms streaming latency, keyterm prompting for domain vocabulary, and native code-switching across 50+ languages.

The model itself is good. What matters more is that it streams. For voice agents, batch transcription is useless. You need words as they are spoken, not a transcript delivered after the call ends. Nova-3 sends interim results over a WebSocket connection and lets you decide when a turn is complete. Or you pair it with Flux, which makes that decision for you.

The keyterm prompting feature deserves more attention than it gets. You pass a list of domain terms at connection time and the model biases toward them. Product names, acronyms, medical terms, anything the generic model would mangle. No retraining needed.

2. Deepgram Flux

Flux is the turn detection layer that pairs with Nova-3. It is a separate model that runs alongside transcription and emits structured events: StartOfTurn, Update, and EndOfTurn.

Before Flux, voice agents used endpointing, which is basically a silence timer. If the user pauses for 400ms, assume they are done. But people pause mid-sentence all the time. They pause to think. They pause because the dog barked. An endpointing-based agent cuts them off or sits in awkward silence.

Flux models turn-taking directly. The EndOfTurn event fires when the model decides the utterance is complete, not when a timer expires. The EagerEndOfTurn variant can fire 150 to 250ms earlier than a silence-based approach, which makes voice agents feel responsive rather than impatient.

3. WebSocket debugging tools

You cannot test a streaming voice agent with curl. You need WebSocket tooling, and the standard HTTP tools do not help.

websocat became my default. It pipes audio files into a streaming STT connection and prints the transcript to stdout. The workflow looks like this:

websocat -t "wss://api.deepgram.com/v1/listen?model=nova-3&interim_results=true" \
  -H "Authorization: Token $DG_API_KEY" \
  - raw < audio.raw

For interactive testing I used a small Python script with the websockets library. It reads microphone input, sends chunks to the API, and prints intermedi results. That let me test latency, turn detection, and accuracy in real time without building a UI.

If you work with voice agents and do not have a WebSocket debug harness, build one. It saves more time than any other tool on this list.

4. FFmpeg

Every audio file arrives in a different format: 48kHz stereo WAV, 16kHz mono FLAC, 8kHz mulaw from a phone system, or Opus from a browser. WebRTC outputs in yet another encoding.

FFmpeg normalises all of it. The pipeline I used most was:

ffmpeg -i input.mp3 -acodec pcm_s16le -ar 16000 -ac 1 output.raw

16kHz, 16-bit, mono, raw PCM. That is the format every STT API expects, and FFmpeg gets you there in one command. For batch testing, a simple shell loop processes an entire directory.

5. Docker and NVIDIA tooling

Deepgram runs self-hosted for regulated industries. Healthcare, finance, government. The on-prem deployment uses Docker containers with NVIDIA GPU passthrough, and testing it locally means running the same containers on your own hardware.

The tooling that mattered: nvidia-smi for GPU monitoring, docker compose for multi-service orchestration, and the NVIDIA Container Toolkit (nvidia-container-runtime) for GPU passthrough. A typical test stack ran the API gateway, the engine, and the license proxy as three containers.

If you are shipping a voice AI product that any regulated customer will use, Docker is not optional. It is the delivery mechanism.

6. Python asyncio

Voice agents are concurrent by nature. Audio streams in, transcripts come out, LLM calls happen, TTS streams back, all simultaneously. Python’s asyncio event loop handles this pattern well.

The pattern I used across several prototypes: one coroutine reads audio chunks from a microphone or file and sends them over a WebSocket. Another coroutine reads messages from the same WebSocket and processes transcripts. A third coroutine manages the LLM call and TTS playback. They all run on the same event loop, coordinated with asyncio queues.

async def stream_audio(ws, audio_source):
    async for chunk in audio_source:
        await ws.send(chunk)

async def process_transcripts(ws):
    async for msg in ws:
        transcript = parse_transcript(msg)
        if transcript.is_final:
            await handle_turn(transcript)

async def main():
    async with websockets.connect(STT_URL) as ws:
        await asyncio.gather(
            stream_audio(ws, mic),
            process_transcripts(ws)
        )

The async pattern is not new, but for voice agents it is the right abstraction. Blocking IO kills the latency budget.

FAQ

What is the difference between Nova-3 and Flux?

Nova-3 handles speech-to-text transcription. Flux handles turn detection. They run together: Nova-3 transcribes each word, Flux decides when the utterance is complete and signals the application to respond. You can use Nova-3 without Flux by implementing your own endpointing, but Flux makes the agent feel more natural.

Why use websocat instead of a library?

websocat lets you test a streaming connection from the terminal in one command. No Python script, no SDK, no build step. It is useful for quick smoke tests before writing any integration code.

What sample rate do STT APIs expect?

16kHz, 16-bit, mono, raw PCM. Most STT APIs accept this format directly. FFmpeg converts any audio file into this format. Phone audio is often 8kHz mulaw and needs upsampling.

Can you build voice agents without Docker?

Yes, for cloud-only deployments you use the hosted API. Docker is needed when customers require on-prem deployment for data residency or compliance reasons. Deepgram’s self-hosted option runs in Docker with NVIDIA GPU passthrough.

Is Python the only language for voice agent backends?

No. Node.js with async/await works well. Go is popular for high-throughput streaming services. Python is common for prototyping because asyncio is expressive and the ML ecosystem runs on it.