---
title: Six tools that power production voice agents
description: Building a production voice agent takes more than an API key. Here are six tools I reached for daily at Deepgram, from streaming STT to async Python.
canonical: https://lukeocodes.dev/six-tools-production-voice-agents
type: article
locale: en_GB
published: 2026-06-06T00:00:00.000Z
modified: 2026-11-02T00:00:00.000Z
authors:
  - Luke Oliff
section: Developer Experience
tags:
  - Developer Experience
  - Engineering
  - AI
  - API
feed: https://lukeocodes.dev/rss.xml
search: https://lukeocodes.dev/search.json?q={query}
schema:
  - "@context": https://schema.org
    "@graph":
      - "@type": Organization
        "@id": https://ably.com/#organization
        name: Ably
        url: https://ably.com
      - "@type": WebSite
        "@id": https://lukeocodes.dev/#website
        url: https://lukeocodes.dev
        name: Luke Oliff
        alternateName: lukeocodes
        inLanguage: en
        publisher:
          "@id": https://lukeocodes.dev/authors/luke#person
        description: Personal blog and writing by Luke Oliff — developer relations, voice AI, developer experience.
        potentialAction:
          - "@type": SearchAction
            target:
              "@type": EntryPoint
              urlTemplate: https://lukeocodes.dev/blog/search?q={search_term_string}
            query-input: required name=search_term_string
      - "@type": Person
        "@id": https://lukeocodes.dev/authors/luke#person
        name: Luke Oliff
        url: https://lukeocodes.dev/authors/luke
        description: Staff engineer and Staff Tech Lead for Developer Experience at Ably. Writing about developer relations, AI, and the voice industry, from the UK.
        mainEntityOfPage: https://lukeocodes.dev/authors/luke
        givenName: Luke
        familyName: Oliff
        image: https://lukeocodes.dev/authors/luke.jpg
        jobTitle: Staff Tech Lead, Developer Experience
        worksFor:
          "@id": https://ably.com/#organization
        sameAs:
          - https://www.linkedin.com/in/lukeocodes
          - https://github.com/lukeocodes
          - https://bsky.app/profile/lukeocodes.dev
          - https://x.com/lukeocodes
          - https://hashnode.com/@lukeocodes
          - https://dev.to/lukeocodes
        knowsAbout:
          - Developer Relations
          - Developer Experience
          - Voice AI
          - Text-to-Speech
          - Answer Engine Optimisation
          - Search Engine Optimisation
          - Open Source
          - Developer Tooling
          - API Design
          - Speech-to-Text
          - Voice Agents
          - Community Building
        knowsLanguage:
          - English
        nationality: British
        alumniOf:
          - "@type": Organization
            name: The Open University
  - "@context": https://schema.org
    "@graph":
      - "@type": Organization
        "@id": https://ably.com/#organization
        name: Ably
        url: https://ably.com
      - "@type": Person
        "@id": https://lukeocodes.dev/authors/luke#person
        name: Luke Oliff
        url: https://lukeocodes.dev/authors/luke
        description: Staff engineer and Staff Tech Lead for Developer Experience at Ably. Writing about developer relations, AI, and the voice industry, from the UK.
        mainEntityOfPage: https://lukeocodes.dev/authors/luke
        givenName: Luke
        familyName: Oliff
        image: https://lukeocodes.dev/authors/luke.jpg
        jobTitle: Staff Tech Lead, Developer Experience
        worksFor:
          "@id": https://ably.com/#organization
        sameAs:
          - https://www.linkedin.com/in/lukeocodes
          - https://github.com/lukeocodes
          - https://bsky.app/profile/lukeocodes.dev
          - https://x.com/lukeocodes
          - https://hashnode.com/@lukeocodes
          - https://dev.to/lukeocodes
        knowsAbout:
          - Developer Relations
          - Developer Experience
          - Voice AI
          - Text-to-Speech
          - Answer Engine Optimisation
          - Search Engine Optimisation
          - Open Source
          - Developer Tooling
          - API Design
          - Speech-to-Text
          - Voice Agents
          - Community Building
        knowsLanguage:
          - English
        nationality: British
        alumniOf:
          - "@type": Organization
            name: The Open University
      - "@type": BlogPosting
        "@id": https://lukeocodes.dev/six-tools-production-voice-agents#article
        headline: Six tools that power production voice agents
        alternativeHeadline: Six tools that power production voice agents
        description: Building a production voice agent takes more than an API key. Here are six tools I reached for daily at Deepgram, from streaming STT to async Python.
        datePublished: 2026-06-06T00:00:00.000Z
        dateModified: 2026-11-02T00:00:00.000Z
        url: https://lukeocodes.dev/six-tools-production-voice-agents
        mainEntityOfPage: https://lukeocodes.dev/six-tools-production-voice-agents
        inLanguage: en
        author:
          - "@id": https://lukeocodes.dev/authors/luke#person
        isPartOf:
          "@id": https://lukeocodes.dev/#website
        keywords: Developer Experience, Engineering, AI, API
        articleSection: Developer Experience
        about:
          - "@type": Thing
            name: Developer Experience
        abstract: I spent my last few months at Deepgram building real-time voice agent demos and SDK examples. The part that surprised me most was not the models. It was how many supporting tools you need to make a voice agent reliable in production. An API key gets you a transcript. It does not get you a product.
        speakable:
          "@type": SpeakableSpecification
          cssSelector:
            - .post-tldr
            - h1
        image: https://lukeocodes.dev/og.png?h=Six+tools+that+power+production+voice+agents&d=Building+a+production+voice+agent+takes+more+than+an+API+key.+Here+are+six+tools+I+reached+for+daily+at+Deepgram%2C+from+streaming+STT+to+async+Python.&e=Writing
        wordCount: 1026
        timeRequired: PT6M
        publisher:
          "@id": https://ably.com/#organization
      - "@type": BreadcrumbList
        itemListElement:
          - "@type": ListItem
            position: 1
            name: Home
            item: https://lukeocodes.dev
          - "@type": ListItem
            position: 2
            name: Writing
            item: https://lukeocodes.dev/blog
          - "@type": ListItem
            position: 3
            name: Six tools that power production voice agents
            item: https://lukeocodes.dev/six-tools-production-voice-agents
---

[Luke Oliff.](/)

- [Blog](/blog)
- [CV](/cv)
- [Uses](/uses)
- [User manual](/user-manual)
- [Living with ADHD](/living-and-working-with-adhd)
- [RSS](/rss.xml)

[Luke Oliff · Writing](/blog)

# Six tools that power production voice agents

Jun 2026 · Updated Nov 2026 · [Developer Experience](/blog/category/developer-experience) · 6 min read · [Luke Oliff](/authors/luke)

I spent my last few months at Deepgram building real-time voice agent demos and SDK examples. The part that surprised me most was not the models. It was how many supporting tools you need to make a voice agent reliable in production. An API key gets you a transcript. It does not get you a product.

Here are six tools that shaped how I build voice agents, ordered from most foundational to most surprising.

## 1. Deepgram Nova-3

Nova-3 is the streaming STT model that runs beneath most of the voice agents I worked on. Sub-300ms streaming latency, keyterm prompting for domain vocabulary, and native code-switching across 50+ languages.

The model itself is good. What matters more is that it streams. For voice agents, batch transcription is useless. You need words as they are spoken, not a transcript delivered after the call ends. Nova-3 sends interim results over a WebSocket connection and lets you decide when a turn is complete. Or you pair it with Flux, which makes that decision for you.

The keyterm prompting feature deserves more attention than it gets. You pass a list of domain terms at connection time and the model biases toward them. Product names, acronyms, medical terms, anything the generic model would mangle. No retraining needed.

## 2. Deepgram Flux

Flux is the turn detection layer that pairs with Nova-3. It is a separate model that runs alongside transcription and emits structured events: StartOfTurn, Update, and EndOfTurn.

Before Flux, voice agents used endpointing, which is basically a silence timer. If the user pauses for 400ms, assume they are done. But people pause mid-sentence all the time. They pause to think. They pause because the dog barked. An endpointing-based agent cuts them off or sits in awkward silence.

Flux models turn-taking directly. The EndOfTurn event fires when the model decides the utterance is complete, not when a timer expires. The EagerEndOfTurn variant can fire 150 to 250ms earlier than a silence-based approach, which makes voice agents feel responsive rather than impatient.

## 3. WebSocket debugging tools

You cannot test a streaming voice agent with curl. You need WebSocket tooling, and the standard HTTP tools do not help.

websocat became my default. It pipes audio files into a streaming STT connection and prints the transcript to stdout. The workflow looks like this:

```bash
websocat -t "wss://api.deepgram.com/v1/listen?model=nova-3&interim_results=true" \
  -H "Authorization: Token $DG_API_KEY" \
  - raw < audio.raw
```

For interactive testing I used a small Python script with the `websockets` library. It reads microphone input, sends chunks to the API, and prints interim results. That let me test latency, turn detection, and accuracy in real time without building a UI.

If you work with voice agents and do not have a WebSocket debug harness, build one. It saves more time than any other tool on this list.

## 4. FFmpeg

Every audio file arrives in a different format: 48kHz stereo WAV, 16kHz mono FLAC, 8kHz mulaw from a phone system, or Opus from a browser. WebRTC outputs in yet another encoding.

FFmpeg normalises all of it. The pipeline I used most was:

```bash
ffmpeg -i input.mp3 -acodec pcm_s16le -ar 16000 -ac 1 output.raw
```

16kHz, 16-bit, mono, raw PCM. That is the format every STT API expects, and FFmpeg gets you there in one command. For batch testing, a simple shell loop processes an entire directory.

## 5. Docker and NVIDIA tooling

Deepgram runs self-hosted for regulated industries. Healthcare, finance, government. The on-prem deployment uses Docker containers with NVIDIA GPU passthrough, and testing it locally means running the same containers on your own hardware.

The tooling that mattered: `nvidia-smi` for GPU monitoring, `docker compose` for multi-service orchestration, and the NVIDIA Container Toolkit (`nvidia-container-runtime`) for GPU passthrough. A typical test stack ran the API gateway, the engine, and the license proxy as three containers.

If you are shipping a voice AI product that any regulated customer will use, Docker is not optional. It is the delivery mechanism.

## 6. Python asyncio

Voice agents are concurrent by nature. Audio streams in, transcripts come out, LLM calls happen, TTS streams back, all simultaneously. Python's asyncio event loop handles this pattern well.

The pattern I used across several prototypes: one coroutine reads audio chunks from a microphone or file and sends them over a WebSocket. Another coroutine reads messages from the same WebSocket and processes transcripts. A third coroutine manages the LLM call and TTS playback. They all run on the same event loop, coordinated with asyncio queues.

```python
async def stream_audio(ws, audio_source):
    async for chunk in audio_source:
        await ws.send(chunk)

async def process_transcripts(ws):
    async for msg in ws:
        transcript = parse_transcript(msg)
        if transcript.is_final:
            await handle_turn(transcript)

async def main():
    async with websockets.connect(STT_URL) as ws:
        await asyncio.gather(
            stream_audio(ws, mic),
            process_transcripts(ws)
        )
```

The async pattern is not new, but for voice agents it is the right abstraction. Blocking IO kills the latency budget.

## FAQ

### What is the difference between Nova-3 and Flux?

Nova-3 handles speech-to-text transcription. Flux handles turn detection. They run together: Nova-3 transcribes each word, Flux decides when the utterance is complete and signals the application to respond. You can use Nova-3 without Flux by implementing your own endpointing, but Flux makes the agent feel more natural.

### Why use websocat instead of a library?

websocat lets you test a streaming connection from the terminal in one command. No Python script, no SDK, no build step. It is useful for quick smoke tests before writing any integration code.

### What sample rate do STT APIs expect?

16kHz, 16-bit, mono, raw PCM. Most STT APIs accept this format directly. FFmpeg converts any audio file into this format. Phone audio is often 8kHz mulaw and needs upsampling.

### Can you build voice agents without Docker?

Yes, for cloud-only deployments you use the hosted API. Docker is needed when customers require on-prem deployment for data residency or compliance reasons. Deepgram's self-hosted option runs in Docker with NVIDIA GPU passthrough.

### Is Python the only language for voice agent backends?

No. Node.js with async/await works well. Go is popular for high-throughput streaming services. Python is common for prototyping because asyncio is expressive and the ML ecosystem runs on it.

[← Newer: Sunday roundup: afterthoughts on productivity and pace](/sunday-roundup-afterthoughts-on-productivity-and-pace)

[All posts](/blog) →

## Search this site

`GET https://lukeocodes.dev/search.json?q=<words>` returns matching posts as JSON, each with its page URL and its markdown URL. Add `&limit=<1-50>` for more than 10. How it works: [/blog/search](https://lukeocodes.dev/blog/search.md).

```sh
curl "https://lukeocodes.dev/search.json?q=developer+relations"
```

---

© 2026 Luke Oliff. Posts live at the root of this site.

- [GitHub](https://github.com/lukeocodes)
- [LinkedIn](https://www.linkedin.com/in/lukeocodes)
- [X](https://x.com/lukeocodes)
- [Bluesky](https://bsky.app/profile/lukeocodes.dev)
- [Email](mailto:luke@lukeoliff.com)
- [RSS](/rss.xml)

## Cookies

This site sets one analytics cookie, from Google Analytics, and only in a browser that accepts the cookie banner. A client that does not run scripts sets no cookies, so there is nothing to accept or decline here.

[Cookie policy](/cookies)
