What nobody tells you about audio in speech-to-text

Speech-to-text APIs take raw audio and return text. What happens between those two events involves a pipeline most developers never see. Sample rate conversion, channel mixing, normalization, encoding negotiation, chunk timing. The docs list the parameters but they don’t tell you which ones will silently break your results.

I spent years at Deepgram helping developers debug their STT integrations. The most common thread across hundreds of support tickets was not “the model is wrong.” It was “the audio is wrong in a way nobody warned me about.” The model was almost always fine. The audio pipeline, less so.

This is the pipeline the docs skip.

Why sample rate mismatches are the most common silent failure

Someone connects their audio source, sends 44.1kHz audio to a streaming endpoint configured for 16kHz, and gets back a transcript that looks like a modem fell in a river. The API returns no error. The connection succeeds. The transcript is garbage.

Sample rate is the single most common support issue I saw. The API expects 16kHz mono PCM linear16 by default. Most consumer microphones and audio files are 44.1kHz or 48kHz. The API does not resample for you. If you send 44.1kHz audio to a 16kHz endpoint, the model interprets every sample as a 16kHz signal. Speech gets compressed. Frequencies shift. The transcript becomes nonsense.

The fix is simple but not obvious the first time. Convert to the expected sample rate before you send anything.

import soundfile as sf

audio, sample_rate = sf.read("input.wav")
if sample_rate != 16000:
    from scipy import signal
    audio = signal.resample(
        audio, int(len(audio) * 16000 / sample_rate)
    )

If you are sending containerized audio like WAV or WebM, the API can read the sample rate from the header. You omit the sample_rate parameter and let the server handle it. But if you are sending raw PCM chunks over a WebSocket, you must match the sample rate in the URL parameter exactly. A mismatch produces zero errors and zero usable transcripts.

Encoding: the parameter that breaks without an error message

Every streaming STT API I have used requires you to declare your audio encoding upfront. The parameter is required in the URL or connection config. Get it wrong and the connection works but the transcription is noise.

Deepgram supported linear16, linear32, mulaw, alaw, opus, and ogg-opus for raw streaming. The default in most SDKs was linear16. That meant you had to send 16-bit signed little-endian PCM samples. If you sent 32-bit float audio and declared linear16, the API decoded each four-byte sample as a two-byte sample. Half your data got interpreted as audio. The rest became garbage or silence.

I once spent an afternoon with a developer who had a working curl command but a broken Python script. They were sending the same audio file to both. The curl command passed. The Python script produced nonsense. The difference was that curl was sending the raw WAV file as a binary payload while the Python script was reading the PCM samples into numpy arrays and re-saving them without preserving the bit depth. The curl command worked because the file was WAV-containerized. The Python script streamed raw arrays with the wrong encoding declaration.

The lesson: know whether you are sending containerized or raw audio. Pick one and configure the encoding parameter to match. If you use the SDK, it handles this for you. If you build your own WebSocket client, you own every byte.

Chunk size and timing: the invisible latency drivers

Streaming STT works by sending audio in small chunks over a persistent WebSocket connection. The size and timing of those chunks directly affect latency and accuracy.

Send chunks that are too small, like 100 bytes every 10 milliseconds, and you create more overhead than the model can process efficiently. Send chunks that are too large, like 100 kilobytes every 5 seconds, and you introduce visible latency because the model has to wait for the full chunk before it can return results.

The sweet spot for most STT APIs was around 100 to 200 milliseconds of audio per chunk. At 16kHz with linear16 encoding, 16000 samples per second times 2 bytes per sample equals 32000 bytes per second. A 100 millisecond chunk is about 3200 bytes. A 200 millisecond chunk is about 6400 bytes.

Most APIs recommended 4096 bytes per chunk. That mapped to roughly 128 milliseconds of audio at 16kHz. It balanced network efficiency with real-time responsiveness.

import asyncio
import websockets

async def stream_audio(uri, audio_generator):
    async with websockets.connect(uri) as ws:
        for chunk in audio_generator:
            await ws.send(chunk)
            # give the server time to process
            await asyncio.sleep(0.05)

The sleep after each send is important. If you send chunks as fast as you can read them from disk, you overwhelm the server’s input buffer and increase end-to-end latency. Space them at roughly real-time intervals. Read a chunk. Send it. Wait 50 to 100 milliseconds. Repeat.

Channel confusion: when stereo becomes gibberish

A developer in the Deepgram Discord once opened a ticket. Their audio file transcribed perfectly in the Deepgram dashboard but returned garbage through the API. They had checked sample rate. They had checked encoding. Both matched.

The file was stereo. The dashboard detected and downmixed automatically. The raw API did not.

Most STT models expect mono audio. Two channels of 16-bit PCM interleaved looks like double the sample rate to a model expecting mono. Every other sample belongs to the other channel. The model tries to transcribe a signal that alternates between two voices. The result is a mess.

Downmixing is straightforward:

import numpy as np

def to_mono(audio, channels):
    if channels == 1:
        return audio
    frames = audio.reshape(-1, channels)
    return np.mean(frames, axis=1).astype(audio.dtype)

The mistake people made was assuming the API would handle this. Some providers did. Most did not. If your audio source might output stereo, always downmix before you stream.

Normalization: why quiet audio produces empty transcripts

Normalization is the step everyone forgets. You record a conversation in a quiet room. The audio levels are low. The STT model processes the signal and returns empty results or partial words.

Audio normalization adjusts the amplitude so the signal uses the full dynamic range. Without it, quiet speech sits below the model’s detection threshold. The model hears silence interrupted by fragments.

def normalize(audio):
    max_val = np.abs(audio).max()
    if max_val > 0:
        return (audio / max_val * 0.95).astype(audio.dtype)
    return audio

The 0.95 multiplier gives headroom to avoid clipping. Normalize after you resample and before you encode. The order matters. Resampling changes the signal amplitude slightly. Normalize after resample, not before.

The production checklist the docs never give you

Every time I helped a developer debug a production STT integration, we walked through the same checklist. In order:

1. Sample rate: matches the API parameter (16kHz for most, confirm yours).
2. Encoding: matches what you are actually sending (linear16, mulaw, etc).
3. Channels: mono. Downmix stereo before streaming.
4. Normalization: signal uses the full dynamic range.
5. Chunk size: 4096 bytes or 80 to 200ms of audio per chunk.
6. Chunk timing: spaced at real-time intervals, not burst.
7. Container vs raw: pick one. Containerized audio detects parameters from headers. Raw audio requires explicit parameters.

The order is intentional. If sample rate is wrong, nothing else matters. If encoding is wrong, nothing matters. Fix from the top down.

I wrote this list on a sticky note and kept it on my monitor at Deepgram. Every support ticket I could not immediately diagnose started with a walk through these seven items. It solved the problem about 80 percent of the time. The other 20 percent was genuinely interesting model behavior. But that is another post.

FAQ

Why does my STT connection succeed but return garbage?

The most common cause is a sample rate or encoding mismatch. The API does not validate your audio parameters against the actual audio you send. If you declare linear16 but send mulaw, or declare 16kHz but send 44.1kHz audio, the connection works because the server trusts your parameters. The resulting transcript will be nonsense with no error. Check your audio preprocessing before you check your API key.

Do I need to resample every audio file before sending it?

Only if you are sending raw PCM audio and your source sample rate does not match the API parameter. If you are sending containerized audio like WAV or WebM, the server reads the sample rate from the file header and you can omit the parameter. For raw streaming, match the parameter to your audio or convert to 16kHz.

What happens if I send stereo audio to a mono STT model?

The model processes interleaved stereo channels as if they were a single mono signal. Every other sample belongs to the opposite channel. The transcript alternates between the two audio sources at sample rate speed. The result is unusable. Downmix to mono before streaming.

How does chunk size affect STT latency?

Smaller chunks reduce the time until the first transcript arrives but increase overhead per chunk. Larger chunks reduce overhead but delay results because the model has to buffer the full chunk before processing. The most widely recommended chunk size is 4096 bytes at 16kHz, which gives roughly 128ms of audio per chunk.

Do STT SDKs handle audio preprocessing automatically?

Most SDKs handle encoding and connection management but do not resample or downmix your audio. They assume the audio you give them is already in the correct format. The Python and Node SDKs for Deepgram, for example, will stream whatever bytes you give them with the parameters you specify. Audio preprocessing is your responsibility.