How voice AI SDKs handle things REST clients never have to

Whitepaper Wednesday: maintaining SDKs across five languages for a streaming speech API taught me that most of the patterns REST SDKs rely on don’t survive first contact with real-time audio. A streaming voice SDK isn’t a fancier HTTP client. It is a state machine that happens to send and receive audio.

REST SDKs have it easy. You build a client, map endpoints to methods, serialise a request, and deserialise a response. Errors come back as HTTP status codes. Retries are loops and timeouts are simple. The whole thing fits on a whiteboard in about ten minutes.

Voice AI SDKs do not fit on that whiteboard. A streaming speech-to-text session opens a WebSocket, manages a connection that can live for hours, buffers audio in both directions, decides what to do when the network drops, and classifies errors into categories that determine whether the session lives or dies. The HTTP client patterns that every developer learns break immediately.

Here is what makes them different.

Why HTTP client patterns break for real-time audio

The fundamental difference is that HTTP is transactional and WebSocket is continuous. An HTTP request has a start, a middle, and an end. You send a payload, you wait, you get a response, you close the connection. Every SDK written for a REST API inherits that simplicity by default.

A streaming speech session has no natural end. A meeting transcript session runs for an hour. A live captioning pipeline runs all day. A voice assistant session lives as long as the user is talking. The SDK cannot assume the connection will terminate. It has to manage a persistent, stateful connection that can degrade, recover, or fail at any point, and the developer using the SDK should not have to think about any of it.

That changes everything about how the SDK is designed.

The three-state connection model

The first thing you learn building a streaming voice SDK is that a WebSocket is not simply open or closed. It is in one of several states, and the SDK has to know which one at every moment.

The simplest model has three states: connected, degraded, and disconnected.

Connected is what every demo tests. The WebSocket handshake succeeded, audio is flowing, transcriptions are arriving. Everything works. The SDK is happy.

Degraded is the state that production reveals. The network hiccuped. The STT provider is overloaded and sending partial responses slower than expected. The client is on a mobile connection that keeps switching between towers and dropping packets. The SDK is still connected but cannot guarantee smooth delivery.

Disconnected is what happens when degradation does not recover. The connection dropped and the SDK needs to decide whether to reconnect or fail.

REST SDKs do not have a degraded state. An HTTP request either succeeds or fails. There is no in-between. Streaming voice SDKs have to recognise degradation early, buffer audio locally, and attempt recovery before the user notices anything wrong. That means the SDK needs internal health checks, heartbeat monitoring, and a state machine that transitions between these three states without leaking that complexity to the developer.

Audio buffering is not the same as data buffering

Every developer understands buffering at a high level. You queue data when production outpaces consumption, and you drain it when the consumer catches up. Audio buffering in a voice SDK is that pattern with three extra dimensions: latency sensitivity, memory constraints, and temporal ordering.

The latency constraint is the hardest. A standard data buffer can grow to megabytes without anyone noticing. An audio buffer that exceeds a few hundred milliseconds of speech becomes a noticeable delay. The user hears their own voice delayed and starts speaking more slowly, which makes the delay worse. This is the latency spiral, and it is the single most common production failure in voice AI deployments.

The memory constraint follows from the first one. A voice SDK on a mobile device does not have gigabytes of memory to throw at a buffer. A five-second circular buffer for 16kHz mono 16-bit PCM audio is about 160 kilobytes. That is the budget. Exceed it and the SDK starts dropping audio frames, which means the transcription loses words.

The temporal ordering constraint is the one that trips up developers new to voice. Audio chunks cannot be reordered. If chunk 5 arrives before chunk 4 because of a network jitter event, the SDK cannot deliver chunk 5 to the speech model before chunk 4 without corrupting the transcript. The SDK has to hold chunk 5, wait for chunk 4, resequence them, and then deliver them in the correct order. This is not something a REST SDK ever has to think about.

Reconnection with audio replay

A REST SDK that loses its connection retries the request. Simple. A voice SDK that loses its connection during a live session faces a harder problem. The user kept talking during the disconnection. That audio is gone. When the connection restores, the SDK needs to reconstruct what was said in the gap.

The pattern that works is a circular audio buffer that keeps the last N seconds of audio on the client. When the SDK detects a disconnection, it pauses sending audio and starts buffering locally. When the connection restores, it replays the buffered audio starting from the oldest chunk, then resumes live streaming from where the buffer ends. The server sees a brief gap and then a catch-up stream. The transcript picks up where it left off, minus whatever audio exceeded the buffer window.

The buffer window size is the key tradeoff. Too small and the gap in the transcript is noticeable. Too large and the client runs out of memory on constrained devices. Five seconds is the sweet spot for most voice applications. Long enough to cover a typical network interruption. Small enough to fit in 160 kilobytes of memory.

Error classification for streaming sessions

REST SDKs classify errors by HTTP status code. 4xx is client error, 5xx is server error, everything else is success. The retry strategy follows naturally from the status code.

Streaming voice SDKs need a different classification system because the errors are different. A WebSocket close code of 1006 (abnormal closure) means the network dropped the connection mid-session. A code of 1003 (unsupported data) means the SDK sent audio in the wrong format. Both produce a closed connection, but one is recoverable and the other is not.

The useful classification splits errors into three categories: connection errors, data errors, and protocol errors.

Connection errors include close codes 1006 (abnormal closure), 1012 (service restart), and 1013 (try again later). These mean the infrastructure had a hiccup. The SDK should buffer locally, wait a configured backoff interval, and reconnect with audio replay.

Data errors include close codes 1003 (unsupported data), 1007 (invalid frame payload), and 1009 (message too big). These mean the SDK itself made a mistake. Reconnecting will produce the same error. The SDK should surface a clear error to the developer explaining what format or size constraint was violated.

Protocol errors cover everything else. The server sent an unexpected message. The session timed out. The authentication token expired mid-session. These need a mix of retry and fail logic depending on the specific error.

A well-designed streaming SDK routes every connection closure through this classification before deciding what to do. The developer who writes client.transcribe(audioStream) should never see a raw WebSocket close code. They see a ConnectionLostError with a clear message about what happened and what the SDK is doing about it.

Backpressure in two directions

Backpressure in a REST SDK is straightforward. If the server responds too slowly, the client times out and the developer tunes the timeout value. If the client sends requests too quickly, the server rate-limits and the client retries with exponential backoff.

A streaming voice SDK has backpressure in both directions simultaneously. The client sends audio to the server for transcription while the server sends transcripts back to the client. Either direction can stall independently.

The more common problem is server-to-client backpressure. The TTS provider generates audio faster than the network can deliver it to the client. The client’s receive buffer grows until it runs out of memory or the user hears audio that is seconds behind the conversation. The fix is application-layer flow control: the client tells the server how much audio it can buffer, measured in milliseconds of playable audio, and the server pauses sending when the client’s credit runs out.

The less common but more destructive problem is client-to-server backpressure. The client’s network degrades and audio chunks queue up in the send buffer faster than they can be transmitted. The WebSocket’s internal buffer grows until memory is exhausted and the process crashes. The fix here is not flow control but drop policy. When the send buffer exceeds a threshold (typically 64 kilobytes per connection), the SDK starts dropping audio frames. Stuttered audio is better than a crashed process.

Chunk sizing and the latency tradeoff

Every streaming voice SDK has to decide how big each audio chunk should be. REST SDKs never make this decision because they send the entire payload at once.

Smaller chunks mean lower latency. The server receives audio faster and starts transcribing sooner. The first transcript token arrives earlier. But smaller chunks also mean more overhead per chunk and less acoustic context for the speech model, which can reduce accuracy on short utterances.

Larger chunks mean better accuracy and less overhead, but they add latency. The server waits longer to receive enough audio to make an accurate guess at what was said.

The practical range for speech-to-text is 100 to 125 milliseconds of audio per chunk. At 16kHz mono 16-bit PCM, that is about 3,200 to 4,000 bytes. Chunks in this range give the speech model enough context for accurate transcription while keeping time-to-first-token under 300 milliseconds in most network conditions.

Text-to-speech has a different optimal range because the generation is not monotonically aligned with the input. TTS models generate audio in bursts. A chunking strategy that works for speech-to-text can produce uneven latency for speech synthesis. The optimal for TTS is sentence-level or phrase-level chunks, which are variable in size but produce the most natural playback experience.

The SDK should expose these as configuration options but provide sensible defaults. Most developers do not want to think about chunk sizing. They want to pass in audio and get out text. The defaults should work for the common case.

What SDK abstractions should hide

The list of things a streaming voice SDK should abstract from the developer is long. WebSocket connection lifecycle, buffering, reconnection, error classification, chunk sizing, backpressure, audio format negotiation. Every one of these is an implementation detail that the developer should not have to think about to get a working integration.

The list of things a streaming voice SDK should surface to the developer is short.

Connection state changes, exposed as events or callbacks. The developer should know when the session is healthy, degraded, or lost, because that informs their own application logic.

Error events that include a recovery suggestion. Not a raw WebSocket close code, but a message that says “the audio format you sent is not supported. Supported formats are 16kHz mono 16-bit PCM WAV.”

Latency metrics. The developer should be able to observe time-to-first-token and inter-arrival times to know whether their integration is performing within expected bounds.

That is it. Everything else is inside the SDK, hidden behind a simple transcribe or synthesize call that behaves predictably across network conditions, audio sources, and deployment environments.

What this means for API design

If you are designing a voice API, the SDK is not an afterthought. It is the execution environment for every architectural decision you make about your wire protocol. A streaming API with a poorly designed SDK is a bad API regardless of how good the model is.

The best streaming APIs I have worked with share a common pattern. They define a clear, deterministic message protocol over WebSocket. Every message has a type, a sequence number, and a well-documented schema. The SDK maps that protocol to language-native primitives. The developer never constructs a raw WebSocket frame. They call a method and get back a stream of typed events.

The worst streaming APIs I have worked with share a different common pattern. They document the WebSocket endpoint and tell developers to figure out the protocol from a single curl example. The developer writes their own WebSocket client code, guesses at message formats, and produces integrations that break the first time the network jitters.

The protocol is the contract. The SDK is the guarantee that the contract is honoured. Designing the first without the second produces an API that works in a demo and fails in production.

FAQ

Why can’t I use a regular HTTP client for streaming STT?

HTTP is request-response by design. The client sends a request, the server sends a response, and the connection is done. Streaming speech requires a persistent, bidirectional connection where both sides send data concurrently. HTTP/2 and HTTP/3 support server push and multiplexing, but they do not give you the same control over message ordering, backpressure, and keep-alive that WebSocket provides.

How do I choose the right audio chunk size for my application?

Start with 100-millisecond chunks at the audio format your model expects. Measure time-to-first-token and transcription accuracy. If latency is too high, decrease the chunk size. If accuracy suffers on short utterances, increase it. The optimal range for most speech-to-text applications is 100 to 125 milliseconds per chunk.

What should I do when my streaming SDK disconnects mid-session?

First, check whether your SDK exposes connection state events. If it does, listen for the degraded event and decide whether your application can tolerate a few seconds of buffered catch-up. If your SDK does not expose state events, that is a gap in the SDK design, and you should consider a SDK that handles reconnection transparently with an audio replay buffer.

Does WebSocket backpressure work the same for TTS and STT?

No. STT backpressure is primarily about send buffers. The client produces audio faster than the network can deliver it, and the SDK must drop frames to stay within memory limits. TTS backpressure is about receive buffers. The server produces audio faster than the client can consume it, and the SDK must apply credit-based flow control to prevent buffer bloat.

How much memory should a voice AI SDK use on a mobile device?

A well-designed streaming SDK should keep its working set under 256 kilobytes per active connection on a mobile device. That covers a five-second circular audio buffer for reconnection, a small send buffer for backpressure tolerance, and the audio format conversion pipeline. Any more than that and the SDK is either buffering too aggressively or leaking memory.