SDKs for Streaming APIs Are Different

Throwback Thursday to when I first realised that everything I knew about SDK design was built on an assumption I had never questioned: every API call ends.

REST APIs taught us a pattern that works beautifully for CRUD. You build an HTTP client, wrap it in a function call, handle the response or the error, and move on. The connection opens, data travels, the connection closes, and that is the whole transaction.

Then I started building SDKs for streaming audio APIs.

The connection does not close. It stays open for minutes or hours. Audio arrives in chunks, not responses. Messages flow in both directions simultaneously. And every assumption the REST SDK pattern made about lifecycle, error handling, and resource management turns into a question you have to answer from scratch.

How REST SDKs handle API calls

A REST SDK is simple because HTTP is simple. The client opens a connection, sends a request, waits for the response, and parses it. Error handling is a status code. Retry is a for loop with a sleep.

class RestClient:
    def __init__(self, base_url, api_key):
        self.base_url = base_url
        self.headers = {"Authorization": f"Bearer {api_key}"}

    def transcribe(self, audio_file):
        resp = requests.post(
            f"{self.base_url}/v1/transcribe",
            headers=self.headers,
            files={"audio": audio_file},
        )
        resp.raise_for_status()
        return resp.json()

This works because HTTP gives you a natural transaction boundary. The request starts, the response ends, and between them you have a clean unit of work. If the network drops mid-request, you retry the whole thing. If the server returns a 500, you retry. If the client crashes, no state is lost because the state was in that one request.

None of this survives contact with a streaming API.

What changes when your API streams audio

A streaming audio connection is a persistent WebSocket. Audio chunks flow from client to server, and transcription results flow from server to client, all on the same connection. The connection can last for the entire duration of a conversation, which could be hours.

This changes everything about the SDK design.

No natural transaction boundary. Audio is continuous. You cannot retry “the last second” of audio the way you retry an HTTP request. The server has already received and processed some of it. The client has already sent it and moved on. Reconnection is not about replaying a request, it is about resuming from a known state.

Two-way data flow. The SDK is not just sending data, it is receiving data at the same time. Messages arrive asynchronously. The SDK needs to dispatch them to the right handler without blocking the send path.

The connection is stateful. The server holds session state, audio context, and model state on your behalf. If the connection drops, that state may be lost. The SDK needs to decide whether to reconnect, resend context, or start fresh.

No natural error surface. A dropped WebSocket is not a 4xx or 5xx. It is a close frame with a code, or a timeout, or a reset. The SDK needs to classify the failure and decide whether to reconnect, report an error, or silently recover.

Pattern 1: Connection lifecycle as a first-class concept

In a REST SDK, the connection lifecycle is invisible. The HTTP client handles it. You never think about opening or closing connections.

In a streaming SDK, the connection lifecycle is the most important thing the SDK manages. It needs a connect method, a disconnect method, and internal state tracking for every state the connection can be in.

class StreamClient:
    def __init__(self, api_key):
        self.api_key = api_key
        self.ws = None
        self.connected = False
        self.closing = False

    async def connect(self, url):
        self.ws = await websockets.connect(
            url,
            extra_headers={"Authorization": f"Token {self.api_key}"},
        )
        self.connected = True
        self.closing = False

    async def disconnect(self):
        self.closing = True
        if self.ws:
            await self.ws.close()
            self.ws = None
            self.connected = False

The state machine is not optional. The SDK has to know whether it is connecting, connected, reconnecting, or closed because every method on the SDK behaves differently depending on the state. Send audio during reconnection? Buffer it. Send audio while closing? Drop it.

Pattern 2: Reconnection with exponential backoff and jitter

The next thing you learn is that WebSocket connections drop. Not if. When. The server restarts, the network blips, the load balancer cycles. The SDK has to reconnect, and it has to do it in a way that does not make things worse.

Exponential backoff with jitter is the standard approach. Start with a small delay, double it each time, cap it, and add random jitter so reconnecting clients spread out instead of hitting in waves.

async def connect_with_backoff(self, url):
    delay = 1
    max_delay = 30
    while not self.closing:
        try:
            await self.connect(url)
            return
        except ConnectionError:
            jitter = random.uniform(-delay / 2, delay / 2)
            await asyncio.sleep(delay + jitter)
            delay = min(delay * 2, max_delay)

The REST equivalent of this is a retry loop around requests.post(). But the streaming version is different in a critical way: while the SDK is reconnecting, audio is still arriving from the microphone. The SDK needs to buffer that audio, reconnect, and then send the buffered audio without the caller noticing a gap.

That requirement does not exist in REST SDKs.

Pattern 3: Backpressure from the streaming end

REST SDKs do not think about backpressure. The client sends a request and waits. The server either responds or does not. There is no scenario where the server sends data faster than the client can consume it because the server sends exactly as much as the client asked for.

Streaming is different. Audio can arrive faster than the client can process it, or the network can be slower than the audio source. The SDK has to handle both directions.

class BoundedSender:
    def __init__(self, max_buffer=65536):
        self.max_buffer = max_buffer
        self.buffer = b""

    async def send(self, ws, chunk):
        self.buffer += chunk
        if len(self.buffer) > self.max_buffer:
            self.buffer = self.buffer[-self.max_buffer:]
            await asyncio.sleep(0.05)
        await ws.send(self.buffer)
        self.buffer = b""

The key insight is that backpressure operates differently for STT and TTS. For speech-to-text, dropping a few milliseconds of audio is acceptable. The model catches up. For text-to-speech, dropping a chunk creates an audible glitch that the listener notices. The SDK needs to know which direction it is handling and adjust its backpressure strategy accordingly.

REST SDKs never have to make this distinction.

Pattern 4: Graceful shutdown with drain and finalise

A REST SDK shuts down by closing the HTTP session. There is nothing to finalise. The server does not care that the client disconnected because the transaction was already complete.

A streaming connection needs an orderly shutdown. The client tells the server the stream is done, waits for the server to process the final audio and return any remaining results, and only then closes the WebSocket.

async def graceful_shutdown(self, timeout=5):
    await self.ws.send({"type": "EndOfStream"})
    try:
        async with asyncio.timeout(timeout):
            async for msg in self.ws:
                self._handle_result(msg)
    except asyncio.TimeoutError:
        pass
    finally:
        await self.ws.close()
        self.connected = False

If the SDK just closes the WebSocket without the drain phase, the server might be mid-transcription when the connection drops. The transcript cuts off. The user sees half a sentence and assumes the system is broken. The drain phase prevents that.

What this means for SDK API design

The differences have real consequences for how the SDK surface looks.

A REST SDK exports functions or methods that map to API endpoints. The user calls client.transcribe(audio) and gets a result back. Simple.

A streaming SDK exports a stateful object that the user instantiates, connects, and then interacts with through events and callbacks. The user calls client.connect(), registers event handlers, streams audio, and handles results as they arrive. It is a fundamentally different programming model.

# REST style
client = RestClient(api_key)
result = client.transcribe("audio.wav")
print(result["transcript"])

# Streaming style
client = StreamClient(api_key)
client.on("transcript", lambda t: print(t))
await client.connect()
await client.stream_mic()  # runs until interrupted
await client.disconnect()

The streaming version requires the user to think about connection state, event handling, and lifecycle management. The REST version does not. A good streaming SDK hides as much of that complexity as possible, but some of it is inherent. The connection is stateful. The user has to deal with that.

FAQ

When do I need a streaming SDK instead of a REST client?

If your use case involves real-time audio, live transcription, or bidirectional communication, you need a streaming SDK. If you are batch-processing pre-recorded files, a REST client is fine. The dividing line is whether the connection outlives a single request-response cycle.

Can I build streaming patterns on top of a REST client?

Technically yes, but you will be fighting the abstraction. Polling an endpoint every 100 milliseconds is not streaming, it is polling with extra steps. A WebSocket connection built on a persistent TCP socket is fundamentally different from repeated HTTP requests, and the SDK should reflect that difference.

What is the hardest part of shipping a streaming SDK?

Connection state management. REST SDKs have one state (ready). Streaming SDKs have connecting, connected, reconnecting, draining, and closed. Every method on the SDK has different behaviour in each state. Getting the state machine wrong causes bugs that are hard to reproduce because they depend on network timing.

Should I use a streaming SDK in a serverless function?

Probably not. Serverless runtimes have short timeouts and cold starts that do not play well with persistent connections. A REST client or a job-based pattern is better for serverless. Streaming SDKs belong in long-running processes like desktop apps, mobile clients, and backend services.

Do all streaming APIs use WebSockets?

Most use WebSocket or WebRTC. Some use HTTP/2 server-sent events for one-way streaming. The SDK patterns apply regardless of the underlying transport. The need for connection lifecycle management, reconnection, backpressure, and graceful shutdown is the same whether the transport is WebSocket, SSE, or a custom TCP protocol.