TIL: Streaming JSON Parsing With ijson for Large Responses

JSON responses from streaming APIs can be gigabytes. Parsing the whole thing with json.load before you can use any of it wastes time and memory. ijson parses incrementally.

import ijson

with open('large_response.json', 'rb') as f:
    for item in ijson.items(f, 'results.item'):
        print(item['transcript'])

ijson yields items as it encounters them in the stream. The prefix ‘results.item’ tells it what path in the JSON tree to emit. For an STT response with hundreds of utterances, you get each transcript as it appears in the file without loading the whole thing.

# Parse from a streaming HTTP response
import requests
response = requests.get(url, stream=True)
for item in ijson.items(response.raw, 'item'):
    process(item)

ijson works with any file-like object. Pass a streaming HTTP response body directly and process results as they arrive.

Does ijson support different JSON formats?

It handles standard JSON, but not JSON Lines. For newline delimited JSON, iterate over lines and json.loads each one.

What prefix syntax does ijson use?

Dot separated paths. results.item matches {results: {item: […]}}. Use item for arrays at the root level.