TIL: Read speech marks from the Speechify API response

TIL Tuesday: the Speechify API returns word-level timing in every speech response, and I almost missed it. The first time I called POST /v1/audio/speech I went straight for the audio data and ignored everything else. Turns out the response carries a speech_marks object with a chunks array containing start and end times for every word.

{
  "audio_data": "<base64-encoded-audio>",
  "audio_format": "mp3",
  "speech_marks": {
    "start": 0,
    "end": 28,
    "start_time": 0,
    "end_time": 1850,
    "value": "Here is the speech marks example.",
    "chunks": [
      { "start": 0, "end": 4, "start_time": 125, "end_time": 375, "value": "Here" },
      { "start": 5, "end": 7, "start_time": 375, "end_time": 500, "value": "is" },
      { "start": 8, "end": 11, "start_time": 500, "end_time": 750, "value": "the" },
      { "start": 12, "end": 18, "start_time": 750, "end_time": 1250, "value": "speech" },
      { "start": 19, "end": 24, "start_time": 1250, "end_time": 1600, "value": "marks" },
      { "start": 25, "end": 32, "start_time": 1600, "end_time": 1850, "value": "example." }
    ]
  }
}

Each entry has five fields. value is the spoken text. start_time and end_time are times in milliseconds from the beginning of the audio. start and end are character indices into the original input text. The outer speech_marks object wraps the full sentence, and the chunks array has per-word entries.

The first word does not always start at start_time 0. There is often a few milliseconds of silence before speech starts, so the first chunk’s start_time is typically offset from zero.

Why speech marks matter

Word timing unlocks things raw audio alone cannot do. Subtitles synced to spoken audio, karaoke-style highlighting where each word lights up as it is spoken, and cursor-based navigation in long audio where you jump to the exact second a specific word starts.

For voice agents, speech marks let you render a transcript with live word highlighting while the agent is still speaking. The client gets the full response at once and the timing data means you align the visible text with the audible speech without guessing.

How to read them

The response from POST /v1/audio/speech includes speech_marks automatically. No extra parameter needed. The official SDKs expose it as .speechMarks or .speech_marks. If you call the API directly with curl, decode the JSON response and read speech_marks.chunks.

curl -s -X POST "https://api.speechify.ai/v1/audio/speech" \
  -H "Authorization: Bearer $SPEECHIFY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "input": "This is the Speechify text to speech API.",
    "voice_id": "george",
    "model": "simba-english",
    "audio_format": "mp3"
  }' | jq '.speech_marks.chunks'

That pipe to jq .speech_marks.chunks filters out everything but the word timing. Handy for debugging or prototyping.

What speech marks are not

They are not phoneme-level timestamps. Each entry is a full word, not individual sounds within a word. They are also not guaranteed to cover every word in the input. Punctuation, pauses inserted by SSML tags, and silence between sentences may not all appear as entries. If you need per-character timing or silence detection, that is a separate concern.

The response also carries character index gaps between words. The start and end values of adjacent words may not connect cleanly. When looking for a word at a specific character index, check if start is greater than or equal to your target rather than checking if the index falls within both bounds.

For everything else, speech marks are free and they come with every response. I wish I had spotted them sooner.

FAQ

Do speech marks work with streaming responses?

Not in June 2026 for the basic streaming endpoint. The batch POST /v1/audio/speech always returns speech marks, but the streaming POST /v1/audio/stream at this point returns plain audio. A dedicated endpoint for streamed speech marks shipped later. For now, the batch endpoint is the reliable path for word timing.

How accurate are the word timings?

Accurate enough for subtitle syncing and word highlighting, within a few milliseconds of the actual audio. The timing comes from the model’s duration prediction rather than post-hoc alignment, so it matches the generated audio by construction.

Do speech marks work with SSML input?

Yes. SSML tags do not break speech marks, but inserted pauses like <break time="500ms"/> create gaps in the timing sequence. The entries still cover the spoken words. The gap between one word’s end_time and the next word’s start_time reflects the SSML pause.

Can I get character-level instead of word-level timing?

The Speechify API returns word-level timing. For per-character alignment you would need an external forced aligner such as Montreal Forced Aligner or WhisperX after the fact.

Do cloned voices return speech marks?

Yes. Speech marks work the same way for catalog voices and cloned voices. The model predicts timing regardless of which voice ID you use.