TTS Quality Has No Single Number Yet

Three weeks ago I moved from Deepgram to Speechify. I spent four years on the STT side where quality is simple. You measure Word Error Rate, you optimise it, and if your WER is low enough the model is good. Simple is not the word for how TTS quality works.

TTS has no equivalent of WER. There is no single number that, when you optimise it, guarantees the output sounds good. The industry has been trying to invent one for decades and it keeps failing, because speech synthesis quality is not one thing. It is voice character, naturalness, prosody, pronunciation accuracy, latency, consistency at scale, and a dozen other dimensions that matter differently depending on what you are building.

I spent the last three weeks reading every benchmark, leaderboard, and evaluation methodology I could find in the TTS world. The honest answer is that none of them work the way developers expect them to, and most people evaluating TTS providers are picking the wrong metric for their use case.

Why MOS died and nobody admitted it

Mean Opinion Score is the oldest method for measuring TTS quality. You gather a group of listeners, play them audio samples, and ask them to rate each one on a scale of 1 to 5. Average the scores and you get a number that supposedly represents how natural the speech sounds.

MOS was designed for telecom codecs in the 1990s, where the question was whether a digital phone line sounded better than an analogue one. The range of variation was small and the listening task was simple. Apply it to modern neural TTS and it breaks in three ways.

First, listeners anchor their scores differently. One person’s 4 is another person’s 3. You can control for this with training sessions and reference samples, but the MOS literature is full of studies where the same model scores 4.2 in one lab and 3.8 in another because of listener population effects.

Second, the scale compresses at the top. Modern TTS models all cluster between 3.8 and 4.5 on a five-point scale. A 0.3 MOS difference between two models might be statistically significant but audibly meaningless. The scale cannot resolve what it is supposed to measure anymore.

Third, MOS measures reaction not perception. A listener hears a short utterance and assigns a number. That number reflects whether the voice sounded pleasant on the first listen, not whether it holds up across a 30-minute conversation or handles domain-specific vocabulary correctly.

Many TTS providers still publish MOS numbers. Some fabricate them from internal tests with no methodological disclosure. Others run legitimate studies that produce numbers so clustered they tell you nothing. If a vendor leads their pitch with a MOS score, treat that the same way you treat a restaurant that brags about its Yelp rating from 2018.

The Elo leaderboard is better but still not a purchasing decision

The Artificial Analysis Speech Arena runs the most transparent quality benchmark in TTS today. It uses pairwise blind comparisons. Listeners hear two samples from different models and pick which one sounds better. The results feed an Elo rating, the same system chess uses. The current leader has Simba 3.2 at 1,234 Elo and Qwen-Audio-3.0-TTS-Plus at 1,236, a statistical tie inside overlapping confidence intervals of about plus or minus 17 points.

Elo fixes the anchoring problem. Listeners compare two samples side by side instead of assigning absolute scores. The ranking is relative, so it survives changes in the listener pool. And because it runs continuously, it captures quality shifts when new models land.

But Elo has a blind spot the size of the use case. The Speech Arena tests English audio on default catalogue voices with short utterances. It does not measure latency, language coverage, voice cloning fidelity, pronunciation of domain-specific terms, or consistency across thousands of requests. A model that tops the Elo leaderboard could still be the wrong choice for a multilingual customer support voice agent or a medical dictation app that needs to say “hydrochlorothiazide” correctly every time.

The Arena is honest about these limits. Its methodology page lists what it measures and what it does not. The problem is that developers treat the leaderboard ranking as a buying signal when it is really a naturalness signal for a narrow slice of usage.

Think of Elo as the TTS equivalent of a GPU benchmark. A high frame rate in a controlled test tells you the card is fast, but it does not tell you whether it fits your case, your power budget, or your cooling setup. The benchmark is a filter, not a decision.

Seven dimensions of TTS quality that matter in production

After three weeks of reading evaluation methodologies and talking to people who deploy TTS at scale, I keep coming back to seven dimensions. Every use case weights them differently, but ignoring any of them creates a blind spot.

Naturalness is the one everyone measures. It is what the Elo leaderboard captures and what MOS was designed for. It matters most for long-form content like audiobooks and narration where the listener has time to notice if something sounds off. For short transactional utterances in a voice agent, naturalness is less important than speed.

Pronunciation accuracy matters for every use case but especially for names, product terms, and domain language. Most TTS providers publish a combined WER that includes both TTS errors and transcription errors. Isolated TTS WER is harder to find. The Coval benchmark reports it for streaming providers, and Gradium currently leads with 3.3 percent WER on a multilingual test set. But this is measured differently from the WER you know from STT, because the reference is a human transcript of the generated audio, not a transcript of ground-truth speech.

Voice character is the subjective quality that separates the models at the top of the leaderboard. When Simba 3.2 and Qwen-Audio-3.0-TTS-Plus sit two Elo points apart with overlapping confidence intervals, the difference is not quality but preference. One voice sounds warmer, another sounds crisper. Average listeners in the Edison study from this week could not tell an AI voice from a human narrator, but they could still prefer one synthetic voice over another. Voice character is a taste call, and it only matters once you have filtered by the other dimensions.

Prosody and expressiveness is where the gap between open-weight models and closed APIs is widest. Open TTS models around 1,118 Elo on the Speech Arena can sound robotic on complex sentences. The top closed models handle emphasis, pacing, and emotional tone more naturally. If your content has questions, exclamations, or quoted speech, evaluate this dimension specifically rather than relying on a single Elo score.

Consistency at scale is the dimension nobody tests in benchmarks but everybody discovers in production. Models that sound great on the first ten requests can drift on the thousandth. Voice cloning that works on a clean studio recording can fail on noisy input. The only way to measure this is to run your production traffic through the API for a week and audit samples.

Language and accent coverage varies wildly. ElevenLabs supports 70 plus languages with v3. Gradium has strong French. Simba 3.2 has competitive English and Mandarin. The leaderboard tests English only. If your users speak Hindi, Thai, or Swahili, the Elo ranking for English tells you close to nothing about your experience.

Latency is not quality the way naturalness is quality, but it determines user perception of quality more than any other single factor in real-time applications. A perfect voice that takes 800 milliseconds to start speaking feels worse than a good voice that starts in 150 milliseconds. The Coval benchmark tracks time-to-first-audio for streaming providers, and the spread is enormous. Gradium runs 155 milliseconds P50 with a tight IQR. Other providers diverge by hundreds of milliseconds at p95.

The production test that matters more than any benchmark

Everything I have read in the last three weeks points to the same conclusion. The only evaluation that matters is the one you run yourself with your own content, your own latency budget, and your own users.

Benchmarks are useful for filtering. If a model falls below a certain Elo threshold it probably will not sound natural enough for your case. If its TTFA exceeds your latency budget you can eliminate it. But the shortlist of candidates after filtering is where the real work starts.

Take your ten hardest sentences. The ones with technical jargon, unusual names, quoted speech, and mixed punctuation. Run them through each candidate provider. Listen to the outputs side by side. Check whether the model emphasises the right words. Check whether it handles the quoted section with a different intonation or reads it flat.

Then run a thousand requests at production volume and audit the tail. Sample the hundredth, the five hundredth, and the thousandth response. Listen for drift, dropped words, and weird prosody. This takes an afternoon and it catches problems no leaderboard can show you.

And treat long-term contracts with suspicion. The TTS market is repricing every quarter. Google cut Flash TTS pricing 70 percent between April and July. Simba 3.2 costs $10 per million characters against rivals at $85 and $100. A three year commitment locks you into today’s pricing against a market that keeps getting cheaper and better. Keep your integration thin enough that switching providers takes a config change, not a rewrite, and requote every quarter.

What I wish I had known three weeks ago

If you are evaluating TTS providers today, stop looking for a single quality score. It does not exist. The leaderboard tells you who sounds best on short English clips. Elo tells you who won the pairwise popularity contest. MOS tells you what a group of listeners thought in a lab last year. None of them tells you whether the voice works for your specific use case.

Filter by naturalness, then evaluate on pronunciation accuracy, voice character, language coverage, consistency at scale, and latency under your own traffic. Pick the one that passes all six and price-compare the survivors.

I spent four years measuring STT quality where one number tells you most of the story. TTS is not that world. The sooner you accept that and build your own evaluation pipeline, the better your shipping product will sound.

FAQ

What is the best way to measure TTS quality in 2026?

Run a blind preference test with your own content, ideally with your target audience. Filter providers by Elo and TTFA first, then compare the shortlist on your ten hardest sentences under production conditions. No single benchmark gives you the full picture, and the provider that scores highest on a generic leaderboard may not be the one that works best for your specific domain, language, or latency budget.

Why can’t TTS use a single metric like STT uses WER?

Speech synthesis quality is multi-dimensional. A model can sound natural on short sentences but fail on complex prosody or domain vocabulary. There is no agreed reference transcript to measure against the way STT has a human transcript. Each dimension (naturalness, pronunciation, voice character, latency) needs its own evaluation method.

What does the Speech Arena Elo rating actually measure?

Pairwise listener preference on short English utterances using default catalogue voices. It is the most transparent quality benchmark in TTS and the closest thing the industry has to a standard comparison. But it excludes latency, multilingual quality, voice cloning fidelity, consistency at scale, and pronunciation of domain-specific terms. Treat it as a naturalness filter, not a purchasing decision.

Which TTS API has the best quality in 2026?

The top of the blind-vote leaderboard is a statistical tie between Simba 3.2 (1,234 Elo) and Qwen-Audio-3.0-TTS-Plus (1,236 Elo) inside overlapping confidence intervals. Quality at the frontier has converged. The deciding factors are now voice character preference, latency profile, language coverage, and price per million characters.

How often should I re-evaluate my TTS provider?

Quarterly as a minimum. The market is repricing faster than any other API category I have seen. Google cut pricing 70 percent in one quarter. New models land on the leaderboard every few weeks and providers ship silent infrastructure changes that shift latency rankings overnight. An evaluation from January is stale by April, and a contract signed on July’s rates locks you into pricing against a market that keeps deflating.

[link: listeners-voted-ai-voices-won for the Edison and Azerion studies on AI voice perception] [link: voice-ai-worth-billions-speech-nearly-free for the market pricing analysis]