Why do speech-to-text accuracy claims disagree?

Speech-to-text accuracy claims disagree because they are rarely measured the same way. One test uses read sentences in a quiet room, another uses a phone call, and a third uses a clip the model may already have seen during training. Each can be honest and still produce a different number. The fix is not to pick the lowest figure. It is to check whether two numbers share the same audio, the same reference, the same scoring rules and a test set nobody tuned on.

The same model can produce two very different numbers

Parakeet TDT 0.6B v2 had an average word error rate of 6.05 percent ¹ on the English short-form track of the Open ASR Leaderboard. In a 2025 test by 9to5Mac, one 7-minute-31-second episode gave the Parakeet v2 model a word error rate of 12.3 percent ². The two numbers come from different audio, a different reference and a different scoring rule, so neither one is wrong. They are not the same measurement.

The 9to5Mac test was a single episode, which is a small sample. A different episode could reorder the results. Read a single-episode test as a data point, not a ranking.

Four things change the number

The audio. Harder audio raises the error rate, and distance from the microphone matters too. In a 2017 study, a recognizer reached 12.0 percent word error rate ³ on real recordings made with a single distant ceiling microphone in an apartment. A test made in a studio cannot predict the result in a living room. For the metric itself, see word error rate.

The reference and the normalization. For its normalized scores, the Open ASR Leaderboard removes punctuation, casing and disfluencies such as fillers before it counts errors ⁴. A vendor that scores raw text with punctuation and fillers will get a different number from the same output. Our guide to word error rate explains how the arithmetic changes with the rules.

The version and the date. Models change between releases, and a leaderboard row may describe an older version than the one on sale. Check the version named in the claim.

The test set itself. A public test set can leak into the data used to train a model. A 2025 study found that a substantial amount of the LibriSpeech and Common Voice evaluation sets appear in public large language model pretraining corpora ⁵. In controlled experiments, recognizers built on contaminated language models showed only subtle differences in error rate, but assigned significantly higher probability to transcriptions they had seen during training ⁶. Those were controlled experiments, not the commercial systems on the leaderboards. A low score on a leaked test is not evidence of skill.

Leakage can make a result look like progress

Contamination is not only a theory. A 2026 preprint on Swiss German argues that published results are inflated by it. Its authors report that a Whisper model trained on the test set, with no Swiss German data, reached 13.88 percent ⁷. That is one preprint on one benchmark, but it shows how a score can improve without any real gain in the language. Appen, a data vendor, defines "benchmaxxing" as optimizing models to score well on public test sets without equivalent gains in real-world performance ⁸.

Our guide to speech recognition benchmarks explains how the leaderboards try to limit this, including private test sets. The speech recognition overview covers the wider subject.

What to ask before you trust a number

1. Which test set, and how big? A small or easy set can flatter any system.
2. Is the test set public, private, or the vendor's own? Public sets can leak into training.
3. How was the text scored? Ask whether punctuation, casing and fillers were removed first.
4. Which version and which date? Systems change between releases.
5. What audio conditions? Clean, read speech is the easiest case for every system.

A reputable vendor can answer all five. If it cannot, treat its number as a claim you have not yet verified.

What to do

Compare only numbers that share the same method, and be wary of a comparison that mixes tests. If you are choosing a system, build a small test from your own use: record real speech in the room where you will dictate, transcribe it by hand, and score each candidate the same way. That result will be more useful than any published table. For the question of which model leads, read the most accurate speech-to-text model. For the leaderboard itself, read what the Open ASR Leaderboard is.

If you want to see how a single transcript scores, our free audio to text tool transcribes a file in the browser, and you can compare its output with your own reference text.