Speech Recognition Benchmarks and Leaderboards

A speech recognition benchmark is a fixed set of recordings, each with a human-made reference transcript, that every system is scored on in the same way. The score is almost always word error rate (WER): the number of substituted, deleted and inserted words divided by the number of words in the reference. The best-known public scoreboard is the Open ASR Leaderboard, which compares ¹. Its numbers are useful, but they answer a narrower question than most buyers assume: how a model does on the particular recordings in its test sets, under the leaderboard's own scoring rules.

This guide explains what the main test sets measure, why clean read speech flatters every system, and how to read a leaderboard without being misled. For the metric itself, see word error rate and our guide to how WER is calculated and read.

A benchmark is a fixed set of recordings with reference transcripts, scored by word error rate

A benchmark has three parts: test audio that no system was tuned on, a reference transcript written by a person, and a scoring rule that says what counts as an error. Change any one of the three and the score changes, even when the models stay the same.

The scoring rule is the step most readers never see. For its normalized scores, the Open ASR Leaderboard removes punctuation, casing and disfluencies such as fillers, repetitions and repairs before it counts errors ². That makes systems comparable, but it means the score says nothing about whether a model writes "um" or places commas correctly. Numbers are another trap: a system that writes "200" and a reference that says "two hundred" can be scored as two errors unless the pipeline converts spoken forms first, a step called inverse text normalization.

Two scores are comparable only when the audio, the reference and the normalization are the same. A WER on a leaderboard and a WER in a vendor's brochure often are not.

When you open a leaderboard, read a row in four steps. First, find the track: short-form English, long-form audio and multilingual tracks are different tests. Second, check the date and version, because the same model name can hide a retrained system. Third, check whether the system is open or proprietary and whether it runs on your device or in the cloud. Fourth, ask whether the score is a single number or an average across several test sets, and which ones. A row that answers none of these questions tells you a model's rank on that day, not how it will hear you.

LibriSpeech, built from read audiobooks, is the most quoted test and now the least informative

LibriSpeech is the most quoted English test, and the best systems have nearly exhausted it. It contains ³, drawn from audiobooks read aloud. A Conformer model reached 2.1 percent WER on its clean test set and 4.3 percent on its harder "other" set without a language model ⁴.

When the best systems sit near 2 percent, the gap between them says little about how they will handle a meeting, a phone call or a speaker with an accent. A test built from read sentences cannot show the errors that appear in conversation, and the next section shows how large those errors are.

The problem goes beyond difficulty. The same test set may also have leaked into the data used to build the models, which the contamination section below covers.

Conversational, meeting and noisy test sets expose the errors LibriSpeech hides

Harder audio produces much higher error rates, and studies from 1997 and 2017 show the same pattern. In a 1997 review, a high-performance hidden Markov model (HMM) recognizer had a word error rate of 43 percent on spontaneous telephone conversations from the Switchboard corpus ⁵. Human transcribers had 4 percent on the same kind of speech ⁶, though the two figures come from different tests.

The setup changes the result as well. In a 2017 study, decoding close-talking sentences recorded in a studio gave a word error rate of 3.7 percent ⁷. With a single distant ceiling microphone in an apartment and real recordings, a DNN recognizer from the same study reached 12.0 percent ⁸. A studio test does not predict the result in a room with a distant microphone.

Speech style also matters. A 2026 study of commercial systems found that every system exceeded 51 percent WER on severe dysarthria ⁹, a speech disorder caused by brain or nerve damage ¹⁰. Accent is a further test. The Earnings-22 corpus is a free benchmark for accented English in real recordings, and it contains ¹¹. Its authors say that systems have achieved superhuman error rates on many common corpora while lacking adequate performance on speech in the wild ¹².

If your dictation happens at a desk with a headset, LibriSpeech-like scores may be a fair guide. If it happens in an office, a car or a noisy kitchen, you need test audio from those places, not from a studio.

Common Voice, FLEURS and Multilingual LibriSpeech measure languages beyond English

Non-English benchmarks exist, but each covers fewer hours of speech, and in many languages less variety, than the English tests. Mozilla's Common Voice corpus held ¹³ at the time of its 2019 paper, collected from volunteers. The contamination discussion below applies to it as well.

FLEURS is built on the same sentences translated into many languages, so the same content can be compared across languages. It is an n-way parallel speech dataset in 102 languages, with the machine translation sentences read aloud ¹⁴. Results on it vary widely. In the Whisper paper's FLEURS table, the large-v2 model's error rate was 3.0 percent in Spanish ¹⁵ and 99.0 percent in Telugu ¹⁶, which is why a language-specific score matters more than any single English figure.

Multilingual LibriSpeech extends the audiobook design to eight languages, with about 44.5 thousand hours of English and about 6 thousand hours in the other languages combined ¹⁷. Read audiobook speech is the genre there, so it inherits the same limits as LibriSpeech ¹⁷.

For the language-by-language picture, see our guide to Whisper supported languages and accuracy, and for accents, speech recognition and accents.

The Open ASR Leaderboard scores open and closed models on the same data with the same code

The Open ASR Leaderboard is the most convenient public comparison because it runs every system on the same audio, with the same normalization and code. Its paper describes a reproducible platform that compares ¹, and it reports two measures for every system: "word error rate (WER) for comparing transcription quality, and inverse real-time factor (RTFx) for comparing inference speed" ¹⁸.

The paper also draws a conclusion about architecture: Conformer-based encoders paired with transformer decoders achieve the best average WER ¹⁹. Among the snapshot results in its table, the English short-form figures look like this.

The chart shows three strong open models close together. Treat that order as the leaderboard's snapshot, not as a ranking that will hold. Leaderboards change as new systems are added, and the paper's authors note that private evaluation sets could help minimize the risk of test-set contamination ²³. In response, Appen says it provides a suite of new, private English-language audio datasets to the leaderboard ²⁴. Appen is a data vendor, so its announcement is a vendor's claim about its own product.

To see the answer to the most common version of this question, read what the Open ASR Leaderboard is and the most accurate speech-to-text model.

Speed is scored too: inverse real-time factor counts seconds of audio processed per second

Speed on a benchmark is reported as the inverse real-time factor, RTFx: seconds of audio processed for each second of compute. A higher number means a faster system. The Open ASR figures show how wide the range is. Parakeet TDT 0.6B v2 reached an RTFx of 3,390 ²¹, while Whisper Large v3 reached 146 ²², so Parakeet processed audio about 23 times faster on the leaderboard's hardware ²⁵.

The speed numbers come from batched processing on the leaderboard's own GPU setup, so they describe throughput, not the delay you feel. For a single live utterance, the number that matters is latency, the time from the end of your sentence to the text on the screen. Our guide to real-time speech recognition covers that, and the real-time factor entry defines the ratio.

Accuracy and speed can trade off. A faster decoder may not be the most accurate, so a benchmark that reports both is more useful than one that reports only the lowest WER.

For your own use, measure latency separately. Start a stopwatch when you finish a sentence and stop it when the last word appears on screen. Repeat the test with the network off if the app runs on your device, because a cloud system's delay includes the trip to a server and back. Batch throughput tells a data team how fast a backlog clears; it does not tell a writer how long the cursor waits.

A model trained on a test set's audio scores better than it really is

A test set stops being a test once a system has seen it. Contamination can take two forms: the test text ends up in the text used to train a language model, or the test audio and transcripts are used to train the recognizer itself.

The first form is the one with the strongest evidence. A 2025 study found that a substantial amount of the LibriSpeech and Common Voice evaluation sets appear in public LLM pretraining corpora ²⁶. The authors trained models with and without that overlap. The contaminated models produced only subtle differences in error rate, but they assigned significantly higher probabilities to transcriptions they had seen during training ²⁷. The lesson is that WER can look normal while the model has memorized the test. A low score on a benchmark that has leaked is not evidence of skill.

The second form has a clear example. A 2026 preprint on Swiss German states that published results of 17.1 to 17.5 percent WER are inflated by benchmark contamination. A vanilla Whisper model trained on the test set, with no Swiss German data, reached 13.88 percent WER, which beat all published systems ²⁸. That is a single preprint on one benchmark, but it shows how a score can be produced without real gains in the language.

Contamination is hard to see from the outside. A model card that lists training data without saying which sets were held out makes it impossible to tell. Good documentation names the training corpora, states whether any evaluation set was excluded, and reports results on sets created after the training data was collected. When a vendor cannot say these things, treat its benchmark as unverified.

When you read a new accuracy claim, check four things:

1. Is the test set public, private, or created by the vendor?
2. Does the model card or paper say what training data was used?
3. Was the test set used to choose settings or tune the model?
4. Has anyone reproduced the result on fresh audio you can listen to?

A vendor's own benchmark is a claim until someone else reproduces it

Vendors publish benchmarks because they are useful marketing. The problem is that a single test can produce very different results when the method changes. In a 2025 test, 9to5Mac ran one 7-minute-31-second episode through several engines. Apple's speech API had a word error rate of 10.3 percent ²⁹, and Whisper Large V3 Turbo had 1.5 percent on the same episode ³⁰. One episode is a small sample, and a different episode could give a different order.

A good benchmark report says what was tested, how it was scored, and when, and it leaves room for someone else to rerun it. A report that gives one number with no test set, no normalization and no date tells you almost nothing.

Lower WER does not always mean a better experience. In a 2017 study of automatic captions for deaf and hard-of-hearing viewers, word error rate correlated only 0.11 with user ratings ³¹. That is one study, with its own limits, but it shows that the standard metric may not capture what a reader notices.

Before trusting a vendor's figure, ask for the following. A reputable vendor can answer each question.

A small benchmark built from your own audio predicts your results better than any leaderboard

The most reliable benchmark is the one you build from your own speech. The leaderboards cover many systems, but they cannot cover your microphone, your room, your vocabulary or your way of speaking. The evidence above shows why: accented speech, a distant microphone and noise each change the error rate, and a clean test says nothing about them.

Your own test answers the question a leaderboard cannot, which is how a system handles your voice, your words and your room. It also protects you from a result you cannot reproduce. A simple method works for a single person or a team:

1. Record real use, with the same microphone and in the same room where you will dictate.
2. Transcribe the recordings by hand, deciding in advance whether fillers and punctuation count.
3. Run each candidate system on the same recordings, with its default settings.
4. Compute WER with the same normalization for every system.
5. Repeat with a second speaker, or in a noisier room, to see how stable the ranking is.

Keep the recordings private and small. The aim is to predict your own use, not to publish a result. Store the reference transcripts and the exact normalization script with the scores, so anyone can rerun the comparison later and get the same numbers. A result that cannot be rerun is no better than a vendor's brochure, however careful the test felt at the time. A small set in your own conditions tells you more than a large leaderboard built from other people's audio. You can test a clip without installing anything with our free audio to text tool. On-device dictation apps such as Yap, which we make, can be checked with the same recordings in the same way, so the comparison is fair. If you want to try a dictation app on your own Mac or Windows PC, see the download page.

Common questions this guide answers

• What is the Open ASR Leaderboard?
• What is the most accurate speech-to-text model?
• Why do speech-to-text accuracy claims disagree?

Where to go next

Start with how speech recognition works if the models are new to you. To compare products, read speech-to-text APIs compared on price, accuracy and speed and how accurate speech recognition really is. For the open models themselves, see open-source speech recognition and how to run Whisper locally. The whole subject is covered on the speech recognition overview, and the word error rate and LibriSpeech entries define the two terms this guide uses most.