AssemblyAI vs OpenAI Whisper: which is more accurate?

AssemblyAI's top model is more accurate than Whisper large-v3 on both public English leaderboards we cite, but the size of the gap changes with the test. On the Artificial Analysis three-dataset index, Universal-3.5 Pro scores 3.0 percent word error rate ¹ and Whisper large-v3 scores 4.1 percent ². On the Open ASR Leaderboard's English short-form results, the figures are 5.80 percent ³ against 7.84 percent ⁴. Lower is better. Both pages were read in 2026 and scores change with each model version.

Both leaderboards put AssemblyAI ahead, by 1 to 2 points

Word error rate (WER) is the share of words that are wrong, as defined in the glossary entry on word error rate. The two leaderboards use different test sets, so compare within a row pair, not across them.

The direction is the same in both, and AssemblyAI's model is ahead. A one to two point gap in WER is real: at 4 percent, about one word in 25 is wrong, and at 3 percent it is about one in 33. For a transcript someone must read, that is a visible difference. For a search index or a rough summary, it may not matter.

The Whisper number depends on who hosts it

"Whisper" is a model, not a service, and the same weights give different scores in different hosts. On the Artificial Analysis index, Whisper large-v3 scored 4.1 percent on fal.ai ², 4.5 percent on Together AI ⁵ and 10.1 percent on Replicate ⁶. Settings, chunking and decoding choices explain such spreads. If you read "Whisper is X percent," ask which host and which settings.

Leaderboards are not your audio

Three cautions apply before you pick by a leaderboard.

• Normalized text. The Open ASR scores are computed after punctuation, casing and disfluencies such as fillers and repetitions are removed ⁷. A product that must deliver punctuated, capitalized text is not scored on those parts.
• Versions move. Both models are updated. A figure from last year may describe a different model.
• Different audio. Phone calls, accented speech, noise and technical vocabulary can reorder the ranking. Our guide to why accuracy claims differ explains how, and the question on a good word error rate says what to aim for.

Whisper can still be the right choice

"More accurate on a leaderboard" is not the only factor. Weigh these.

• Price. AssemblyAI's Universal-3.5 Pro lists at 0.21 dollars per hour ⁸, about 42 percent below OpenAI's whisper-1 API ⁹. Self-hosted Whisper has no per-hour fee but needs hardware and engineering time. See how much AssemblyAI costs.
• Privacy and offline use. You can run Whisper on your own computer so that audio never leaves it, which a cloud API cannot match. The guide to running Whisper locally covers it.
• Languages. Universal-3.5 Pro supports 18 languages ¹⁰, and AssemblyAI's cheaper Universal-2 supports 99 ¹¹. Check both for your language, since accuracy varies by language.
• Hallucinations. In a 2024 study of Whisper as it stood in 2023, roughly 1 percent of transcriptions contained entire phrases that were not in the audio ¹². That describes one version, and our question on why Whisper hallucinates covers the causes. We cite no matching study for AssemblyAI, so this is not a claim that it does better.

What to do

1. Collect 30 to 60 minutes of audio like your real use, and write correct transcripts for it.
2. Run it through AssemblyAI's free tier (see whether AssemblyAI is free) and through the Whisper setup you would actually use, with the same files.
3. Compute WER for each, deciding first whether punctuation and capitals count.
4. Compare cost for your monthly hours, and retention terms if audio is sensitive.

The guide to AssemblyAI walks through the test, the comparison of speech-to-text APIs puts both beside other services, and the speech recognition overview covers the wider subject. For dictation on your own device, Yap, which we make, is an on-device option, and its details are on the download page.