Which neural network is best for speech recognition?

No single neural network is best for speech recognition, because the right choice depends on the job. For dictation that must appear while you speak, a streaming transducer is the usual answer. For transcribing files, an encoder-decoder such as Whisper or a Conformer encoder is a strong first choice. The benchmark numbers below show the trade-offs, but your own recordings should make the final call.

Live dictation needs a network that streams, which points to transducers

A network streams when its text appears as the audio arrives, instead of after the speaker finishes. A transducer does this by pairing a prediction network, which reads the words written so far, with a joint network that combines it with the audio at each step. In the paper behind Google's on-device recognizer, He and colleagues report that a recurrent network transducer can outperform a conventional CTC-based model in terms of both latency and accuracy across a number of evaluation categories ¹.

Yap, which we make, runs its dictation on the device for Mac and Windows. On-device dictation is the setting where streaming matters most, because the text has to keep up with your voice. The guide to on-device and cloud recognition explains the trade-offs in more detail.

On the public leaderboard, the top models are close, and the families trade speed for accuracy

The Open ASR Leaderboard reports two numbers for each system on its English short-form track: the average word error rate, where lower is better, and the inverse real-time factor, where higher means faster. The two best-known open models in the table below show the trade-off.

On this track the transducer was both more accurate and much faster. That is one benchmark on English short-form audio, and a model that leads it can trail on other tracks. The guide to speech recognition benchmarks explains how to read these tables, and the word error rate entry defines the measure.

Conformer encoders score well, and the leaderboard authors point to them

The Conformer combines convolution with attention, the convolution-augmented transformer design that Gulati and colleagues proposed ⁴. Convolution captures short patterns in the sound, and attention captures context over longer spans. On LibriSpeech, the standard English benchmark, the Conformer reached a 2.1% word error rate on test-clean and 4.3% on test-other, without a language model ⁵. The authors of the leaderboard paper observe that Conformer-based encoders paired with transformer-based decoders achieve the best average word error rate across the systems they compared ⁶. A Conformer is therefore a strong default when you build or choose an encoder, though the best system for your audio may differ.

Encoder-decoders such as Whisper suit files, and they can invent words

An encoder-decoder reads the clip, then writes text one token at a time. That design suits recordings you can process after they are finished, and it does not suit text that must appear as you talk. Its main risk is fluent text that the audio never contained. In a 2024 study of Whisper, roughly 1 percent of audio transcriptions contained entire hallucinated phrases or sentences that did not exist in the audio ⁷. Read a Whisper transcript against the recording, and be most careful in silent or noisy passages.

How to choose: four steps

1. Decide whether the text must stream. If words must appear as you speak, choose a streaming recognizer, usually a transducer-based one.
2. For recordings, test two or three models on your own audio. Use the same few recordings for each, and include a noisy one.
3. Compare the error rates and read the output. Measure the word error rate on your recordings, then check for sentences that were never spoken.
4. Check the language. An English leaderboard says little about other languages, and accuracy can differ a great deal from one language to the next. Read the Whisper language accuracy guide before you pick a model for another language.

The network is only part of the result. The data it was trained on, covered in how speech recognition models are trained, and the decoding setup in your app also change what you hear. For what large language models add, read whether large language models can do speech recognition. For the Whisper family in more depth, read the guide to Whisper AI.