Speech Recognition Models: How Neural ASR Works

Modern automatic speech recognition (ASR) runs on neural networks, and three architecture families cover most of the field: connectionist temporal classification (CTC), RNN transducers and attention-based encoder-decoders. Many of the strongest recent systems put a Conformer or transformer encoder in front of one of these decoders, and they learn from large collections of transcribed audio. The choice between families is mostly a trade-off between streaming, speed and accuracy. Transducers are the design behind on-device dictation that appears as you speak. Encoder-decoders such as OpenAI's Whisper write only after they have read the clip, so they are better suited to files.

This guide explains each family, the research behind it and where each one fits. It is part of our speech recognition overview. For the full pipeline from sound to text, read how speech recognition works. For the measure behind most of the numbers here, read word error rate.

From about 2012, neural networks began to outperform the statistical models behind speech recognition

Around 2012, most recognizers were built from statistical parts. The review by Hinton and colleagues describes the usual setup: hidden Markov models handle the timing of speech, and Gaussian mixture models score how well each state fits a short window of sound ¹.

The same review reports that deep neural networks with many hidden layers, trained with new methods, had been shown to outperform those mixtures on a variety of benchmarks, sometimes by a large margin ². The review was written by several research groups that had recent successes with the approach.

Recurrent networks came next. In 2013, Graves, Mohamed and Hinton reported that deep long short-term memory networks, trained end-to-end with suitable regularization, reached a 17.7% test set error on TIMIT phoneme recognition ³. TIMIT is a small phoneme benchmark, so that number cannot be compared with the word error rates of today's systems. It did show that a recurrent network trained end-to-end could reach a strong score on this task.

Hybrid systems kept the old pipeline but replaced its acoustic model with a neural network

A hybrid system keeps the classic structure: a timing model, a pronunciation lexicon and a separate language model. It replaces only the part that scores the sound. In the 2012 review, that scoring is the Gaussian mixture model ¹, and neural networks have been shown to outperform those mixtures ².

Because only the acoustic model changes, a better scorer can be swapped in without redesigning the decoder, the lexicon or the language model. The history of speech recognition covers the decades before this shift.

End-to-end models map audio straight to text with a single network

An end-to-end model learns the whole mapping from audio features to text in one network. It has no separate step that turns sounds into phonemes and then into words, and it has no hand-written alignment between audio and text.

The obstacle was alignment. Speech arrives as a long stream of sound, while the text is a short sequence of characters with no timing marks. Recurrent networks needed training data that was already segmented, and their outputs needed post-processing before they became labels ⁴. Connectionist temporal classification addresses that problem by training on unsegmented audio directly, and it is the first of the three families in this guide.

The input is usually a mel spectrogram, which shows the energy of the sound across frequency bands over time. The network turns that picture into text.

CTC labels every slice of audio with a character or a blank, which is simple but ignores context between outputs

A CTC network emits, for each short frame of audio, a probability over the characters plus one special blank symbol. The blank lets the network say "nothing new here" while it waits for the next sound. A decoding rule then merges repeated characters and removes blanks, so the frames that spell "hello" become the word "hello".

The simplicity has a cost. Each frame's output is decided largely on its own, given the audio. The Listen, Attend and Spell paper, which compares itself with CTC, says that its network produces character sequences without any independence assumptions between the characters, and calls that the key improvement over previous end-to-end CTC models ⁵. Attention models, covered below, are built to remove that gap.

Transducers add a prediction network and stream naturally, which is why on-device dictation uses them

A transducer adds two pieces to the CTC idea. A prediction network reads the labels written so far, and a joint network combines it with the audio encoder at each step. The output grows as the audio arrives, which is the property called streaming. Because the prediction network sees the earlier labels, the transducer keeps some of the context that CTC gives up, while still producing text before the speaker finishes. That combination is the reason the design is common in dictation. In 2012, Graves wrote that recurrent networks traditionally required a pre-defined alignment between input and output, and he presented an end-to-end transduction system that removes that need ⁶.

The case for phones comes from Google's own measurements. In a 2019 paper by Google researchers, He and colleagues report that their recurrent network transducer can outperform a conventional CTC-based model in terms of both latency and accuracy across a number of evaluation categories ⁷. Google's post says the Gboard recognizer is a model trained with RNN transducer technology that is compact enough to reside on a phone ⁸. In 2019, the Gboard recognizer, which works offline, was 80 MB after compression, and it first launched on Pixel phones in American English only ⁹.

A phone cannot wait for the end of a sentence, and it cannot always reach a server. A transducer that streams and fits in a small model addresses both needs. Yap, which we make, runs its dictation on the device for Mac and Windows, which is the setting where streaming and a small model matter most. For what that means in practice, read on-device vs cloud speech recognition. For how Apple's dictation runs on a Mac, see how Apple dictation works.

Attention encoder-decoders such as Whisper read a whole clip before writing, which suits files more than live typing

An attention encoder-decoder has two parts. The encoder turns the audio into a sequence of hidden states. The decoder writes text one token at a time, and at each step it attends to the parts of the audio it needs, while also conditioning on the tokens it has already written. The Listen, Attend and Spell paper is an early example of this design for speech. On a subset of Google's voice search task, it reached a 14.1% word error rate with no dictionary and no language model ¹⁰.

Whisper uses the same family. The Whisper authors chose an encoder-decoder Transformer because they judged that architecture well validated to scale reliably ¹¹. Its models are trained on 30-second audio chunks, and the paper states that they cannot consume longer audio inputs at once ¹². The paper therefore needs a separate long-form procedure for recordings longer than that window, which is a file-processing strategy rather than a live one. For more on the model itself, see what Whisper AI is.

The decoder writes from what it has already written, so it can also produce fluent text that the audio never contained. In a 2024 study of Whisper, roughly 1 percent of audio transcriptions contained entire hallucinated phrases or sentences that did not exist in the audio ¹³. Checking output on your own recordings matters for that reason.

The Conformer combined convolution and attention, and it is a strong default encoder

The Conformer is an encoder design that combines convolution with attention. Gulati and colleagues propose the convolution-augmented transformer for speech recognition, named Conformer ¹⁴. Convolution captures short local patterns in the sound, and attention captures context over longer spans, so the encoder gets both.

On LibriSpeech, the standard English benchmark, the Conformer reached a 2.1% word error rate on test-clean and 4.3% on test-other, without a language model ¹⁵. On the Open ASR Leaderboard, its 2025 paper observes that Conformer-based encoders paired with transformer-based decoders achieve the best average word error rate ¹⁶. That makes the Conformer a strong default when you build or choose an encoder, though the best system for your audio may still differ.

Self-supervised learning (wav2vec 2.0) lets a model learn from untranscribed audio

Transcribed audio is expensive, so the wav2vec 2.0 approach learns in two stages. It first learns speech representations from unlabeled audio, then fine-tunes on a much smaller labeled set. This is the idea behind wav2vec 2.0.

With all of LibriSpeech's labeled training data, wav2vec 2.0 reached a 1.8% word error rate on the clean test set and 3.3% on the harder other test set ¹⁷. The more striking result uses little labeled audio. With just ten minutes of labeled data and pre-training on 53,000 hours of unlabeled audio, wav2vec 2.0 still reached 4.8% on LibriSpeech clean and 8.2% on other ¹⁸.

These results are on read English, and ten minutes of labels is a research setting. They do not show that a small labeled set is enough for dictation in a noisy room. What they do show is that unlabeled audio is a useful resource, which is why most large speech models are pre-trained this way.

Scale changed results: Whisper trained on 680,000 hours of multilingual audio

Whisper was trained on 680,000 hours of multilingual and multitask audio with transcripts ¹⁹. The authors estimate that a language's word error rate halves for every 16-fold increase in its training data ²⁰. Scale therefore matters, and the authors' estimate helps explain the differences between languages.

The gap is wide. On the Fleurs benchmark, Whisper large-v2 had a 4.2% word error rate for English ²¹ and 39.3% for Swahili ²². Accuracy also falls away from the clean read speech that most benchmarks use. On the CHiME6 test set, the same model had a 25.5% word error rate ²³, which is a warning for anyone who measures only on clean read speech. For how models are trained and what datasets they use, read how speech recognition models are trained.

Audio-language models that pair a speech encoder with an LLM are the newest branch

The newest branch connects a speech or audio encoder to a large language model. The SALMONN paper describes a model that lets an LLM process and understand general audio inputs directly, with competitive performance on tasks such as automatic speech recognition ²⁴.

The branch has also reached the leaderboards. NVIDIA's Canary Qwen 2.5B, a speech encoder with a language-model decoder, had a 5.63% average word error rate and an inverse real-time factor of 418 ²⁵. Its numbers appear with the other systems below.

A second use keeps an ordinary recognizer and lets an LLM correct its output. The HyPoradise benchmark shows why this helps. A large language model can correct words that are missing from the recognizer's N-best list, because it generates text rather than only choosing among candidates ²⁶. Whether LLMs can replace a recognizer outright is a separate question, covered in can large language models do speech recognition.

Each architecture has a best use: a comparison of speed, streaming and accuracy

The three families differ most in streaming. The table below summarizes the trade-offs, and the figures that follow compare three current systems on the same leaderboard.

On the Open ASR Leaderboard's English short-form track, NVIDIA's Parakeet TDT 0.6B v2, a transducer, had a 6.05% average word error rate and an inverse real-time factor of 3,390 ²⁷. Whisper Large v3, an encoder-decoder, had 7.44% and 146 ²⁸. The Canary Qwen model had 5.63% and 418 ²⁵. The leaderboard paper says token-and-duration transducer (TDT) decoders offer superior speed, which makes them better suited to long-form and batched processing ²⁹.

On these numbers, Parakeet was about 23 times faster than Whisper Large v3 ³⁰, and it was also more accurate. The leaderboard reports accuracy and speed side by side. A fast model is only useful if its accuracy holds, so read the two columns together. These results come from one benchmark on English short-form audio. For how to read such leaderboards, see speech recognition benchmarks and leaderboards, and for the measure itself, see the real-time factor and the word error rate entries.

Read the leaderboard numbers with their conditions

The leaderboard scores are not the same as the text you would see in an app. Before it computes word error rate, the leaderboard's normalized transcripts remove punctuation and casing, along with disfluencies such as fillers ³¹. The score therefore measures the words, not whether a sentence starts with a capital letter or contains an "um". A dictation user who cares about punctuation and cleanup should test on their own audio and their own style of speech.

The leaderboard also covers English short-form audio. Long recordings, other languages, and noisy rooms each need their own test, and a model that leads one track can trail on another.

For a dictation app, the deciding factor is streaming, which points to a transducer. For batch transcription of recordings, accuracy and throughput matter most, and an encoder-decoder such as Whisper is a reasonable choice to test first. To compare the options available to you, read open-source speech recognition and real-time speech recognition.