What role do hidden Markov models play in speech recognition?

Hidden Markov models ( HMMs) gave speech recognizers their sense of timing. Speech is a stream of short sounds, and the same word takes a different number of audio frames each time someone says it. An HMM models a word as a sequence of hidden states, one for each unit of sound, and it moves from state to state with set probabilities. The recognizer never sees the states. It sees the audio, and it searches for the most likely path through the states, which gives the most likely sequence of sounds and words. For about four decades this was the standard design. A 2012 review still described most systems as using HMMs ¹, until neural networks outperformed the GMM-HMM baseline ².

This page explains the job the HMM did, what neural networks replaced, and what is left of the HMM today. For the wider picture, read our guide to speech recognition models or the history of speech recognition.

HMMs were built to handle the timing of speech

Researchers seem to have applied HMMs to speech independently at two laboratories around 1972 ³. The model gave them a principled way to answer three questions. Rabiner's 1989 tutorial lists them: how likely a sequence of observations is, which sequence of states is best, and how to adjust the model's parameters to fit the signal ⁴. The second question is the search that a recognizer runs each time it turns audio into words.

In the classic system, the HMM handles timing and a second model scores the sound. A 2012 review describes the division: HMMs handle the timing of speech, and Gaussian mixture models score how well each HMM state fits a short window of acoustic input ¹. A language model then favors word sequences that make sense, so a word that sounds right but reads badly is less likely to win.

Neural networks replaced the scoring and left the timing to the HMM

The HMM with Gaussian mixtures spread through the field and became the dominant approach by the 1990s ⁵. Neural networks then entered as a replacement for the mixtures. A hybrid system keeps the HMM and puts a neural network in place of the Gaussian scorer. In the textbook's definition of the hybrid HMM/MLP architecture, the network's outputs serve as the probability estimates for an HMM-based system ⁶.

The gain was large. In a 2012 review, a deep neural network reduced the word error rate of a GMM-HMM baseline from 27.4 to 18.5 percent on a Switchboard telephone task ². Word error rate counts the substitutions, deletions and insertions in a transcript, as explained in the word error rate entry.

The table shows which job each design gave to each part.

Deep recurrent networks made the HMM optional

Graves, Mohamed and Hinton reported in 2013 that deep long short-term memory networks, trained end-to-end, reached a 17.7% test set error on TIMIT phoneme recognition ⁷. TIMIT is a small benchmark of phonemes, so that score cannot be compared with the word error rates of today. It showed that a recurrent network could learn phoneme recognition without a separate HMM for timing.

The problem that the HMM had solved was alignment. Graves wrote in 2012 that recurrent networks traditionally required a pre-defined alignment between input and output sequences ⁸. The connectionist temporal classification (CTC) and RNN transducer models that followed learn the mapping from audio to text in one network. They do not need a separate HMM for timing. Our question on how speech recognition models are trained covers how those networks learn.

It depends: when an HMM still matters

Older products and documents are where you are most likely to meet an HMM. A product that describes an acoustic model, a timing model and a separate decoder is probably built on the older design. A product that describes one neural network is probably end-to-end. Vendors do not always say which, so the wording of the documentation is a clue, not proof.

Old results do not compare with new ones. The 2012 Switchboard numbers and the 2013 TIMIT number come from older test sets and systems. Read them as evidence of a trend. Compare error rates only when they come from the same benchmark, and use our guide to speech recognition benchmarks for how to read leaderboards.

The HMM is not the choice you make. When you pick a dictation app, the questions that matter are whether it streams, how accurate it is on your kind of speech, and where the audio is processed. The timing model inside the engine matters much less. Our question on the best neural network for speech recognition covers those trade-offs.

What to do with this

1. Read the engine description. Look for whether the product names a separate acoustic model and decoder, or one neural network.
2. Translate the old vocabulary. In older papers, the HMM handles timing, the Gaussian mixture scores sound, and the decoder searches for the best word sequence.
3. Test on your own audio. A benchmark score is measured on a specific corpus and test set. Your room, microphone and speaking style may differ.
4. Ask whether the model streams. If you need text as you speak, look for a streaming design such as a transducer. An HMM alone does not settle that question.

Related questions

• Can large language models do speech recognition?
• How are speech recognition models trained?
• Which neural network is best for speech recognition?