Glossary
Language model (in speech recognition)
In speech recognition, a language model (LM) scores how probable a word sequence is in a language. The recognizer combines that score with the acoustic evidence to choose between words that sound alike, such as homophones.
Updated October 17, 20265 cited facts, each traced to its source
Also called LM, n-gram language model.
In speech recognition, a language model (LM) gives a probability to a sequence of words. It answers the question: how likely is this phrase in this language? The recognizer combines that score with the acoustic model, which scores how well the audio matches. When two transcripts sound almost the same, the language model usually decides which one is written.
Older systems used n-gram counts
The classic form is the n-gram language model, which estimates the probability of each word from the few words before it. A trigram model, for example, looks at the previous two words. It is built by counting word sequences in a large body of text. The method is simple and fast, which is why it ran on early hardware, and it is still used in some systems and as a rescoring step. Neural language models have since replaced most n-gram models in research, because they handle longer context.
A concrete example: homophones
Consider the audio for "recognize speech" and "wreck a nice beach". The sounds are close enough that the acoustic evidence alone may not separate them. A language model strongly prefers "recognize speech" in ordinary English, so the recognizer chooses it. Without that prior, the same audio could produce a plausible but wrong sentence. The same effect fixes many homophones such as "their", "there" and "they're" in dictation.
How much it helps
The gain can be large. In a 2015 simulation of touchscreen typing, combining a spatial model with a language model reduced the word error rate from a 38.4 percent baseline to 5.7 percent 1. That is a typing study rather than a dictation study, but it shows how much the choice of likely words can matter.
A language model is not always required. In a 2015 Google study on voice search, the LAS model reached a word error rate of 14.1 percent on a subset of the task without a dictionary or a language model 2. End-to-end neural systems learn an implicit language model from their training transcripts 3. An external language model is then mainly useful where the training text is limited.
Domain text is one reason to add one. In a 2022 study of UK Supreme Court hearings, adding an in-domain language model and common legal phrases improved word error rate on the court audio 4.
Why it matters for dictation
Language models affect how sensible your text looks, and some dictation products expose them as a separate stage. Superwhisper's documentation says a dictation can skip its language model stage, and that its Voice to Text mode outputs raw transcribed text with no language model involved 5. Raw output is faithful to what the recognizer heard, but it may keep odd word choices. Stages that rewrite text can fix those choices but may also change what you said. Our guide to speech recognition models explains where the language model sits in each design.
Related terms
- Acoustic model: the sound side of the same decision.
- Automatic speech recognition (ASR): the task that combines both models.
- Word error rate (WER): the measure that shows whether a language model helped.
- Inverse text normalization (ITN): the formatting step that turns spoken words into digits and symbols.
Guides that cover it
Evidence
Every number on this page, with where it comes from. Open a fact to see the exact words in its source.
- 1.
- 2.
- 3.
- 4.
- 5.
Related terms
- Speech Recognition: What It Is and How Well It WorksThe topic this term belongs to
- Acoustic modelAn acoustic model is the part of a speech recognizer that scores how well a short window of audio matches the sounds of speech. Since about 2012 most acoustic models are deep neural networks, and their training data strongly affects accuracy.
- Automatic speech recognition (ASR)Automatic speech recognition (ASR) is software that converts spoken audio into written words. Modern systems are neural networks trained on large amounts of transcribed speech, and they run in the cloud or on the device.
- ConformerA Conformer is a neural encoder for speech recognition that combines self-attention, which sees the whole input, with convolution, which captures local sound patterns. FastConformer is NVIDIA's faster variant of the same design, used in streaming models.
- Connectionist temporal classification (CTC)Connectionist temporal classification (CTC) is a way to train a neural network to turn audio into text without knowing in advance which moment of sound matches which letter. The network labels every short slice of audio with a character or a blank, and the blanks and repeated labels are then removed to leave the transcript.
- Hidden Markov model (HMM)A hidden Markov model (HMM) is a statistical model in which a sequence of hidden states produces the observations you can measure, and each state changes to the next with set probabilities. In speech recognition, the hidden states are units of sound and the observations are short frames of audio.
- Inverse text normalization (ITN)Inverse text normalization (ITN) is the step that converts a speech recognizer's spoken-form output into written form, so that 'two hundred' becomes 200. It changes how the text reads, not which words the recognizer heard.
- LibriSpeechLibriSpeech is a free corpus of about 1,000 hours of read English speech, taken from LibriVox audiobooks, and it is one of the most widely used benchmarks for English speech recognition. Its scores describe clean, read speech, which differs from everyday dictation.
- Mel spectrogramA Mel spectrogram is a grid of sound energy over time, with frequency bands spaced to match human hearing. Many speech recognizers compute its logarithm, the log-Mel spectrogram, and feed it to a neural network instead of the raw waveform.
- Real-time factor (RTF)The real-time factor (RTF) is the time a system takes to process audio divided by the length of that audio. An RTF of 1 or lower means the system keeps up with real-time speech. Some leaderboards report the inverse, RTFx, where higher is faster.
- RNN transducer (RNN-T)An RNN transducer (RNN-T) is a neural speech model that reads audio in order and emits words as it goes, so text can appear while the speaker is still talking. It has an encoder for the sound, a prediction network for the words already written, and a joint network that chooses the next output.
- Speaker diarizationSpeaker diarization is the task of working out who spoke when in a recording, by splitting the audio into turns and grouping turns that come from the same voice. It is separate from transcribing the words, and a transcript gains speaker labels only when diarization is run alongside recognition.
- Voice activity detection (VAD)Voice activity detection (VAD) is a component that decides whether a stretch of audio contains speech. Speech recognizers use it to split audio into turns, skip silence and decide when a speaker has finished.
Pages that use this term
- Speech to Text and LLMs: What Language Models ChangeGuide
- Speech Recognition Models: How Neural ASR WorksGuide
- How Apple Dictation Works: On-Device, Server and AIGuide
- Can speech to text work without an LLM?Question
- Is speech recognition part of NLP?Question
- What role do hidden Markov models play in speech recognition?Question
- Acoustic modelGlossary
- Automatic speech recognition (ASR)Glossary
- Connectionist temporal classification (CTC)Glossary
- Generative error correction (GER)Glossary
- Hidden Markov model (HMM)Glossary
- Large language model (LLM)Glossary