Glossary
Acoustic model
An acoustic model is the part of a speech recognizer that scores how well a short window of audio matches the sounds of speech. Since about 2012 most acoustic models are deep neural networks, and their training data strongly affects accuracy.
Updated October 17, 20266 cited facts, each traced to its source
An acoustic model is the part of a speech recognizer that connects the audio to the sounds of speech. For each short window of audio, it estimates how likely that window is to contain each speech unit, such as a phoneme or a sub-word piece. The rest of the system combines those scores with a language model to choose the most probable words. The acoustic model hears the sound. It does not know what the sentence means.
The older design: HMMs and Gaussian mixtures
In the classic design, speech recognition systems used hidden Markov models (HMMs) to handle the timing of speech, and Gaussian mixture models (GMMs) to determine how well each state of each HMM fits a short window of acoustic input 1. The HMM part is described in the hidden Markov model entry. The GMM was the acoustic model that scored sound.
Neural networks replaced the GMM
In a 2012 experiment on the Switchboard telephone task, a deep network with seven hidden layers and 2,048 units in each layer replaced the Gaussian mixture model in the acoustic model 2. The same review reported that the network reduced the word error rate of the GMM-HMM baseline from 27.4 to 18.5 percent 3. That result started the move to neural acoustic models that most systems use today.
Modern end-to-end systems often merge the acoustic model and the decoder into one network, so a separate acoustic model is no longer visible in the product. The job still exists inside the network. Our guide on speech recognition models explains the architectures that replaced it.
A concrete example: why accents cause errors
Suppose a recognizer's acoustic model was trained mostly on speech from one group of people. Its sound scores will be most reliable for speakers who sound like that group. A 2020 study of five commercial recognizers found an average word error rate of 0.35 for Black speakers and 0.19 for white speakers 4. The authors traced the gap to the acoustic models that the systems used 5. The practical lesson is that accuracy depends on how well the acoustic training data matches your voice.
How the acoustic model is improved
Adapting an acoustic model to a speaker or accent can help a great deal. In a 2007 study of foreign-accented English, the best combination of pronunciation modelling and acoustic adaptation cut word error rate by 46 to 71 percent relative 6. That is a relative figure, so the absolute change depends on the starting error rate.
Why it matters for dictation
For dictation, the acoustic model decides whether the sounds you make are heard correctly. It is the first place errors begin: a misheard vowel can turn "wreck" into "rack" before the language model has a chance to fix it. When a dictation tool is wrong on your voice, the cause is often the acoustic side, which is why our guide to speech recognition accuracy covers speaker and microphone effects separately from vocabulary.
Related terms
- Automatic speech recognition (ASR): the full task that the acoustic model serves.
- Language model (in speech recognition): the component that chooses among words the acoustic model finds plausible.
- Hidden Markov model (HMM): the timing model used alongside acoustic scores in classic systems.
- Connectionist temporal classification (CTC): a neural training method that aligns audio with letters or tokens.
Guides that cover it
Evidence
Every number on this page, with where it comes from. Open a fact to see the exact words in its source.
- 1.
- 2.
- 3.
- 4.
- 5.
- 6.
Related terms
- Speech Recognition: What It Is and How Well It WorksThe topic this term belongs to
- Automatic speech recognition (ASR)Automatic speech recognition (ASR) is software that converts spoken audio into written words. Modern systems are neural networks trained on large amounts of transcribed speech, and they run in the cloud or on the device.
- ConformerA Conformer is a neural encoder for speech recognition that combines self-attention, which sees the whole input, with convolution, which captures local sound patterns. FastConformer is NVIDIA's faster variant of the same design, used in streaming models.
- Connectionist temporal classification (CTC)Connectionist temporal classification (CTC) is a way to train a neural network to turn audio into text without knowing in advance which moment of sound matches which letter. The network labels every short slice of audio with a character or a blank, and the blanks and repeated labels are then removed to leave the transcript.
- Hidden Markov model (HMM)A hidden Markov model (HMM) is a statistical model in which a sequence of hidden states produces the observations you can measure, and each state changes to the next with set probabilities. In speech recognition, the hidden states are units of sound and the observations are short frames of audio.
- Inverse text normalization (ITN)Inverse text normalization (ITN) is the step that converts a speech recognizer's spoken-form output into written form, so that 'two hundred' becomes 200. It changes how the text reads, not which words the recognizer heard.
- Language model (in speech recognition)In speech recognition, a language model (LM) scores how probable a word sequence is in a language. The recognizer combines that score with the acoustic evidence to choose between words that sound alike, such as homophones.
- LibriSpeechLibriSpeech is a free corpus of about 1,000 hours of read English speech, taken from LibriVox audiobooks, and it is one of the most widely used benchmarks for English speech recognition. Its scores describe clean, read speech, which differs from everyday dictation.
- Mel spectrogramA Mel spectrogram is a grid of sound energy over time, with frequency bands spaced to match human hearing. Many speech recognizers compute its logarithm, the log-Mel spectrogram, and feed it to a neural network instead of the raw waveform.
- Real-time factor (RTF)The real-time factor (RTF) is the time a system takes to process audio divided by the length of that audio. An RTF of 1 or lower means the system keeps up with real-time speech. Some leaderboards report the inverse, RTFx, where higher is faster.
- RNN transducer (RNN-T)An RNN transducer (RNN-T) is a neural speech model that reads audio in order and emits words as it goes, so text can appear while the speaker is still talking. It has an encoder for the sound, a prediction network for the words already written, and a joint network that chooses the next output.
- Speaker diarizationSpeaker diarization is the task of working out who spoke when in a recording, by splitting the audio into turns and grouping turns that come from the same voice. It is separate from transcribing the words, and a transcript gains speaker labels only when diarization is run alongside recognition.
- Voice activity detection (VAD)Voice activity detection (VAD) is a component that decides whether a stretch of audio contains speech. Speech recognizers use it to split audio into turns, skip silence and decide when a speaker has finished.
Pages that use this term
- Speech Recognition Models: How Neural ASR WorksGuide
- How Apple Dictation Works: On-Device, Server and AIGuide
- History of Speech Recognition: From Audrey to WhisperGuide
- Is speech recognition less accurate for some people?Question
- What role do hidden Markov models play in speech recognition?Question
- Automatic speech recognition (ASR)Glossary
- Connectionist temporal classification (CTC)Glossary
- Hidden Markov model (HMM)Glossary
- Language model (in speech recognition)Glossary
- LibriSpeechGlossary
- Mel spectrogramGlossary