Glossary
Mel spectrogram
A Mel spectrogram is a grid of sound energy over time, with frequency bands spaced to match human hearing. Many speech recognizers compute its logarithm, the log-Mel spectrogram, and feed it to a neural network instead of the raw waveform.
Updated October 17, 20263 cited facts, each traced to its source
Also called log-Mel spectrogram, filter bank features, MFCC.
A Mel spectrogram is a grid of sound energy. Each column is a short slice of time, each row is a band of frequencies, and the bands are spaced on the Mel scale, which spaces them more closely at low pitches, where human hearing is finer. Speech recognizers compute this grid from the waveform before any neural network sees the sound, and many of them take its logarithm, which gives a log-Mel spectrogram.
How a Mel spectrogram is computed
The steps are common across recognizers, even when the numbers change:
- Cut the audio into short, overlapping windows. Whisper uses windows of 25 milliseconds, moved forward every 10 milliseconds 1.
- Measure the energy at each frequency in each window, using a Fourier transform.
- Add up the energy in each Mel band with a filter bank, which is a matrix of weights, one band per row. The Mel scale makes the low bands narrow and the high bands wide.
- Take the logarithm of each band's energy. The log turns loudness into a scale closer to how people perceive it.
The Whisper code uses 80 Mel bands by default and accepts 80 or 128 2. Each band is one row, so one second of audio becomes a grid of 80 rows by 100 columns. The step between windows sets the number of columns: a 10-millisecond stride gives 100 frames for every second of audio 3.
Three names for related features
The terms overlap, and tools use them loosely:
| Feature | What it is | Where you see it |
|---|---|---|
| Mel spectrogram | Sound energy in Mel bands, before the logarithm | Analysis and visualization |
| Log-Mel spectrogram, or filter bank features | The logarithm of those band energies | The input to most neural recognizers, including Whisper |
| MFCC (mel-frequency cepstral coefficients) | A cosine transform of the log-Mel energies, keeping the leading coefficients | Older hidden Markov model systems and some classic pipelines |
The filter bank is the common step. MFCC adds one more transform that decorrelates the bands, which suited older statistical models. Neural networks handle correlated inputs well, so many modern systems skip that step and feed the log-Mel features directly.
Why it matters for dictation
The Mel spectrogram is where the recognizer's view of your voice begins, so its limits set the limits of everything after it:
- Noise shares the same bands. A fan, a keyboard or traffic puts energy into the same frequency bands as your voice. Once they are mixed in the grid, a model has to separate them, so a close microphone and a quiet room often matter more than the choice of model.
- The stride sets the pace. Frames arrive at the stride's rate, so a 10-millisecond stride sets how often new frames can reach a streaming recognizer, and therefore how quickly text can appear. See real-time speech recognition for how latency adds up.
- The features are invisible to you. You never see the grid. You see its result as words, which is why a bad microphone can look like a bad model.
Where it fits in a recognizer
In the classic pipeline, the features feed an acoustic model. In a neural end-to-end system, a network maps the same features to text directly. The guide on how speech recognition works walks through each stage.
Related terms
- Automatic speech recognition (ASR): the full task that these features feed.
- Acoustic model: the model that reads the features in classic systems.
- Voice activity detection (VAD): often uses the same frame energies to decide when speech starts and stops.
- LibriSpeech: the corpus whose audio is usually converted into these features for testing.
Guides that cover it
Evidence
Every number on this page, with where it comes from. Open a fact to see the exact words in its source.
- 1.
- 2.
- 3.With a 10-millisecond stride, Whisper's spectrogram has 100 frames for every second of audio.
Calculated from 1 other fact
Related terms
- Speech Recognition: What It Is and How Well It WorksThe topic this term belongs to
- Acoustic modelAn acoustic model is the part of a speech recognizer that scores how well a short window of audio matches the sounds of speech. Since about 2012 most acoustic models are deep neural networks, and their training data strongly affects accuracy.
- Automatic speech recognition (ASR)Automatic speech recognition (ASR) is software that converts spoken audio into written words. Modern systems are neural networks trained on large amounts of transcribed speech, and they run in the cloud or on the device.
- ConformerA Conformer is a neural encoder for speech recognition that combines self-attention, which sees the whole input, with convolution, which captures local sound patterns. FastConformer is NVIDIA's faster variant of the same design, used in streaming models.
- Connectionist temporal classification (CTC)Connectionist temporal classification (CTC) is a way to train a neural network to turn audio into text without knowing in advance which moment of sound matches which letter. The network labels every short slice of audio with a character or a blank, and the blanks and repeated labels are then removed to leave the transcript.
- Hidden Markov model (HMM)A hidden Markov model (HMM) is a statistical model in which a sequence of hidden states produces the observations you can measure, and each state changes to the next with set probabilities. In speech recognition, the hidden states are units of sound and the observations are short frames of audio.
- Inverse text normalization (ITN)Inverse text normalization (ITN) is the step that converts a speech recognizer's spoken-form output into written form, so that 'two hundred' becomes 200. It changes how the text reads, not which words the recognizer heard.
- Language model (in speech recognition)In speech recognition, a language model (LM) scores how probable a word sequence is in a language. The recognizer combines that score with the acoustic evidence to choose between words that sound alike, such as homophones.
- LibriSpeechLibriSpeech is a free corpus of about 1,000 hours of read English speech, taken from LibriVox audiobooks, and it is one of the most widely used benchmarks for English speech recognition. Its scores describe clean, read speech, which differs from everyday dictation.
- Real-time factor (RTF)The real-time factor (RTF) is the time a system takes to process audio divided by the length of that audio. An RTF of 1 or lower means the system keeps up with real-time speech. Some leaderboards report the inverse, RTFx, where higher is faster.
- RNN transducer (RNN-T)An RNN transducer (RNN-T) is a neural speech model that reads audio in order and emits words as it goes, so text can appear while the speaker is still talking. It has an encoder for the sound, a prediction network for the words already written, and a joint network that chooses the next output.
- Speaker diarizationSpeaker diarization is the task of working out who spoke when in a recording, by splitting the audio into turns and grouping turns that come from the same voice. It is separate from transcribing the words, and a transcript gains speaker labels only when diarization is run alongside recognition.
- Voice activity detection (VAD)Voice activity detection (VAD) is a component that decides whether a stretch of audio contains speech. Speech recognizers use it to split audio into turns, skip silence and decide when a speaker has finished.