Glossary
Connectionist temporal classification (CTC)
Connectionist temporal classification (CTC) is a way to train a neural network to turn audio into text without knowing in advance which moment of sound matches which letter. The network labels every short slice of audio with a character or a blank, and the blanks and repeated labels are then removed to leave the transcript.
Updated October 17, 20261 cited fact, each traced to its source
Also called CTC.
Connectionist temporal classification (CTC) is a training method for neural networks that map audio to text when nobody has marked where each letter sits in the recording. Graves and colleagues introduced it in 2006 because earlier recurrent networks needed pre-segmented training data and post-processing to produce label sequences 1. CTC removes that need, so a network can learn from an audio clip and its written transcript alone.
CTC labels every slice of audio, then collapses the labels
A speech model cuts the audio into a long series of short frames. At each frame, the network gives a probability for every character in the alphabet and for one extra symbol called the blank. The network does not have to say where a letter begins or ends.
To turn those frame labels into text, the decoder applies two rules. It first merges runs of the same symbol into one, then deletes every blank. For example, the frame labels h, h, blank, e, e, blank, l, l, blank, l, o, o collapse to h, blank, e, blank, l, blank, l, o. Removing the blanks leaves hello.
The blank does real work. It keeps the two l's in "hello" apart. Without it, the run l, l, l would merge into a single l and the decoder would produce "helo".
Training adds up every alignment that spells the right words
Many different frame sequences collapse to the same transcript, and CTC does not need to know which one is correct. During training, it adds up the probability of all of them and tries to make the total probability of the correct transcript as high as possible. The timing is learned implicitly rather than given as a label.
Why it matters for dictation
CTC is cheap to run. Each frame is labeled with little reference to the others, so the output can be produced as audio arrives. The cost is that each label has limited knowledge of the words before it. Two words that sound alike, such as "their" and "there", can be decided without much context, so CTC models usually rely on a language model to choose the right spelling.
That limit is why later models, including the RNN transducer, were built to condition each output on the labels already written. When a dictation product says it is fast and light, CTC is often part of the design, but the accuracy you see depends on the whole system, not on the training method alone.
Related terms
- RNN transducer (RNN-T): an extension that also conditions each output on the labels already emitted.
- Acoustic model: the part of a recognizer that scores sound against speech units.
- Language model: the part that scores which word sequences are likely.
- Hidden Markov model (HMM): the earlier approach that CTC and transducers replaced in many systems.
Guides that cover it
Evidence
Every number on this page, with where it comes from. Open a fact to see the exact words in its source.
- 1.Graves and colleagues wrote that recurrent networks needed pre-segmented training data and post-processing to produce label sequences, and presented a method that trains them to label unsegmented sequences directly.
Alex Graves et al., Proceedings of the 23rd International Conference on Machine Learning (ICML 2006), pp. 369-376, 2006 · Original
Related terms
- Speech Recognition: What It Is and How Well It WorksThe topic this term belongs to
- Acoustic modelAn acoustic model is the part of a speech recognizer that scores how well a short window of audio matches the sounds of speech. Since about 2012 most acoustic models are deep neural networks, and their training data strongly affects accuracy.
- Automatic speech recognition (ASR)Automatic speech recognition (ASR) is software that converts spoken audio into written words. Modern systems are neural networks trained on large amounts of transcribed speech, and they run in the cloud or on the device.
- ConformerA Conformer is a neural encoder for speech recognition that combines self-attention, which sees the whole input, with convolution, which captures local sound patterns. FastConformer is NVIDIA's faster variant of the same design, used in streaming models.
- Hidden Markov model (HMM)A hidden Markov model (HMM) is a statistical model in which a sequence of hidden states produces the observations you can measure, and each state changes to the next with set probabilities. In speech recognition, the hidden states are units of sound and the observations are short frames of audio.
- Inverse text normalization (ITN)Inverse text normalization (ITN) is the step that converts a speech recognizer's spoken-form output into written form, so that 'two hundred' becomes 200. It changes how the text reads, not which words the recognizer heard.
- Language model (in speech recognition)In speech recognition, a language model (LM) scores how probable a word sequence is in a language. The recognizer combines that score with the acoustic evidence to choose between words that sound alike, such as homophones.
- LibriSpeechLibriSpeech is a free corpus of about 1,000 hours of read English speech, taken from LibriVox audiobooks, and it is one of the most widely used benchmarks for English speech recognition. Its scores describe clean, read speech, which differs from everyday dictation.
- Mel spectrogramA Mel spectrogram is a grid of sound energy over time, with frequency bands spaced to match human hearing. Many speech recognizers compute its logarithm, the log-Mel spectrogram, and feed it to a neural network instead of the raw waveform.
- Real-time factor (RTF)The real-time factor (RTF) is the time a system takes to process audio divided by the length of that audio. An RTF of 1 or lower means the system keeps up with real-time speech. Some leaderboards report the inverse, RTFx, where higher is faster.
- RNN transducer (RNN-T)An RNN transducer (RNN-T) is a neural speech model that reads audio in order and emits words as it goes, so text can appear while the speaker is still talking. It has an encoder for the sound, a prediction network for the words already written, and a joint network that chooses the next output.
- Speaker diarizationSpeaker diarization is the task of working out who spoke when in a recording, by splitting the audio into turns and grouping turns that come from the same voice. It is separate from transcribing the words, and a transcript gains speaker labels only when diarization is run alongside recognition.
- Voice activity detection (VAD)Voice activity detection (VAD) is a component that decides whether a stretch of audio contains speech. Speech recognizers use it to split audio into turns, skip silence and decide when a speaker has finished.
Pages that use this term
- Speech Recognition: What It Is and How Well It WorksTopic
- Speech Recognition Models: How Neural ASR WorksGuide
- Real-Time Speech Recognition: Streaming and LatencyGuide
- History of Speech Recognition: From Audrey to WhisperGuide
- How are speech recognition models trained?Question
- What are the most important speech recognition research papers?Question
- What role do hidden Markov models play in speech recognition?Question
- Who invented speech recognition?Question
- Acoustic modelGlossary
- Automatic speech recognition (ASR)Glossary
- ConformerGlossary
- Hidden Markov model (HMM)Glossary
- RNN transducer (RNN-T)Glossary
- wav2vec 2.0Glossary