Glossary
wav2vec 2.0
wav2vec 2.0 is a speech model that first learns the structure of speech from large amounts of unlabeled audio, then is fine-tuned on a much smaller set of transcribed recordings. Because transcribed speech is expensive to make, this approach helps when labeled data is scarce.
Updated October 17, 20262 cited facts, each traced to its source
Also called wav2vec.
wav2vec 2.0 is a speech model that learns from audio that has no transcript, then uses a small amount of labeled speech to learn to write text. The method is called self-supervised learning, because the training signal comes from the audio itself. The model is described by Baevski and colleagues in 2020 1.
Pre-training uses unlabeled audio, fine-tuning uses a little text
The training has two stages:
- Pre-training. The model hides short spans of its internal speech representations and learns to pick the correct representation for each hidden span from a set of distractors. This is a contrastive objective, and it requires no written transcripts.
- Fine-tuning. A small output head is added and trained on transcribed speech, usually with a CTC loss over characters.
The pre-training stage is where most of the learning happens, and it can use very large collections of recordings.
Ten minutes of labels can still give a usable model
The paper's most striking result uses very little labeled data. With ten minutes of labeled audio and pre-training on 53,000 hours of unlabeled audio, wav2vec 2.0 still reached 4.8 percent word error rate on LibriSpeech clean and 8.2 percent on the harder other set 2. Those numbers are for clean, read English, so treat them as a measure of what the method can do, not as the error rate you will get on everyday speech.
With all of the labeled LibriSpeech training data, the same model reached 1.8 percent on the clean test set and 3.3 percent on the other set 1. The gap between those two results shows how much the labeled data still matters.
Why it matters for dictation
Dictation quality depends on training data. Many languages, dialects and accents have much less transcribed audio than English. A method that learns from unlabeled audio lowers the number of transcribed hours a team needs before a model is usable, which is why it is attractive for languages with little labeled speech.
It is also useful when a voice differs from the typical training set, such as speech affected by a condition that changes how people talk. In those cases, a small set of labeled recordings from the target speaker can adapt a pre-trained model more cheaply than training one from scratch.
Related terms
- Connectionist temporal classification (CTC): the loss used when fine-tuning for text output.
- LibriSpeech: the benchmark corpus used in the results above.
- Word error rate (WER): the score these results are reported in.
- Automatic speech recognition (ASR): the field this model belongs to.
Guides that cover it
Evidence
Every number on this page, with where it comes from. Open a fact to see the exact words in its source.
- 1.
- 2.
Related terms
- Speech Recognition: What It Is and How Well It WorksThe topic this term belongs to
- Acoustic modelAn acoustic model is the part of a speech recognizer that scores how well a short window of audio matches the sounds of speech. Since about 2012 most acoustic models are deep neural networks, and their training data strongly affects accuracy.
- Automatic speech recognition (ASR)Automatic speech recognition (ASR) is software that converts spoken audio into written words. Modern systems are neural networks trained on large amounts of transcribed speech, and they run in the cloud or on the device.
- ConformerA Conformer is a neural encoder for speech recognition that combines self-attention, which sees the whole input, with convolution, which captures local sound patterns. FastConformer is NVIDIA's faster variant of the same design, used in streaming models.
- Connectionist temporal classification (CTC)Connectionist temporal classification (CTC) is a way to train a neural network to turn audio into text without knowing in advance which moment of sound matches which letter. The network labels every short slice of audio with a character or a blank, and the blanks and repeated labels are then removed to leave the transcript.
- Hidden Markov model (HMM)A hidden Markov model (HMM) is a statistical model in which a sequence of hidden states produces the observations you can measure, and each state changes to the next with set probabilities. In speech recognition, the hidden states are units of sound and the observations are short frames of audio.
- Inverse text normalization (ITN)Inverse text normalization (ITN) is the step that converts a speech recognizer's spoken-form output into written form, so that 'two hundred' becomes 200. It changes how the text reads, not which words the recognizer heard.
- Language model (in speech recognition)In speech recognition, a language model (LM) scores how probable a word sequence is in a language. The recognizer combines that score with the acoustic evidence to choose between words that sound alike, such as homophones.
- LibriSpeechLibriSpeech is a free corpus of about 1,000 hours of read English speech, taken from LibriVox audiobooks, and it is one of the most widely used benchmarks for English speech recognition. Its scores describe clean, read speech, which differs from everyday dictation.
- Mel spectrogramA Mel spectrogram is a grid of sound energy over time, with frequency bands spaced to match human hearing. Many speech recognizers compute its logarithm, the log-Mel spectrogram, and feed it to a neural network instead of the raw waveform.
- Real-time factor (RTF)The real-time factor (RTF) is the time a system takes to process audio divided by the length of that audio. An RTF of 1 or lower means the system keeps up with real-time speech. Some leaderboards report the inverse, RTFx, where higher is faster.
- RNN transducer (RNN-T)An RNN transducer (RNN-T) is a neural speech model that reads audio in order and emits words as it goes, so text can appear while the speaker is still talking. It has an encoder for the sound, a prediction network for the words already written, and a joint network that chooses the next output.
- Speaker diarizationSpeaker diarization is the task of working out who spoke when in a recording, by splitting the audio into turns and grouping turns that come from the same voice. It is separate from transcribing the words, and a transcript gains speaker labels only when diarization is run alongside recognition.