Glossary
Speaker diarization
Speaker diarization is the task of working out who spoke when in a recording, by splitting the audio into turns and grouping turns that come from the same voice. It is separate from transcribing the words, and a transcript gains speaker labels only when diarization is run alongside recognition.
Updated October 17, 20264 cited facts, each traced to its source
Also called diarization, diarisation.
Speaker diarization answers the question "who spoke when?" in a recording. Google's documentation describes it as detecting when speakers change and labeling each voice detected in the audio with a number 1. It does not produce the words. Speech recognition writes what was said, and diarization adds which speaker said it.
Diarization groups the audio into turns, then labels the voices
A typical system works in four steps:
- Find the speech. A voice activity detector separates speech from silence and noise.
- Split into segments. The speech is cut at pauses and at likely speaker changes.
- Describe each segment's voice. The system computes a numerical summary of how each segment sounds, often called a speaker embedding.
- Cluster the voices. Segments whose summaries are close are grouped, and each group becomes one speaker.
For a two-person interview, the output might read "Speaker 1: Thanks for joining us. Speaker 2: Happy to be here." The labels are arbitrary numbers, so the system does not know the speakers' names unless they are supplied separately.
Joint models attach speaker labels to the words
Some systems run diarization as a separate stage after recognition. Others predict words and speakers together. Shafey and colleagues at Google described a joint model based on sequence transduction, and reported that it improved word-level diarization error rate from 15.8 percent to 2.2 percent, against a conventional baseline, on physician-patient conversations 2.
That score counts a word as an error when it is recognized correctly but tagged with the wrong speaker, or when it is a substitution with a wrong speaker label 3. The same authors note that this measure must be read together with word error rate, because it does not count deleted or inserted words.
Why it matters for dictation
If you dictate alone, you do not need diarization, because there is only one speaker. It matters for meeting notes, interviews, lectures and any recording with several voices, where a transcript without speaker labels is hard to follow.
Diarization is harder when people talk over one another, when voices sound alike, when the room is noisy, or when a speaker is brief. Language support is also uneven. Google's documentation warns that speaker diarization is not available for every language and points to its language support page 4, so check that page before you rely on speaker labels in a language you need.
Related terms
- Automatic speech recognition (ASR): the word-by-word transcription that diarization is added to.
- Voice activity detection (VAD): the step that finds speech before speakers are separated.
- Word error rate (WER): the score that must be read alongside diarization error rate.
Guides that cover it
Evidence
Every number on this page, with where it comes from. Open a fact to see the exact words in its source.
- 1.
- 2.
- 3.
- 4.
Related terms
- Speech Recognition: What It Is and How Well It WorksThe topic this term belongs to
- Acoustic modelAn acoustic model is the part of a speech recognizer that scores how well a short window of audio matches the sounds of speech. Since about 2012 most acoustic models are deep neural networks, and their training data strongly affects accuracy.
- Automatic speech recognition (ASR)Automatic speech recognition (ASR) is software that converts spoken audio into written words. Modern systems are neural networks trained on large amounts of transcribed speech, and they run in the cloud or on the device.
- ConformerA Conformer is a neural encoder for speech recognition that combines self-attention, which sees the whole input, with convolution, which captures local sound patterns. FastConformer is NVIDIA's faster variant of the same design, used in streaming models.
- Connectionist temporal classification (CTC)Connectionist temporal classification (CTC) is a way to train a neural network to turn audio into text without knowing in advance which moment of sound matches which letter. The network labels every short slice of audio with a character or a blank, and the blanks and repeated labels are then removed to leave the transcript.
- Hidden Markov model (HMM)A hidden Markov model (HMM) is a statistical model in which a sequence of hidden states produces the observations you can measure, and each state changes to the next with set probabilities. In speech recognition, the hidden states are units of sound and the observations are short frames of audio.
- Inverse text normalization (ITN)Inverse text normalization (ITN) is the step that converts a speech recognizer's spoken-form output into written form, so that 'two hundred' becomes 200. It changes how the text reads, not which words the recognizer heard.
- Language model (in speech recognition)In speech recognition, a language model (LM) scores how probable a word sequence is in a language. The recognizer combines that score with the acoustic evidence to choose between words that sound alike, such as homophones.
- LibriSpeechLibriSpeech is a free corpus of about 1,000 hours of read English speech, taken from LibriVox audiobooks, and it is one of the most widely used benchmarks for English speech recognition. Its scores describe clean, read speech, which differs from everyday dictation.
- Mel spectrogramA Mel spectrogram is a grid of sound energy over time, with frequency bands spaced to match human hearing. Many speech recognizers compute its logarithm, the log-Mel spectrogram, and feed it to a neural network instead of the raw waveform.
- Real-time factor (RTF)The real-time factor (RTF) is the time a system takes to process audio divided by the length of that audio. An RTF of 1 or lower means the system keeps up with real-time speech. Some leaderboards report the inverse, RTFx, where higher is faster.
- RNN transducer (RNN-T)An RNN transducer (RNN-T) is a neural speech model that reads audio in order and emits words as it goes, so text can appear while the speaker is still talking. It has an encoder for the sound, a prediction network for the words already written, and a joint network that chooses the next output.
- Voice activity detection (VAD)Voice activity detection (VAD) is a component that decides whether a stretch of audio contains speech. Speech recognizers use it to split audio into turns, skip silence and decide when a speaker has finished.
Pages that use this term
- Speech Recognition: What It Is and How Well It WorksTopic
- Transcription: How to Turn Audio and Video Into TextTopic
- Meeting Transcription: How It Works and What to UseGuide
- What Is Whisper AI? OpenAI's Speech Model ExplainedGuide
- Speech-to-Text APIs Compared: Price, Accuracy, SpeedGuide
- Video to Text: Transcribe Videos and YouTubeGuide
- Word Error Rate (WER): How to Calculate and Read ItGuide
- Voice Memo to Text: Built-In and Free OptionsGuide
- Transcription Software: How to Choose OneGuide
- How to Run Whisper Locally on Your Own ComputerGuide
- Transcribe Audio With Whisper, ChatGPT or GeminiGuide
- How to Transcribe an Interview: Audio to TextGuide
- Verbatim vs Clean Verbatim TranscriptionGuide
- Timestamps and Speaker Labels in TranscriptsGuide
- Can Dragon transcribe audio files?Question
- Can Gemini transcribe audio?Question
- Can Microsoft Word transcribe audio files?Question
- Can Whisper identify different speakers?Question
- Can you use AI transcription for research interviews?Question
- Does Gemini have a speech-to-text API?Question
- Does speech to text collect biometric data?Question
- How do you find and download a Zoom transcript?Question
- How do you prepare interview transcripts for qualitative analysis?Question
- How do you transcribe a meeting on a Mac?Question