Can Whisper identify different speakers?

No. Whisper turns audio into a stream of text, and it does not tag each part with the person who said it. OpenAI's speech-to-text guide says to use its separate hosted model, gpt-4o-transcribe-diarize, only when you need to identify who speaks during a recording ¹. If you need to know who said what, you add a second job called speaker diarization, and you can get it from an open-source tool that runs beside Whisper or from a hosted speech API that includes it.

Whisper's transcript has no speaker tags, so diarization is a separate job

Speaker diarization answers the question "who spoke when." It finds the points where the voice changes, groups the audio by voice, and joins those groups to the words. The labels it produces are usually numbers such as speaker 1 and speaker 2, not names. Matching a number to a person is a separate step that someone has to do or confirm.

This split matters when you choose a tool. Transcription and diarization are separate models, so a transcript can be accurate and still assign the wrong speaker to the words. A tool that advertises both needs to be tested on both.

Our guide to what Whisper AI is explains the model's other limits, and the speech recognition overview places diarization among the related tasks.

You can add speaker labels with an open-source tool or a hosted API

Two routes work, and they differ in who runs the software.

WhisperX is a third-party project built on Whisper. Its README describes speaker diarization as part of its pipeline ². Because you run it yourself, you also own the setup, the model downloads and the updates. Read its installation notes before you plan a project around it.

Google's diarization detects when speakers change and labels each voice by number ³. Google also says to review its language support page to see whether the feature is available for your language ⁴, so check that before you build around it.

ElevenLabs says its Scribe API returns structured JSON transcripts with speaker diarization and word-level timestamps ⁵. That describes the feature, not how accurate its speaker labels are. Treat it as a claim to test.

Word diarization error rate measures the speaker errors you care about

The usual way to score speaker labels is word diarization error rate (WDER). It counts the words that carry the wrong speaker tag. The 2019 paper that defines it counts two kinds of word: a correctly recognized word with the wrong speaker, and a substituted word with the wrong speaker ⁶.

The same paper reports a joint model that transcribes and assigns speakers in one network. On the authors' own test corpus, word diarization error rate fell from 15.8 percent to 2.2 percent ⁷. Those numbers come from one corpus, so read them as evidence of what the approach can do, not as a score for any product you can buy.

WDER does not measure missed or extra words, so the paper says it must be used together with word error rate (WER) ⁸. A system can keep its speaker errors low by dropping hard words, so look at both numbers.

What to do if you need speaker labels from Whisper

1. Decide what you need. Speaker numbers tell you who spoke when. A name needs a mapping from number to person, which you supply or confirm.
2. Pick a test recording. Use 10 to 20 minutes of audio with the same number of speakers, the same overlap and the same room you will use in practice. Mark the speaker turns by hand so you have a reference.
3. Check language support first. Confirm that the diarization feature covers your language before you spend time on the test ⁴.
4. Score both measures. Compute WER for the words and WDER for the speaker tags, and compare the two numbers between routes.
5. Review the labels by hand if the transcript is used for records, legal work or anything where a wrong speaker changes the meaning.

It depends on how many people talk and how clean the audio is

Two people taking turns on a phone line is the easiest case. A meeting with several people who talk over each other is harder, because the diarizer has to decide where one voice ends and the next begins. Noisy rooms and far-away microphones make both the words and the speaker tags less reliable, so the same tool can perform very differently across these cases.

If your audio is a single speaker, you do not need diarization at all. Whisper's transcript is enough, and a separate step only adds setup and cost.

For more on the choice between hosted and self-run tools, see our comparison of speech-to-text APIs. If you are weighing Whisper against a cloud service on the words alone, read whether Whisper is more accurate than Google Speech-to-Text. For the free model itself, whether Whisper AI is free answers the cost question.