Is Whisper a large language model?

No. Whisper is a speech recognition model. Its authors built it as an encoder-decoder Transformer and trained it to turn audio into text, not to answer questions or hold a conversation. It does contain a decoder that writes words one after another, so it shares some machinery with a large language model (LLM). But an LLM in the usual sense is a general-purpose text model that follows instructions, and Whisper is not one. Whether you call it a language model at all depends on how broadly you define the term, so the useful question is what the model is trained to do.

Whisper's architecture is built for transcription

The Whisper authors chose an encoder-decoder Transformer for their speech recognition models, because they judged it "well validated to scale reliably" ¹. The design maps a sound input onto a text output. It is not a chat design.

Its training data reflects the same purpose. The Whisper models were trained on 680,000 hours of multilingual and multitask audio with transcripts ². The training pairs audio with transcripts, which is a different kind of data from the general text an LLM learns from.

Whisper also works on short windows of sound. The paper says its models are trained on 30-second audio chunks and cannot take longer audio as a single input ³. Longer recordings are split and processed piece by piece. That is a design for audio segments, not for open-ended text.

Size does not settle the question either. The largest Whisper model has 1,550 million parameters ⁴. Parameter count tells you how big a model is, not what it was trained to do, so it cannot make Whisper an LLM or rule it out.

Whisper does not follow instructions

The most practical difference is instruction following. OpenAI says Whisper does not follow instructions the way a general-purpose text model does, and that its prompt accepts up to 224 tokens ⁵. That prompt is a hint about spelling, names or style, not a command. You cannot ask Whisper to summarize the recording, answer a question about it, or rewrite it in a formal tone.

This is the test to use. If the model's job is to take an instruction and produce a reply, it behaves like an LLM. If its job is to produce the words that were spoken, it is a recognizer, even when it has a text decoder inside.

Where Whisper and LLMs blur together

Two things make the question confusing. First, Whisper writes text, and it can write fluent text the audio never contained. Researchers call this hallucination, and Whisper's authors list complete hallucination, a transcript entirely unrelated to the actual audio, among its known failure modes ⁶. A model that writes the most likely next words can produce text that sounds right and is wrong, which is a language-model behavior. For the causes, see why Whisper makes up words.

Second, LLMs can now hear. The SALMONN authors describe a model that lets a large language model process and understand general audio inputs directly, with competitive performance on speech tasks such as automatic speech recognition ⁷. That is the reverse design: a language model with an audio encoder added to it. NVIDIA's Canary-Qwen maps its audio encoder's output into the language model's embedding space through a linear projection ⁸, so the audio becomes input the language model can read.

So the line is not "speech model or LLM." It is whether the system starts from a language model that learns to hear, or from a speech model that is built to transcribe. Whisper is the second kind. Canary-Qwen is closer to the first. The speech language model entry explains the newer design in more detail.

It depends on which claim you are testing

• If you mean "does Whisper use language modeling?" Its decoder is trained to predict text conditioned on audio, so it has language knowledge in that narrow sense. Calling that an LLM is a stretch, because the model has no general instruction-following ability.
• If you mean "does Whisper translate?" It can translate speech into English, which is still a speech task, with the output tied to the audio.
• If an app says it uses "Whisper and an LLM," the two are separate steps. Whisper turns the audio into text, and the LLM, if there is one, edits or answers from that text. The LLM is the app's addition, not part of Whisper.

What to do

1. To transcribe speech, use a speech model. Whisper is built for that job, and its output is the words the audio contains.
2. To summarize, rewrite or answer from what you said, send the transcript to a language model. Check the reply against the transcript, because the LLM step can change words too.
3. Ask an app which step uses the LLM. If the answer is unclear, assume the recognizer is Whisper or another speech model and the LLM edits the text afterward. Test the output on your own speech before you rely on it.

For the wider question of how the two fit together, read do LLMs make speech recognition more accurate and can large language models do speech recognition. Our overview of the subject is speech to text and LLMs, and the wider topic is in Voice and AI.