Speech to Text and LLMs: What Language Models Change

A large language model (LLM) can change speech to text in three ways. It can edit the text a recognizer has already produced, which is called post-correction. It can work as a speech-aware model that reads the audio and writes the words itself. Or it can take your spoken words as a prompt to act on, rather than as text to type. The first two can lower the word error rate on hard audio. Both can also change words that were already right, and both add delay and cost. Whisper, the open recognizer most people mean by the name, is not an LLM, although it is a Transformer trained on speech.

Speech recognition always had a language model, LLMs make it far larger

Recognizers have long used a language model to choose between words that sound alike. The classic French case is et (and) and est (is). A 2008 study of these homophones notes that ASR systems rely on language model weights to tell them apart ¹. The real question is therefore not whether a recognizer has language knowledge. It is how much of that knowledge sits inside the recognizer, and whether a second, much larger model is added on top.

Whisper is the clearest example of a recognizer that is not an LLM. Its authors chose an encoder-decoder Transformer as the architecture, because it has been well validated to scale reliably ². OpenAI says Whisper does not follow instructions the way a general-purpose text model does, and that its prompt accepts up to 224 tokens ³. Our full answer is on the page Is Whisper an LLM?.

When a large language model joins a recognizer, three things change. The model knows far more about how words follow one another, which helps it choose between words that sound alike. It writes words rather than choosing among them, so it can produce words the recognizer never proposed. And it is a second model, which adds its own delay, its own cost and its own way of going wrong. Each of these changes is a reason to test the combination, not a reason to assume it is better.

LLM post-correction lowers error rates on hard audio, within limits

Post-correction is the simplest design. A recognizer produces its best guess, and an LLM rewrites that guess. On hard audio the gains can be large. In the HyPoradise benchmark, a correction model called H2T-LoRA reduced relative word error rate by 77.1 percent on the ATIS test set and by 55.1 percent on WSJ ⁴. Because an LLM generates text rather than only choosing among the recognizer's candidates, it can supply words that were missing from the candidate list ⁵. Researchers call this generative error correction. In a typical setup the recognizer does not hand over a single transcript. It passes a short list of its best alternatives, and the language model uses the context of the whole list to decide which words fit. That is why a rewrite can do better than simply picking the best candidate, and also why it can go wrong, since it is no longer limited to what the recognizer proposed.

The limit shows up when you split the audio by how well the recognizer did. A 2026 study of dysarthric speech, which is speech affected by a motor disorder, reports an overall word error rate of 13.63 percent for the recognizer. That average hides a split: 0 percent on the sentences it transcribed without error, and 21.98 percent on the sentences that had errors ⁶. Correction only has room to help in the second group. After supervised fine-tuning, the error rate on the erroneous sentences fell from 22 to 19 percent across all the models tested ⁷. The sentences that were already right had no errors for correction to fix.

Two limits follow. Correction helps only where the recognizer makes errors, so on clean audio there is little to gain and some risk of damaging correct text. The results above depend on labeled data: the HyPoradise benchmark contains more than 334,000 pairs of recognizer output and correct text ⁸, which most people do not have. For a broader view, see do LLMs improve speech recognition accuracy.

Speech-aware LLMs transcribe audio directly

A speech-aware model skips the text draft. It adds an audio encoder that turns sound into vectors the language model can process, so the model hears the audio itself. The SALMONN authors describe a model that lets a large language model process and understand general audio inputs directly, with competitive performance on speech tasks such as automatic speech recognition ⁹. This family is often called a speech language model.

The top path in the diagram is post-correction. The lower path is a speech-aware model, which needs no draft. NVIDIA's Canary-Qwen is one published design. Its model card says the audio encoder's output is mapped into the language model's embedding space through a linear projection ¹⁰. Mistral describes its Voxtral models as trained to understand both spoken audio and text documents ¹¹. OpenAI's speech-to-text guide does not say what architecture its transcribe models use, so this guide does not classify them. For the wider background, see can LLMs do speech recognition.

The appeal of the speech-aware design is that the model can use what a text draft throws away, such as hesitation, emphasis or a word that was hard to hear. The benchmarks in this guide do not test that claim directly. Treat it as a hypothesis until you measure it on your own speech, and remember that a speech-aware model is still a model that can get things wrong.

On the English leaderboard, the speech-aware design is competitive with the best conventional recognizers. On the Open ASR Leaderboard's English short-form track, Canary-Qwen 2.5B had an average word error rate of 5.63 percent ¹². Whisper Large v3 had 7.44 percent ¹³, which makes the Canary-Qwen figure about 24 percent lower ¹⁴. NVIDIA's Parakeet TDT 0.6B v2 reached 6.05 percent ¹⁵.

The leaderboard has a limit you should know. Its normalized scoring removes punctuation, casing and fillers before it counts errors ¹⁶, so it measures the words, not how they are formatted. A low leaderboard number is a good signal for a model, not a guarantee for your own voice and microphone.

LLM transcription spells and formats better but can invent text

Speech-aware models do not win in every language. In the Voxtral paper's FLEURS comparison, GPT-4o mini Transcribe had a Hindi word error rate 57 percent lower than Whisper large-v3's ¹⁷. In Arabic the order changes. Gemini 2.5 Flash had a word error rate 64 percent above Whisper's ¹⁸. These come from one team's comparison on one benchmark, so the ranking depends on the language and should be tested on yours.

Formatting is the part an LLM is most likely to improve. Word error rate, as the leaderboard scores it, does not reward punctuation, paragraph breaks or the removal of fillers. Those are the things a language model can add to a raw transcript, and they are the things a reader notices first. You should judge them by reading the output, not by a score.

A single score also treats every error alike, and they are not alike. A wrong filler word is a small annoyance. A wrong name in a client email, a wrong number in a dosage, or a homophone such as their for there can matter far more. When you run the test below, count those important words on their own, because an average can hide the one error that matters.

The larger risk is invented text. Whisper's authors list complete hallucination, a transcript unrelated to the actual audio, among its known failure modes ¹⁹. In a 2024 study, about 1 percent of Whisper transcriptions contained entire phrases or sentences that did not exist in the audio ²⁰. That study tested Whisper as it stood in 2023 ²¹, so its rate does not describe today's models. A model that writes fluent text can also write text the speaker never said, and the output looks just as confident.

A rewrite can also change words that were right. Pu and colleagues report that LLMs make many unnecessary changes to the input transcription ²². Yamashita and colleagues argue that existing LLM-based correction relies primarily on textual information and neglects phonetic cues, which leads to over-correction, meaning changes to words that were already right ²³. For the everyday version of this trade-off, see AI cleanup for dictation.

Because none of these numbers is about your voice, test the step yourself before you rely on it:

1. Record 20 to 30 sentences of your own dictation, in the room and with the microphone you will use.
2. Transcribe them with the recognizer alone, and save the text.
3. Run the same audio through the LLM step, with the prompt you would use in daily work, and save that text.
4. For each sentence, count the words the LLM changed where the recognizer was already right, and the words it added that you did not say.
5. Keep the LLM step only if the errors it fixes clearly outnumber the new errors it introduces.

Delay and cost decide whether an LLM belongs in a live dictation pipeline

For live dictation, delay comes first. The useful measure is time to first token: the wait between sending a request and the first output token arriving. An LLM step placed after the recognizer adds that wait on top of the recognizer's own delay. VoiceAgentBench measured both designs on one H100 GPU. The end-to-end speech model Qwen2.5-Omni 7B had a 90th percentile time to first token of about 40 milliseconds ²⁴. A pipeline of Whisper large v3 and the Qwen3 8B language model had about 800 milliseconds under the same conditions ²⁵, which is about 20 times as long ²⁶.

Streaming recognizers give a different trade-off, with partial text that appears while you speak and is then revised. In Artificial Analysis's 2026 streaming benchmark, OpenAI's hosted GPT Realtime Whisper went from a 7.5 percent word error rate on its first partial transcript to 5.1 percent on the final transcript ²⁷. So waiting for the final text buys accuracy at the cost of time. An LLM step that waits for finished text adds to that wait.

Cost depends on the minutes of audio you send, not on the words you get back. OpenAI's list price puts one hour of audio at about $0.18 with gpt-4o-mini-transcribe ²⁸ and about $0.36 with gpt-4o-transcribe ²⁹. Live transcription costs more: at $0.017 per minute ³⁰, an hour of gpt-realtime-whisper comes to about $1.02 ³¹. These are transcription prices only. A cleanup pass that sends the text to a model adds its own token cost, and prices change, so check the price list on the day you decide.

Three questions settle most decisions:

1. Must the text appear before you finish speaking? If yes, a streaming recognizer is the base, and any LLM step has to be fast enough to fit inside the wait you accept.
2. Would an invented or over-edited word be costly? If yes, keep the recognizer's text as the record, and use the LLM only as a visible suggestion you can reject.
3. Are you willing to pay per minute of audio, and wait for a second model? If not, built-in dictation or an on-device recognizer is the simpler choice.

The choice follows from where the words go. If text must appear as you speak, a streaming recognizer is the usual base, and an LLM edit fits best on finished text, where its delay matters less. If plain text is enough, built-in dictation needs no LLM at all, as covered in speech to text without AI. For agents that answer rather than type, the delay is part of the product, as covered in AI voice agents. The question of which speech to text to pair with an LLM is covered in the best speech to text for LLM apps.

Yap, which we make, is an on-device dictation app for Mac and Windows. Its speech recognition runs on your device, so the per-minute cloud prices above do not apply to it. You can download it and try it on your own speech.

Other guides in this series cover AI dictation and how it works, whether speech to text counts as AI, spoken prompts for AI chatbots, and the cost of AI dictation. Our overview of the whole subject is Voice and AI. For the recognizers themselves, see speech recognition, and for why a model can produce invented text, see why Whisper hallucinates.