Do LLMs make speech recognition more accurate?

Sometimes. A large language model (LLM) can make speech recognition more accurate, but only in a narrow place. It helps when it fixes errors the recognizer has already made, on audio that is hard to recognize. It cannot improve sentences the recognizer already got right, and the same step can change correct words into wrong ones. Whether an LLM helps your dictation depends on how often your recognizer fails, and on how the LLM is used.

Correction helps most where the recognizer fails

The clearest gains come from post-correction, where an LLM rewrites the recognizer's draft. In the HyPoradise benchmark, a correction model called H2T-LoRA reduced relative word error rate by 77.1 percent on the ATIS test set and by 55.1 percent on WSJ ¹. Those are research results on benchmark data, so your own gain may differ.

Part of the reason is that an LLM generates text rather than only choosing among the recognizer's candidates. It can supply words that were missing from the candidate list ². That generative step may account for part of the gain, and it is also a risk, because a generated word is not guaranteed to be one the speaker said. This technique is called generative error correction.

The gain sits in the sentences with errors

The effect is concentrated, and the split matters more than the average. A 2026 study of dysarthric speech, which is speech affected by a motor disorder, reports an overall word error rate of 13.63 percent for the recognizer alone ³. The average hides the pattern: sentences transcribed without error scored 0 percent, and sentences with errors scored 21.98 percent ³. Correction only has room to help in the second group.

After supervised fine-tuning, the error rate on the erroneous sentences fell from 22 to 19 percent, across all the models the authors tested ⁴. That is a real gain on the sentences that needed it, and no gain on the rest. If your recognizer is already accurate on your speech, the room for improvement is small.

Correction can over-edit words that were right

The main risk is over-correction. Pu and colleagues found that when an LLM corrects speech recognition output, it makes many unnecessary changes to the input transcription ⁵. Yamashita and colleagues say that existing LLM-based correction methods rely mainly on the text and neglect phonetic cues, which leads to over-correction ⁶.

Their fix shows what good correction looks like. The authors add the recognizer's N-best hypotheses, the ranked alternative transcripts, and phonetic context to the language model to mitigate over-correction ⁷. The lesson from that approach is that a correction step may do better when it sees how the audio sounded and what else the recognizer considered, not just the final text.

A model that hears the audio is a different route

The other design skips the draft. A speech-aware model reads the audio directly. On the Open ASR Leaderboard's English short-form track, NVIDIA's Canary-Qwen 2.5B had an average word error rate about 24 percent lower than OpenAI's Whisper Large v3 ⁸. That is a strong result on benchmark audio. It is still a leaderboard, built from test sets that may not match your microphone, your room or your vocabulary. The speech language model entry covers this design in more detail.

It depends on the audio, the design and the cost

• Clean, accurate recognition: an LLM step adds little, and it can add over-editing. Keep the recognizer's text as it is.
• Hard audio with many errors: correction has the most room to help, provided you check its changes.
• Live dictation: correction waits for finished text, so it adds delay. Whether that delay is acceptable depends on your app, as covered in which speech to text to pair with an LLM.
• Formatting and cleanup: an LLM can add punctuation and remove fillers, which benchmarks such as word error rate do not score. Judge that by reading the output, not by a number.

What to do

1. Measure your recognizer first. Record 20 to 30 of your own sentences and check how many words it gets wrong. If it is already accurate, an LLM step has little to fix.
2. Run the LLM step on the same audio. Save both outputs, so you can compare the recognizer's text with the corrected text.
3. Count the new errors. Note every word the LLM changed where the recognizer was right, and every word it added that you did not say.
4. Keep the step only if it fixes more than it breaks. A step that fixes few errors and changes correct words is not worth its delay.
5. Keep the raw transcript. Use the LLM output as a suggestion you can reject, so an over-edit does not replace the record of what you said.

The guide speech to text and LLMs gives the full test. For how models that hear audio differ from text-only models, read can large language models do speech recognition. For why an LLM can be a poor fit for some error types, see why Whisper makes up words. The overview of the whole subject is Voice and AI.