Why does Whisper transcribe in the wrong language?

Whisper transcribes in the wrong language for one of four main reasons. It guessed the language from the opening of the audio, the speech mixes two languages, the language had little training data, or the translate task was on and it produced English. In most cases, the first step is to set the language yourself. The steps below take each cause in turn.

Whisper chooses the language from the first 30 seconds

If you do not set a language, the open-source Whisper code detects it from the audio. Its transcribe function looks at up to the first 30 seconds ¹. The check happens once, at the start, so a recording whose language changes after that opening is not re-checked by that step. A short intro in one language can therefore set the wrong language for the rest of a longer recording in another.

The fix is to name the language. The README's example, whisper japanese.wav --language Japanese, shows the option in the command-line tool ( README on GitHub). In an app, look for the language setting and set it to the language you speak rather than leaving it on automatic.

Mixed-language speech is the hardest case

Speech that moves between two languages is where detection breaks down most. On two conversational corpora of code-switching, Whisper's mixed error rate on the code-switched utterances was about 39 percent ². The same study found that speech recognition with Whisper significantly altered and reduced the speakers' original code-switching ³.

Setting one language helps with the main language but does not solve the mixed words. If you regularly mix languages, expect to correct the switched phrases by hand, and check whether a multilingual option or a different tool handles it better for your speech. The guide to multilingual dictation covers two-language setups in more detail.

Languages with little training data get a poor guess

Detection is good, but not perfect. In the Whisper paper's language identification test on FLEURS, the best model picked the right language 80.3 percent of the time, across the 82 languages it had training data for ⁴. The paper says Whisper's training data has no speech for 20 of the 102 FLEURS languages ⁵.

So a language with thin training data can be misidentified, and its speech is also transcribed less accurately once it is identified ⁶. The guide to Whisper's languages and accuracy shows how error rates differ from one language to the next. For these languages, setting the language by hand is the best step you can take, even though it cannot make the model more accurate.

A transcript in English often means the translate task

Some people ask why the output is in English when they expected the spoken language. Whisper's translate task produces English. The Whisper paper's translation training data is 125,000 hours of speech translated into English ⁷, and the README's translate example outputs English text.

Check the task setting or the command first. If you passed --task translate, remove it to get a transcript in the original language. The reverse case also exists. The README says the turbo model returns the original language even when the translate task is set ⁸, so an English request can come back in the spoken language. If you need English, use a multilingual model instead, since turbo is not trained for translation ⁹.

How to fix the wrong language

1. Set the language explicitly before you transcribe, instead of relying on detection.
2. Remove the translate task if you want the original language back, and keep it only for an English output on purpose.
3. Use a multilingual model for translation, since turbo is not trained for it.
4. Test with a short clip in your language and read the result before you run a long recording.

If the wrong language appears only in mixed speech, that is the expected limit rather than a setting error. The question on whether Whisper can translate audio explains the two tasks. The question on how good Whisper is at Hindi shows how these causes affect one language in practice, and the overview of languages and accents lists the other guides on language settings across apps.