How good is Whisper at Japanese?

Whisper is good at Japanese when it runs a large model on clean audio, and much weaker with small models or everyday speech. On the FLEURS benchmark, the large-v2 model from 2022 made errors on 5.3 percent of Japanese characters ¹. That is a character error rate, not a word error rate, so it cannot be set directly against the English figures from the same test. Set the language yourself, use the largest model your app offers, and test with your own voice.

Japanese is scored by character, so the number looks different from English

Japanese is written without spaces between most words, so a word is not a clear unit to count. The FLEURS paper says that Japanese and Korean transcripts may or may not contain spaces, depending on who transcribed them ². To score by character, the Whisper paper puts a space between every letter for Chinese, Japanese, Thai, Lao and Burmese ³.

The practical point is that a Japanese score of 5.3 percent ¹ and an English score of 4.2 percent ⁴ are not the same kind of measure. Read the Japanese figure as "about one character in twenty was wrong on read sentences," and do not treat it as a rank against other languages.

The large model is accurate, and the smallest model is not

The smallest model, tiny, had a character error rate of 37.0 percent on the same test ⁵, which means more than one character in three was wrong. The large-v2 model, the largest in the paper, had 5.3 percent on that test ¹.

The model you run matters as much as the language setting. Many apps offer a choice between a small, fast model and a large, slower one, so check which one is running before you blame the audio. The paper's figures are for large-v2, so treat them as the pattern for the family rather than a guarantee for each app release.

Fine-tuning on Japanese audio closes much of the gap

Fine-tuning means training a model further on one language. In a 2024 study, the untuned Whisper-Tiny model had a character error rate of 32.7 percent on Japanese, and end-to-end fine-tuning on Japanese data lowered it to 14.7 percent ⁶. The study used its own Japanese test sets, so its numbers are not the FLEURS figures above, but the direction is the same.

A fine-tuned model is a separate download. An app that says it uses Whisper for Japanese may run the standard model, so check which one before you assume the gain.

Translating Japanese speech into English is a separate step with lower scores

Whisper can translate speech into English, but only into English. On the CoVoST 2 translation test, the large-v2 model scored 26.1 BLEU for Japanese speech ⁷. BLEU runs from 0 to 100 and measures how closely a machine translation matches one reference translation, so a score of 26 is a useful translation, not a perfect one, and it cannot be compared with the error rates above.

The turbo model is not trained for translation ⁸, so do not rely on it for an English version. For an English version of a Japanese recording, first keep the Japanese transcript, then translate it with a multilingual model. The question on whether Whisper can translate audio explains the two tasks, and the guide to translating audio to English covers the tools.

Automatic detection can pick the wrong language

If you do not set a language, Whisper decides from the audio, and it looks at only the first 30 seconds to do so ⁹. A recording that opens with a short English greeting before Japanese speech can be detected as English, and the rest is then transcribed with the wrong language. The question on Whisper transcribing in the wrong language lists the usual causes.

What to do if you need Japanese

1. Set the language to Japanese before you start, rather than leaving detection on.
2. Choose the largest model your app offers, and check that the app is not running a smaller default.
3. Test two or three minutes of the speech you actually produce, such as meetings, dictation or interviews, and check the output by eye.
4. Keep the Japanese transcript as the record, and translate it as a second step if an English version is needed.

For the wider picture, see the guide to Japanese, Chinese and Korean speech to text and the guide to Whisper's languages and accuracy. The question on Whisper and Korean and the question on Arabic compare the other large Asian and Middle Eastern languages. The overview of languages and accents covers the rest.