How good is Whisper at Hindi?

Whisper is usable in Hindi, but it is clearly weaker than it is in English. On the FLEURS benchmark, the large-v2 model from 2022 had a word error rate of 21.5 percent in Hindi ¹. Its English rate on the same test was 4.2 percent ². A Hindi model fine-tuned on Indian-language audio did better on a comparable test, but the result depends heavily on which test you read.

Whisper's Hindi error rate is about five times its English rate

The ratio makes the gap concrete. Dividing the Hindi rate by the English rate gives 21.5 divided by 4.2, or about 5.1 times ³. More than one word in five is wrong in Hindi on that test ¹, against 4.2 percent of English words ².

The gap fits a trend the paper reports. The authors estimate that a language's word error rate halves for every 16-fold increase in its training data ⁴, so the trend predicts a higher rate for any language with less training data than English. The guide to Whisper's languages and accuracy sets Hindi next to the other languages in the same test.

Fine-tuned Hindi models score better on a similar test

Fine-tuning means training the model further on one language's recordings. The Vistaar study fine-tuned Whisper on Indian-language audio and called the result IndicWhisper. On the FLEURS part of the Vistaar benchmark, it scored 11.4 percent ⁵, compared with 21.5 percent for the untuned large-v2 model on the FLEURS test ¹.

Across the Hindi subset of the Vistaar benchmark, which has seven test sets, IndicWhisper averaged 13.6 percent word error rate ⁶. The FLEURS comparison of 11.4 and 21.5 percent is the closest available, though the Vistaar authors' text normalization may not match the Whisper paper's, so read it as close rather than exact.

A fine-tuned model is a different model from the default Whisper models. If an app says it uses Whisper for Hindi, check which model it loads before you assume it has the fine-tuned results.

The Common Voice result shows how much the test matters

A different test gives a very different answer. In the Voxtral paper, Whisper large-v3 had a word error rate of 46.75 percent on the Mozilla Common Voice Hindi test ⁷. Common Voice is built from volunteer recordings, with 2,500 hours from over 50,000 contributors by 2019 ⁸, so its speakers and microphones vary more than in a benchmark of read sentences. Set against these numbers, a single Hindi score is a range, not a fixed value. The guide to Hindi voice typing covers how the tools you might use for Hindi compare in practice, and the guide to speech recognition accents explains why real speakers see higher rates than read sentences.

What to do if you dictate in Hindi

• Set the language to Hindi rather than leaving detection on. Detection looks at only the first 30 seconds of audio ⁹ and can be wrong for mixed speech.
• Test with your own recordings. Dictate two or three minutes of the kind of Hindi you actually write and count the errors before you trust a tool.
• Expect mixed Hindi and English to be harder. Speech that switches languages is a known weak spot, as the question on wrong-language output explains.
• Compare tools on the same test. A score from one benchmark does not transfer to another, as the Common Voice result shows.

The question on whether Whisper can translate audio covers a different case: a Hindi recording you want in English. For the wider subject, see the overview of languages and accents.