Preprint

Robust Speech Recognition via Large-Scale Weak Supervision (full text)

Alec Radford et al., OpenAI (arXiv), 2022 · DOI 10.48550/arXiv.2212.04356

Read the original

Read on October 16, 2026

What it is: The full paper that introduced Whisper. Its appendix tables give word error rates for each size of the model on the Fleurs benchmark in about 100 languages, and a section relates those rates to the hours of training audio per language.
How much weight it carries: A widely cited paper by the model's authors, so the numbers describe their own system under their own evaluation. Fleurs is read speech, and results on casual speech, noise or other accents can differ. Later model versions have changed.

Facts taken from this source

Of Whisper's 680,000 hours of training audio, 117,000 hours cover 96 languages other than English.
“Of those 680,000 hours of audio, 117,000 hours cover 96 other languages.” (Introduction)
Whisper's training data contains no speech for 20 of the 102 languages in the FLEURS benchmark, which caps its language identification accuracy at 80.4%.
“no training data for 20 of the 102 languages in Fleurs” (Section 3.6)
On the 82 languages that Whisper's training data covers, the best Whisper model identified the spoken language with 80.3% accuracy on the FLEURS benchmark.
“the best Whisper model achieves 80.3% accuracy” (Section 3.6)
The Whisper authors say their worst outliers are languages with unique scripts that are distantly related to the Indo-European languages that make up most of the training data.
“languages that have unique scripts and are more distantly related to the Indo-European languages” (Section 3.3)
Whisper's training data includes 125,000 hours of translation data from speech in other languages into English.
“The dataset also includes 125,000 hours of X→en translation data.” (Introduction)
Of the 680,000 hours of audio used to train Whisper, 117,000 hours cover 96 languages other than English.
“Of those 680,000 hours of audio, 117,000 hours cover 96 other languages” (Introduction)
Whisper's training data includes 125,000 hours of audio for translating speech from other languages into English.
“The dataset also includes 125,000 hours of X→en translation data” (Introduction)
On the CallHome test set of conversations between friends and family, Whisper Large V2 had a word error rate of 17.6 percent.
“Switchboard 28.3 13.8 51.2 CallHome 34.8 17.6 49.4” (Table 2, effective robustness across datasets)
The Whisper authors list complete hallucination among known failure modes: the model outputs a transcript entirely unrelated to the actual audio, along with repeat loops and missing words at segment edges.
“complete hallucination where the model will output ... entirely unrelated to the actual audio” (Section 6, Limitations and Future Work)
The Whisper paper concludes that its English speech recognition is not perfect but very close to human-level accuracy.
“English ASR performance is not perfect but very close to human-level accuracy” (Section 3.3, English Speech Recognition)
According to the Whisper paper, the best word error rate on LibriSpeech test-clean fell from 5.3 percent in 2015 to 1.4 percent by 2021.
“Yet seven years later the SOTA WER on LibriSpeech test-clean has dropped another 73% from their 5.3% to 1.4%” (Section 3.3, English Speech Recognition)
The Whisper paper says its models are trained on 30-second audio chunks and cannot take longer audio as a single input.
“Whisper models are trained on 30-second audio chunks and cannot consume longer audio inputs at once.” (Section 3.8, Long-form Transcription)
The Whisper paper says its model predicts the spoken language with one token for each language in its training set, and that there are 99 such tokens, so the training data covers 99 languages.
“for each language in our training set (99 total)” (Section 2.3, Multitask format (language token list))
For the Whisper paper's text scoring, the authors put a space between every letter for Chinese, Japanese, Thai, Lao and Burmese, which means they measured character error rate for those languages instead of word error rate.
“namely Chinese, Japanese, Thai, Lao, and Burmese, effectively measuring the character error rate instead.” (Appendix, text standardization for multilingual WER)
On the CoVoST 2 test of speech translated into English, the Whisper large-v2 model scored 26.1 BLEU for Japanese speech translated into English.
“Model Arabic Catalan Welsh German Spanish Estonian Persian French Indonesian Italian Japanese Latvian Mongolian ... Whisper large-v2 39.7 31.8 21.5 36.3 40.1 15.0 19.3 36.4 48.1 30.9 26.1 13.9 0.1” (Table 15 (BLEU scores on CoVoST2), row Whisper large-v2, column Japanese)
On the CoVoST 2 test of speech translated into English, zero-shot Whisper scored 36.2 BLEU for high-resource languages.
“Maestro 38.2 31.3 18.4 25.2 Zero-Shot Whisper 36.2 32.6 25.2 29.1” (Table 4, X to English speech translation)
On the CoVoST 2 test of speech translated into English, zero-shot Whisper scored 25.2 BLEU for low-resource languages.
“Maestro 38.2 31.3 18.4 25.2 Zero-Shot Whisper 36.2 32.6 25.2 29.1” (Table 4, X to English speech translation)
On the Common Voice 9 test set, the Whisper large-v2 model had a word error rate of 53.8 percent for Arabic.
“Whisper large-v2 53.8 19.9 103.4 14.1 13.5 34.2 14.4 6.4 16.0 9.4 5.6 35.1 39.4” (Table 11 (CommonVoice9), column Arabic)
On the Common Voice 9 test set, the Whisper large-v2 model had a word error rate of 39.4 percent for Persian.
“Whisper large-v2 53.8 19.9 103.4 14.1 13.5 34.2 14.4 6.4 16.0 9.4 5.6 35.1 39.4” (Table 11 (CommonVoice9), column Persian)
Across languages, the log of Whisper's word error rate has a squared correlation of 0.83 with the log of the amount of training data for that language.
“We find a strong squared correlation coefficient of 0.83 between the log of the word error rate and the log of the amount of training data per language.” (Section 3.3)
The Whisper authors chose an encoder-decoder Transformer as the architecture for their speech recognition models because they judged it well validated to scale reliably.
“We chose an encoder-decoder Transformer (Vaswani et al., 2017) as this architecture has been well validated to scale reliably” (Section 2, Approach)
On the FLEURS benchmark, Whisper large-v2 had a word error rate of 140.3 percent for Amharic, meaning more errors than words in the reference.
“Whisper large-v2 36.7 140.3 16.0 106.2 23.4 45.4 14.6 104.1 15.7 7.3 14.7 13.3 33.0 13.8” (Appendix D.2.4, Table 13)
On the Fleurs benchmark, the Whisper large-v2 model had a word error rate of 16.0 percent for Arabic.
“Model Afrikaans Amharic Arabic Assamese Azerbaijani Belarusian Bulgarian Bengali Bosnian Catalan Chinese Czech Welsh Danish ... Whisper large-v2 36.7 140.3 16.0 106.2 23.4 45.4 14.6 104.1 15.7 7.3 14.7 13.3 33.0 13.8” (Table 13 (Fleurs))
On the Fleurs benchmark, the Whisper large-v2 model had a word error rate of 104.1 percent for Bengali.
“Model Afrikaans Amharic Arabic Assamese Azerbaijani Belarusian Bulgarian Bengali Bosnian Catalan Chinese Czech Welsh Danish ... Whisper large-v2 36.7 140.3 16.0 106.2 23.4 45.4 14.6 104.1 15.7” (Table 13 (Fleurs), column Bengali)
On the FLEURS read-speech benchmark, Whisper large-v2 had a word error rate of 6.7 percent in Dutch.
“Whisper large-v2 6.7 75.3 102.4 5.4 93.7 4.3 14.4 5.6” (Appendix D.2.4, FLEURS transcription table, Dutch column)
On the Fleurs benchmark, the Whisper large-v2 model had a word error rate of 4.2 percent for English.
“Model German Greek English Spanish Estonian Persian Finnish Tagalog French Galician Gujarati Hausa Hebrew Hindi ... Whisper large-v2 4.5 12.5 4.2 3.0 21.9 32.9 9.7 13.8 8.3 15.4 102.7 88.9 27.1 21.5” (Table 13 (Fleurs))
On the Fleurs benchmark, the Whisper large-v2 model had a word error rate of 8.3 percent for French.
“Model German Greek English Spanish Estonian Persian Finnish Tagalog French Galician Gujarati Hausa Hebrew Hindi ... Whisper large-v2 4.5 12.5 4.2 3.0 21.9 32.9 9.7 13.8 8.3 15.4 102.7 88.9 27.1 21.5” (Table 13 (Fleurs))
On the Fleurs benchmark, the Whisper large-v2 model had a word error rate of 4.5 percent for German.
“Model German Greek English Spanish Estonian Persian Finnish Tagalog French Galician Gujarati Hausa Hebrew Hindi ... Whisper large-v2 4.5 12.5 4.2 3.0 21.9 32.9 9.7 13.8 8.3 15.4 102.7 88.9 27.1 21.5” (Table 13 (Fleurs))
On the FLEURS benchmark, Whisper large-v2 had a word error rate of 88.9 percent for Hausa.
“Whisper large-v2 4.5 12.5 4.2 3.0 21.9 32.9 9.7 13.8 8.3 15.4 102.7 88.9 27.1 21.5” (Appendix D.2.4, Table 13)
On the Fleurs benchmark, the Whisper large-v2 model had a word error rate of 27.1 percent for Hebrew.
“Whisper large-v2 4.5 12.5 4.2 3.0 21.9 32.9 9.7 13.8 8.3 15.4 102.7 88.9 27.1 21.5” (Table 13 (Fleurs), column Hebrew)
On the Fleurs benchmark, the Whisper large-v2 model had a word error rate of 21.5 percent for Hindi.
“Model German Greek English Spanish Estonian Persian Finnish Tagalog French Galician Gujarati Hausa Hebrew Hindi ... Whisper large-v2 4.5 12.5 4.2 3.0 21.9 32.9 9.7 13.8 8.3 15.4 102.7 88.9 27.1 21.5” (Table 13 (Fleurs))
On the FLEURS read-speech benchmark, Whisper large-v2 had a word error rate of 4.0 percent in Italian.
“Whisper large-v2 13.4 17.0 44.6 7.1 38.2 4.0 5.3” (Appendix D.2.4, FLEURS transcription table, Italian column)
On the Fleurs benchmark, the Whisper large-v2 model had a word error rate of 5.3 percent for Japanese.
“Model Croatian Hungarian Armenian Indonesian Icelandic Italian Japanese Javanese Georgian Kazakh Khmer Kannada Korean Luxembourgish ... Whisper large-v2 13.4 17.0 44.6 7.1 38.2 4.0 5.3 nan 105.0 37.7 99.7 37.0 14.3 88.0” (Table 13 (Fleurs))
On the Fleurs benchmark, the Whisper large-v2 model had a character error rate of 5.3 percent for Japanese, because the paper scores Japanese by characters rather than words.
“Model Croatian Hungarian Armenian Indonesian Icelandic Italian Japanese Javanese Georgian Kazakh Khmer Kannada Korean Luxembourgish ... Whisper large-v2 13.4 17.0 44.6 7.1 38.2 4.0 5.3 nan 105.0 37.7 99.7 37.0 14.3 88.0” (Table 13 (Fleurs), column Japanese; Appendix text on character-level scoring)
On the Fleurs benchmark, the smallest Whisper model, tiny, had a character error rate of 37.0 percent for Japanese, against 5.3 percent for large-v2 on the same test.
“Model Croatian Hungarian Armenian Indonesian Icelandic Italian Japanese Javanese Georgian Kazakh Khmer Kannada Korean Luxembourgish ... Whisper tiny 79.0 83.8 118.6 51.7 113.3 29.8 37.0 107.3 123.0 165.2 100.6 100.7 36.1 99.1” (Table 13 (Fleurs), row Whisper tiny, column Japanese)
On the Fleurs benchmark, the Whisper large-v2 model had a word error rate of 37.0 percent for Kannada.
“Model Croatian Hungarian Armenian Indonesian Icelandic Italian Japanese Javanese Georgian Kazakh Khmer Kannada Korean Luxembourgish ... Whisper large-v2 13.4 17.0 44.6 7.1 38.2 4.0 5.3 nan 105.0 37.7 99.7 37.0 14.3 88.0” (Table 13 (Fleurs), column Kannada)
On the Fleurs benchmark, the Whisper large-v2 model had a word error rate of 14.3 percent for Korean.
“Model Croatian Hungarian Armenian Indonesian Icelandic Italian Japanese Javanese Georgian Kazakh Khmer Kannada Korean Luxembourgish ... Whisper large-v2 13.4 17.0 44.6 7.1 38.2 4.0 5.3 nan 105.0 37.7 99.7 37.0 14.3 88.0” (Table 13 (Fleurs), row Whisper large-v2, column Korean)
On the Fleurs benchmark, the Whisper large-v2 model had a word error rate of 100.7 percent for Malayalam.
“Model Lingala Lao Lithuanian Latvian Maori Macedonian Malayalam Mongolian Marathi Malay ... Whisper large-v2 75.6 101.5 28.1 23.1 38.5 16.5 100.7 110.5 38.3 8.7” (Table 13 (Fleurs), column Malayalam)
On the Fleurs benchmark, the Whisper large-v2 model had a word error rate of 38.3 percent for Marathi.
“Model Lingala Lao Lithuanian Latvian Maori Macedonian Malayalam Mongolian Marathi Malay ... Whisper large-v2 75.6 101.5 28.1 23.1 38.5 16.5 100.7 110.5 38.3 8.7” (Table 13 (Fleurs), column Marathi)
On the Fleurs benchmark, the Whisper large-v2 model had a word error rate of 32.9 percent for Persian.
“Whisper large-v2 4.5 12.5 4.2 3.0 21.9 32.9 9.7 13.8 8.3 15.4 102.7 88.9 27.1 21.5” (Table 13 (Fleurs), column Persian)
On the FLEURS read-speech benchmark, Whisper large-v2 had a word error rate of 4.3 percent in Portuguese.
“Whisper large-v2 6.7 75.3 102.4 5.4 93.7 4.3 14.4 5.6” (Appendix D.2.4, FLEURS transcription table, Portuguese column)
On the Fleurs benchmark, the Whisper large-v2 model had a word error rate of 5.6 percent for Russian.
“Model Dutch Occitan Punjabi Polish Pashto Portuguese Romanian Russian Sindhi Slovak Slovenian Shona Somali Serbian ... Whisper large-v2 6.7 75.3 102.4 5.4 93.7 4.3 14.4 5.6 156.5 11.7 23.1 121.0 102.9 33.9” (Table 13 (Fleurs))
On the Fleurs benchmark, the Whisper large-v2 model had a word error rate of 3.0 percent for Spanish.
“Model German Greek English Spanish Estonian Persian Finnish Tagalog French Galician Gujarati Hausa Hebrew Hindi ... Whisper large-v2 4.5 12.5 4.2 3.0 21.9 32.9 9.7 13.8 8.3 15.4 102.7 88.9 27.1 21.5” (Table 13 (Fleurs))
On the Fleurs benchmark, the Whisper large-v2 model had a word error rate of 39.3 percent for Swahili.
“Model Swedish Swahili Tamil Telugu Tajik Thai Turkish Ukrainian Urdu Uzbek Vietnamese Yoruba ... Whisper large-v2 8.5 39.3 17.5 99.0 85.8 11.5 8.4 8.6 22.6 90.2 10.3 94.8” (Table 13 (Fleurs))
On the Fleurs benchmark, the Whisper large-v2 model had a word error rate of 17.5 percent for Tamil.
“Model Swedish Swahili Tamil Telugu Tajik Thai Turkish Ukrainian Urdu Uzbek Vietnamese Yoruba ... Whisper large-v2 8.5 39.3 17.5 99.0 85.8 11.5 8.4 8.6 22.6 90.2 10.3 94.8” (Table 13 (Fleurs))
On the Fleurs benchmark, the Whisper large-v2 model had a word error rate of 99.0 percent for Telugu.
“Model Swedish Swahili Tamil Telugu Tajik Thai Turkish Ukrainian Urdu Uzbek Vietnamese Yoruba ... Whisper large-v2 8.5 39.3 17.5 99.0 85.8 11.5 8.4 8.6 22.6 90.2 10.3 94.8” (Table 13 (Fleurs))
On the Fleurs benchmark, the Whisper large-v2 model had a word error rate of 22.6 percent for Urdu.
“Model Swedish Swahili Tamil Telugu Tajik Thai Turkish Ukrainian Urdu Uzbek Vietnamese Yoruba ... Whisper large-v2 8.5 39.3 17.5 99.0 85.8 11.5 8.4 8.6 22.6 90.2 10.3 94.8” (Table 13 (Fleurs))
On the Fleurs benchmark, the Whisper large-v2 model had a word error rate of 94.8 percent for Yoruba.
“Model Swedish Swahili Tamil Telugu Tajik Thai Turkish Ukrainian Urdu Uzbek Vietnamese Yoruba ... Whisper large-v2 8.5 39.3 17.5 99.0 85.8 11.5 8.4 8.6 22.6 90.2 10.3 94.8” (Table 13 (Fleurs))
The Whisper paper names five languages whose text it scores by character error rate: Chinese, Japanese, Thai, Lao and Burmese. Korean is not on that list.
“Additionally, we put a space between every letter for the languages that do not use spaces to separate words, namely Chinese, Japanese, Thai, Lao, and Burmese, effectively measuring the character error rate instead.” (Appendix D, text normalization for scoring)
Whisper re-samples all audio to 16,000 Hz and computes its 80-channel log-magnitude Mel spectrogram on 25-millisecond windows with a stride of 10 milliseconds.
“All audio is re-sampled to 16,000 Hz... is computed on 25-millisecond windows with a stride of 10 milliseconds” (Section 2, Approach (data processing))
The Whisper paper says many different tasks can be performed on the same input audio, and lists transcription, translation, voice activity detection and alignment among them.
“There are many different tasks that can be performed on the same input audio signal: transcription, translation, voice activity detection, alignment” (Section 2.3, Multitask Format)
The Whisper paper says a complete speech system has several additional components, which are often handled separately, making the system relatively complex.
“These components are often handled separately, resulting in a relatively complex system around the core speech recognition model.” (Section 2.3, Multitask Format)
The Whisper paper's authors found that most of the training data labeled as Welsh-to-English translation was actually English audio.
“Inspection shows the majority of supposedly Welsh translation data is actually English audio” (Discussion of the Fleurs translation results, after Figure 4)
The Whisper authors estimate that a language's word error rate halves for every 16-fold increase in its training data.
“results in an estimate that WER halves for every 16” (Section 3.3)
On the Fleurs read-speech test, Whisper large-v2 scored 13.2 BLEU when translating Bengali speech into English.
“Model Afrikaans Amharic Arabic Assamese Azerbaijani Belarusian Bulgarian Bengali Bosnian ... Whisper large-v2 34.1 1.9 25.5 5.4 13.7 11.7 28.5 13.2 29.7 34.2 18.4 27.8 13.0 32.7” (Table 14 (D.3.1 Fleurs, BLEU scores))
On the Fleurs read-speech test, Whisper large-v2 scored 34.6 BLEU when translating German speech into English.
“Whisper large-v2 34.6 23.7 80.2 23.3 18.7 19.6 22.1 24.4 32.2 27.9 16.2 0.4 21.8 22.0” (Table 14, BLEU scores on Fleurs (speech translation))
On the Fleurs read-speech test, Whisper large-v2 scored 22.0 BLEU when translating Hindi speech into English.
“Whisper large-v2 34.6 23.7 80.2 23.3 18.7 19.6 22.1 24.4 32.2 27.9 16.2 0.4 21.8 22.0” (Table 14, BLEU scores on Fleurs (speech translation))
On the Fleurs read-speech test, Whisper large-v2 scored 18.9 BLEU when translating Japanese speech into English.
“Whisper large-v2 27.0 21.2 16.0 29.1 9.1 23.6 18.9 6.2 2.4 5.4 6.1 11.6 21.3 16.8” (Table 14, BLEU scores on Fleurs (speech translation))
On the Fleurs read-speech test, Whisper large-v2 scored 23.3 BLEU when translating Spanish speech into English.
“Whisper large-v2 34.6 23.7 80.2 23.3 18.7 19.6 22.1 24.4 32.2 27.9 16.2 0.4 21.8 22.0” (Table 14, BLEU scores on Fleurs (speech translation))
On the Fleurs read-speech test, Whisper large-v2 scored 7.2 BLEU when translating Swahili speech into English.
“Whisper large-v2 35.3 7.2 9.2 12.5 14.5 16.1 26.6 29.4 17.2 6.0 20.4 1.4” (Table 14, BLEU scores on Fleurs (speech translation))
On the Fleurs read-speech test, Whisper large-v2 scored 17.2 BLEU when translating Urdu speech into English.
“Model Swedish Swahili Tamil Telugu Tajik Thai Turkish Ukrainian Urdu Uzbek Vietnamese Yoruba ... Whisper large-v2 35.3 7.2 9.2 12.5 14.5 16.1 26.6 29.4 17.2 6.0 20.4 1.4” (Table 14 (D.3.1 Fleurs, BLEU scores))
On the Fleurs read-speech test, Whisper large-v2 scored 1.4 BLEU when translating Yoruba speech into English.
“Whisper large-v2 35.3 7.2 9.2 12.5 14.5 16.1 26.6 29.4 17.2 6.0 20.4 1.4” (Table 14, BLEU scores on Fleurs (speech translation))
The Whisper paper reports a zero-shot score of 29.1 BLEU on the CoVoST2 speech to English translation benchmark, without training on CoVoST2 data.
“We achieve a new state of the art of 29.1 BLEU zero-shot” (Speech translation results on CoVoST2)

Pages that rest on this source

Preprint

Robust Speech Recognition via Large-Scale Weak Supervision (full text)

Alec Radford et al., OpenAI (arXiv), 2022 · DOI 10.48550/arXiv.2212.04356

Read the original

Read on October 16, 2026

What it is: The full paper that introduced Whisper. Its appendix tables give word error rates for each size of the model on the Fleurs benchmark in about 100 languages, and a section relates those rates to the hours of training audio per language.
How much weight it carries: A widely cited paper by the model's authors, so the numbers describe their own system under their own evaluation. Fleurs is read speech, and results on casual speech, noise or other accents can differ. Later model versions have changed.

Facts taken from this source

Of Whisper's 680,000 hours of training audio, 117,000 hours cover 96 languages other than English.
“Of those 680,000 hours of audio, 117,000 hours cover 96 other languages.” (Introduction)
Whisper's training data contains no speech for 20 of the 102 languages in the FLEURS benchmark, which caps its language identification accuracy at 80.4%.
“no training data for 20 of the 102 languages in Fleurs” (Section 3.6)
On the 82 languages that Whisper's training data covers, the best Whisper model identified the spoken language with 80.3% accuracy on the FLEURS benchmark.
“the best Whisper model achieves 80.3% accuracy” (Section 3.6)
The Whisper authors say their worst outliers are languages with unique scripts that are distantly related to the Indo-European languages that make up most of the training data.
“languages that have unique scripts and are more distantly related to the Indo-European languages” (Section 3.3)
Whisper's training data includes 125,000 hours of translation data from speech in other languages into English.
“The dataset also includes 125,000 hours of X→en translation data.” (Introduction)
Of the 680,000 hours of audio used to train Whisper, 117,000 hours cover 96 languages other than English.
“Of those 680,000 hours of audio, 117,000 hours cover 96 other languages” (Introduction)
Whisper's training data includes 125,000 hours of audio for translating speech from other languages into English.
“The dataset also includes 125,000 hours of X→en translation data” (Introduction)
On the CallHome test set of conversations between friends and family, Whisper Large V2 had a word error rate of 17.6 percent.
“Switchboard 28.3 13.8 51.2 CallHome 34.8 17.6 49.4” (Table 2, effective robustness across datasets)
The Whisper authors list complete hallucination among known failure modes: the model outputs a transcript entirely unrelated to the actual audio, along with repeat loops and missing words at segment edges.
“complete hallucination where the model will output ... entirely unrelated to the actual audio” (Section 6, Limitations and Future Work)
The Whisper paper concludes that its English speech recognition is not perfect but very close to human-level accuracy.
“English ASR performance is not perfect but very close to human-level accuracy” (Section 3.3, English Speech Recognition)
According to the Whisper paper, the best word error rate on LibriSpeech test-clean fell from 5.3 percent in 2015 to 1.4 percent by 2021.
“Yet seven years later the SOTA WER on LibriSpeech test-clean has dropped another 73% from their 5.3% to 1.4%” (Section 3.3, English Speech Recognition)
The Whisper paper says its models are trained on 30-second audio chunks and cannot take longer audio as a single input.
“Whisper models are trained on 30-second audio chunks and cannot consume longer audio inputs at once.” (Section 3.8, Long-form Transcription)
The Whisper paper says its model predicts the spoken language with one token for each language in its training set, and that there are 99 such tokens, so the training data covers 99 languages.
“for each language in our training set (99 total)” (Section 2.3, Multitask format (language token list))
For the Whisper paper's text scoring, the authors put a space between every letter for Chinese, Japanese, Thai, Lao and Burmese, which means they measured character error rate for those languages instead of word error rate.
“namely Chinese, Japanese, Thai, Lao, and Burmese, effectively measuring the character error rate instead.” (Appendix, text standardization for multilingual WER)
On the CoVoST 2 test of speech translated into English, the Whisper large-v2 model scored 26.1 BLEU for Japanese speech translated into English.
“Model Arabic Catalan Welsh German Spanish Estonian Persian French Indonesian Italian Japanese Latvian Mongolian ... Whisper large-v2 39.7 31.8 21.5 36.3 40.1 15.0 19.3 36.4 48.1 30.9 26.1 13.9 0.1” (Table 15 (BLEU scores on CoVoST2), row Whisper large-v2, column Japanese)
On the CoVoST 2 test of speech translated into English, zero-shot Whisper scored 36.2 BLEU for high-resource languages.
“Maestro 38.2 31.3 18.4 25.2 Zero-Shot Whisper 36.2 32.6 25.2 29.1” (Table 4, X to English speech translation)
On the CoVoST 2 test of speech translated into English, zero-shot Whisper scored 25.2 BLEU for low-resource languages.
“Maestro 38.2 31.3 18.4 25.2 Zero-Shot Whisper 36.2 32.6 25.2 29.1” (Table 4, X to English speech translation)
On the Common Voice 9 test set, the Whisper large-v2 model had a word error rate of 53.8 percent for Arabic.
“Whisper large-v2 53.8 19.9 103.4 14.1 13.5 34.2 14.4 6.4 16.0 9.4 5.6 35.1 39.4” (Table 11 (CommonVoice9), column Arabic)
On the Common Voice 9 test set, the Whisper large-v2 model had a word error rate of 39.4 percent for Persian.
“Whisper large-v2 53.8 19.9 103.4 14.1 13.5 34.2 14.4 6.4 16.0 9.4 5.6 35.1 39.4” (Table 11 (CommonVoice9), column Persian)
Across languages, the log of Whisper's word error rate has a squared correlation of 0.83 with the log of the amount of training data for that language.
“We find a strong squared correlation coefficient of 0.83 between the log of the word error rate and the log of the amount of training data per language.” (Section 3.3)
The Whisper authors chose an encoder-decoder Transformer as the architecture for their speech recognition models because they judged it well validated to scale reliably.
“We chose an encoder-decoder Transformer (Vaswani et al., 2017) as this architecture has been well validated to scale reliably” (Section 2, Approach)
On the FLEURS benchmark, Whisper large-v2 had a word error rate of 140.3 percent for Amharic, meaning more errors than words in the reference.
“Whisper large-v2 36.7 140.3 16.0 106.2 23.4 45.4 14.6 104.1 15.7 7.3 14.7 13.3 33.0 13.8” (Appendix D.2.4, Table 13)
On the Fleurs benchmark, the Whisper large-v2 model had a word error rate of 16.0 percent for Arabic.
“Model Afrikaans Amharic Arabic Assamese Azerbaijani Belarusian Bulgarian Bengali Bosnian Catalan Chinese Czech Welsh Danish ... Whisper large-v2 36.7 140.3 16.0 106.2 23.4 45.4 14.6 104.1 15.7 7.3 14.7 13.3 33.0 13.8” (Table 13 (Fleurs))
On the Fleurs benchmark, the Whisper large-v2 model had a word error rate of 104.1 percent for Bengali.
“Model Afrikaans Amharic Arabic Assamese Azerbaijani Belarusian Bulgarian Bengali Bosnian Catalan Chinese Czech Welsh Danish ... Whisper large-v2 36.7 140.3 16.0 106.2 23.4 45.4 14.6 104.1 15.7” (Table 13 (Fleurs), column Bengali)
On the FLEURS read-speech benchmark, Whisper large-v2 had a word error rate of 6.7 percent in Dutch.
“Whisper large-v2 6.7 75.3 102.4 5.4 93.7 4.3 14.4 5.6” (Appendix D.2.4, FLEURS transcription table, Dutch column)
On the Fleurs benchmark, the Whisper large-v2 model had a word error rate of 4.2 percent for English.
“Model German Greek English Spanish Estonian Persian Finnish Tagalog French Galician Gujarati Hausa Hebrew Hindi ... Whisper large-v2 4.5 12.5 4.2 3.0 21.9 32.9 9.7 13.8 8.3 15.4 102.7 88.9 27.1 21.5” (Table 13 (Fleurs))
On the Fleurs benchmark, the Whisper large-v2 model had a word error rate of 8.3 percent for French.
“Model German Greek English Spanish Estonian Persian Finnish Tagalog French Galician Gujarati Hausa Hebrew Hindi ... Whisper large-v2 4.5 12.5 4.2 3.0 21.9 32.9 9.7 13.8 8.3 15.4 102.7 88.9 27.1 21.5” (Table 13 (Fleurs))
On the Fleurs benchmark, the Whisper large-v2 model had a word error rate of 4.5 percent for German.
“Model German Greek English Spanish Estonian Persian Finnish Tagalog French Galician Gujarati Hausa Hebrew Hindi ... Whisper large-v2 4.5 12.5 4.2 3.0 21.9 32.9 9.7 13.8 8.3 15.4 102.7 88.9 27.1 21.5” (Table 13 (Fleurs))
On the FLEURS benchmark, Whisper large-v2 had a word error rate of 88.9 percent for Hausa.
“Whisper large-v2 4.5 12.5 4.2 3.0 21.9 32.9 9.7 13.8 8.3 15.4 102.7 88.9 27.1 21.5” (Appendix D.2.4, Table 13)
On the Fleurs benchmark, the Whisper large-v2 model had a word error rate of 27.1 percent for Hebrew.
“Whisper large-v2 4.5 12.5 4.2 3.0 21.9 32.9 9.7 13.8 8.3 15.4 102.7 88.9 27.1 21.5” (Table 13 (Fleurs), column Hebrew)
On the Fleurs benchmark, the Whisper large-v2 model had a word error rate of 21.5 percent for Hindi.
“Model German Greek English Spanish Estonian Persian Finnish Tagalog French Galician Gujarati Hausa Hebrew Hindi ... Whisper large-v2 4.5 12.5 4.2 3.0 21.9 32.9 9.7 13.8 8.3 15.4 102.7 88.9 27.1 21.5” (Table 13 (Fleurs))
On the FLEURS read-speech benchmark, Whisper large-v2 had a word error rate of 4.0 percent in Italian.
“Whisper large-v2 13.4 17.0 44.6 7.1 38.2 4.0 5.3” (Appendix D.2.4, FLEURS transcription table, Italian column)
On the Fleurs benchmark, the Whisper large-v2 model had a word error rate of 5.3 percent for Japanese.
“Model Croatian Hungarian Armenian Indonesian Icelandic Italian Japanese Javanese Georgian Kazakh Khmer Kannada Korean Luxembourgish ... Whisper large-v2 13.4 17.0 44.6 7.1 38.2 4.0 5.3 nan 105.0 37.7 99.7 37.0 14.3 88.0” (Table 13 (Fleurs))
On the Fleurs benchmark, the Whisper large-v2 model had a character error rate of 5.3 percent for Japanese, because the paper scores Japanese by characters rather than words.
“Model Croatian Hungarian Armenian Indonesian Icelandic Italian Japanese Javanese Georgian Kazakh Khmer Kannada Korean Luxembourgish ... Whisper large-v2 13.4 17.0 44.6 7.1 38.2 4.0 5.3 nan 105.0 37.7 99.7 37.0 14.3 88.0” (Table 13 (Fleurs), column Japanese; Appendix text on character-level scoring)
On the Fleurs benchmark, the smallest Whisper model, tiny, had a character error rate of 37.0 percent for Japanese, against 5.3 percent for large-v2 on the same test.
“Model Croatian Hungarian Armenian Indonesian Icelandic Italian Japanese Javanese Georgian Kazakh Khmer Kannada Korean Luxembourgish ... Whisper tiny 79.0 83.8 118.6 51.7 113.3 29.8 37.0 107.3 123.0 165.2 100.6 100.7 36.1 99.1” (Table 13 (Fleurs), row Whisper tiny, column Japanese)
On the Fleurs benchmark, the Whisper large-v2 model had a word error rate of 37.0 percent for Kannada.
“Model Croatian Hungarian Armenian Indonesian Icelandic Italian Japanese Javanese Georgian Kazakh Khmer Kannada Korean Luxembourgish ... Whisper large-v2 13.4 17.0 44.6 7.1 38.2 4.0 5.3 nan 105.0 37.7 99.7 37.0 14.3 88.0” (Table 13 (Fleurs), column Kannada)
On the Fleurs benchmark, the Whisper large-v2 model had a word error rate of 14.3 percent for Korean.
“Model Croatian Hungarian Armenian Indonesian Icelandic Italian Japanese Javanese Georgian Kazakh Khmer Kannada Korean Luxembourgish ... Whisper large-v2 13.4 17.0 44.6 7.1 38.2 4.0 5.3 nan 105.0 37.7 99.7 37.0 14.3 88.0” (Table 13 (Fleurs), row Whisper large-v2, column Korean)
On the Fleurs benchmark, the Whisper large-v2 model had a word error rate of 100.7 percent for Malayalam.
“Model Lingala Lao Lithuanian Latvian Maori Macedonian Malayalam Mongolian Marathi Malay ... Whisper large-v2 75.6 101.5 28.1 23.1 38.5 16.5 100.7 110.5 38.3 8.7” (Table 13 (Fleurs), column Malayalam)
On the Fleurs benchmark, the Whisper large-v2 model had a word error rate of 38.3 percent for Marathi.
“Model Lingala Lao Lithuanian Latvian Maori Macedonian Malayalam Mongolian Marathi Malay ... Whisper large-v2 75.6 101.5 28.1 23.1 38.5 16.5 100.7 110.5 38.3 8.7” (Table 13 (Fleurs), column Marathi)
On the Fleurs benchmark, the Whisper large-v2 model had a word error rate of 32.9 percent for Persian.
“Whisper large-v2 4.5 12.5 4.2 3.0 21.9 32.9 9.7 13.8 8.3 15.4 102.7 88.9 27.1 21.5” (Table 13 (Fleurs), column Persian)
On the FLEURS read-speech benchmark, Whisper large-v2 had a word error rate of 4.3 percent in Portuguese.
“Whisper large-v2 6.7 75.3 102.4 5.4 93.7 4.3 14.4 5.6” (Appendix D.2.4, FLEURS transcription table, Portuguese column)
On the Fleurs benchmark, the Whisper large-v2 model had a word error rate of 5.6 percent for Russian.
“Model Dutch Occitan Punjabi Polish Pashto Portuguese Romanian Russian Sindhi Slovak Slovenian Shona Somali Serbian ... Whisper large-v2 6.7 75.3 102.4 5.4 93.7 4.3 14.4 5.6 156.5 11.7 23.1 121.0 102.9 33.9” (Table 13 (Fleurs))
On the Fleurs benchmark, the Whisper large-v2 model had a word error rate of 3.0 percent for Spanish.
“Model German Greek English Spanish Estonian Persian Finnish Tagalog French Galician Gujarati Hausa Hebrew Hindi ... Whisper large-v2 4.5 12.5 4.2 3.0 21.9 32.9 9.7 13.8 8.3 15.4 102.7 88.9 27.1 21.5” (Table 13 (Fleurs))
On the Fleurs benchmark, the Whisper large-v2 model had a word error rate of 39.3 percent for Swahili.
“Model Swedish Swahili Tamil Telugu Tajik Thai Turkish Ukrainian Urdu Uzbek Vietnamese Yoruba ... Whisper large-v2 8.5 39.3 17.5 99.0 85.8 11.5 8.4 8.6 22.6 90.2 10.3 94.8” (Table 13 (Fleurs))
On the Fleurs benchmark, the Whisper large-v2 model had a word error rate of 17.5 percent for Tamil.
“Model Swedish Swahili Tamil Telugu Tajik Thai Turkish Ukrainian Urdu Uzbek Vietnamese Yoruba ... Whisper large-v2 8.5 39.3 17.5 99.0 85.8 11.5 8.4 8.6 22.6 90.2 10.3 94.8” (Table 13 (Fleurs))
On the Fleurs benchmark, the Whisper large-v2 model had a word error rate of 99.0 percent for Telugu.
“Model Swedish Swahili Tamil Telugu Tajik Thai Turkish Ukrainian Urdu Uzbek Vietnamese Yoruba ... Whisper large-v2 8.5 39.3 17.5 99.0 85.8 11.5 8.4 8.6 22.6 90.2 10.3 94.8” (Table 13 (Fleurs))
On the Fleurs benchmark, the Whisper large-v2 model had a word error rate of 22.6 percent for Urdu.
“Model Swedish Swahili Tamil Telugu Tajik Thai Turkish Ukrainian Urdu Uzbek Vietnamese Yoruba ... Whisper large-v2 8.5 39.3 17.5 99.0 85.8 11.5 8.4 8.6 22.6 90.2 10.3 94.8” (Table 13 (Fleurs))
On the Fleurs benchmark, the Whisper large-v2 model had a word error rate of 94.8 percent for Yoruba.
“Model Swedish Swahili Tamil Telugu Tajik Thai Turkish Ukrainian Urdu Uzbek Vietnamese Yoruba ... Whisper large-v2 8.5 39.3 17.5 99.0 85.8 11.5 8.4 8.6 22.6 90.2 10.3 94.8” (Table 13 (Fleurs))
The Whisper paper names five languages whose text it scores by character error rate: Chinese, Japanese, Thai, Lao and Burmese. Korean is not on that list.
“Additionally, we put a space between every letter for the languages that do not use spaces to separate words, namely Chinese, Japanese, Thai, Lao, and Burmese, effectively measuring the character error rate instead.” (Appendix D, text normalization for scoring)
Whisper re-samples all audio to 16,000 Hz and computes its 80-channel log-magnitude Mel spectrogram on 25-millisecond windows with a stride of 10 milliseconds.
“All audio is re-sampled to 16,000 Hz... is computed on 25-millisecond windows with a stride of 10 milliseconds” (Section 2, Approach (data processing))
The Whisper paper says many different tasks can be performed on the same input audio, and lists transcription, translation, voice activity detection and alignment among them.
“There are many different tasks that can be performed on the same input audio signal: transcription, translation, voice activity detection, alignment” (Section 2.3, Multitask Format)
The Whisper paper says a complete speech system has several additional components, which are often handled separately, making the system relatively complex.
“These components are often handled separately, resulting in a relatively complex system around the core speech recognition model.” (Section 2.3, Multitask Format)
The Whisper paper's authors found that most of the training data labeled as Welsh-to-English translation was actually English audio.
“Inspection shows the majority of supposedly Welsh translation data is actually English audio” (Discussion of the Fleurs translation results, after Figure 4)
The Whisper authors estimate that a language's word error rate halves for every 16-fold increase in its training data.
“results in an estimate that WER halves for every 16” (Section 3.3)
On the Fleurs read-speech test, Whisper large-v2 scored 13.2 BLEU when translating Bengali speech into English.
“Model Afrikaans Amharic Arabic Assamese Azerbaijani Belarusian Bulgarian Bengali Bosnian ... Whisper large-v2 34.1 1.9 25.5 5.4 13.7 11.7 28.5 13.2 29.7 34.2 18.4 27.8 13.0 32.7” (Table 14 (D.3.1 Fleurs, BLEU scores))
On the Fleurs read-speech test, Whisper large-v2 scored 34.6 BLEU when translating German speech into English.
“Whisper large-v2 34.6 23.7 80.2 23.3 18.7 19.6 22.1 24.4 32.2 27.9 16.2 0.4 21.8 22.0” (Table 14, BLEU scores on Fleurs (speech translation))
On the Fleurs read-speech test, Whisper large-v2 scored 22.0 BLEU when translating Hindi speech into English.
“Whisper large-v2 34.6 23.7 80.2 23.3 18.7 19.6 22.1 24.4 32.2 27.9 16.2 0.4 21.8 22.0” (Table 14, BLEU scores on Fleurs (speech translation))
On the Fleurs read-speech test, Whisper large-v2 scored 18.9 BLEU when translating Japanese speech into English.
“Whisper large-v2 27.0 21.2 16.0 29.1 9.1 23.6 18.9 6.2 2.4 5.4 6.1 11.6 21.3 16.8” (Table 14, BLEU scores on Fleurs (speech translation))
On the Fleurs read-speech test, Whisper large-v2 scored 23.3 BLEU when translating Spanish speech into English.
“Whisper large-v2 34.6 23.7 80.2 23.3 18.7 19.6 22.1 24.4 32.2 27.9 16.2 0.4 21.8 22.0” (Table 14, BLEU scores on Fleurs (speech translation))
On the Fleurs read-speech test, Whisper large-v2 scored 7.2 BLEU when translating Swahili speech into English.
“Whisper large-v2 35.3 7.2 9.2 12.5 14.5 16.1 26.6 29.4 17.2 6.0 20.4 1.4” (Table 14, BLEU scores on Fleurs (speech translation))
On the Fleurs read-speech test, Whisper large-v2 scored 17.2 BLEU when translating Urdu speech into English.
“Model Swedish Swahili Tamil Telugu Tajik Thai Turkish Ukrainian Urdu Uzbek Vietnamese Yoruba ... Whisper large-v2 35.3 7.2 9.2 12.5 14.5 16.1 26.6 29.4 17.2 6.0 20.4 1.4” (Table 14 (D.3.1 Fleurs, BLEU scores))
On the Fleurs read-speech test, Whisper large-v2 scored 1.4 BLEU when translating Yoruba speech into English.
“Whisper large-v2 35.3 7.2 9.2 12.5 14.5 16.1 26.6 29.4 17.2 6.0 20.4 1.4” (Table 14, BLEU scores on Fleurs (speech translation))
The Whisper paper reports a zero-shot score of 29.1 BLEU on the CoVoST2 speech to English translation benchmark, without training on CoVoST2 data.
“We achieve a new state of the art of 29.1 BLEU zero-shot” (Speech translation results on CoVoST2)