Preprint
Svarah: Evaluating English ASR Systems on Indian Accents
Tahir Javed et al., arXiv, 2023
Read the originalRead on October 17, 2026
- What it is
- A 2023 benchmark of 9.6 hours of English from 117 speakers across 65 locations in India, read and conversational, used to test six open source English recognizers and two commercial systems, including Whisper, Wav2Vec2, Azure and Google.
- How much weight it carries
- Preprint from the authors who built the benchmark, with a public dataset and a clear method. It is a 2023 test: newer models and cloud versions may score differently. It covers 117 speakers, which is a modest sample for the many accents of India.
Facts taken from this source
- In the Svarah test, the recognizers performed poorly for speakers whose first language is Assamese, Bodo or Nepali, which the authors say are low-resource languages from northeastern India.
“the models perform poorly for speakers who are native speakers of Assamese, Bodo, and Nepali” (Section 5, Results and Discussions)
- For the open source models in the test, the gap between the error rate on Svarah and on native English read speech from the Speech Accent Archive ranged from 5.65 to 43.03 percentage points.
“the gap between the WERs on Svarah and SAA L1 ranges from 5.65 - 43.03” (Section 5, Results and Discussions)
- The Svarah authors say Indian speakers are very poorly represented in existing English speech recognition benchmarks such as LibriSpeech and Switchboard.
“Indian speakers find a very poor representation in existing English ASR benchmarks such as LibriSpeech” (Abstract)
- The Svarah authors found that the two commercial systems they tested were outperformed by the open source Whisper and NVIDIA Conformer models on Indian English.
“the two commercial models are outperformed by the open-source Whisper and NVIDIA Conformer models” (Section 5, Results and Discussions)
- On Svarah, a 2023 benchmark of English from Indian speakers, the large version of OpenAI's Whisper had a word error rate of 7.2 percent in the Svarah column.
“Whisper large 1550M 7.2 1.6 2.7” (Table 1)
- The Svarah authors say the medium and large versions of Whisper gave single-digit word error rates on Indian English, and note that their size makes them hard to deploy in India.
“the medium and large variants of Whisper give single-digit WERs” (Section 5, Results and Discussions)