Peer-reviewed research
Does Whisper Understand Swiss German? An Automatic, Qualitative and Human Evaluation
Eyal Liron Dolev et al., Proceedings of the 11th Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial 2024), 2024
Read the originalRead on October 17, 2026
- What it is
- Dolev, Lutz and Aepli tested OpenAI's Whisper on Swiss German audio with Standard German reference text, using word error rate, BLEU, a qualitative review and a survey of 28 people. Whisper scored close to published systems and was judged a good tool when the output is Standard German.
- How much weight it carries
- Peer-reviewed workshop paper from linguistics and speech groups at the University of Zurich. It measures Standard German output, not verbatim Swiss German spelling, and each test set is small or specialized.
Facts taken from this source
- The Swiss German corpus authors fine-tuned a Wav2Vec2 XLS-R model on their data and reached a word error rate of 14.0 percent on their test set, as Dolev and colleagues report.
“They fine-tuned a Wav2Vec2 XLS-R model on the corpus and reached a WER of 14.0 and a BLEU score of 74.7 on their test set.” (Section 2, related work on Swiss German corpora)
- Dolev and colleagues conclude that Whisper can be used for transcribing Swiss German without adaptation, provided the desired output is Standard German.
“We are therefore convinced that Whisper can be used, as is and out-of-the-box, without any further adaptations, for transcribing Swiss German, providing that the desired output is Standard German” (Section 8, Conclusion)
- In Dolev and colleagues' tests, Whisper large-v3 used without any training on Swiss German had a word error rate of 23.0 percent on the STT4SG-350 test set.
“Whisper large-v3 zero-shot 0.230 63.1 This work” (Table 2, STT4SG-350 row)