Preprint
Evaluation of LLMs in Speech is Often Flawed: Test Set Contamination in Large Language Models for Speech Recognition
Yuan Tseng et al., arXiv (Audio and Speech Processing), 2025
Read the originalRead on October 17, 2026
- What it is
- A 2025 study that checked whether the LibriSpeech and Common Voice evaluation sets, the two corpora most often quoted for speech results, appear in the text used to pretrain public large language models. It then trained models with and without that overlap and compared their transcription errors.
- How much weight it carries
- A preprint on arXiv (version 2 is dated June 2025). It is not peer reviewed in the text we checked. It measures contamination in LLM-based recognizers, so its finding applies most directly to systems built on LLMs. The effect on conventional recognizers is not measured.
Facts taken from this source
- In the 2025 study, LLM-based recognizers from contaminated language models showed only subtle error rate differences, but assigned much higher probability to transcriptions they had seen during training.
“They show only subtle error rate differences if the LLM is contaminated, but assign significantly higher probabilities to transcriptions seen during LLM training.” (Abstract)
- A 2025 study found that a substantial amount of the LibriSpeech and Common Voice evaluation sets appear in public large language model pretraining corpora.
“this work finds that a substantial amount of the LibriSpeech and Common Voice evaluation sets appear in public LLM pretraining corpora” (Abstract)