Fact · Speech recognition
In the 2025 study, LLM-based recognizers from contaminated language models showed only subtle error rate differences, but assigned much higher probability to transcriptions they had seen during training.
Checked against its source on October 17, 2026
In the source's own words
“They show only subtle error rate differences if the LLM is contaminated, but assign significantly higher probabilities to transcriptions seen during LLM training.”
Evaluation of LLMs in Speech is Often Flawed: Test Set Contamination in Large Language Models for Speech Recognition, Yuan Tseng et al., arXiv (Audio and Speech Processing), 2025, Abstract.
Open the original- What it measures
- The authors trained language models with and without the contamination and then built speech recognizers on top of each.
- What it does not show
- The probability effect was measured in controlled experiments, not in the commercial systems on the leaderboards. The authors conclude that held-out data is needed to evaluate LLM-based speech systems.