Peer-reviewed research
Are LLMs Robust for Spoken Dialogues?
Seyed Mahed Mousavi et al., Proceedings of IWSDS 2024 (arXiv preprint 2401.02297), 2024
Read the originalRead on October 17, 2026
- What it is
- A study that transcribed spoken task-oriented dialogues with a speech recognizer, measured the errors it made, and simulated those errors in a larger set of dialogues. It tests fine-tuned GPT-2 and T5 models on the transcripts.
- How much weight it carries
- Peer-reviewed at a speech dialogue workshop and written by speech researchers. The models are small fine-tuned models from before the current generation of chatbots, so the result is about the direction of the effect and how errors were simulated, not the size of the effect in today's chatbots.
Facts taken from this source
- In a 2024 study, fine-tuned GPT-2 and T5 models were not robust to the errors of a speech recognizer by default, but training them on spoken dialogue data made them more robust.
“The results show that LLMs are not robust to spoken noise by default, however, fine-tuning/training such models on a proper dataset of spoken TODs can result in a more robust performance.” (Abstract)