Preprint
Reasoning Robustness of LLMs to Adversarial Typographical Errors
Esther Gan et al., arXiv preprint 2411.05345, 2024
Read the originalRead on October 17, 2026
- What it is
- A study of how a few typed errors affect the step-by-step reasoning of language models on math and reasoning questions. The authors pick the typo most likely to derail each question, which is a worst case rather than a typical typo, and report the accuracy after one, eight and more edits.
- How much weight it carries
- A preprint that has not been through peer review as far as the arXiv record shows. The errors are chosen adversarially, so the drops are a worst-case measure, not the average effect of a typo. The figures are from the abstract.
Facts taken from this source
- With eight adversarially chosen character edits, Mistral-7B-Instruct's accuracy on the GSM8K math benchmark fell to 19.2 percent.
“with 8 character edits the performance further drops to 19.2%” (Abstract)
- A single adversarially chosen character edit lowered Mistral-7B-Instruct's accuracy on the GSM8K math benchmark from 43.7 percent to 38.6 percent.
“with 1 character edit, Mistral-7B-Instruct's accuracy drops from 43.7% to 38.6% on GSM8K” (Abstract)