Fact · Voice and AI
19.2percent accuracy after eight edits
With eight adversarially chosen character edits, Mistral-7B-Instruct's accuracy on the GSM8K math benchmark fell to 19.2 percent.
Checked against its source on October 17, 2026
In the source's own words
“with 8 character edits the performance further drops to 19.2%”
Reasoning Robustness of LLMs to Adversarial Typographical Errors, Esther Gan et al., arXiv preprint 2411.05345, 2024, Abstract.
Open the original- What it measures
- The same model and benchmark as the one-edit result. The edits are chosen to hurt the model most, so the drop is the worst case for the number of errors.
- What it does not show
- The adversarial choice of edits makes this an upper bound on harm from typos. Ordinary dictation errors are not chosen to hurt the model, so they do less damage on average.