Fact · Voice and AI
38.6percent accuracy after one edit
A single adversarially chosen character edit lowered Mistral-7B-Instruct's accuracy on the GSM8K math benchmark from 43.7 percent to 38.6 percent.
Checked against its source on October 17, 2026
In the source's own words
“with 1 character edit, Mistral-7B-Instruct's accuracy drops from 43.7% to 38.6% on GSM8K”
Reasoning Robustness of LLMs to Adversarial Typographical Errors, Esther Gan et al., arXiv preprint 2411.05345, 2024, Abstract.
Open the original- What it measures
- The attack picks the typo most likely to make the model fail for each question. The starting accuracy was 43.7 percent, measured on the same questions without the edit.
- What it does not show
- This is a worst case. Random typos, such as the ones a speech recognizer makes, would usually do less harm than a chosen edit. The model is an older open-source model.