Fact · Voice and AI
76accuracy points
In few-shot prompts that differed only in formatting, LLaMA-2-13B's accuracy varied by up to 76 percentage points, according to the study.
Checked against its source on October 17, 2026
In the source's own words
“performance differences of up to 76 accuracy points when evaluated using LLaMA-2-13B”
Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting, Melanie Sclar et al., ICLR 2024 (arXiv preprint 2310.11324), 2024, Abstract.
Open the original- What it measures
- The formats differed in small, meaning-preserving ways, such as separators and layout, in few-shot prompts for classification tasks. The authors then proposed FormatSpread to report a range instead of one format.
- What it does not show
- The number is the worst case across the formats the authors tried, for an open-source model, in few-shot settings. It does not describe a typical chat prompt typed into ChatGPT or Claude.