Peer-reviewed research
Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting
Melanie Sclar et al., ICLR 2024 (arXiv preprint 2310.11324), 2024
Read the originalRead on October 17, 2026
- What it is
- A study that changes only the formatting of few-shot prompts, not their meaning, and measures how much the answers of several open-source language models move. It proposes FormatSpread, a method that reports a range of accuracy across plausible formats instead of one number.
- How much weight it carries
- Peer-reviewed at ICLR 2024. It measures few-shot prompts with open-source models of 2023 and 2024, and formatting changes of a kind that a person typing a chat message would not make. It shows that prompt wording and layout matter, not how a typical chat prompt compares with a spoken one.
Facts taken from this source
- In few-shot prompts that differed only in formatting, LLaMA-2-13B's accuracy varied by up to 76 percentage points, according to the study.
“performance differences of up to 76 accuracy points when evaluated using LLaMA-2-13B” (Abstract)