Preprint
Scalable Extraction of Training Data from (Production) Language Models
Milad Nasr et al., arXiv, 2023
Read the originalRead on October 17, 2026
- What it is
- A preprint from Google DeepMind, the University of Washington, Cornell, ETH Zurich and other institutions. It shows that a simple prompting attack can make ChatGPT repeat verbatim pieces of its training data, and it estimates the query cost of the attack.
- How much weight it carries
- Peer-reviewed venue not confirmed at the time of checking, but the authors include researchers who published the 2021 USENIX Security memorization study. The attack was tested on ChatGPT as it stood in 2023, so current models may behave differently.
Facts taken from this source
- The 2023 attack that recovered over ten thousand ChatGPT training examples used queries costing $200 USD.
“we recover over ten thousand examples from ChatGPT’s training dataset at a query cost of $200 USD” (Abstract)
- In a 2023 study, researchers recovered over ten thousand examples from ChatGPT's training dataset by prompting the model to diverge from normal output.
“we recover over ten thousand examples from ChatGPT’s training dataset at a query cost of $200 USD” (Abstract)