Peer-reviewed research
On Faithfulness and Factuality in Abstractive Summarization
Joshua Maynez et al., Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020 · DOI 10.18653/v1/2020.acl-main.173
Read the originalRead on October 17, 2026
- What it is
- A large human evaluation from Google Research and Google DeepMind of neural abstractive summarization systems, which rewrite a document rather than copy from it. The annotators checked each summary for content that the source document does not support.
- How much weight it carries
- Peer-reviewed and widely cited. It tested 2020-era models on news articles, so modern models may hallucinate less, but the paper's finding that rewriting can add unsupported content is the basis of later summary benchmarks.
Facts taken from this source
- In a 2020 human evaluation, annotators found substantial amounts of hallucinated content in every summary that the neural abstractive models generated.
“Our human annotators found substantial amounts of hallucinated content in all model generated summaries.” (Abstract)