Fact · Voice and AI
In a 2020 human evaluation, annotators found substantial amounts of hallucinated content in every summary that the neural abstractive models generated.
Checked against its source on October 17, 2026
In the source's own words
“Our human annotators found substantial amounts of hallucinated content in all model generated summaries.”
On Faithfulness and Factuality in Abstractive Summarization, Joshua Maynez et al., Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020, Abstract.
Open the original (DOI 10.18653/v1/2020.acl-main.173)- What it measures
- The authors evaluated several neural abstractive summarization systems on news articles, asking annotators whether each summary was faithful to its source document. The models were trained before the large chat models of 2023 onward.
- What it does not show
- A 2020 study of news summaries. It does not measure the current chat models, a voice memo, or a meeting, and it does not give a single percentage for all summaries.