Dataset or benchmark
Vectara Hallucination Leaderboard: LLM hallucination rates when summarizing a document
Vectara, Vectara, GitHub, 2026
Read the originalRead on October 17, 2026
- What it is
- A public leaderboard that asks each language model to summarize documents using only the facts in them, then scores each summary with Vectara's Hallucination Evaluation Model. The README was last updated on September 22, 2026.
- How much weight it carries
- Published by a company that sells retrieval and search products, with its method and prompt in the same file. Rates are for one dataset at temperature 0, and models refuse some prompts, so a low rate does not prove a summary is complete. Vendor-run, but the method is open.
Facts taken from this source
- On the same Vectara leaderboard table, the highest hallucination rate is 24.2 percent, for the mistralai/ministral-3-3b-2512 model, which answered only 74.3 percent of prompts.
“mistralai/ministral-3-3b-2512|24.2 %|75.8 %|74.3 %|119.4” (Leaderboard table, lower rows)
- On the Vectara leaderboard updated September 22, 2026, the lowest hallucination rate in the table is 1.8 percent, for the antgroup/finix_s1_32b model.
“antgroup/finix_s1_32b|1.8 %|98.2 %|99.5 %|172.4” (Leaderboard table, first row)
- The Vectara leaderboard asks each model to summarize every document using only the facts in that document, then scores the summary for factual consistency.
“fed the full set of documents in the dataset to each of the LLMs and asked them to summarize each document, using only the facts presented in the document.” (Methodology section)