Peer-reviewed research
A framework to assess clinical safety and hallucination rates of LLMs for medical text summarisation
Asgari E et al., npj Digital Medicine, 2025 · DOI 10.1038/s41746-025-01670-7
Read the originalRead on October 16, 2026
- What it is
- The authors propose an error taxonomy, a clinical safety framework and a review interface for testing large language model (LLM) notes sentence by sentence against clinician annotations. Across 18 experimental configurations and 12,999 annotated sentences, they report a 1.47 percent hallucination rate and a 3.45 percent omission rate.
- How much weight it carries
- Peer-reviewed and widely cited. All authors were employees of Tortus AI, the company that sells the system tested, at the time of writing; the paper declares no competing interests. The rates reflect the authors' prompts and workflows and LLM note generation in general, not an independent test of each commercial scribe.
Facts taken from this source
- The error metrics in the npj Digital Medicine framework study were derived from 18 experimental configurations and 12,999 clinician-annotated sentences.
“Our clinical error metrics were derived from 18 experimental configurations involving LLMs for clinical note generation, consisting of 12,999 clinician-annotated sentences.” (Abstract)
- In the npj Digital Medicine framework study, LLM-generated clinical notes had a hallucination rate of 1.47 percent, measured against clinician-annotated sentences.
“We observed a 1.47% hallucination rate and a 3.45% omission rate.” (Abstract)
- In the same npj Digital Medicine study, LLM-generated clinical notes had an omission rate of 3.45 percent, meaning clinically relevant content from the visit was left out.
“We observed a 1.47% hallucination rate and a 3.45% omission rate.” (Abstract)
Pages that rest on this source
- Human vs AI Transcription: Accuracy, Cost, SpeedGuide
- AI Dictation: How AI Voice Typing WorksGuide
- AI Medical Scribes and Ambient Clinical DocumentationGuide
- Transcribe Audio With Whisper, ChatGPT or GeminiGuide
- AI Voice Notes: From Rambling to Summaries and To-DosGuide
- Can ChatGPT transcribe audio files?Question
- How accurate are AI medical scribes?Question
- How do you turn a meeting transcript into minutes?Question
- Medical scribeGlossary
- Are AI summaries of voice recordings accurate?Question
- What is the difference between a meeting transcript and meeting minutes?Question