This research evaluates how large language models can assess factual consistency in automatically generated summaries, with a focus on medical applications.

• Introduces TreatFact, a clinical text summary dataset evaluated by domain experts
• Analyzes both open-source and proprietary LLMs for factual consistency evaluation
• Identifies key factors affecting LLM performance in detecting factual inconsistencies
• Highlights unique challenges in evaluating clinical text summaries

For healthcare organizations, this research provides crucial insights into mitigating misinformation risks in AI-generated medical content, helping ensure patient safety and maintaining trust in automated documentation systems.

Factual consistency evaluation of summarization in the Era of large language models