This research introduces a novel framework for evaluating healthcare LLMs that goes beyond traditional question-answering methods.

• Identifies limitations in current evaluation approaches that rely solely on close-ended (factuality) or open-ended (expressiveness) assessments
• Introduces CareQA, a new medical benchmark designed to provide more comprehensive evaluation
• Demonstrates that combining both evaluation types provides a more complete picture of model capabilities
• Proposes automatic evaluation techniques that reduce dependency on human reviewers

This matters for healthcare because proper evaluation ensures medical LLMs provide factual, contextually appropriate information - critical for patient safety and clinical decision support.

Automatic Evaluation of Healthcare LLMs Beyond Question-Answering