This research introduces a novel approach to evaluate fairness in LLMs when applied to high-stakes medical decisions like organ allocation.

• Introduces Borda scores from voting theory to create nuanced fairness metrics
• Evaluates how LLMs may perpetuate or mitigate biases in organ allocation systems
• Addresses the limitations of traditional accuracy-based evaluation methods
• Provides a framework for assessing group fairness when ground truth is unclear

This work is critical for the medical field as it helps ensure equitable outcomes in life-or-death scenarios where AI assists human decision-makers.

Original Paper: Ethical AI on the Waitlist: Group Fairness Evaluation of LLM-Aided Organ Allocation