This research introduces a novel dataset of clinically uncommon patient cases to evaluate how well large language models perform on complex medical diagnoses that deviate from textbook scenarios.

• Features real-world cases with rare diseases and atypical presentations
• Evaluates LLMs' ability to handle diagnostic challenges beyond standard training data
• Addresses critical gap in current medical benchmarks that rely on exam questions
• Demonstrates the need for more diverse clinical training data

This work matters because it helps identify limitations in medical AI systems when faced with the complex, ambiguous cases that frequently challenge human physicians in practice.

CUPCase: Clinically Uncommon Patient Cases and Diagnoses Dataset