This research introduces the Medical Abstraction and Reasoning Corpus (M-ARC) to evaluate LLM reasoning capabilities in clinical scenarios, revealing significant limitations despite strong benchmark performance.

• LLMs demonstrate inflexible reasoning patterns when facing novel medical scenarios
• Models perform substantially worse than physicians on M-ARC tasks requiring abstraction and adaptability
• Research identifies specific failure modes in LLMs' clinical reasoning abilities
• Findings suggest caution is needed when deploying LLMs in real-world medical settings

These limitations highlight the gap between benchmark success and practical clinical reasoning, emphasizing the need for more robust evaluation frameworks before deploying AI in healthcare contexts.

Limitations of Large Language Models in Clinical Problem-Solving Arising from Inflexible Reasoning