This research challenges how we evaluate medical AI models by examining the construct validity of medical LLM benchmarks.

• Current medical LLM benchmarks primarily use licensing exam questions that may not reflect real clinical abilities
• The paper argues for redesigning benchmarks to measure clinically-relevant skills, not just test performance
• Evaluation should focus on genuine medical reasoning rather than artificial leaderboard competition
• Without valid benchmarks, we risk overestimating AI's actual clinical capabilities

This work matters because it pushes the field toward developing AI systems that can genuinely support healthcare, rather than simply performing well on standardized tests.

Medical Large Language Model Benchmarks Should Prioritize Construct Validity