GPBench offers the first detailed evaluation framework for assessing how Large Language Models perform in general practitioner (GP) roles across diverse medical scenarios.

• Evaluates LLMs on real-world clinical tasks including disease diagnosis, treatment planning, and medication management
• Reveals significant performance gaps between current LLMs and human GP capabilities
• Identifies specific areas where LLMs need improvement to become reliable medical assistants

This research matters because it establishes clear metrics for measuring AI readiness in primary healthcare settings, potentially addressing medical resource gaps in underserved communities while highlighting necessary safety improvements before deployment.

GPBench: A Comprehensive and Fine-Grained Benchmark for Evaluating Large Language Models as General Practitioners