This research introduces a nonparametric contextual ranking framework that provides statistical confidence measures when comparing large language models.

• Addresses the critical challenge of quantifying uncertainty in LLM evaluations
• Enables hypothesis testing between different model rankings
• Provides domain-specific insights particularly valuable for medical applications
• Supports better-informed decisions when selecting models based on the best-of-N policy

For medical applications, this framework helps identify which models can be trusted in high-stakes healthcare scenarios, potentially reducing harmful hallucinations and improving clinical decision support.

Confidence Diagram of Nonparametric Ranking for Uncertainty Assessment in Large Language Models Evaluation