This research introduces the first comprehensive benchmark dataset specifically for evaluating language models on mental health questions.

• Addresses critical need for standardized evaluation in mental health AI applications
• Covers diverse mental health topics including anxiety, depression, and trauma
• Incorporates medical literature from PubMed abstracts for knowledge-intensive testing
• Provides a foundation for developing more reliable mental health support solutions

Why it matters: Mental health conditions affect millions worldwide, and AI-powered systems that can accurately answer mental health questions could significantly expand access to reliable information and support.

MHQA: A Diverse, Knowledge Intensive Mental Health Question Answering Challenge for Language Models