This research introduces a comprehensive evaluation framework for mental health chatbots, validated by healthcare professionals to ensure safe and reliable AI-powered mental health support.

• Created a benchmark with 100 test questions and ideal responses
• Developed five guideline questions to assess chatbot response quality
• Validated the framework with mental health experts
• Tested the approach on a GPT-3.5-turbo-based chatbot

This research addresses critical safety concerns in deploying AI for mental health support, providing healthcare organizations with practical tools to evaluate and improve chatbot implementations before patient interaction.

Building Trust in Mental Health Chatbots: Safety Metrics and LLM-Based Evaluation Tools