This research introduces EvalPrompt, a novel methodology to assess large language models' accuracy and safety in medical self-diagnosis contexts.

• LLMs passed medical licensing exams but showed concerning rates of misinformation when used for self-diagnosis
• The study revealed dangerously inaccurate advice when presenting symptoms of serious conditions
• Models were found to provide incomplete disclosures about limitations in medical applications
• Urgent need for specialized evaluation frameworks identified before LLMs are deployed in healthcare settings

This research is critical for healthcare professionals, AI developers, and policymakers as it highlights significant patient safety concerns with current AI models while providing a structured evaluation framework for future testing.

Medical Misinformation in AI-Assisted Self-Diagnosis: Development of a Method (EvalPrompt) for Analyzing Large Language Models