This research introduces an automated framework to evaluate consistency in large language models (LLMs) specifically for cybersecurity applications, addressing a critical trustworthiness gap.

• Developed methods to detect and quantify response inconsistencies across multiple LLM queries
• Evaluated LLMs against a specialized cybersecurity benchmark
• Identified key factors that influence consistency in security-related responses
• Proposed strategies to improve LLM reliability for security tasks

These findings are crucial for organizations implementing LLMs in security operations, where inconsistent responses could lead to vulnerabilities or misconfigurations in security systems.

Automated Consistency Analysis of LLMs