This research introduces a method to predict when black-box LLMs will make mistakes by using follow-up prompts as feature extractors, addressing a critical gap in LLM reliability assessment.

• Creates reliable performance predictors without requiring access to internal model data
• Extracts features through strategic self-queries to the model
• Enables detection of adversarial prompts and misrepresented model architectures
• Provides a practical approach for security teams to identify potential LLM failures

Why it matters: As organizations increasingly deploy third-party LLMs, having mechanisms to predict failures becomes essential for risk mitigation, security compliance, and maintaining user trust in AI systems.

Predicting the Performance of Black-box LLMs through Self-Queries