This study introduces the Critical Foreign Policy Decisions Benchmark to evaluate diplomatic biases in seven leading LLMs, assessing how they handle complex international relations scenarios.

• Models showed inconsistent preferences across diplomatic scenarios, with notable variance in military escalation and alliance building
• AI systems demonstrate tendencies toward certain diplomatic approaches that may not align with national security objectives
• Claude 3.5 Sonnet exhibited the most consistent positions, while other models showed variable policy preferences
• These biases have significant implications as LLMs are increasingly integrated into national security decision-making

Understanding these inherent biases is crucial for security institutions as they adopt AI for diplomatic analysis and content generation, requiring careful evaluation of models before deployment in sensitive contexts.

Critical Foreign Policy Decisions (CFPD)-Benchmark: Measuring Diplomatic Preferences in Large Language Models