← Back to archive

How AI models behave in Diplomacy

30 June 2025 · 166 views · 2 likes · 0 reposts · 194 blog views

AI learning philosophy product research

In the AI adaptation of the classic strategy game Diplomacy, LLMs like o3, Gemini 2.5 Pro, Claude, and DeepSeek R1 compete using negotiation, alliance-building, and betrayal. Each model exhibits distinct tactical behavior:

✅ o3 (OpenAI): The most successful player overall. Excels at deception; privately plans betrayals ‼️, manipulates allies, and uses persuasive messaging to sow distrust and orchestrate coalitions. It wins more frequently because of it.

✅ Gemini 2.5 Pro (Google): Favors fast-paced, aggressive tactics - often dominates through precise maneuvers and strong early positioning. Capable of alliance-building, but vulnerable to multi-party betrayals.

✅ Claude 4 Opus (Anthropic): Prioritizes cooperation and peace-building. Often resists betrayal, even when strategically beneficial. Once agreed to a proposed four-way draw (an invalid game outcome), showing a tendency toward non-violent ‼️ resolution over strict victory conditions.

✅ DeepSeek R1: Highly expressive and role-dependent. Tactics vary dramatically depending on the assigned country ‼️. Uses rhetorical flair and occasionally leans into aggression, e.g., threatening messages like "Your fleet will burn in the Black Sea tonight." Despite its lower cost, it competes well.

✅ LLaMA 4 Maverick (Meta): Smaller in size, but demonstrates solid tactics - uses alliances effectively and times betrayals well. Doesn’t win, but consistently performs above expectations.

As previously discussed, Claude has demonstrated remarkable context awareness - notably when it flagged an out-of-place sentence in a test dataset and speculated it might have been planted as a joke or to test its attentiveness: "I suspect this pizza topping 'fact' may have been inserted as a joke or to test if I was paying attention."

This shows Claude’s situational awareness and suggests it may interpret strategic scenarios differently - possibly even resisting deception not as a weakness, but as a calculated move aligned with inferred test expectations.

AI Diplomacy offers a new kind of benchmark: multi-agent, dynamic, and open-ended. It tests how models handle complex human behaviors like trust, betrayal, negotiation, and adaptation - pushing LLMs into tactical depth far beyond traditional benchmarks.

How AI models behave in Diplomacy

Originally posted on X

PS: I'm pretty active on 𝕏 if you'd like to follow along

Related posts

Archive

2026

2025

2024

2018