Age of LLM: A Strategic 1v1 Benchmark for Reasoning, Diplomacy and Reliability of Large Language Models under Fog of War

A New Kind of AI Benchmark: Testing LLMs Under Strategic Fog of War

The research community has introduced "Age of LLM," a novel benchmark that pits two large language models against each other in a turn-based strategy game on a 13x7 grid. The objective is straightforward—destroy the opponent's base—but the conditions are deliberately adversarial. The benchmark introduces three specific stressors: fog of war (incomplete information), full diplomatic communication (including messages, ceasefires, and ultimatums), and secret resources (uranium kept hidden from the opponent). This moves beyond static question-answering or code generation tasks into a dynamic, interactive environment where models must reason, negotiate, and adapt under uncertainty.

Why This Matters

Most existing benchmarks test isolated capabilities—math, logic, or factual recall—in clean, single-turn settings. Age of LLM instead evaluates what might be called "strategic coherence": the ability to maintain a goal over multiple turns while managing incomplete information and social interaction. This is a fundamentally different challenge. A model that scores highly on GSM8K or MMLU might still fail here because it cannot handle the combinatorial complexity of simultaneous reasoning, deception detection, and long-term planning.

The inclusion of diplomacy is particularly significant. It forces models to engage in a form of theory of mind—inferring what the opponent knows, wants, or might bluff about. This is a capability that remains poorly understood and inconsistently measured in current LLMs. The secret uranium mechanic adds another layer: models must decide when to reveal or conceal information, a skill that touches on trust, negotiation, and strategic disclosure.

Implications for AI Practitioners

For developers and deployers of LLMs, this benchmark highlights several practical gaps. First, multi-turn strategic reasoning remains a weak point for most models. Even advanced systems often lose track of long-term objectives when faced with distracting diplomatic overtures or partial information. Practitioners should consider testing their models in similar adversarial, multi-step environments before deploying them in settings like automated negotiation, game AI, or military simulations.

Second, diplomacy and deception are not just "nice-to-have" features—they are core reasoning challenges. If your model cannot detect a bluff or formulate a credible ultimatum, it may be unsuitable for any application involving competitive or cooperative human interaction. This includes customer service, legal negotiation, and even some educational tools.

Finally, the benchmark underscores the importance of evaluation beyond accuracy. Age of LLM measures reliability under stress—how often does the model make catastrophic mistakes when information is hidden? This is closer to real-world deployment conditions than any static test. Practitioners should incorporate similar stress-testing into their own evaluation pipelines, especially for high-stakes applications.

Key Takeaways

• Age of LLM introduces a dynamic, adversarial benchmark that tests strategic reasoning, diplomacy, and decision-making under incomplete information—capabilities poorly captured by existing static benchmarks.
• The inclusion of fog of war and secret resources forces models to handle uncertainty and strategic disclosure, revealing weaknesses in long-term planning and theory of mind.
• For practitioners, this benchmark highlights the need to evaluate models in multi-turn, interactive settings before deployment in negotiation, simulation, or competitive environments.
• Reliability under stress—not just accuracy—emerges as a critical metric, especially for applications where information is partial or opponents are adversarial.