arxiv-2609-02459 SRC-0548 · paper

CivBench: A Long-Horizon Benchmark for Tool-Mediated Agents in Civilization VI

Austin Tudor David Andrews, Liam Wilkinson, Jamie Heagerty, Harry Coppock, Jakob Nicolaus Foerster, Rui Ponte Costa

Created: 2026-09-02 · Ingested: 2026-09-28

We present CivBench, an open-source benchmark for evaluating language model agents in long-horizon, tool-mediated environments through the Model Context Protocol (MCP). A single episode spans 300+ turns and produces thousands of tool calls over a large action space, requiring sustained planning, state monitoring, and execution under partial observability. The environment exposes 76 MCP tools and a narration layer that converts visual game state into structured text. We use CivBench to characterise agent behaviour across four model families in 23 admissible runs. The sample is a pilot, not a mo

Referenced by

Claims in this catalog that draw on this source, and whether as support or counterpoint.

supportsIn long-horizon tool-mediated environments where strategically relevant state is available only on explicit query (300+ turn Civilization VI via MCP, 23 runs, four model families under a shared playbook), agents allocate only about 1-2% of non-infrastructure tool calls to strategic monitoring and execute only about 48-66% of the near-term commitments they state in their own planning diaries, even though the playbook told them to query victory progress every 20 turns.