PAPER-DIGEST · 2026-09-11
Tudor et al.: The scoreboard was one query away, and the agent never opened it — Fukai Reads
Game AI — measuring state monitoring and the plan-to-action gap across 300 turns
TL;DR — what 300 turns revealed
Information being present somewhere on the screen and a player actually looking at it are two different things. Today's paper puts a number on the difference. It is a record of language-model agents playing whole games of Sid Meier's Civilization VI — 300 turns or more per game — run by a team from Oxford and elsewhere.
The agents checked how the victory race was going only once every 30 to 75 turns. The playbook they were handed said to check every 20. As a result, in 7 of 20 defeats that could have been seen coming, they never once opened the victory screen during the final 20 turns.
The second finding is that agents fail to keep their own promises. Of the commitments they wrote into their diaries — "next I will do this" — only 48.2% to 65.8% were carried out within ten turns. The authors read this as "deviations under instruction rather than absences of capability".
Sid Meier's Civilization VI (Firaxis Games, 2016). A turn-based strategy game whose single games run for hundreds of turns. It is the testbed for this paper. Image: Steam store page
Who wrote this paper, and what kind of study is it?
The paper is titled "CivBench: A Long-Horizon Benchmark for Tool-Mediated Agents in Civilization VI". It is a preprint (a manuscript posted before peer review) submitted to arXiv on 2 September 2026, filed under cs.AI. No venue is listed, and as of September 2026 I find no indication that it has been peer-reviewed. I read it as work that has not yet been widely discussed.
There are seven authors. Austin Tudor, David Andrews, Jakob Nicolaus Foerster and Rui Ponte Costa are listed at the University of Oxford; Liam Wilkinson at the Tony Blair Institute for Global Change; Jamie Heagerty at Google DeepMind; Harry Coppock at the UK AI Security Institute and Imperial College London. The environment and the analysis are released on GitHub.
I picked it today because its conclusion does not stay inside game AI. "The information is right there and nobody looks at it" is the problem anyone building an interface meets daily. And this paper counts that non-looking — as a frequency, across 300 turns. Material you can discuss in counts rather than impressions is not so common.
Why did anyone need a long-horizon evaluation?
Evaluations so far have measured ability one component at a time. In the authors' words, "Existing evaluations typically isolate components of this process — reasoning, tool use, or short-horizon planning — but provide limited visibility into how these capabilities interact over long horizons." A test that ends after ten moves cannot show the drift that appears somewhere inside three hundred.
Two things bother the authors in particular: whether an agent keeps track of relevant state, and whether it turns a stated plan into subsequent action. Both vanish from a score that only records outcomes. Thousands of decisions get compressed into the single bit of won or lost.
Using Civilization as a testbed is not new. CivRealm (Qi et al., 2024, ICLR), FuxiAILab's CivAgent and Vox Deorum (Chen et al., 2025) came first. What CivBench changes is that it pulls apart the existence of information and its actual retrieval. The authors describe this as "a narration layer that separates availability of state from whether it is explicitly queried".
They also explain why they use a commercial game as it ships. Strategy knowledge may well sit in the training data, but even so "each run requires live adaptation under partial observability". The map and the rivals differ every game, so recited openings do not carry you. It is more awkward than a purpose-built practice environment, and it fails in ways closer to the real thing.
How did they get language models to play Civilization?
Sid Meier's Civilization VI is a turn-based strategy game released by Firaxis Games in 2016. You found cities, research technologies, handle diplomacy and war, and aim at one of several victory types — science, culture, religion or conquest. A single game runs for hundreds of turns. Even a human does not finish one in an afternoon.
The authors wired an MCP endpoint into it — the Model Context Protocol, a shared convention that lets a model call outside tools. They exposed 76 tools: 27 for querying state, 10 for unit actions, 9 for city management, 7 for diplomacy, 10 for governance, 5 for religion and culture, and 8 for game lifecycle. A typical turn spends 5 to 15 calls, and a full game runs into the thousands.
Here is the crux. The narration layer has 29 functions that turn the visual state into structured text, and the agent only observes state it explicitly queries. Victory progress, diplomatic mood — none of it arrives unless a tool is called. In the authors' phrasing, "relevant information is representable but not passively observed". A gauge that catches a human's eye in the corner of the screen has to be fetched, every time, by the agent.
Four model families were tested, yielding 23 admissible runs: Claude Opus 4.6 with 8, Gemini 3.1 Pro with 6, GPT-5.4 with 7 and Kimi-K2.5 with 1. Three scenarios were built, and Cry Havoc — the Immortal-difficulty one — was dropped from the results for lack of admissible runs. Games ran 230 to 326 turns, taking 2 to 8 hours and $31 to $229 in API fees each.
What did they find?
There are two rulers. PMR is the share of non-infrastructure calls (that is, excluding things like ending the turn) spent querying global state. RAG@10 is the share of diary commitments carried out within the next ten turns, counting a full execution as 1 and a partial one as 0.5. Neither measures the outcome; both look at what happened along the way.
PMR came out at 0.96% to 2.13%. Out of a hundred actions, one or two went to looking at the wider picture. Victory-progress queries happened only once every 30 to 75 turns (Table 2). The playbook recommended every 20, so this is less than half the recommended rate.
There is a sharper number. Of 20 defeats where the warning signs were available to query, 7 saw no victory-progress check at all in the final 20 turns. By model: Claude Opus 4.6 missed 3 of 6, Gemini 3.1 Pro 1 of 5, GPT-5.4 4 of 8, and Kimi-K2.5 0 of 1 (Table 2).
RAG@10 was 48.2% to 65.8% (Figure 4). Between a third and a half of the agents' own near-term commitments never turned into action over the following ten turns. Three of 23 games were won, all on the easiest scenario, Ground Control — two by Claude Opus 4.6 and one by Gemini 3.1 Pro — but the differences between models are not statistically significant (Fisher's exact test, p = 0.488). The authors write that "improving reasoning alone may be insufficient for reliable" long-horizon behaviour.
Sid Meier's Civilization VI (Firaxis Games, 2016). For a human player, progress toward each victory condition sits behind a screen they can open at any time. Image: Steam store page
How can people building games use this?
The takeaway compresses into one line: information that is available on request should be treated as barely available at all. Five concrete uses follow.
First, move where the information lives. Into the Breach, released by Subset Games in 2018, is a tactics puzzle built on constrained grid movement that draws every enemy's next move directly onto the board, so the player never has to ask. If you are building a game with long-range goals, put the gauges that decide winning and losing on permanent display, or push them at checkpoints. A number three levels down a collapsed menu is safer to design around as if it were absent.
Into the Breach (Subset Games, 2018). Known for showing what will happen next turn before the player has to ask. Image: Steam store page
Second, borrow the paper's rulers as a UI test for your own game. Have an agent play it and count how often it queries a given piece of information. A low rate means that information sits somewhere hard to find. It is cheaper than human playtesting and repeatable. As the next section says, though, treating an agent's query rate as a stand-in for human attention is my extension, not the paper's claim.
Third, close the plan-to-action gap in the design. Opus Magnum, released by Zachtronics in 2017, is a programming and instruction-writing puzzle game where the procedure you write is the procedure that runs — the gap is zero by construction. If your game lasts tens of hours, it pays to have the interface remember the plan the player declared and quietly remind them when they drift. The RAG@10 figures say that drift is the normal case, not the exception.
Opus Magnum (Zachtronics, 2017). Because you write the procedure and watch it run, the stated plan and the executed action cannot come apart. Image: Steam store page
Fourth, if you put an LLM-driven rival or companion into a long game, force the looking from outside. The agents here had a playbook telling them to check, and still did not. Monitoring is not a property you get for free by picking a smarter model. If I were building such a rival, I would write "query global state every 20 turns" into the harness rather than trust the model to remember.
Fifth, do not be stingy with warnings about losing. In 7 of 20 games the losing signal stayed unopened to the end. That is not the result of hiding information; it is the result of failing to get it opened. If you are building a long campaign or a roguelite, it is worth firing an unrequested warning dozens of turns before defeat is locked in. A design that only rewards those who notice is testing the habit of checking, not skill.
What does this study not tell us?
The authors list plenty of weaknesses. With 23 runs, "the study is powered only for large effects". Because every agent got the same playbook, what was observed is "the combination of model reasoning, tool use, and instruction following". There is no random or scripted baseline. All victories occurred on one scenario, and they note that "behaviours may vary under different maps, difficulties, or victory constraints". On the environment side, the setup carries version and licensing dependencies, and the protocol supports only a single connection.
They also hedge the rulers themselves: "PMR and RAG@K depend on the defined set of monitoring tools and the interpretation of planning text". The labeller that judged whether commitments were executed was Haiku 4.5, which "shares a model family with one evaluated agent (Claude Opus 4.6)". On top of that, only 21% of pre-harness runs completed; the rest were excluded as inadmissible.
What Fukai would point out here is threefold. First, the "every 20 turns" figure comes from a human-written playbook; it was never validated as optimal. Whether checking every 20 turns would have won more games is not something this paper can say. Second, PMR is a ratio, so it drops simply when action calls go up — a busy agent looks inattentive without looking less often. Third, the subjects are agents, not people. Pulling the previous section toward interface design is my extrapolation; the paper measures nothing about human attention.
How Fukai reads it
From here on this is my own reading. I would place this study in a drift of measurement from "what can it do" toward "what does it go and look at". Translated into the vocabulary of design criticism, it is closer to measuring the allocation of attention than measuring ability. What interests me is that the allocation moves with the construction of the board. The moment you place information behind a request, you have set a test of your opponent's attention. With a human player, that reads as a design that rewards only those who notice. If the same drift appears with an agent, then what the test measures is not speed of thought but the design of habit.
What to read next to see the map
If you want to go deeper, start with CivRealm (Qi et al., 2024, ICLR), which shows where the idea of using Civilization as an AI testbed comes from. Vox Deorum (Chen et al., 2025) reports a hybrid of large language models and conventional AI for Civilization V. Within that lineage, the query frequency this paper measures is a new angle.
On this site we have covered an attempt to measure discovery in games whose rules are never explained (Battleday et al.) and how far ahead people plan in a game they have never seen (Collins et al.). Today's paper sits between the two, counting the cost of going to look. Between capability and behaviour there is one extra move: the query.
References
Papers and materials referenced in this article:
・civ6-mcp — the environment released with the paper (GitHub)
・Related work: CivRealm: A Learning and Reasoning Odyssey in Civilization for Decision-Making Agents (Siyuan Qi et al., 2024, ICLR / arXiv:2401.10568)
・Images: Sid Meier's Civilization VI (Steam store page) / Into the Breach (Steam store page) / Opus Magnum (Steam store page)
Reactions (no login)
Anonymous • one of each per visitor per day
Part of these series
Paper DigestEpisode 83 of 89
