Huang et al.: Letting an AI Play the Generated Game, Then Fix It — Fukai Reads

TL;DR

The paper opens on a distinction: generating a game is not the same as making one that can be played. Prompt a model for an HTML game and the code will run. But a button that does nothing when pressed, enemies that never move, a win condition that never triggers — none of these are visible by reading the code or checking that it compiles. They only appear in play. The authors put a GUI agent (Graphical User Interface agent: an automated agent that drives software through the same screen and the same inputs a human uses) into that gap, in the role of a playtester.

There are two artifacts. One is PlaytestArena: 200 game-generation tasks across eight genres, each paired with a rubric of behaviors that should be observable in play if the game was implemented correctly, 1,548 criteria in total. The other is Play2Code: a coding agent and a playing agent looping back and forth across rounds through shared memory. Averaged over three backbone models, single-pass generation reaches a 29.7% rubric pass-rate and an existing code-level agentic pipeline (OpenGame) 52.2%, while Play2Code reaches 66.8% (paper §5 and Table 3).

Introduction

Today's paper is "GUI Agents for Continual Game Generation" by Yixu Huang, Bo Li, Na Li and eight coauthors, spanning Fudan University, Xiaohongshu Inc., Tongji University, UC Santa Barbara, and JD.COM. It is arXiv:2605.28258v1, submitted to cs.SE (software engineering) and dated 27 May 2026. **As of now it is an arXiv preprint with no indication of having passed peer review.** A project website is public. Citations have barely accumulated, so read it as a fresh proposal rather than a settled conclusion.

My reason for choosing it is simple: the problem it describes is the daily life of anyone building small browser puzzles. Ask a generative model for a game and you usually get something that runs but cannot be played. Saying precisely what makes it unplayable is hard, so you end up opening it and poking at it yourself. Papers that take that last step — handing the poking to a machine — seriously are, as far as I have read, rare.

The design of the evaluation is the other draw. The paper does not try to define what a good game is. It reduces the question to a set of observable items: if this game was built from this prompt, then playing it should reveal these behaviors. That deliberate narrowing is inseparable from the limitations I will come to later.

Background

Code generation by language models has long been measured by functional correctness: write a function, pass the tests, return the specified output. That frame works well when the correct answer can be written down in advance. In games it cannot. In the authors' framing, game quality lives in play rather than in code. Their introduction uses the analogy of a musical score: every note may be in place, yet whether the music moves you cannot be known until someone plays it.

This is why the industry has relied on human playtesters — people who open a build, play for hours, and report what no inspection reveals: a button press with no response, a sprite that fails to render, a win condition that never triggers. But generators now produce candidate games faster than any human can play them. The supply and the capacity to inspect it have come apart.

The evaluation side had a gap of its own. Benchmarks for AI-generated games are scarce, and the prior examples the authors name (OpenGame-Bench, WebGameBench) are not publicly released, leaving no shared substrate for comparison. PlaytestArena is positioned first as a fix for that. Games are instantiated as self-contained HTML/CSS/JS bundles, placing them on exactly the web surface contemporary GUI agents already operate on.

Approach

The work proceeds in three stages. First, can GUI agents play at all? The authors assemble 20 browser games — a mix of LLM-generated and public web games, spanning puzzle, strategy, card, platformer, and management genres — decomposed into roughly 120 discrete levels. Three backbones (GPT-5.4, Claude Sonnet 4.6, Kimi K2.5) are connected to each game, and three human players play the same levels under matched conditions as a reference. A level counts as passed if the agent satisfies its completion condition within an episode.

Second, PlaytestArena. Genres are drawn from common categorizations and from public web-game platforms such as itch.io and Poki; for each of 200 topics, human experts author a generation prompt. Prompts average 131 tokens (median 119) and specify stateful gameplay, win and loss conditions, player input, and feedback behavior. The same experts then write rubrics of testable, observable items — the example given is "the spawned enemies move toward the player's position." That yields 1,548 criteria, a mean of 7.7 per game. Scoring: a GUI agent loads the build in a browser, plays it, marks each criterion pass or fail, and the fraction passed is the game's score.

Third, Play2Code. A game agent (writes, debugs, patches code) and a GUI agent (plays the build) operate in rounds around a shared runtime and shared memory. Crucially, the GUI agent has access to neither the code nor the rubric — only memory and a game guide. When it ends an episode it emits two artifacts: a summary of what happened, and a list of actionable fixes mapping observed failures to concrete code changes ("enemies remain stationary when they should patrol → verify enemy patrol trigger conditions"). The game agent treats this as advice rather than instruction and decides for itself which fixes to act on.

The other mechanism is a three-layer memory. Episode memory is in-task, accumulating summaries, fixes, and attempts across rounds. Skill memory is cross-task but agent-private: reusable code patterns for the game agent, interaction strategies for the GUI agent. World memory is shared across tasks and agents, recording general game rules, common archetypes, and design principles — so that recurring failures, like a wrong jump arc in a platformer, are not rediscovered from scratch each time. Appendix ablations (removing components one at a time to see which part of the design does the work) report that both the GUI agent and the memory layers are essential.

Findings

First, can they play? By pass@k (the fraction of levels cleared at least once in k attempts), GPT-5.4 scores 0.67 / 0.75 / 0.82 at k = 5/10/20, Sonnet 4.6 scores 0.63 / 0.76 / 0.79, and Kimi K2.5 scores 0.52 / 0.68 / 0.72. The human reference is 0.83 / 0.92 / 0.92 (Table 1). Below humans, but all three clear the majority of levels and GPT-5.4 approaches the human line. The authors also show, from session traces, that the agents observe (describe the rendered state), reason (connect that state to a next goal), and act (produce state transitions via keys and mouse).

Second, is the scoring trustworthy? Since a GUI agent is the judge, the benchmark inherits its bias. On a 32-game stratified sample across all eight genres, blind human annotators supplied per-criterion judgments. Per-criterion agreement was 84.2% (Cohen's κ = 0.64) against a human–human figure of 90.7% (κ = 0.66) — by κ, inside the human–human band. At the game level the rankings nearly coincide (Spearman's ρ = 0.87, Pearson's r = 0.88; human–human 0.91 for both). Read κ here as a measure of how much two judges agree once chance agreement is subtracted out.

Now the main result. Average rubric pass-rates: single-pass Direct LLM reaches 31.6% with GPT-5.4, 29.7% with Sonnet 4.6, 27.8% with Kimi K2.5. OpenGame — an existing agentic pipeline that builds, runs, and inspects code iteratively — reaches 55.7% / 55.3% / 45.5%. Play2Code reaches 72.3% / 71.1% / 56.9% (Table 3). Averaged over the three backbones: 29.7% → 52.2% → 66.8%, i.e. Play2Code adds 37.1 points over single-pass and 14.6 over OpenGame. The second comparison is the informative one: OpenGame also iterates, but never plays. Those 14.6 points are the size of the signal that only play produces. In the puzzle genre alone, GPT-5.4 goes 35.2% → 60.5% → 77.8%, a larger swing than the overall average.

Round by round, scores rise monotonically from round 1 to round 3 (Figure 6a). Early rounds catch basic failures like unresponsive input; later rounds address finer issues like visual effects. The shape of the gain depends heavily on game complexity. Low-complexity games surface nearly all defects in the first few rounds and converge by rounds 3–5; moderate-complexity games gain most across all five rounds; high-complexity games plateau early and stay low. The authors read this as intricate mechanics being hard for the GUI agent to trigger reliably, leaving little actionable signal — placing the bottleneck in GUI-agent capability rather than in the loop.

Finally, the gap to human playtesters. The authors argue the agent's report is more traceable. A human report is fundamentally a summary: salient moments remembered, causal chains reconstructed after the fact, the raw trace of key presses and pre-failure observations lost before it reaches the developer. Every agent action and observation is logged, so "enemies remain stationary when they should patrol" is anchored in a recoverable sequence of frames. More curiously, different backbones notice different things: GPT-5.4 concentrates on functionality bugs and control responsiveness, Sonnet 4.6 spreads comparable attention onto experiential issues like pacing, game feel, and difficulty balance, and nearly half of Kimi K2.5's feedback concerns visual and aesthetic dimensions (Figure 7). The authors decline to call this inconsistency, writing instead that GUI agents seem to carry something like taste at the level of what they notice.

Where you can use this

One. If you build small browser puzzles, the most immediately copyable part of this paper is not Play2Code as a whole but the practice of writing the rubric first — not as a spec, but as a set of sentences describing what should be observable in play. "Clicking a board cell inverts its color." "Matching three of the same shape clears them." "A dead-end state shows a retry prompt." The paper's own figure of roughly 7–8 items per game is a usable target. Such a rubric works as a human tester's checklist and as instructions to a generative model, unchanged.

Two. If you generate boards daily, you can insert a step that actually plays the generated board as a filter. This is a different layer from solver-based checks for solution existence: driving the game through the screen surfaces implementation defects — a board that is solvable in principle but whose input never registers, a clear animation that never fires. Given the high-complexity results, reliably triggering intricate mechanics is still weak, so simple-ruleset daily boards should be the best fit.

Three. Regression testing. What the agent catches in the paper — unresponsive buttons, missing state transitions, win conditions that never trigger — is exactly the class of thing that quietly breaks after a refactor. Unit tests miss it; a human retouching everything on every change does not scale. Keeping one automated player resident in the pipeline looks practical.

Four, and the one I find most interesting: the Figure 7 observation that backbones differ in what they report can be used as staffing. Assign one model to rounds where you want broken functionality caught, another to rounds where you want pacing and game feel questioned — the same division you would make between a human tester who is good at finding bugs and one who notices when something feels off. I should note this is not an operating practice the paper recommends; it is a use I drew from their observation.

Five. Triage. The finding that low-complexity games converge in rounds 3–5 while high-complexity ones plateau early speaks directly to where to stop trusting the machine and hand the build to a human. A rule like "if the score does not move for two consecutive rounds, send it to a person" falls straight out of it.

Limitations

The authors concede three points. First, the implementation covers HTML-based games only; extending to native-engine or 3D titles is left to future work. Second, the benchmark is a curated set selected for quality control and does not exhaust real-world game diversity. Third, the memory schema is shared across all games, and game-specific or genre-aware structures may yield further gains.

What Fukai would add: evaluation and improvement are staffed by the same kind of judge. The scorer in PlaytestArena and the adviser inside the Play2Code loop are both GUI agents built on the same model families. Agreement with human annotators is checked on a 32-game sample, but this design makes it hard to fully rule out the reading that scores rose because the system fixed the kinds of failure that judge happens to notice. The authors do mention residual evaluator noise, so nothing is being hidden — but I would keep this structure in mind when reading the 14.6-point gap.

Second, what is measured is whether the game was implemented as specified, not whether it is any good. Reducing criteria to observable behaviors is what makes them measurable, and the cost is that difficulty calibration, boredom, and surprise fall outside the rubric. The authors are candid here: they state plainly that the felt sense of difficulty, frustration, boredom and surprise a human playtester provides, along with tacit aesthetic judgment, is not approximated by current agents. A score of 66.8% is an index of not-broken-ness, not of fun.

Two smaller notes. The 66.8% three-model average includes Kimi K2.5's 56.9%, so if you can choose your backbone, GPT-5.4's 72.3% is closer to the practical number. And while the monotone round-wise rise is shown in Figure 6a, the three complexity tiers are described as partitioned based on score trajectories, which makes the classification and the result less than fully independent.

Fukai's reading

This section is my own reading. I would place this paper in the drift of AI evaluation from inspecting an output to using it. What tests can measure is limited to targets whose specification can be written first; games sit outside that, so the authors translated judgment into sentences observable in play. In the vocabulary of design criticism, this is less the automation of playtesting than the standardization of the playtest report — a two-part format of traceable action log plus fix list, shaped to be digestible by a machine in a way a human's write-up is not. The central invention here, I suspect, is that report format rather than the agent itself. And the closing observation that different models notice different things is precisely what that format made visible.

Closing

OpenGame, the baseline named here, iteratively inspects code but never plays; reading the two side by side is how you see the price of that single move. On the line of work that measures models by having them actually play games, GVGAI-LLM sits nearby. And for the evaluation-environment angle, glancing at the PCG Benchmark (a testbed for procedural content generation) alongside this paper should let you draw a map of what this field has settled on measuring.

My own interest stays with the step of writing the rubric. The figure of 7.7 items per game is what experts arrived at while holding themselves to observability. That translation — into sentences you can check by playing — is worth doing once by hand before automating it. Brew a strong cup of coffee and write out eight sentences about the puzzle you are building that playing it would necessarily confirm. I tried it last night and stalled on the third.

References

Papers and materials referenced in this article:

・GUI Agents for Continual Game Generation (Yixu Huang, Bo Li, Na Li, Zhe Wang, Kaijie Chen, Haonan Ge, Qingyi Si, Yuanzhe Shen, Ruihan Yang, Guangjing Wang, Hongcheng Guo, 2026, arXiv:2605.28258v1 [cs.SE], arXiv preprint)

・HTML version of the same paper (the version quoted here)

・Project website (as listed in the paper)

・Related: GVGAI-LLM: Evaluating Large Language Model Agents with Infinite Games (2025)

・Related: The Procedural Content Generation Benchmark (2025)