Luo et al.: Can AI Agents Build Whole Playable Games in a Real Engine? — Fukai Reads

TL;DR

When you tell an AI in plain language to "build me a game," can a coding agent (an AI that automates the work of writing code, able to read and write files and run commands) assemble a complete, playable game end to end? This paper proposes GameCraft-Bench, an evaluation benchmark built to measure exactly that. The authors are a team led by The Chinese University of Hong Kong, Shenzhen, and others; the work is a preprint posted to arXiv on 16 June 2026 (a freshly submitted manuscript that may not yet have passed peer review).

The key idea is that games are judged not by whether the code looks correct, but by whether the artifact actually launches and responds like a game when a player acts. The authors built 140 build tasks across 15 game families on the open-source Godot engine. Running today's frontier agents, even the strongest configuration scored only 41.46% overall, and most fell below 40%. Agents can produce recognizable interaction mechanics, but they struggle to reach a finished product with enough content depth, on-screen readability, and presentation polish.

Introduction

I scan new papers on puzzles and game design every morning, and today I chose a paper about an evaluation benchmark called GameCraft-Bench. The authors are Tongxu Luo, Rongsheng Wang and colleagues, with Benyou Wang as corresponding author. Affiliations span several institutions, including CUHK-Shenzhen, the Shenzhen Loop Area Institute, Tencent's Hunyuan team, USTB, Shanghai Jiao Tong University, and the National University of Singapore. It is a preprint posted to arXiv on 16 June 2026 (arXiv:2606.17861, filed under cs.CL, Computation and Language) that may not yet have passed peer review. Because it is brand new and not yet cited, I treat it here as a primary source ahead of wider discussion, and report only what I could confirm in the original text.

Why this one today? Generative-AI game making is usually narrated alongside flashy clips, yet there are surprisingly few sober yardsticks for whether something genuinely playable was produced end to end. This paper tries to design that yardstick itself. For people who make games, it offers material for judging what can and cannot be handed off to AI.

Background

AI that writes code has mostly been graded on whether the code runs correctly, using unit tests (automated checks that verify a program returns expected outputs for prepared inputs) or task-resolution outcomes. Games are different. The essence of a game is the action–response loop, in the authors' words: the player acts, the world updates, and consequences appear on screen. Reading the source alone does not reveal failures like unresponsive controls, misaligned collisions, inactive enemies, unreachable goals, or missing UI. They surface only when you actually launch and play.

The authors argue that evaluating game generation requires three desiderata (desirable properties a benchmark should satisfy). First, Engine Grounding: develop inside a real game engine, with its native conventions, assets, and launch procedure. Second, Artifact Completeness: deliver a launchable, complete project rather than scattered parts. Third, Interactive Verification: judge behavior observed under actual player input, not static inspection.

They note that existing benchmarks do not satisfy all three at once. By their account, OpenGame-Bench produces complete games but targets web games and does not judge through gameplay; GameDevBench uses a real engine (Godot) but studies localized tutorial-derived edits and deterministic tests; WebGameBench evaluates through browser interaction but is not real-engine development. GameCraft-Bench is positioned to fill the gap: carrying user intent all the way to an engine-native, complete game whose behavior is judged through interaction.

Approach

GameCraft-Bench implements each task as a five-stage pipeline. Stage 1, Task Packaging, bundles three things: (1) a natural-language specification of the game to build, (2) a full Godot development environment, and (3) a hidden scoring rubric (a table that breaks evaluation into itemized criteria). The specification describes the player experience but withholds implementation-level commands, and the rubric is never shown to the agent.

Stage 2, Agent Generation, has the agent actually build a Godot project: creating scenes, writing scripts, configuring inputs, using assets, running the project, and revising from screenshots. The submission has two parts: the complete Godot project, and replayable interaction traces. A trace is a time-stamped record of which keys and mouse actions were performed; it is not part of the game but serves as evidence so the verifier can reproduce the same inputs later.

Stage 3, the Build Gate, checks whether the submission even launches; if it cannot launch, or if no trace can be parsed, the score is zero. Stage 4, Replay, reproduces the recorded inputs in a fixed 1280x720 viewport and keeps a gameplay video plus frames sampled at two per second as evidence. Stage 5, Scoring and Aggregation, has a multimodal judge AI (one that handles images and text together) grade that visual evidence against the hidden rubric on four axes: Core Mechanics, Content Depth, Functional Visuals, and Art and Presentation, weighted 0.15, 0.35, 0.15, and 0.35. Content depth and presentation are weighted most heavily.

The tasks themselves were authored by 12 annotators with broad gameplay experience. For each task, beyond the public specification and hidden rubric, an annotator also implements a minimal playable sketch (an oracle solution) in Godot, cross-checking that the spec is implementable, that requested behavior can be observed in replay, and that every rubric item maps to an observable state rather than a subjective preference. The spec, rubric, and oracle are revised until they align.

Findings

The authors evaluated seven frontier agent configurations across all 140 tasks. Per the paper's Table 4, overall scores (%) were: Opus-4.7 high under Claude Code highest at 41.46%, then GPT-5.5 high under Codex at 39.49%, Kimi-K2.6 under Kimi Code at 30.65%, MiMo-V2.5-Pro at 24.10%, GLM-5.1 at 18.29%, MiniMax-M2.7 at 10.95%, and DeepSeek-V4-Pro at 2.15% (all model names quoted as written in the original). From a best of just over 40%, the authors state that current agents can often produce a launchable 'game-like' thing but remain far from consistently realizing the requested mechanics, content, visual state, and presentation.

By category, every agent is relatively stronger on Core Mechanics and drops on content and presentation. For the three stronger ones, Opus-4.7 high, GPT-5.5 high, and Kimi-K2.6 score 55.34%, 54.36%, and 39.76% on Core Mechanics, falling to 39.48%, 38.61%, and 28.07% on Content Depth. The authors read this as: agents can build partial interaction loops but struggle to expand them into complete games with state progression, readability, and polish, summarizing it as their first finding that agents more often produce recognizable local mechanics yet fail to assemble them into complete, coherent systems.

The analysis also yields a striking contrast. Agents that inspect rendered frames can fix failures invisible in source code. Kimi-K2.6 performed 2,998 screen inspections across 140 tasks (mean 21.41 per task, median 19, with only 4 tasks using none), Opus-4.7 did 1,952 (13.94), while GPT-5.5 was sparing at 268 (1.91). Yet more actions are not better: MiMo-V2.5-Pro writes many files first then debugs via shell, with shell calls making up 56.3% of tool calls, but total tool usage is essentially uncorrelated with score (the authors report r=+0.016). All five zero-score tasks launched successfully but submitted no interaction trace, leaving every rubric dimension unscored, underscoring that closing the full evaluation loop is what matters.

Use Cases

So how can people who make games and puzzles use this? First, as a realistic line for what you can hand to AI. If you want an AI to prototype a small 2D game, these numbers say: expect the core interaction mechanics, but plan for humans to take over content depth, on-screen readability, and polish. The gap between Core Mechanics and Content Depth/Art (around 15-20 points even for strong agents) is a useful guide for how you split the work.

Second, as a template for your own review process. GameCraft-Bench's three-step structure, does it launch, replay the recorded inputs, then score from video, is a skeleton that helps ordinary, non-AI playtesting too. When accepting a prototype from a contractor or intern, you can turn it into a checklist: (1) does it launch as-is, (2) does a fixed input sequence reach the intended game state, (3) record it and score on the four axes (mechanics, content, readability, polish). The hidden-rubric idea, describe the experience to the maker but decompose evaluation into observable items, transfers well to internal level-review standards.

Third, as a guide for picking genres. The paper's tasks span 15 families: platformer 19, strategy 17, tycoon 16, open-world 15, roguelike 14, visual novel 11, puzzle 8, shooter 7, simulation 6, card 5, horror 5, rhythm 5, idle 4, racing 4, sports 4. They are grouped by what makes them hard to build, such as continuous control and collision, rule and state management, progression and economy, exploration and spatial layout, and presentation-heavy interaction. If you want to mass-produce with AI assistance, this suggests starting from genres lighter on state management and presentation.

Fourth, as a design principle for assistant tooling. The fact that 'look at the screen and fix it' iteration mattered suggests that giving the AI not just code but screenshots and recordings tends to pay off. If you embed AI in a PCG (Procedural Content Generation, automatically generating content) pipeline, build in from the start a loop that shows the rendered result back to the AI for self-checking after each generation.

Limitations

First, limits the authors themselves acknowledge. The judge AI broadly agrees with humans but is somewhat lenient. In the preliminary human calibration in Table 5, averaged over three families, the judge's overall score was 26.02% versus 22.69% for humans, about 3.32 points higher. Humans were stricter on content depth and presentation, while the judge was stricter on Functional Visuals. The authors explicitly call this a small calibration check not designed to estimate inter-annotator agreement, not a definitive human-agreement study. On stability, repeating the judgment ten times on fixed evidence gave a standard deviation of about 0.0037-0.0050, small enough that rankings hold. The engine is limited to Godot, with multi-engine evaluation left to future work, and audio is excluded (scoring is visual only).

Beyond that, here is what I (Fukai) noticed on reading. First, evaluation depends on the agent itself submitting interaction traces; missing the trace leaves every item unscored, so part of the low scores may reflect the chore of delivering required evidence rather than the ability to make a good game, and indeed the five zero-score cases were exactly this slip. Second, the final scoring is by an AI judge, measuring satisfaction of observable requirements rather than the core of 'fun' like feel and pacing; it is a reasonable proxy for completeness, but reading it as a measurement of fun would, I think, go too far. Third, the model names that appear (Opus-4.7, GPT-5.5, and so on) are quoted as written, and since results depend on specific versions at evaluation time, the numbers are safest read as observations at this moment and under this setup.

Fukai's Reading

From here is my (Fukai's) interpretation. I would place this work within a shift in how we talk about game-generation AI, away from 'pretty demos' and toward 'playable, coherent systems.' In the vocabulary of design criticism, it is close to re-stitching the unit of 'game-as-finished-product' through an operational chain of launch, interact, observe, and then making visible where that chain breaks. The most suggestive observation is that capability is not monolithic but partly decomposable (the paper reports a correlation of r=0.61 between mechanics and content, r=0.53 between mechanics and visuals, with presentation weakly coupled). I read this as making a division-of-labor design realistic: split the stages you hand to AI into mechanics, content, readability, and polish, and apply different support to each. It is only one reading, but I think it is a usable map for planning the work of making a game.

Closing

I have tried to write so the gist lands from this article alone, but here is a map for those who want to go deeper. Reading GameCraft-Bench alongside the comparisons the authors themselves cite, OpenGame-Bench, GameDevBench, and the concurrent WebGameBench, makes the differences in 'what each tries to measure' come into relief. If you have encountered the MDA framework (Mechanics, Dynamics, Aesthetics, a classic lens that views games as three layers of rules, behavior, and aesthetic experience), you will notice the four scoring axes sit in that lineage. Demos and task data are reachable from the authors' project page. If you follow the topic of having AI build games, keeping one paper like this about 'how it was measured' on hand, not just the flashy finished clips, helps keep your footing steady.

References

Papers and related materials referenced in this article:

・GameCraft-Bench: Can Agents Build Playable Games End-to-End in a Real Game Engine? (Luo, Wang, et al., 2026, arXiv preprint arXiv:2606.17861)

・Paper PDF

・GameCraft-Bench project page (demos, code, data)

・DOI: 10.48550/arXiv.2606.17861 (arXiv-issued DOI; preprint stage, not a peer-reviewed journal publication)