Zhou et al.: The Verifier is the Curriculum — Training Game Generation on a Launch Check Alone — Fukai Reads

TL;DR

When you post-train a language model to generate whole games, what decides whether an output is good? The prevailing answer is to have another AI score it (a learned judge). This paper first confirms that such a judge can be gamed, then adopts the opposite signal: a black-and-white check of whether the generated project launches cleanly in the Godot engine. The authors call it strict-launch.

Using only that gate to run three rounds of "generate, keep what launches, retrain," clean generation on four completely held-out families (horror, rhythm, puzzle, shooter) rises from 8.8% to 42.2% per candidate, and best-of-16 coverage of the 25 held-out tasks goes from 18 to 25 — the same ceiling as the gold references.

Swap the gate for a lenient check and the entire gain disappears. The claim is the title: the verifier is the curriculum — what it certifies is what the model learns.

Introduction

Today's paper is "The Verifier is the Curriculum: Execution-Gated Self-Distillation for Cross-Family Game Generation" by Chenyu Zhou, Qiliang Jiang, Shuning Wu and Xu Zhou, from the School of Engineering at Institute of Science Tokyo, the College of Control Science and Engineering at Zhejiang University, and the Department of Electrical and Computer Engineering at the National University of Singapore. Xu Zhou is the corresponding author. It is an arXiv preprint (arXiv:2607.09709) submitted on 23 June 2026 — not peer-reviewed — released under CC BY 4.0.

It is new enough to have no citations yet, so this article should be read as a look at a fresh primary source that the field has not yet argued about. I picked it because its subject is not model size or a new architecture but the design of the acceptance criterion inside a production loop — which is exactly what people who build games and puzzles by hand do every day.

I read the arXiv HTML version, all sections plus appendices A, B and C. Everything below comes from what I actually read. I did not view the figure images themselves; I only used the numbers stated in their captions.

Background

The testbed is GameCraft-Bench (Luo et al., 2026): hand the model a short natural-language design brief and it must emit a complete, runnable Godot 4 project — at minimum an engine config file, scene files, the GDScript that implements behavior, and a recorded input trace that exercises the game. Correctness is conjunctive: there is no partial credit. If a script fails to parse, the engine refuses the whole scene.

That conjunctive property makes scoring hard. GameCraft-Bench's official metric, Overall, is a GPT-5.5 visual judge that renders the running game and rates it on several qualitative axes. In Appendix A the authors measure a hole in that judge: with the code entirely frozen, swapping solid-color placeholders for real sprites raises the art score — and it still rises when the replacement sprites are chosen at random from comparable sources. What moves the judge, they report, is the presence of bitmap assets, not their fitness.

At the same time the judge's absolute scores are tiny. Across 30 measured candidates in Appendix A, even strict-launch-clean ones average Overall 0.064 (median 0.052, max 0.233), while every candidate that fails to launch scores exactly 0.000. Even the held-out gold references average only 0.197, with large family variation (horror around 0.31 versus rhythm around 0.08). The judge is both gameable and coarse. That is the starting point for asking what else could serve as the signal.

Approach

The authors organize verification into a four-rung ladder: (1) BUILD — can the project be opened headless (run from the command line with no window) and quit; (2) strict-launch — does it launch headless with exit code 0 and no parse, load or runtime error; (3) execution grounding — actually drive it and measure whether state changes; (4) the multimodal visual judge. Higher rungs certify more, but the top one is gameable. Only rung two is used as the training gate.

The load-bearing number is the gap between rungs one and two. Of 888 candidates the supervised-only model produced for training tasks, BUILD accepts 887 (99.9%) while strict-launch accepts 73 (8.2%). Nearly everything "builds"; almost nothing runs clean. And all 815 that fail strict-launch contain a GDScript or config parse error. Appendix B lists the typical modes: scripts that never load because they fail to parse, malformed config files that make the engine open its project manager instead of the game, and broken external resource references in scene files.

The loop itself is plain rejection-sampling self-distillation (sample many candidates, keep only those meeting a condition). For each brief the model samples K=8 candidates, each is materialized as an actual file tree, and a candidate counts as clean if it passes strict-launch and has at least three files. After deduplication, at most two are kept per brief. This is the round's "fuel": it is appended to the pool of 111 gold projects, and a fresh LoRA (training only a small add-on module rather than the whole model) is trained from the base model each round. The fuel grew 67 → 123 → 186.

Setup: base Qwen3-14B; LoRA r=16, alpha=32; 8 epochs; learning rate 1e-4; batch size 1 with gradient accumulation 8; max sequence length 24576. Generation via vLLM 0.11.2 serving the LoRA unmerged at bfloat16. Evaluation is K=8 on the 25 held-out tasks across four seeds, i.e. 800 candidates per model, with significance from cluster-permutation testing at the task level (30,000 sign-flip permutations). Sampling temperature and the compute used are not stated in the paper.

Findings

The headline result is Table 1. Per-candidate clean-launch on the 25 held-out tasks across four unseen families: 8.8% for supervised-only, then 13.6%, 30.9% and 42.2% (Wilson 95% CI 38.9–45.7) over three rounds. Best-of-16 coverage goes 18/25 → 19/25 → 22/25 → 25/25, reaching the gold ceiling. Every round is a significant improvement: round 1 over supervised gives p=0.0019 (effect +4.9pp, 95% interval 1.9–8.1), and rounds 2 and 3 each give p<1e-4. Round 3 beats the supervised model on all 25 tasks (25 wins, 0 losses). Samples needed for one clean held-out game falls from about 11 to about 2.4, roughly a 5x improvement.

The controls are the interesting part. CTRL matches round 1's fuel per task but appends 67 exact duplicates of gold training projects — same quantity, same task coverage, only content diversity differs. It lands at 5.6%, significantly below the base model's 8.8% (7 wins / 13 losses, p=0.019), with coverage collapsing to 13/25. Round 1, at the same budget but with self-generated fuel, beats CTRL decisively (16 wins / 5 losses, p<1e-4), and that holds in every leave-one-family-out fold (p ≤ 0.017). Adding data is not what helps; adding different content is.

The direct evidence that the gate's precision is the cause is the filter swap. Rerun the same loop with the lenient BUILD check — which passes 99.9% of generations — instead of strict-launch, and 81% of the collected fuel (54 of 67) is strict-launch-broken. A model trained on it reaches 8.6% clean-launch: indistinguishable from the base 8.8% (paired p=1.0) and far below round 1's 13.6% (5 wins / 16 losses, p=1e-3). Change nothing about the optimizer, budget or task distribution — only the filter's precision — and the entire first-round gain vanishes.

The gain is also decomposed. Holding budget and task coverage fixed and swapping only the generator from round 1 to round 2 gives a quality channel of +8.8pp (p=1e-4); the extra fuel round 2 actually collected (67 → 123) gives a quantity channel of +8.5pp (p=2e-4). The quality channel alone exceeds round 1's entire +4.9pp gain over supervised. Two audits confirm the outputs are not empty shells that merely launch. Statically, clean RFT projects have 43% more GDScript than clean SFT ones (median 331 → 473 lines), roughly three times as many signal connections, and more files (8 → 9.5). Dynamically, the number of held-out tasks with at least one grounded clean candidate rises monotonically 10 → 12 → 14 → 15 out of 25, and at a matched budget round 1 yields 16 grounded candidates against CTRL's 5. As validation of the grounding measure itself, 15 of 15 gold projects are grounded (median 0.75, minimum 0.58) while 0 of 16 known-bad ones are (an empty shell scores 0.013).

By family (Table 2, best-of-K), rhythm — weakest for the supervised model — goes 2/5 → 5/5, puzzle 5/8 → 8/8, shooter 6/7 → 7/7, and horror stays saturated at 5/5. By round 3 every family is at full coverage, though the path is not monotone: shooter dips from 6/7 to 5/7 at round 1. CTRL regresses in three of the four families. On training randomness, retraining round 1 under three independent seeds gives 13.25 ± 0.65% (mean ± s.d.).

Where you can use this

Read from the perspective of someone building puzzles or games, the most portable idea is a design rule: make the gate on your generator a mechanically decidable property, not a matter of taste. If you are building a Sokoban-like puzzle generator, put deterministic predicates at the gate before any "is this fun" scoring model — does a solver find a solution, is it unique, is the number of wasted moves under threshold. In this paper's terms, whatever you accept becomes the distribution your next model learns.

For hyper-casual PCG (Procedural Content Generation, the automatic creation of content), the gate can be cruder still. Push generated level data through the real runtime and check only that it initializes without crashing; feed just those back as training data. The 99.9% versus 8.2% gap between BUILD and strict-launch reads as a warning: if "it loaded, sort of" counts as passing, your gate is doing essentially nothing. If your pipeline's pass rate is above 90%, suspect a lenient gate before congratulating yourself on quality.

Third, for teams already having an LLM draft levels: calibrate the gate. Before trusting their verifier, the authors checked that it separates known-good from known-bad (15/15 gold grounded, 0/16 bad). You can do the same — run both your shipped levels and the levels you once rolled back on a bug report through the checker. If it passes the latter, it is not safe to drive generation with.

Fourth, copy the two-tier split between the signal you train on and the signal you merely watch. The authors measure execution grounding but do not train against it. On a daily-puzzle site, restrict the generation gate to hard conditions like solution uniqueness, and keep soft metrics such as drop-off rate or completion time observation-only. Optimize a soft metric and you invite exactly what Appendix A showed: assets swapped to raise the art score, only now on your own dashboard.

Limitations

The authors list four limitations in Section 10. First, validation covers four unseen families and 25 tasks within a single benchmark; broader families and larger models are future work. Second, the diversity control is exact duplication, because each task carries only one gold reference — so no control with different content at equal quality was possible. Third, training-seed robustness is established only for round 1. Fourth, execution grounding is a functional lower bound, not a measure of playability. In Section 11 they add their own hedge: this is one instance, and testing the principle outside game generation is the next step.

What I would point out here is, first, the granularity of what was learned. Appendix B's observation that all 815 strict-launch failures contain a parse error reads as evidence that this gate may mostly be teaching the model to write valid GDScript and config files. In an environment with no partial credit that is a real achievement, since nothing else can happen until the syntax is right. But it is a different axis from "the games got better," and readers should not merge the two.

Second, no visual-judge scores are reported for the RFT rounds. Dropping the judge from the training signal is defensible given Appendix A, but the consequence is that we learn launch rates went up, code got longer, and more tasks became grounded — while the paper can say nothing about whether the resulting games are any fun. The authors write in Section 8 that a clean launch certifies that a project runs, not that it does anything. I read that as the most honest sentence about the paper's scope.

Smaller points: the grounding weights (0.35 / 0.25 / 0.40) and the 0.50 threshold are hand-chosen. The authors argue insensitivity from the gap between gold's minimum 0.58 and an empty shell's 0.013, which looks reasonable. Also, only one base model (Qwen3-14B) was used, the optimizer is fixed to rejection sampling (they flag policy-gradient methods such as GRPO/RLVR as future work), and three rounds do not reveal where exploration runs out. Since round 3 already hits the gold ceiling of 25/25, this metric structurally cannot measure any further gain.

How Fukai reads it

This paragraph is my own interpretation. I would place this study less in the lineage of generative-AI work and more in that of acceptance criteria on a production floor. In the vocabulary of design criticism, strict-launch is not quality assurance; it is an acceptance spec that defines the minimum condition for passing extremely narrowly. What I find interesting is the authors' argument that this narrow spec is precisely what produced diversity. Optimizing against a judge concentrates outputs on the surface features the judge likes, whereas a binary launch check demands nothing but "it runs," leaving a wide set of functionally distinct solutions admissible. Translated into hand-building experience, it resembles the phenomenon where narrowing a review criterion from "is this good" to "is this broken" widens the range of proposals a team brings in. That said, this is my own restatement of Appendix A and Section 7 — the authors do not go that far.

Closing

To place this paper on the map, start with its testbed, GameCraft-Bench (Luo et al., 2026), which we have covered in a separate article here. On the axis of verifying generated artifacts by executing them, the closest neighbors among the works they cite are GameGen-Verifier (Jia et al., 2026, runtime verification of generated games via state injection), OpenGame (Jiang et al., 2026) and GameDevBench (Chi et al., 2026).

For the lineage of the self-distillation loop itself, reading STaR (Zelikman et al., 2022) and ReST (Gulcehre et al., 2023) alongside V-STaR (Hosseini et al., 2024), which trains a verifier of its own, makes clear why this paper holds the loop fixed and varies only the nature of the gate. From the other side — judges being gamed — Skalse et al. (2022) on formalizing reward hacking and Zheng et al. (2025) on null models winning automatic benchmarks are the natural entry points.

My own lingering question is how far the claim that the verifier's precision sets the ceiling holds in studios where the gate is human review rather than a machine check. That is outside this paper's scope, but it is what stayed with me after reading.

References

Papers and materials referenced in this article:

・The Verifier is the Curriculum: Execution-Gated Self-Distillation for Cross-Family Game Generation (Chenyu Zhou, Qiliang Jiang, Shuning Wu, Xu Zhou, 2026, arXiv preprint)

・DOI: 10.48550/arXiv.2607.09709 (arXiv-issued; not peer-reviewed)

・Full HTML text (including Appendices A, B, C)

・Related work (the paper's testbed): GameCraft-Bench (Luo et al., 2026)

・Citation check: Semantic Scholar / Google Scholar