Siper et al.: Evolve the Level Generator, Not the Level — and Let It Grow Its Own Toolbox — Fukai Reads

TL;DR

Automatically generating puzzle levels is a decades-old line of research. What today's paper moves is the target of that search. Instead of searching for levels, the authors search for the programs that produce levels. They call this "metageneration".

The method runs like this. A large language model (an LLM: a system trained on large amounts of text so that it can continue text, or write code) is asked to write a level generator as a Python program. That program is actually executed and scored, and the good ones become parents that the LLM rewrites. So far this is ordinary evolutionary search. What the paper adds is Continual Abstraction Discovery (CAD): reusable pieces are carved out of the high-scoring programs into a shared toolbox that later generations can call.

Across four domains — Sokoban, Zelda, Dangerous Dave and Lode Runner — the CAD version ended higher. All eight comparisons favour CAD, an exact two-sided sign test giving p=0.008. The programs also got shorter (roughly 370 lines down to roughly 296), and the extracted helpers kept being called in later generations, at an adoption rate above roughly 80 percent. One of the paper's main domains is Sokoban-style box pushing. Pictured: Patrick's Parabox (Patrick Traynor, 2022), screenshot from its Steam store page.

Introduction — who wrote it, and where

Three authors. Matthew Siper and Julian Togelius are at the Game Innovation Lab, New York University; Ahmed Khalifa is at the Institute of Digital Games, University of Malta. The arXiv identifier is 2608.17947, submitted 18 August 2026, listed under cs.AI, cs.LG and cs.NE.

One point deserves precision. The arXiv comment field states that the paper is accepted for publication at the IEEE Conference on Games 2026. So this is not a preprint thrown over the wall; it is the author version of a peer-reviewed paper. That said, it has been public for about three weeks and has almost no citations yet. It has not been widely discussed, and I want that on the record.

I picked it today for a simple reason: it makes you want to go and build something. If you maintain your own level-generation tooling, the skeleton of this paper transfers directly. And you do not need an LLM to transfer it — as I will argue below, the real lesson here is about the order in which work is done, not about machine learning.

Background — why search for programs instead of levels

Procedural content generation (PCG: making game content automatically with code) comes in two broad flavours. One searches for the artifact directly: generate a board, score it, perturb the good ones, repeat. The other searches for the maker: the object of search is the generating procedure itself.

The second has long been argued to be the more valuable one. A procedure that emits good boards forever beats one good board. Procedures are readable, editable and reusable on other content. Direct artifact search nonetheless stayed dominant, because getting a machine to write the procedure was hard.

Then LLMs arrived. Models that write code as fluently as prose can serve as the mutation operator of an evolutionary loop: show them the parent program, ask for a change, and a syntactically intact child comes back. The authors place this in the lineage of work that puts an LLM inside an evolutionary loop, of which AlphaEvolve is the best-known example.

But that creates a new problem. If every child is rewritten from scratch, the same tools get rewritten over and over. A function that checks whether a coordinate is inside the board; a function that ensures exactly one player exists; a function that checks reachability — every generation reinvents the same foundations. What to do about that waste is the paper's starting point. Genetic programming has long had Automatically Defined Functions (naming and reusing sub-procedures discovered during search), but the authors' aim is that the helpers should be discovered during the run rather than written in advance by a human.

Approach — the evolutionary loop, and how the toolbox grows

Start with the loop. An individual is a single Python program which, when run, emits a batch of levels. Parents are drawn from an archive ("the archive retains every successfully evaluated individual") by fitness-proportional selection, and the LLM rewrites them into children. Crossover between two parents fires with probability 0.25 once the archive holds at least two individuals. The prompt carries the parent program, a description of the domain, the run's accumulated memory, and the contract the generator must satisfy.

A child is not scored as-is. First it has to run. It must compile and complete at least three of five smoke-test executions. When it fails, the error text is handed straight back to the LLM for a correction, up to four attempts. This is quietly important: most accidents when putting generative AI inside evolutionary search come from broken children slipping into the population.

Scoring is the mean of three things: validity (the fraction of emitted levels that are actually well-formed), quality (the score under the shared PCG Benchmark yardstick) and diversity (mean pairwise difference between emitted levels), averaged over 30 executions per candidate. No formulas here, but the shape reads as: "well-formed", "interesting" and "different every time" are weighted equally, one third each.

Then CAD. It fires starting at generation 8 and every five generations thereafter. It gathers the programs at or above the 75th percentile of fitness in that run and asks the LLM to propose reusable utility functions extracted from them. Each candidate helper must compile and pass a smoke test, with up to two correction attempts. (Diagram) The paper's procedure on one page. Top row: the per-generation loop. Bottom: CAD, firing every five generations. Only accepted helpers enter the library on the left, where later programs can call them.

The acceptance test is my favourite part of the paper. After a program is refactored to call the new helper, it is run on a fixed seed and accepted only when the refactored program matches the original program's behaviour exactly. Any rewrite that reads better but behaves differently is thrown away. Only the survivors join the run-specific helper module that later generations can call.

Findings — a clean sweep, and shorter programs

The experiment is a 2×2: CAD on/off, crossed with access to a fixed hand-written domain API (ready-made primitives for entity normalisation, reachability, repair and domain-specific structural operations). That grid runs over four domains (Sokoban 5×5, Zelda 7×11, Dangerous Dave 7×11, Lode Runner 11×16, all from the PCG Benchmark), ten runs per condition, fifty generations each — 160 completed runs. The LLM is GLM 5.2, at temperature 0.2 for mutation and 0.1 for correction and refactoring.

The headline: "CAD raises mean final best fitness in all eight domain and API comparisons." Four domains times API-present/absent gives eight cells, and CAD is ahead in every one. Put through an exact two-sided sign test (a plain test that looks only at the count of wins and losses), that gives p=0.008.

Let me be honest about what is not there. The paper's main text does not include a numeric table of mean final fitness per domain and condition. What it shows is a figure of fifty-generation trajectories with mean ± standard error bands, so the numbers I can pin down are "eight out of eight" and "p=0.008". On effect size the authors say only this, in words: "The magnitude varies across games. The Base contrast is small in Lode Runner and larger in Sokoban. Under the fixed API, the largest separation occurs in Lode Runner, where the CAD trajectory continues improving while the no-CAD trajectory plateaus earlier." Lode Runner (11×16) is the largest board among the four domains. Pictured: the Steam release of Lode Runner Legacy (Tozai Games), screenshot from its store page.

The toolbox results are the more interesting half. Libraries "expand sharply during the early extraction cycles and then stabilize". In later generations, the share of generated programs that actually call the library stays above roughly 80 percent, and a program contains about 15 to 20 helper calls on average. What goes on the shelf is not decoration; it gets used.

Program length tells the same story. Without the expert API, programs run to roughly 370 lines without CAD and roughly 296 lines with it. Handing over the fixed API shrinks the baseline much further, to roughly 160 lines without CAD and 173 with. Move the shared parts out and the body gets shorter — obvious, but here it is with numbers attached.

The paper also reports which abstractions kept being rediscovered. By call count: in_bounds, which checks whether a coordinate lies inside the board, at 4,693 calls; entity count normalisation at 3,848; ensuring exactly one player at 2,611; finding the player's position at 2,370; ladder placement at 768. And in_bounds and entity count normalisation each appear in 28 of the CAD runs. Separate runs keep arriving at the same tools.

Use cases — five things a designer can take away tomorrow

The implementation assumes an LLM, but most of what transfers does not need one. In order:

(1) Make the deliverable a level-making function, not a level. If you are building a box-pusher and hand-authoring sixty levels, you are certainly repeating the same operations along the way: wall off the border, match crate count to goal count, place exactly one player. Name those and lift them out. What the paper had a machine do is, in essence, that tidying.

(2) Score a generator on validity, quality and diversity, weighted equally. The common failure in home-grown level generation is to watch validity alone and stop at "it produced a solvable board". This paper's fitness splits the weight three ways. To copy it, start by running your generator 30 times rather than once, and reporting the same three things every time: the well-formed rate, an interest metric, and how different the outputs are from each other. Box pushing is easy for a machine to check for well-formedness and hard to check for interest. Pictured: Patrick's Parabox, screenshot from its Steam store page.

(3) Accept a refactor only when the output does not change by a single bit. This is not specific to generators. When tidying a board generator, fix the seed, emit thirty boards, and accept the change only on an exact match before and after. Half the reason this method climbs steadily, as I read it, is that gate: only meaning-preserving rewrites get through, so the toolbox never fills with junk.

(4) Add tools on an interval, from the top slice only. CAD fires from generation 8, every five generations, and looks only at the top 25 percent. Extract from everyone every time and unlucky, badly-shaped code ends up on the shelf. Translated to human practice: every ten levels or so, look back at the two or three that worked best and pull the shared operations out of those.

(5) Hand-written APIs and machine-discovered helpers are not mutually exclusive. To me this is the most practical result in the paper. Even in the condition where a hand-written expert API was supplied, adding CAD improved things further. Even with a toolbox already in hand, shared operations you have not yet named remain. Building the shelf is not the end of the job.

Limitations — the authors', and mine

The authors name three weaknesses. First, the yardstick: "Solvability, benchmark quality, and diversity do not fully capture visual style, pacing, novelty, or designer intent." This experiment cannot separate the score went up from the levels got better.

Second, cost. One run consumes approximately 9.5 million input tokens and 2.6 million output tokens, costs approximately $25, and takes approximately 2.5 hours of wall-clock time — times 160. The authors call this substantial themselves.

Third, and the one that nags at me most: "CAD is evaluated as a complete pipeline. Helper extraction, module correction, and source refactoring are not separated into individual ablations." (An ablation study removes one component at a time to see which part is doing the work.) So whether the gain comes from the extraction or from the behaviour-matching gate cannot be told from this paper.

What I would add is about how the wins are counted. The support for "CAD is ahead in all eight" is a sign test at p=0.008, but those eight cells are four domains counted twice, once with the API and once without. Measurements sharing a domain, an implementation and an LLM are being counted as independent trials, so the effective independence may be lower than eight. And with no per-condition table in the text, a reader cannot check the size of the gap. The direction is consistently one way — I am willing to believe that — but how much it helps is not something this paper lets you take home.

How Fukai reads it

I would rather read this not as a paper about generative AI but as a gauge of how far the automation of design work has come. Automatic level generation has been piling up for at least twenty years. What this paper automates is the tidying of the tools that generate levels — that is, the judgement call inside a refactor. And it anchors that judgement not on "does it read better" but on "does the output stay identical", precisely where a human tends to get lazy. In the language of design criticism, this is not automated creativity; it is closer to automating the habit of putting your toolbox back in order. Unglamorous — which, as I read it, is usually where the gains are.

Closing — what to read next

Read it alongside the reinforcement-learning side of level generation and a map starts to form. On this site we have covered Earle et al. on multi-agent level generation and Bhaumik et al. on combining wave function collapse with reinforcement learning. Today's paper searches one level above those: for the maker, not the made.

If what caught your interest is the idea of shifting what gets searched, the sibling paper is Nasir et al. on evolving the rules of play themselves. Search for levels, for the procedure that makes levels, or for the rules of the game — these are different answers to one question about where to put the search. On the narrower problem of reliably emitting solvable levels, Xu et al. on promoting mechanics to coordinates sits close as well.

And look once more at the names of the tools the machine kept rediscovering. Is this inside the board? Do the entity counts match? Is there exactly one player? Every one of them is something a human puzzle author checks without thinking, every single time. Half the pleasure of reading automation research is that flicker of recognition: so that is what I have been doing all along.

References

Papers and materials referenced in this article:

・Procedural Content Metageneration via Program Search and Continual Abstraction Discovery (Matthew Siper, Ahmed Khalifa, Julian Togelius, 2026, arXiv:2608.17947, cs.AI / cs.LG / cs.NE)

・Full text of the same paper (all figures and quotations here were checked against it; submitted 2026-08-18)

・Affiliations: Matthew Siper and Julian Togelius, Game Innovation Lab, New York University; Ahmed Khalifa, Institute of Digital Games, University of Malta. The arXiv comment field records acceptance at the IEEE Conference on Games 2026 (checked 2026-09-06).

・Related: Earle et al. on multi-agent level design / Bhaumik et al. on stitching WFC and RL / Nasir et al. on evolving the rules of play / Xu et al. on promoting mechanics to coordinates