Earle et al.: Recasting Level Design from a One-Person Job to a Multi-Agent Collaboration — Fukai Reads

TL;DR

PCGRL (Procedural Content Generation via Reinforcement Learning; reinforcement learning is a framework in which an agent learns, through trial and error, the actions that raise a reward) is a way to make an agent design game levels automatically, guided by a reward function rather than by human-made example data. This paper reframes PCGRL, which used to mean 'a single agent edits the whole map one tile at a time,' as a multi-agent problem in which several agents divide the work and edit in parallel.

The result: the more agents there are, the better the quality, the generalization to unseen map shapes, and the computational efficiency all become. And agents that each see only a 3x3 patch around themselves ('near-sighted') beat agents that see the whole map. I introduce this paper as a study that rereads level generation, shifting it from 'one long solo job by a single designer' to 'the collaboration of a team, each member holding a small assignment.'

Introduction

Today I introduce 'Video Game Level Design as a Multi-Agent Reinforcement Learning Problem' by Sam Earle, Zehua Jiang, Eugene Vinitsky, and Julian Togelius. It was posted to arXiv on October 6, 2025 (arXiv:2510.04862) and at the same time accepted as a full technical paper at AIIDE 2025 (the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, a peer-reviewed venue for game AI). So it is both a preprint and a peer-reviewed paper.

The author team includes people who launched the PCGRL framework in 2020. Togelius is one of the central figures in game AI research, and Earle is the one who recently ported PCGRL to JAX (a library for running computation fast on the GPU). So this is less an outside critique than a record of researchers remodeling their own tool.

I chose it today because its claim is modest and close to implementation, and because its experiments to isolate 'why does this work' are careful. For anyone who wants to generate levels automatically, the lessons here carry directly into design decisions. The code is open at github.com/smearle/pcgrl-jax.

Background

Automatic level generation is called PCG (Procedural Content Generation) and has a long research history. Broadly, there is a 'learn from data' family that collects many existing levels and produces similar ones (GANs, VAEs, and recently large language models), and a family that uses no example data and relies only on a reward. PCGRL is the latter. It treats design itself as a kind of game: an agent edits one tile at a time and is rewarded by how much closer the level gets to a target property (for example the length of a corridor, or whether the paths form a single connected whole).

The strength of PCGRL is that once training is done, generation finishes in an instant. Because it can build a board in real time at run time, it has been hoped for in uses like roguelikes that show a new stage on every play, or as a co-creation tool that offers candidates alongside a human designer.

But there was a weakness: training takes a long time, and the cause is the reward computation. To measure a property like 'corridor length,' you must repeatedly compute shortest paths across the whole board. This computation grows heavier as the board grows (the paper states it scales roughly with the square of the map width), and it runs every time the agent edits a single tile. In other words, 'make one move, then re-measure the whole board' repeats endlessly, and most of the time is soaked up there. How to lighten this weight is the starting point of this work.

Approach / Method

The authors' idea is simple: stop making a single agent do everything, and let several agents divide the work. Each agent is a 'turtle' that walks over the board and rewrites the tile under its feet into wall or floor. All of them edit one tile at a time simultaneously, and the sum of their edits earns everyone the same reward based on 'how much closer to the target' it got (this is called a shared reward). Training uses MAPPO (Multi-Agent PPO, a representative reinforcement learning algorithm in its multi-agent version).

Here is the core of the efficiency gain. The reward needs to be computed once over the whole board. So if there are three agents, three tiles' worth of edits advance per reward computation. You reduce the number of heavy computations relative to the number of edits. The authors also add a knob (reward frequency) that computes the reward once every N steps, thinning out the source of the cost further. The implementation runs on the GPU via JAX and is said to train roughly 17 times faster than the earlier numpy version.

Another key point: each agent sees only 'a small window around itself.' The paper compares three fields of view: 3x3 tiles, 16x16 tiles, and 31x31 (the whole board). Counter to intuition, the near-sighted 3x3 beats seeing the whole. Building on prior work, the authors explain that narrowing the view makes agents learn local, reusable design policies, and that this is what leads to strength on unseen boards.

Validation uses two testbeds. One is the 'binary maze,' where the task is to keep the corridors as a single connected whole while making the longest path between two points as long as possible. The other is the 'dungeon,' where you must place a player, a key, a door, and enemies in the correct counts, make the player-to-key and key-to-door paths long, and keep the nearest enemy a little away, all at once. Training is on 16x16 boards, and evaluation also tries various sizes from 8 to 32 and rectangles distorted in shape. To keep the experiments fair, the authors define 'the number of moves needed to trace the whole board once' as one unit (a board scan), so that the total number of edits can be matched even when the number of agents differs.

Findings

The first finding is clear: the more agents, the higher both quality and generalization. In the maze experiment with a fixed move budget (paper Table 1), on a fixed-shape map of width 32 the mean reward rose from 156.10 +/- 6.73 for one agent to 181.81 +/- 11.80 for three, and on width 32 with randomly distorted shapes from 68.36 +/- 7.90 to 88.43 +/- 5.68. In the dungeon task (Table 4) the gap is larger: on fixed-shape width 32, 213.78 +/- 17.55 -> 364.47 +/- 26.86, and on random shape 49.33 +/- 4.00 -> 101.63 +/- 6.09, roughly doubling. In each case, stated as 'compared to what and by how much,' this is the improvement of multiple agents over a single agent.

Second, the authors separate out that this improvement is not merely 'because there were more moves.' Even holding the total number of edits equal (Table 2), and even holding both the total number of edits and the number of reward computations equal (Table 3), the generalization advantage on large, distorted boards remained. Conversely, on small, fixed-shape boards the multi-agent advantage shrinks to about the single-agent level. So the true value of multiple agents, one can read, shows up in unseen situations different from training.

Third, a narrower view is better. In the 3-agent setting (Table 6), the local 3x3 view showed the highest reward and generalization, followed by 16x16, and last the full view (31x31). In addition, the reward computation could be thinned to at least once every 10 steps without dropping performance (single agent, Table 5). But the authors caution that thinning itself does not clearly raise performance; rather, 'adding agents' is what combines reduced computation with quality.

The authors interpret this as multiple agents learning local, modular (recombinable like parts) policies. Indeed, they report a loose division of labor emerging naturally, where separate agents take charge of separate regions of the board and edits that are not optimal alone combine into a good board overall. Note that PCGRL is originally a framework that prioritizes 'is it functionally valid' over visual beauty, and the reward here measures that functional side.

Where you can use this

If you are generating maze-like or dungeon levels in real time for a roguelike, the direct lesson is: don't train your generator as a single agent, train it as a collaboration of several. Especially if you want to serve boards at run time whose size differs from what you trained on, the generalization strength of multiple agents applies directly. The implementation is public, and mazes and dungeons are a ready starting point to try.

If you are doing hyper-casual PCG and running heavy checks of shortest paths or connectivity on every generation, the thing to watch is thinning the reward computation. This paper showed that batching the check once every few steps instead of every move does not hurt quality. Where check cost is the bottleneck of generation, there is room to speed up training and generation just by lowering the check frequency.

If you are building a co-creation tool that offers candidates alongside a designer, the observation that 'a division of labor emerges naturally' is useful. If regions get split among agents, it connects easily to a workflow where a human draws some regions by hand and leaves the rest to another agent. The authors themselves suggest extensions like fixing one agent's edits so others cannot overwrite them, or giving each agent a different 'brush size' (one tile vs. 3x3) to build a hierarchical division of labor.

And the lesson about the field of view transfers beyond generation. When you have a machine build grid-like content, 'showing the whole makes it smarter' is not always true. Showing only a narrow local region can make it learn a general, robust policy. Beyond level layout, tasks where 'local rules stack up to make the whole valid' - tile terrain, pipe puzzles, wiring puzzles - are worth trying with a narrowed view first.

Limitations

Start with the limits the authors themselves acknowledge. The experiments are confined to two very bare-bones domains, 'binary maze' and 'dungeon,' and training is on small 16x16 boards. Real game levels are larger and contain diverse mechanics and assets. RL training still takes several hours, and it requires humans to carefully design the reward (heuristics) that gives agents a learnable signal. The authors honestly write that if the goal is simple enough to be expressed as constraint satisfaction, or if ample human-authored data exists, those routes might make good levels faster.

What I, Fukai, point out here is what this paper's 'generalization' refers to. Generalization here is measured by how high the functional reward (path length, connectivity) is on boards of unseen size or shape. It is not a measure of whether humans find it 'fun' or 'beautiful.' Because PCGRL prioritizes function to begin with, this is natural, but whether the output is fun to play lies outside this number.

One more point, which the authors also raise as future work: a user study with domain experts has not yet been done. The future vision of 'becoming a co-creation tool usable alongside a human designer' is attractive, but at this paper's stage it is fair to read it as having shown efficiency and generalization on functional metrics, and no further. Because the reward is shared, credit assignment to individual agents is coarse, and the division of labor 'emerged naturally' rather than being 'designed and enforced' - both worth receiving cautiously.

Fukai's Reading

From here I note this is my own interpretation. I want to read this work as one that recaptures the question 'what is design?' through the old metaphor of the division of labor. Rather than a single craftsperson managing a long process end-to-end in their head, several hands each holding a narrow assignment weave the whole together by relying on each other's traces (the authors call this stigmergy, indirect coordination through clues left in the environment). The result that near-sightedness is stronger looks to me like a fable of design organization: a team of many locally faithful ordinary workers is harder to break in an unfamiliar setting than one all-seeing genius. That said, this was observed only on the limited stage of grid-based level generation, and generalizing it into a fable is my own overreach, I should add.

Closing

If you want to know the PCGRL framework itself, reading the same authors' original Khalifa et al. (2020) alongside gives you the map. That one provides the starting point of 'turning level design into a game for a single reinforcement learning agent,' and this paper measures 'what happens if you make it a team.'

If you read with an interest in reconciling generation quality and playability, it is also interesting to place it beside the study stitching WFC and reinforcement learning that I introduced yesterday (the Bhaumik et al. piece on this site). That one tries to reconcile look and function by 'narrowing actions with local rules,' while this one seeks efficiency and generalization through 'multiple agents with a local view.' The angles differ, but both face the same direction: 'stack up the local rather than decide the whole at once.'

References

Papers and related material referenced in this article:

・Video Game Level Design as a Multi-Agent Reinforcement Learning Problem (Sam Earle, Zehua Jiang, Eugene Vinitsky, Julian Togelius, 2025, arXiv preprint / AIIDE 2025 full technical paper)

・DOI: 10.48550/arXiv.2510.04862

・Authors' open-source code: smearle/pcgrl-jax (JAX PCGRL implementation)

・Related work (origin of the framework): PCGRL: Procedural Content Generation via Reinforcement Learning (Ahmed Khalifa, Philip Bontrager, Sam Earle, Julian Togelius, 2020, AIIDE 2020)