Gao & Dubé: Letting a machine do the first read of player-made math levels — Fukai Reads

TL;DR

Add a level editor to a children's math game and your best players will start building levels for you. For anyone teaching, that is welcome news: practice levels are always in short supply. But as submissions pile up, a new job appears — checking, one level at a time, whether this thing is safe to hand to another child — and that job does not scale.

Jie Gao and Adam K. Dubé of McGill University tried handing that first read to a machine. From 206 levels — 86 built by experts and 120 by players — they turned "what is placed on the board, and how many of each" into numbers, and trained a classifier to predict whether a level is worth publishing.

Of the four methods compared, random forest discarded the fewest good levels. That said, the authors repeatedly call these findings preliminary, and the paper names no venue. I read it less as a report on a finished tool and more as a design memo on where to start if you want to triage a flood of player-made levels.

Introduction: who wrote this

The paper is "Personalizing Mathematical Game-based Learning for Children: A Preliminary Study", by Jie Gao and Adam K. Dubé. Both are at McGill University in Montreal, Canada; the paper prints jie.gao3@mail.mcgill.ca and adam.dube@mcgill.ca.

No venue is named. Neither the body nor the footnotes mention a conference or a journal. What I read is the version posted to arXiv (arXiv:2603.25925v1), whose identifier places the submission in March 2026. So in this article I treat it as a preprint whose peer review I could not confirm. The authors say as much themselves: these are preliminary findings, and further exploration is needed.

I picked it anyway, because the problem it tackles lands squarely on the desk of anyone running a puzzle site. The moment you let players build levels, you hit the same wall: who checks them before they go out? This paper is an honest, small-scale report on lowering that wall a little with a machine.Portal 2 (Valve), as an example of player-built test chambers arriving in bulk. It is not a game studied in this paper. Image from its Steam store page.

Background: too few levels and too many are the same problem

Teaching mathematics through play goes by the name GBL (game-based learning), and it is no longer unusual in classrooms. Citing prior work, the authors note that GBL has a clear overall positive effect on critical thinking, and that games where the play itself is the arithmetic outperform games where you play first and then solve sums. The former is called intrinsic integration: the learning content is baked into the mechanics.

The trouble is that the better form is the harder one to build. In the authors' framing, each level costs a dedicated designer hours of labour — yet mastering a maths skill takes cumulative practice across dozens of levels. Supplying enough high-quality levels is itself the binding constraint.

The authors also fault existing learning apps for adapting too coarsely. Citing reviews, they observe that most implement only low-resolution adaptivity at the level of the individual player, and argue for designs that are micro-adaptive, dynamic, fine-grained and player-centred.

Solve both at once and a Creative Mode starts to look attractive: players supply volume, and the act of building is itself creative practice for the builder. But as the authors write, there is limited research on filtering player-generated levels with machine learning models. How to choose among the levels you now have was left blank.

Approach: translating a board into counts of things

The setting is a maths learning app used around the world by children aged 5 to 10. The paper does not name the app. It has a Creative Mode: a tool that lets advanced players design new levels from their own ideas.

The dataset is 206 distinct levels: 86 designed by experts and 120 generated by players. Of the 120 player levels, 44 were validated and selected by experts at the game company for other players to play; the rest were excluded. That selection is the label the machine has to predict.

What gets pulled off the board is the inventory: which objects are present and how many. In the paper's words, the counts of features number 25 and their associated values 12. The groups named are the player character and its value, the goal, physics objects (ice blocks, lava blocks, bubbles, one-way platforms, slimy platforms, sticky platforms) and obstacles (clouds, doors, spiky platforms, breakable walls). The level is treated not as a picture but as a parts list.

From there the procedure is conventional. Lasso regression (which shrinks the weights of unhelpful variables to zero, thinning a long list down) reduces the dimensionality. Then four methods are compared: k-nearest neighbours, decision trees, support vector machines and random forests. Scoring uses nested cross-validation — running the split-train-test loop inside another one, so that tuning cannot quietly flatter the result.(Diagram) The four-stage flow the paper describes. The machine is not the final judge; it is the first reader, seated before the expert.

Findings: random forest throws away the fewest good levels

Here are the numbers as printed. Inner loop first. KNN: accuracy 81.42±1.53%, precision 73.85±3.92%, recall 65.32±3.13%, F1 66.41±3.53%, ROC-AUC 84.77±4.71%. Decision tree: 83.89±1.54% / 78.06±3.57% / 73.40±4.72% / 73.63±3.96% / 81.58±3.66%. SVM: 82.42±1.19% / 75.90±3.14% / 66.67±3.23% / 68.11±3.64% / 86.84±2.20%. Random forest: 82.07±1.37% / 74.43±2.64% / 70.46±2.18% / 71.12±2.27% / 85.86±1.85%.

In the outer loop, in the same order: KNN 79.90±4.01% / 68.60±10.79% / 62.97±8.19% / 63.66±9.10% / 84.12±8.45%; decision tree 82.23±3.88% / 75.45±7.54% / 71.72±7.39% / 71.86±6.60% / 81.69±7.62%; SVM 81.93±2.53% / 74.90±6.93% / 65.52±4.98% / 67.50±5.19% / 86.22±4.87%; random forest 82.42±5.71% / 74.53±8.91% / 72.69±9.75% / 72.70±9.22% / 86.57±6.32%. The authors write that among the models tested, RF demonstrated the strongest overall performance, achieving the highest recall and F1-score while maintaining high overall discrimination.

What matters is why recall is the metric to watch. A miss here means the machine throws out a level that should have been published. The confusion matrix showed random forest was the least likely to discard expert-validated levels. For a Creative Mode, a child's level vanishing for no visible reason is a fatal experience — which is why this number deserves to be read before accuracy.

The variables Lasso kept are the other interesting result. The strong predictors were the number of player characters and goals, plus the presence of one-way platforms, platform bubbles and poppable bubbles. The authors read the character and the goal as the essential interactive vessel for the learning content, and conclude that the structural integrity of the game is a prerequisite for effective maths learning during play. They further note that physics objects mattered more than obstacles, suggesting the model prioritises features allowing constructive manipulation over passive barriers.

Where you can use this: if you let players submit levels

One. Put the machine in front of the reviewer, not in front of the player. The shape of this paper is not a machine that decides what gets published; it is a machine that decides what the expert looks at first. If I ran a puzzle site that accepted submissions, I would have the model push only the obviously broken levels to the back of the queue and let a human read the rest. Even ranking by confidence instead of issuing a verdict cuts the reviewing load a lot. The instinct to avoid false rejections is exactly why the paper leans on recall.

Two. Start with the parts list. This study used neither the board image nor the solution trace — only which objects are present and how many. That alone reaches the low eighties in accuracy. If I were building a Sokoban-like, the first features I would compute are the ones you get by counting: number of boxes, number of goals, share of wall tiles, whether any box sits where it can no longer be pushed. Running a solver to measure solution length can wait for the second pass.

Three. Make "not broken" the first gate. That Lasso kept the counts of characters and goals is an unglamorous result with practical teeth. The most common failure in a player-made level is probably not that it is too hard but that something structural is missing: no goal, a blocked entrance, no solution at all. Ask the machine about structural soundness before you ask it about fun. Do it the other way round and you lose the ability to explain any verdict.

Four. If the game teaches, map parts to content up front. The authors read the dominance of manipulable physics objects over obstacles as a fact about that app: the objects are where the calculation and the reasoning live. If you want to screen learning levels, tabulating which parts carry the learning during design is already most of the feature engineering.Human Resource Machine (Tomorrow Corporation), as an example where the act of playing is the learning content. It is not a game studied in this paper. Image from its Steam store page.

Limitations: what the authors admit, and what I noticed

The authors name three limitations, all short and candid. First, the size of the dataset constrains the model comparison. Second, the Creative Mode includes other game modules with different features — so these 206 levels represent only part of it. Third, as these are preliminary findings, further exploration is needed. On top of that, no player study and no deployment evaluation were carried out. Whether this classifier actually improves a child's learning is simply not addressed.

What Fukai would add is this: do not take an accuracy in the low eighties at face value. If all 86 expert levels sit on the valid side, the valid-to-invalid split is roughly 130 to 76. In that case, a model that blindly answers "valid" every time is already right about sixty per cent of the time. The paper does not report that baseline. So the numbers worth reading are not accuracy but recall, F1 and ROC-AUC, which sit further from it.

One more. The outer loop is noisy. Random forest's F1 is 72.70±9.22% — a standard deviation of nearly nine points. At 206 levels, that is wide enough for the ranking to change places. This is presumably what the authors mean when they say the dataset size constrains the comparison. The decision tree scored higher on both accuracy and precision in the inner loop; the authors still recommend random forest, and they do so for an operational reason — recall matters more here — not because the statistics settled the question. That distinction is worth keeping straight.

Fukai's reading

I would place this work not in the lineage of automated level generation but in the lineage of automated editing. Where generation research hands the machine the job of making levels, this paper hands it the job of reading the levels that were made — and leaves the final call with a human expert. In the vocabulary of design criticism, this is an attempt to build not a generator but a gatekeeper, and I suspect that a submission culture needs the gatekeeper first. Precisely because the results are preliminary, I expect the design decision — keeping the machine as a sorter rather than a discarder — to outlast the numbers.

Closing

If you want to go deeper: on how machines evaluate player-made content, Pfau et al. on player-generated content sits nearby. On scaffolding in games built to teach, see Wermann et al., and on learning environments for children, the KidGym write-up makes a decent map.

Asked long enough, "how do we choose among submitted levels" becomes "what is a good level, in words". For the discomfort that question produces, Petri Purho's philosophy shifts the angle. If you would rather have machines measure difficulty itself, read Shyne et al.; for the current thinking on the generation side, Xu and Verbrugge's HDPCG. Put together, generation and selection start to look like two sides of one problem.

Sources

Papers and materials referenced in this article:

・Personalizing Mathematical Game-based Learning for Children: A Preliminary Study (Jie Gao, Adam K. Dubé, 2026, arXiv preprint / arXiv:2603.25925v1)

・The HTML version of the same paper (all quoted text and table figures come from this version)

・Adam Dubé — Associate Professor and Director of the Technology, Learning, & Cognition (TLC) Lab, McGill University (faculty page)

・Images are from Steam store pages: Portal 2 (Valve) and Human Resource Machine (Tomorrow Corporation). None of these games is studied in the paper; they illustrate the topic only.

・The flow diagram was drawn for this article and is not a reproduction of any figure in the paper.