Ye et al.: Measuring Image-Capable AI (MLLMs) with Children’s Intelligence Tests — Fukai Reads

TL;DR

KidGym borrows the idea behind children’s intelligence tests (the Wechsler Intelligence Scales, a long-used battery that breaks a child’s intelligence into several interpretable sub-abilities) to measure the “smarts” of Multimodal Large Language Models (MLLMs, models that process images and text together) by splitting them into five abilities. Execution, Perception Reasoning, Memory, Learning, and Planning are each tested on a 2D grid across 12 tasks (6 single-ability, 6 composite), from an easy L1 to a hard L3.

A team at ShanghaiTech University evaluated nine recent models. In my own summary: top models like o3, GPT-5, and Gemini-2.5-Pro score near-perfect on classification and recall, yet on a puzzle task that asks them to assemble non-semantic abstract shapes the best was only 0.30, and on counting the best was 0.72 against a human 1.00. The point of the paper is that current models break down not on flashy language tasks but quietly, when asked to count, to assemble, and to juggle several rules at once.

Introduction

Today I am reading “Children’s Intelligence Tests Pose Challenges for MLLMs? KidGym: A 2D Grid-Based Reasoning Benchmark for MLLMs,” by Hengwei Ye and colleagues at ShanghaiTech University. The source is an arXiv preprint (arXiv:2603.20209, submitted March 2026; a preprint is a manuscript that has not yet formally passed peer review, i.e. expert vetting), so I am careful not to treat it as peer-reviewed here. The corresponding author is Zheng Tian, and the benchmark is released on the authors’ project page.

I chose it today because it bridges two worlds: AI evaluation and the psychological tradition of measuring children’s cognitive development. For people who make games and puzzles, the question of “which ability, in what order, at what difficulty” is the daily craft itself. KidGym formalizes that question for AI, and it reads directly back into human-facing difficulty design. It is built on top of the Gym API (a common programmatic interface for interacting with environments, widely used in reinforcement learning), so anyone can add new tasks.

Background

Most existing MLLM benchmarks (standard task sets for measuring performance) were “static”: the on-screen information never changed, and the model just answered a one-shot question. The authors see two gaps here. First, real tasks are “dynamic” — you interact continuously and stack actions until the task is done — yet few benchmarks measure this. Second, many benchmarks test only a single ability, telling us nothing about how several abilities interact.

So the authors ground their work in psychometric AI (adapting ability-specific tests validated on humans to evaluate AI) and the Wechsler Intelligence Scales, which treat intelligence not as one number but as a set of interpretable sub-abilities. They are explicit, however, that copying human abilities onto AI is inappropriate, and say they worked with child-brain-science experts (co-authors) to redefine five abilities for models. In their comparison table (Table 1) against frameworks such as ARC-AGI-2 (a hard abstract-visual-reasoning benchmark) and MiniGrid (goal-oriented grid tasks), KidGym claims a position as “difficulty-graded, user-extensible, all abilities, dynamic.”

Approach

In plain terms, the five abilities are: Execution (turning a goal into concrete, correct actions), Memory (retaining previously seen information and using it as ongoing context), Learning (taking in a new rule on the spot — even one that conflicts with prior knowledge — and applying it), Planning (looking ahead and sequencing steps), and Perception Reasoning (drawing logic and choosing answers directly from the image).

Of the 12 tasks, here are the six single-ability ones described visually: Classification (put items into containers as instructed = Execution), Selection (recall items shown before they were hidden = Memory), Sorting (rank things by a rule that clashes with reality, like “faster animals are heavier” = Learning), Maze (collect colored keys to open doors and reach a diamond in the fewest steps = Planning), Filling (restore an image like “a goldfish missing its head” by picking the right piece = Perception Reasoning), and Puzzle (reassemble a 4-piece target made of non-semantic random shapes = abstract Perception Reasoning). The other six are composite — Placement, Counting, Decode Maze, Memory Maze, Memory Filling, Memory Decode — layering a memory or learning load onto a single task.

The tricks that keep evaluation fair double as game-design tools. Scenes come in different semantic contexts (supermarket, canteen, farm) and layouts are randomly initialized each round, to prevent memorization of training data (data contamination). A backpack and a hint bar are kept as part of the state so the agent can retain crucial information, and actions are abstracted to high-level moves like “pick up the basketball” rather than step-by-step navigation. Each task has three difficulty levels; nine models were run 100 zero-shot rounds (solving cold, with no examples) per task, and some were also compared with chain-of-thought (CoT, verbalizing intermediate reasoning) and in-context learning (ICL, showing a few worked examples first).

Findings

The big picture: closed models (commercial ones accessed via API) far outperformed open ones (with public weights), and single-ability tasks scored higher than composite tasks. Success generally fell from L1 to L3, which the authors take as evidence their taxonomy grades difficulty correctly. o3, GPT-5, and Gemini-2.5-Pro stood out across all dimensions, sometimes near-perfect on Classification, Selection, and Memory Decode. Still, the human baseline sits near 1.00 on every task, so the gap remains large.

The failure the authors stress first is “non-semantic abstract visual information.” Filling (FI) uses nameable images like animals, whereas Puzzle (PU) asks for reconstructing an arbitrary shape of random blocks. At the easiest L1, the best FI score was 0.83 (o3) but the best PU score was only 0.30 (GPT-5) — just 5 percentage points above random. Next, “counting.” Humans score 1.00 across all three levels of the Counting task (CO), yet the best model, Gemini-2.5-Pro, reached only 0.72 at the easiest L1, with many failures mistaking a small cluster of two or three items for one. Raising image resolution helped, though humans need no such clarity; the authors read this as a sign models lack a robust representation of number and lean on high-resolution cues.

Third is “composite tasks.” Memory Maze (MMA) and Memory Filling (MFI) merely add a memory load to Maze (MA) and Filling (FI), yet success dropped markedly across all models — juggling several kinds of information, or interlocking rules, at once is still hard. Finally, reasoning methods helped unevenly: Gemini-2.5-Flash improved a lot with CoT, while o3 (which already integrates CoT) barely differed from zero-shot, and ICL could even underperform zero-shot on memory/learning tasks with randomized layouts, because — the authors say — the model over-attends to the examples and misses the changing scene. Overall, Perception Reasoning and Planning were the weakest of the five abilities across all models.

How designers can use this

First. If I were making a Sokoban-like game (pushing boxes and keys around) or a key-and-door maze, KidGym’s three-stage maze ladder — plain Maze (same-color key opens same-color door), Decode Maze (learn a mapping like “blue key opens yellow door” first), Memory Maze (remember the diamond’s location shown earlier) — is a ready-made recipe for stacking difficulty one layer at a time. It is a template for reasoning about which added load bites, and in what order, for the solver.

Second. If I were doing hyper-casual PCG (Procedural Content Generation, the automatic generation of content), KidGym’s method of “swap in different semantic scenes and randomly re-initialize the layout every round” is a concrete procedure for spinning infinite variations from one task. That mechanism is meant to stop AI memorization, but it reads just as well as a way to stop human players from memorizing patterns and to keep each run fresh.

Third. If I were embedding AI as a solver, a tutor, or an NPC’s brain, this paper is a map of “what to trust AI with and what not to.” Since counting and abstract shape-assembly are known weak spots, I can divide labor: surface quantities in the UI, and leave abstract shape-matching to humans.

Fourth, as a supporting use: tagging tasks by the three levels L1-L3 and the five ability dimensions is an idea you can apply to targeted questioning by a player’s weak points, or to selecting tasks by cognitive profile in accessibility and educational games. Writing down which ability a given move tests gives you a handle when you later tune difficulty.

Limitations

As for weaknesses the authors themselves acknowledge: the task count of 12 is limited. They state in the conclusion that “the current number of tasks is limited,” leaving expansion to future work on the strength of the extensible framework. They also caution in the text that an MLLM’s “memory” is unlike a human’s, since it can reread the entire history at every step, and repeatedly warn against mapping the ability names one-to-one onto human concepts.

What I, Fukai, would flag is this. First, this is a “test,” not a “game”: it does not measure fun, motivation, or flow (immersion). When bringing it into game design, I read it strictly as a story about capability profiles. Second, the human baseline lives in an appendix (Appendix C); since I have not read that appendix, I withhold judgment on the number of subjects or the procedure. Third, the reported figures are mostly zero-shot success rates, and numbers can move with real-world prompt engineering or extra tooling. Finally, this is a freshly posted preprint with few citations so far, so I add the caveat that its conclusions have not yet been widely debated.

Fukai’s reading

Let me flag this as my own interpretation. I want to place this work in a shift: from an era that measured game AI on a single “smart / not smart” axis, toward one that makes visible, ability by ability, which capabilities are missing and in what combination. In the vocabulary of design criticism, what KidGym does reads as close to an AI version of automatic difficulty-curve profiling — drawing the curve while tagging each move with “which ability does this test.” A tool built to measure human children ends up measuring artificial solvers, and its design philosophy returns to human-facing puzzle design. That loop is exactly why this paper drew me in.

Closing

If you want to go deeper, read this alongside ARC-AGI-2, which confronts the difficulty of abstract visual reasoning; MiniGrid, a classic of grid environments; and SmartPlay, which measures LLMs across several classic games — together they reveal the map of this field. Set beside the LLM-and-games benchmarks I have already covered on this site (such as GVGAI-LLM and SokoBench), the differences in design philosophy — “what to measure, at what granularity” — come into sharp relief. Today too, with a mug of strong drip coffee, marking up a printed PDF in colored pen, I will go looking for the next one.

References

Papers and related material referenced in this article:

・Children’s Intelligence Tests Pose Challenges for MLLMs? KidGym: A 2D Grid-Based Reasoning Benchmark for MLLMs (Hengwei Ye et al., 2026, arXiv preprint arXiv:2603.20209)

・KidGym project page (benchmark release)

・Related: Leveraging Procedural Generation to Benchmark Reinforcement Learning / Procgen (Cobbe et al., 2019) (inspiration for the maze task)

・Related: MiniGrid / MiniWorld (Chevalier-Boisvert et al., 2023, NeurIPS), ARC-AGI-2 (abstract visual reasoning benchmark), SmartPlay (Wu et al., 2024)