Ying et al.: Measuring AI's General Intelligence Through Every 'Human Game' — Fukai Reads

TL;DR

The paper I am translating today runs in the opposite direction from usual. This column often covers research that uses AI to make games. This one uses games to measure AI. A team from MIT, Harvard, Princeton, Cambridge and others proposes the space of all games that humans design for humans — which they call the 'Multiverse of Human Games' — as a yardstick for machine general intelligence (intelligence that copes broadly with many kinds of tasks).

As a demonstration, they rebuilt 100 popular titles from the App Store and Steam using an LLM (a large language model, trained on vast amounts of text) and had seven frontier vision-language models (AI models that handle both images and words) play them. The strongest model scored only 8.5 when the human median is set to 100 (source: Figure 5 in the paper). The authors report that memory, planning, and the ability to infer a game's rules while playing were the weakest points.

Introduction

The authors are twelve researchers led by Lance Ying and Ryan Truong, spanning MIT, Harvard University, the University of British Columbia, Princeton University, the University of Cambridge, and the Universitat Politecnica de Valencia. The list includes the cognitive scientists Joshua B. Tenenbaum, Thomas L. Griffiths, and Samuel J. Gershman, and José Hernández-Orallo, who has long studied the measurement of AI ability.

It is an arXiv preprint (a manuscript before peer review; arXiv:2602.17594, submitted 19 February 2026), so I note upfront that it may not yet have passed peer review. The project website (aigamestore.org) hosts the actual games that were built and their gameplay videos.

Why did I pick it today? The PCG (Procedural Content Generation) and level-generation papers I usually browse are mostly about the making side. This one flips the vocabulary of game design — difficulty, cognitive load, what makes something fun — and uses it as a ruler for AI ability. I found it striking that the very knobs game makers tune every day could double as a map of intelligence itself.

Background

Measuring intelligence with games is an old idea. Many AIs have shown superhuman strength in a specific game — chess, Go, Atari titles, StarCraft. But what the authors ask is not 'how well can it master one game' but 'how quickly can it learn a wide variety of never-before-seen games, given the same time budget as a human.'

Why 'human games'? The authors frame play as a mechanism for learning that has been conserved through evolution (a claim grounded in prior work). Humans have long abstracted real challenges — strategy, resource management, social maneuvering, spatial reasoning — into small games. So the whole of 'games humans make and enjoy' can be read as a concentrated library of the cognitive skills people need to live in the world.

Conventional benchmarks (standard tests of AI performance) have a weakness, the authors note. Most measure only narrow tasks and, being static, saturate quickly once developers optimize for them. An open space of all human games stays fresh as long as new titles keep appearing — that, the authors argue, is the novelty here.

Approach

At the center is the concept of the 'Multiverse of Human Games.' It denotes the space of all games humans could create, together with the distribution of how likely each is to be made and spread. In principle it is infinite and open-ended. But using real commercial games directly runs into walls, the authors list: copyright, engine diversity, collecting human play data, response latency, and training-data contamination (the risk that a model has already seen the benchmark games during training).

So the authors built a semi-automated pipeline (a chain of stages) called AI GAMESTORE, in four stages. First, they gathered 7,500 titles from the App Store and Steam, kept those with at least 10,000 reviews and an average rating above 4.5, and had Gemini 2.5 Flash score each for suitability — can it be played in minutes, expressed in p5.js (a tool for making visuals and motion with short programs), and given a numeric score — narrowing to 100.

Second, from each description an LLM (here Claude Sonnet 4.5) writes a JavaScript game. Another LLM first writes a test script, plays it to catch bugs, and fixes them; then a human actually plays and gives feedback in words until it is fun to play. On average this took 4.7 refinement steps, about 2 minutes each, roughly 30 minutes end to end per game. A human can also propose new mechanics to spawn 'variants.'

Third, each finished game is annotated for cognitive demand. Seven axes — visual processing, spatial-temporal coordination (acting with good timing), memory, planning, world-model learning (inferring unstated rules through play), physical reasoning, and social reasoning — were scored on a 0-to-5 six-point scale by three of the authors, who deliberated to resolve disagreements. Fourth, humans and models play with the same time budget and their scores are compared. For models, the game is paused every second to query the next second of actions.

Findings

On the human side, 106 participants (mean age 38.81) were recruited via Prolific (a platform for recruiting study participants), each playing 10 games for 120 seconds apiece. On the model side, seven models — GPT-5.2, GPT-5-mini, Gemini-2.5-Pro, Gemini-2.5-Flash, Claude-Opus-4.5, Qwen-3-VL-32B, and Llama-4-Maverick — were each run three times. Scores set each game's human median to 100 and are summarized across the 100 games with a geometric mean (an average less swayed by extreme values).

The strongest model, GPT-5.2, reached only 8.5 against the human baseline of 100 (source: Figure 5). All the top models fell below a tenth of human performance, and the differences among the top six were reported as not statistically significant (p < 0.05). The score distribution was bimodal. On about two-thirds of the games the models made some progress (around 10-30% of the human median, occasionally beating humans on very simple games); on the remaining 30-40% they got almost nowhere, under 1% of the human median.

Where do they stumble? The authors report memory, planning, and world-model learning as the three biggest bottlenecks. Moreover, the more distinct cognitive capabilities a game demanded, the more the models' relative scores fell. To the suspicion that slowness was to blame, the authors respond that restricting to games needing no fast reaction (puzzles and turn-based games) made little difference. On time, models took on average 12-18 times longer than the two minutes humans needed, often more than 20 minutes.

Where to Use It

This is an evaluation paper, but there is plenty for the making side to take home. I will name at least three.

First, use the seven-axis 'map of cognitive demand' to audit your own puzzle's design. Score, from 0 to 5, how much your level demands of visual processing, memory, planning, world-model learning, and so on. If it leans on a single axis, that may be another name for 'monotonous.' The paper shows AI collapses fastest on games that demand several axes at once — read the other way, the depth humans feel as 'bite' lives in the interplay of axes.

Second, use a frontier model as an 'unbreakable playtester' to get a rough read on difficulty. If a frontier model maxes out your casual game instantly, that stage may be too shallow. If instead the AI cannot make even 1% progress while humans breeze through, the stage may hold the Baba Is You kind of fun — discovering hidden rules through play. The shape of the AI's failure is a design clue.

Third, borrow the generation pipeline itself as a prototyping template. Description, then an LLM writes a JavaScript build, then automated tests catch breakage, then a human refines it in words: this loop runs at a realistic cost of about 30 minutes and 4.7 refinements per game. You can mass-produce puzzle variants quickly and run 'saturation-resistant' content like daily puzzles or roguelike stages.

Fourth, the authors' idea of 'spawning variants to avoid saturation' helps anyone who wants to run a leaderboard or challenge for a long time. Varying the mechanics slightly from one skeleton keeps things fresh even after solutions circulate.

Limitations

Start with the weaknesses the authors themselves acknowledge. The 100 games are mostly simple, short titles learnable in minutes; long, multi-hour timescales are absent. The NPCs (non-player characters, the characters other than the player) are naive, so intricate social reasoning — reading another mind that is reading yours — is not tested. They also candidly write that while LLMs can make playable games, they are poor at making 'interesting levels': generated stages were often too easy or unsolvable. And because a single game tests several abilities at once, it is hard to separate which ability caused a failure.

From here on is what Fukai points out. First, every evaluation is confined to '2 minutes of rapid learning' — an arena that favors human intuition's speed, and the paper cannot tell us whether models would catch up given more time. Second, what is being played are LLM-rebuilt approximations, not the original games; how faithfully the original is captured is left to human judgment, which introduces a wobble in the evaluation's foundation. Third, the cognitive annotations were made by three of the authors themselves, and in what I read the inter-annotator agreement is not foregrounded. Contamination is avoided by synthesis, yet the game concepts still come from popular titles, so the models may know them.

Fukai's Reading

From here it is my own reading. I want to place this study in a current where the vocabulary of game-design evaluation is being promoted into 'units for measuring AI.' Difficulty, cognitive load, what is fun — the design variables we have tuned by feel are being used, as they are, as the tick marks on a ruler of intelligence. In the language of design criticism, I read this as backing up an old intuition — that a good game weaves several abilities together and asks for them without strain — with outside evidence in the shape of how AI breaks down. The implication for makers is quiet but heavy: when you polish a single stage, you may, without knowing it, be writing a problem that measures some corner of human intelligence.

Closing

For those who want to go deeper: this paper sits atop the lineage of Kelsey Allen and colleagues' 'Using games to understand the mind' (Nature Human Behaviour, 2024) and Hernández-Orallo and colleagues' work on measuring AI ability axis by axis. Read together, the map of 'games as instruments for measuring intelligence' comes into view.

Pulled toward the maker's interest, reading it alongside prior work on generating games with LLMs (the papers by Todd and by Nasir are cited in the text) and the generation-and-difficulty papers I have translated earlier in this column shows that 'AI that makes' and 'AI that is measured' are two sides of the same pipeline. If you carry home one question — for the next game you design, what human ability is that stage measuring? — then today's translation has done its job.

References

Papers and related materials referenced in this article:

・AI GAMESTORE: Scalable, Open-Ended Evaluation of Machine General Intelligence with Human Games (Ying, Truong, Sharma, Zhao, Cloos, Allen, Griffiths, Collins, Hernández-Orallo, Isola, Gershman, Tenenbaum, 2026, arXiv preprint 2602.17594)

・Project site aigamestore.org (public games and gameplay videos)

・Related work: Allen, K. et al. (2024). Using games to understand the mind. Nature Human Behaviour. (from the paper's references; the lineage of using games to study cognition)