Ahn et al.: Puzzle Difficulty Lives in Concepts, Not Looks — Fukai Reads

TL;DR

Today I read a paper posted to arXiv in February 2026 (still a pre-peer-review preprint) by Caroline Ahn and colleagues at Boston University. They took ARC (the Abstraction and Reasoning Corpus, a set of problems where you infer a grid-transformation rule from a few examples), originally built to test abstract reasoning in AI, and rebuilt it for human experiments as "CogARC," then had 260 people solve grid puzzles. The goal was to capture not just accuracy but how people think and where they go wrong, logged one edit at a time.

The short version: difficulty was driven not by surface features like grid size or number of colors, but by the conceptual complexity of the rule. And when people erred, they did not err randomly; they converged on the same wrong answers. For a puzzle maker, I read this as two concrete lessons: build difficulty in the layer of concepts rather than the flashiness of the board, and treat common wrong answers as a design signal.

Introduction

The lead author is Caroline Ahn, working with colleagues across Boston University's Graduate Program for Neuroscience, the Department of Psychological and Brain Sciences, and the Center for Systems Neuroscience. The work appears on arXiv (identifier 2602.22408) and is, for now, a preprint, meaning it has not yet passed formal peer review. I want to flag that up front (a preprint is a manuscript the authors post themselves before a conference or journal has vetted it).

My reason for picking this one today is simple. ARC keeps making headlines as a test of AI "reasoning," yet its problems are, at root, grid puzzles where you color tiles. So here is material that reads cleanly through a puzzle-design lens, measured carefully with the tools of cognitive science. For anyone building games and asking "what actually makes something hard," it is a rare, empirical map.

One more thing let me read it without over-discounting: the authors run it twice, as a 40-person supervised pilot (Experiment 1) and a 220-person online version (Experiment 2), checking whether the same tendencies reappear. Psychology rightly distrusts one-off flashy results, but because there is an internal replication (rerunning the same procedure to see whether the result holds), I can follow the argument fairly directly.

Background

Abstract reasoning has long been studied with tasks like Raven's Progressive Matrices (RPM, an intelligence test where you infer a rule from a 3x3 array of pictures and pick the missing panel from choices). RPM is widely used as an index of fluid intelligence (the ability to solve novel problems on the spot, distinct from accumulated knowledge). But because it is multiple-choice and scored right-or-wrong, the authors note it reveals very little about how a person actually thought.

That is where ARC comes in. ARC shows a few input-output example pairs, asks you to infer the hidden transformation rule, and then has you draw the correct output for a new input yourself. Being closer to free response than multiple choice, it lets researchers trace what hypotheses people form, how they move their hands, and where they go wrong. The rules draw on objects, number, and geometry, grounded in what developmental psychology calls "core knowledge" (basic early-emerging intuitions about objects, number, and space).

The catch is that most prior human studies of ARC had each person solve only a handful of problems. As a result, the moment-to-moment dynamics stayed largely unknown: how one person's strategy shifts over time, how people behave on the harder problems, and whether, when they fail, they converge on the same answer. Filling that gap is the aim of this study.

Approach

The authors hand-picked 75 problems from ARC's training set and rebuilt them for humans as CogARC. They chose only problems where the grid size stays the same from input to output; then three raters sorted each problem into four categories (objectness, geometry and pattern, number and counting, goal-directedness) and scored rule complexity from 1 to 3 (complexity being set by how many core-knowledge concepts a rule uses and how much its parts interact).

Participants studied the input-output examples on the left, then drew their answer for the central test input by coloring tiles in a grid editor on the right. Each problem allowed up to three attempts, with right/wrong feedback after each. The team defined a "difficulty score" from the number of attempts needed to reach a correct answer, from 1 (solved first try) to 4 (all three attempts failed). They also logged "deliberation time," the gap between seeing a problem and making the first edit.

The interesting part is that every single edit (which tile, which color) was logged at millisecond resolution. That makes it possible to quantify how similar different people's coloring was. Using a measure of how much two people's sets of edits overlap (the Jaccard similarity, which looks at the proportion of overlap), they measured, per problem, whether everyone solved it the same way or diverged. They ran this whole battery twice, in Experiment 1 (40 people, supervised) and Experiment 2 (220 people, online).

Findings

Overall, people solved well. Mean accuracy was 89.5% in Experiment 1 (SD 10.2%) and 80.1% in Experiment 2 (SD 16.6%, median 83.6%, ranging from 13.7% to 100%). But that high average hides wide per-problem variation. In Experiment 2, 78.7% of problems were solved on the first try by at least half of participants, yet only 17.3% were solved first-try by 90% or more, and some were solved by almost no one (all figures as reported in the paper).

So what drove difficulty? Harder problems had longer deliberation times (r=0.69, p<.001 in Experiment 2) and more divergent solving (difficulty vs. edit similarity correlated strongly negatively, r=-0.83, p<.001). Meanwhile difficulty was essentially unrelated to grid size (r=0.05), number of colors (r=-0.05), or the number of edits required (r=0.12). The authors conclude that difficulty lived in inferring the rule itself, not in appearance or manual effort.

They also track change over time. As the task went on, participants got faster to make their first move (deliberation-time slope averaged -0.114, statistically significant). Yet accuracy dipped slightly (difficulty-score slope averaged a small but significant 0.005). The two trends barely tracked each other (r=-0.12, p=.066), and the authors read this less as a speed-for-accuracy trade than as adaptation, growing task familiarity or mild fatigue. They state there was no evidence that practice on earlier problems raised accuracy on later ones.

The most suggestive result is the shape of the errors. Despite the near-free-response format, people did not err randomly; many converged on the exact same wrong answer (a "common error" being an identical output from five or more participants). In one pattern-completion problem, the participants who reached common errors combined (71) outnumbered those who got it right (54). The authors read this as the two-sided nature of inductive biases (systematic tendencies in how people generalize from sparse examples, rooted in core knowledge): they aid fast learning but also produce the same shared misreading.

How Makers Can Use This

Now, from a maker's angle, some concrete uses. First, build difficulty in the layer of concepts. Here, grid size and color count were nearly irrelevant to difficulty; what mattered was how the rules interlocked. If I were building a Sokoban-like (a puzzle where you push boxes onto targets), I read this as: rather than padding with a bigger board or more boxes, add one layer where mechanics interfere (a nested rule like "recolor a group if fewer than three same-colored tiles are connected") to raise difficulty on purpose.

Second, use time-to-first-move as a difficulty sensor. Deliberation time correlated well with difficulty here. Beyond win/lose or total clear time, measuring the silence between opening a level and the first action lets you catch where players get stuck early. If you dynamically tune difficulty in hyper-casual procedural generation (PCG, Procedural Content Generation, the automatic creation of content), that silent interval could be a real-time cue.

Third, collect common wrong answers as a treasure map. This study quantified how many people converge on the same error. If lots of your players fall into the identical mistake, that may be evidence your tutorial or examples are nudging a plausible-but-wrong generalization. Bundling the error logs by a threshold like "five or more identical" shows which example to swap to fix the teaching. That connects directly to onboarding (the opening stretch that teaches how to play).

Fourth, do not count on within-session practice to lift players. Participants got faster over time but did not get better at inferring new rules; if anything, accuracy slipped a little. That fits the background that fluid intelligence resists short-term training. Across a long haul of 75 problems in two hours, fatigue also seems to lurk. When shaping a difficulty curve or placing breaks, I would treat "they'll naturally get better later" as a risky assumption.

Limitations

Let me look at the limits honestly. As the authors themselves acknowledge, ARC measures fluid intelligence and is not the kind of skill learning that improves with practice. They also concede that, given 75 problems over two hours, the late-session dip in accuracy may be partly fatigue or waning motivation. And they are explicit that they chose only problems where input and output grid size match, that CogARC is thus a slice of full ARC, and that it does not directly measure the core-knowledge systems (the results only "suggest").

From here on are things Fukai noticed while reading. Experiment 2's participants were mostly 18-35-year-olds recruited from Amazon's crowd-work market, and 97.7% were native English speakers; mapping this straight onto a real game audience would be premature. The difficulty score is attempt-count based, so it risks counting giving up and failing on the same footing. The over-time accuracy decline (slope 0.005), though significant, is very small, and should not be read as a big practical effect. And, to repeat, this is a pre-peer-review preprint; the internal replication and agreement with prior work like H-ARC lend support, but it is too early to treat as settled fact.

Fukai's Reading

Here, and I mark this as my own reading: I want to take this study as a long-awaited empirical backing for the claim that a puzzle's difficulty lives in the space of meaning, not the surface. In the vocabulary of design criticism, it is close to the old intuition that good difficulty should be built semantically (the conceptual depth of the rule) rather than perceptually (the sheer amount of visual information). And the "shared wrong answers" finding looks, to me, like an attempt to map players' inductive biases. A wrong answer is not the noise of failure but a sample of which hypothesis a person forms first, and the maker who can feed that back into design has an edge, is how I read it.

Closing

For those who want to go deeper: reading Chollet's "On the Measure of Intelligence" gives the foundation for ARC's idea. Pairing it with H-ARC (LeGris et al.), a large-scale measurement of human ARC performance, and ConceptARC (Moskvichev et al.), which organizes problems by rule type, makes the map of this field three-dimensional. If insight and search interest you, this connects to the earlier piece here on Chao et al.'s work on insightful problem-solving. The dataset itself is public, so getting your hands dirty and staring at the clusters of wrong answers may, surprisingly, be the fastest route.

Sources

Papers and materials referenced in this article:

・Exploring Human Behavior During Abstract Rule Inference and Problem Solving with the Cognitive Abstraction and Reasoning Corpus (Ahn et al., 2026, arXiv preprint 2602.22408)

・HTML version of the same paper (text and figures)

・CogARC behavioral dataset (Zenodo, CC-BY, DOI:10.5281/zenodo.18177487)

・On the Measure of Intelligence (Chollet, 2019, the origin of ARC)

・Related work (mentioned in text): H-ARC (LeGris et al., 2024-2025) and ConceptARC (Moskvichev et al., 2023), both on human ARC performance and error structure