Turning “what makes a good puzzle” into a formula: DeepMind quantifies the counter-intuitiveness of chess puzzles

Introduction

Today's Tsumiki roundup. One piece today: I read, in the original English, a Google DeepMind preprint that tries to render “what makes a puzzle creative” into computable metrics, using chess puzzles as its material.

I remain poor at solving puzzles, but the question of how to define the “goodness” of a good puzzle is one no maker can dodge. Today's paper offers one answer to it, drawn from the old material of chess.

Generating Creative Chess Puzzles

The gist first. The authors are Xidong Feng and colleagues at Google DeepMind; the arXiv id is 2510.23881, first posted October 2025 (I flag up front that this is a somewhat older preprint). The premise: generative AI has advanced rapidly across domains, yet producing genuinely creative, aesthetic, counter-intuitive output remains hard. The authors tackle this in the domain of chess puzzles. They first train and benchmark several generative models — autoregressive transformers, diffusion models and others — on the Lichess Puzzler dataset, then propose a reinforcement-learning (RL) framework with rewards derived from chess-engine search statistics (source: arXiv:2510.23881 ↗, English, preprint).

What interests me most as design is not the machine-learning apparatus but the fact that the work renders a long-fuzzy quality — “what makes a good puzzle” — into computable amounts. The authors decompose creativity into three aspects long associated with it in the chess literature: counter-intuitiveness, aesthetics and novelty. The measure of counter-intuitiveness is especially sharp. They look at how much a shallow search (a proxy for intuitive assessment) and a deep search (a proxy for accurate assessment) disagree: the more a move looks bad or beside the point at first glance yet proves best on deeper reading, the more “counter-intuitive” it scores. They express this as a linear combination of search statistics whose dominant term is the “critical depth” at which the engine first finds the solution (this is the paper's definition, not my embellishment).

The other metrics, briefly. Uniqueness follows the Lichess Puzzler method: a position qualifies if the win-chance gap between the best and second-best move exceeds a threshold (puzzles without a single solution are excluded). Novelty is measured via Levenshtein distance over the board FEN and the principal variation (PV), plus the generative model's sequence-level entropy. Aesthetics is auto-classified by detectors consolidating 17 aesthetic elements from prior work and Lichess's own puzzle themes. Folding these into the reward and running RL, they report that the rate of counter-intuitive puzzles rose about tenfold, from 0.22% (supervised) to 2.5%, exceeding even the base rate in the Lichess data (2.1%) — the paper's claim.

More telling for a maker is the finding that naively maximizing “hard/surprising” breaks things. RL with pure puzzle reward tends to collapse into “entropy collapse,” endlessly emitting one high-reward puzzle; or the model inflates reward and entropy by adding pieces (two white queens, three black knights). Against this the authors add a diversity filter (board distance, PV distance, entropy threshold) so that only novel positions earn reward, and impose piece-count regularization, a KL constraint toward the original data, and a mixture of high-quality real data — making “realistic and diverse” explicit constraints (all per the paper).

From here I write as interpretation, flagged as mine. Beyond chess, this is not about whether a puzzle can be solved but about how to define the feel of solving it. The way counter-intuitiveness is measured — the more a player's first instinct rejects a move, the higher its value — looks like a design principle portable to puzzles at large: the heart of a good puzzle should be the move intuition first dismisses as “wrong.” This urge to formalize the very “goodness” of rules and puzzles rhymes with the mathematical systematization of Nikoli-style pencil puzzles I read the other day. Soberly, though, even after RL only 2.5% of generations qualify as counter-intuitive, and aesthetics was not directly optimized — merely “preserved” as a side effect. The authors themselves note these aspects are neither necessary nor sufficient for creativity. Caveats about measuring “creativity” by proxy included, I judged this a first-hand source a maker would bookmark and reread.

A line that stayed with me

From the original English:

“multiple ‘correct’ paths dilute the satisfaction of finding the sharpest, most brilliant line.”

In this one sentence the authors state why they insist on a unique solution. When several answers exist, the conviction of “it can only be this” goes cloudy — translated into design terms, the pleasure of convergence comes from the uniqueness of the option. I read it as a constraint that protects not difficulty but the sharpness of the finishing move.

Reference links

Discussed today:

・Generating Creative Chess Puzzles (Xidong Feng et al., Google DeepMind, arXiv:2510.23881, October 2025; English, pre-review preprint)

Closing

I cannot read a chessboard deeply. Even so, a definition that puts “looks like a blunder at a glance but is in fact best” at the center of “goodness” landed squarely with me as an aspiring designer. What I want to make is probably a design that, at the moment you reach the answer, betrays your own first instinct into “so that was the move.”

I also want to keep in view the limits of measuring creativity by proxy. Tomorrow, again, I will read one trustworthy first-hand source from somewhere in the world, carefully.