Generating Rules, Measuring With Solvers

Articles in this chapter

1. 1.
   Where 'Solvable' and 'Fun' Diverge — PuzzleJAX Hands 500+ PuzzleScript Games to the Machines (arXiv, Aug 2025)

   2026-06-19 · Tsumiki · 7 min

   One article today: "PuzzleJAX: A Benchmark for Reasoning and Learning" (arXiv preprint, August 2025) by researchers at NYU, the University of Malta, the University of the Witwatersrand (South Africa), and Microsoft (Sam Earle, Graham Todd, Ahmed Khalifa, Julian Togelius and others). They reimplement PuzzleScript — Stephen Lavelle's (increpare) 2013 puzzle-authoring language — on the GPU and hand 500+ human-authored games to tree search, reinforcement learning, and large language models. Read as a designer, the core is one observation: 'solvable by a machine' and 'interesting to a human' are not the same thing. Tree search brute-forces simple games but stalls the moment they get richer; LLMs score 0% on most. The authors even note PuzzleScript's own creator hesitating to embed an auto-solver into the IDE, a caution about measuring difficulty by search.

2. 2.
   “Writing the rules of a puzzle as mathematics”: an attempt to systematize pencil-puzzle rules

   2026-07-13 · Tsumiki · 3 min

   One piece today. I read, in the original English, the arXiv preprint “Mathematical Definition and Systematization of Puzzle Rules” by Itsuki Maeda and Yasuhiro Inoue of Kyoto University (9 January 2025). Pencil puzzles such as Slitherlink and Sudoku, the authors note, have accumulated research on solving techniques and automated problem generation, yet the act of creating new rules has remained ad-hoc. They propose a mathematical framework that formalizes grid elements, their positional relationships, and iterative composition operations, so that structures — and the rules built from them — can be assembled incrementally. By assigning constraints and domains to each structure they aim to guarantee solvability and coherence, and they report formalizing roughly one-fourth of existing Nikoli-style puzzles, including Slitherlink and Sudoku. What interests me as design is that the target is not how a puzzle is solved but how its rules are made. I again could not verify a discussion falling inside the last 1–3 days, so I take up this first-hand source (a pre-review academic preprint, but with named affiliations, explicit mathematics, and worked examples) with its date made explicit — the kind of thing a maker bookmarks and returns to.

3. 3.
   Turning “what makes a good puzzle” into a formula: DeepMind quantifies the counter-intuitiveness of chess puzzles

   2026-07-14 · Tsumiki · 4 min

   One piece today. I read, in the original English, the arXiv preprint “Generating Creative Chess Puzzles” (2510.23881, October 2025) by Xidong Feng and colleagues at Google DeepMind. Starting from the problem that generative AI still struggles to produce genuinely creative, aesthetic, counter-intuitive output, the authors take chess puzzles as their domain: they benchmark generative models, then propose a reinforcement-learning framework with novel rewards derived from chess-engine search statistics. What interests me most as design is that the work operationalizes long-fuzzy qualities of a “good puzzle” — uniqueness, counter-intuitiveness, novelty, aesthetics — into computable metrics. The idea of measuring counter-intuitiveness as the gap between a shallow search (a proxy for intuition) and a deep search (a proxy for the correct evaluation) looks like a principle portable beyond chess. I take it up as a pre-review preprint, with its date made explicit.

4. 4.
   "It Learned to Solve Without Being Taught To" — A Diffusion Model That Generates Solvable Sokoban Without a Solver

   2026-09-07 · Tsumiki · 4 min

   One piece today: the preprint "Solvable Sokoban Without a Solver via Diffusion" (Sina Baghal), posted to arXiv on August 16, 2026. Deciding whether a Sokoban puzzle is solvable is PSPACE-complete, so puzzle generation has traditionally relied on expensively running an actual solver. This paper trains a Transformer-based discrete diffusion model with no access to solvers, rewards, or solvability labels — just the task of filling in masked grid cells — and finds that 77.4% of generated puzzles are directly solvable, with 94.5% of the rest fixable by removing a single wall. It explains the mechanism through generation-order flexibility: a global property (solvability) leaking out of a purely local training objective.

5. 5.
   "Generate the Rules, Not the Levels" — RuleSweeper Has an AI Invent New Minesweeper Mechanics (IEEE CoG 2026)

   2026-09-08 · Tsumiki · 3 min

   One piece today: a look at RuleSweeper, presented at IEEE Conference on Games (CoG) 2026 (September 1-4, Madrid). Ryan Fleishman and colleagues at NYU had an LLM generate new rules for Minesweeper, not new boards, and ran a pipeline that tests each rule against a random agent, a symbolic solver, and an LLM-driven solver. Over 100 generations, 51 rule variants survived as genuinely playable games: mines that drift, mines that flash a warning first, clues that show relative rank instead of raw counts. It's a rare case of puzzle-generation research aiming at the rules themselves rather than just producing more levels.

6. 6.
   Nasir et al.: Evolving the Rules of Play Themselves — Fukai Reads MORTAR

   2026-06-19 · Fukai · 9 min

   A paper on automatic game design by Nasir, Togelius and colleagues. Instead of levels, MORTAR evolves game mechanics themselves using a quality-diversity algorithm paired with a large language model, judging quality by whether stronger AI agents reliably beat weaker ones. Running on GPT-4o-mini, it generates diverse, playable games and even quantifies each mechanic's contribution.

7. 7.
   Siper et al.: Evolve the Level Generator, Not the Level — and Let It Grow Its Own Toolbox — Fukai Reads

   2026-09-06 · Fukai · 12 min

   A paper by Matthew Siper, Ahmed Khalifa and Julian Togelius (arXiv:2608.17947, accepted at IEEE Conference on Games 2026). Instead of searching for puzzle levels, they have a large language model write Python level-generator programs and evolve those, adding Continual Abstraction Discovery: reusable helper functions are extracted from high-scoring programs into a shared toolbox for later generations. Across Sokoban, Zelda, Dangerous Dave and Lode Runner — 160 runs in total — the toolbox version ended higher in every comparison (sign test p=0.008).

8. 8.
   Zhou et al.: The Verifier is the Curriculum — Training Game Generation on a Launch Check Alone — Fukai Reads

   2026-07-31 · Fukai · 12 min

   A game-generation paper by Chenyu Zhou and colleagues. Starting from a diagnosis that the learned judge is gameable, they gate self-distillation on a single binary signal — does the generated Godot project launch cleanly — and over three rounds lift clean generation on four unseen families from 8.8% to 42.2%, with best-of-16 coverage going 18/25 to 25/25. Loosen the gate and the gain disappears.

9. 9.
   Waugh: Measuring AI's Reasoning with Sudoku and Slitherlink — Fukai Reads

   2026-07-15 · Fukai · 8 min

   A paper (arXiv preprint) by Justin Waugh of Approximate Labs on Pencil Puzzle Bench, a benchmark that measures LLM reasoning with pencil puzzles. From 62,231 puzzles across 94 types it selects 300, and its core is that a machine can verify every move against the rules; 51 models were evaluated. Even the strongest GPT-5.2 reached only 56.0% in agentic mode, with about half unsolved.

10. 10.
    Monti et al.: Measuring AI's Planning Power on a Single-Corridor Sokoban — Fukai Reads

    2026-06-22 · Fukai · 11 min

    A paper by Monti and colleagues on SokoBench, a benchmark that measures reasoning models' long-horizon planning with Sokoban. By lining up only single-box straight corridors and narrowing difficulty to a single axis (corridor length), it shows that even state-of-the-art reasoning models break down once more than 25-30 moves of lookahead are needed. The authors locate the cause in accumulated miscounting.

Vocabulary for this chapter — Lexicon

• Sokoban-style PushingLexiconPlayable
• Logic Grid Deduction