TAG
#reasoning
0 reviews · 5 essays
Related essays
Mannem et al.: Some Puzzles Are Learnable, Some Are Not — Fukai Reads
An arXiv preprint by Gowrav Mannem and colleagues (Algoverse AI Research). On RecurrReason — Tower of Hanoi, River Crossing, Block World and Checkers Jumping unified under one difficulty knob (N=1-10; 10,817 puzzles, 285,933 moves) — small sequence models reached 97.27% validation and 81.00% out-of-distribution on Block World, but only 11.11%/0.00% on Tower of Hanoi, 1.11%/0.10% on Checkers Jumping, and 0.00% everywhere on River Crossing. A 60M-parameter T5 beat a 124M-parameter GPT-2 on every puzzle, leading the authors to conclude that architecture matters more than scale.
Ye et al.: Measuring Image-Capable AI (MLLMs) with Children’s Intelligence Tests — Fukai Reads
A paper (arXiv preprint) by Hengwei Ye and colleagues at ShanghaiTech University on KidGym, an MLLM evaluation benchmark inspired by children’s intelligence tests (the Wechsler scales). It measures five abilities — Execution, Perception Reasoning, Memory, Learning, Planning — across 12 tasks on a 2D grid at three difficulty levels, evaluating nine models. Even top models reached only 0.30 on abstract-shape puzzles and 0.72 on counting against a human 1.00.
Waugh: Measuring AI's Reasoning with Sudoku and Slitherlink — Fukai Reads
A paper (arXiv preprint) by Justin Waugh of Approximate Labs on Pencil Puzzle Bench, a benchmark that measures LLM reasoning with pencil puzzles. From 62,231 puzzles across 94 types it selects 300, and its core is that a machine can verify every move against the rules; 51 models were evaluated. Even the strongest GPT-5.2 reached only 56.0% in agentic mode, with about half unsolved.
Triebel et al.: Does AI Have Both a Head and a Hand on a Classic Physics Puzzle? — Fukai Reads
A paper by Triebel et al. evaluating VLMs on the classic physics puzzle The Incredible Machine 2. Using VLATIM, a five-stage benchmark, it asks whether screen-operating AI can solve problems like humans; the cleverer large models can plan but cannot click precisely, and no model solved even one puzzle to completion.
Monti et al.: Measuring AI's Planning Power on a Single-Corridor Sokoban — Fukai Reads
A paper by Monti and colleagues on SokoBench, a benchmark that measures reasoning models' long-horizon planning with Sokoban. By lining up only single-box straight corridors and narrowing difficulty to a single axis (corridor length), it shows that even state-of-the-art reasoning models break down once more than 25-30 moves of lookahead are needed. The authors locate the cause in accumulated miscounting.
