TAG
#benchmark
0 篇评论 · 12 篇随笔
相关随笔
Battleday et al.: Measuring AI Discovery With 70 Games That Never Explain Their Rules — Fukai Reads
An arXiv preprint by Ruairidh M. Battleday and fifteen co-authors. On DiG-bench — 70 text-string games with both rules and win conditions hidden, across seven tiers, 21 released publicly — the strongest single model beat 50 games and all models pooled beat 57, while all 70 were beaten by at least one human on a first attempt. Handed the ground-truth rules, the same model jumps from 18 games to 69, and agentic harnesses did not improve on the basic one.
Mannem et al.: Some Puzzles Are Learnable, Some Are Not — Fukai Reads
An arXiv preprint by Gowrav Mannem and colleagues (Algoverse AI Research). On RecurrReason — Tower of Hanoi, River Crossing, Block World and Checkers Jumping unified under one difficulty knob (N=1-10; 10,817 puzzles, 285,933 moves) — small sequence models reached 97.27% validation and 81.00% out-of-distribution on Block World, but only 11.11%/0.00% on Tower of Hanoi, 1.11%/0.10% on Checkers Jumping, and 0.00% everywhere on River Crossing. A 60M-parameter T5 beat a 124M-parameter GPT-2 on every puzzle, leading the authors to conclude that architecture matters more than scale.
Li et al.: Measuring Whether AI Really Sees Shape, via Jigsaw Puzzles — Fukai Reads
A paper by Shawn Li et al. introducing JigShape, a benchmark for spatial reasoning in vision-language models. Interlocking tab-and-blank pieces make the ground truth unique across 95,468 instances from 4x4 to 16x16; only GPT-5.5 beat chance zero-shot (69.65% on 4x4), everything collapses from 8x8 even after fine-tuning, and removing the shapes drops 97% to 10%.
Huang et al.: Letting an AI Play the Generated Game, Then Fix It — Fukai Reads
A game-generation paper by Yixu Huang and colleagues (Fudan University, Xiaohongshu and others). Play2Code puts a screen-driving GUI agent into the generation loop as a playtester, evaluated on PlaytestArena, a new environment of 200 tasks and 1,548 rubric criteria. Averaged over three backbones, rubric pass-rate goes from 29.7% for single-pass generation and 52.2% for a code-inspection-only pipeline to 66.8%.
Ye et al.: Measuring Image-Capable AI (MLLMs) with Children’s Intelligence Tests — Fukai Reads
A paper (arXiv preprint) by Hengwei Ye and colleagues at ShanghaiTech University on KidGym, an MLLM evaluation benchmark inspired by children’s intelligence tests (the Wechsler scales). It measures five abilities — Execution, Perception Reasoning, Memory, Learning, Planning — across 12 tasks on a 2D grid at three difficulty levels, evaluating nine models. Even top models reached only 0.30 on abstract-shape puzzles and 0.72 on counting against a human 1.00.
Waugh：用数独与 Slitherlink 测量 AI 的推理能力——Fukai 解读
Approximate Labs 的 Justin Waugh 撰写的论文(arXiv 预印本)，介绍了以铅笔解谜衡量 LLM 推理能力的基准 Pencil Puzzle Bench。从 62,231 道题、94 种类型中挑选 300 题，核心是机器可以逐步核算每一手是否违反规则，并据此评测了 51 个模型。即使最强的 GPT-5.2，在能动式解法下也只有 56.0%，约一半题目未能解出。
Ying et al.: Measuring AI's General Intelligence Through Every 'Human Game' — Fukai Reads
A preprint from a team at MIT, Harvard and others that measures AI's general intelligence through games humans made. Rebuilding 100 popular App Store and Steam titles with an LLM and having seven frontier vision-language models play them, the best reached only 8.5 against a human median of 100, falling far short on memory, planning and inferring rules.
Triebel et al.: Does AI Have Both a Head and a Hand on a Classic Physics Puzzle? — Fukai Reads
A paper by Triebel et al. evaluating VLMs on the classic physics puzzle The Incredible Machine 2. Using VLATIM, a five-stage benchmark, it asks whether screen-operating AI can solve problems like humans; the cleverer large models can plan but cannot click precisely, and no model solved even one puzzle to completion.
Monti et al.: Measuring AI's Planning Power on a Single-Corridor Sokoban — Fukai Reads
A paper by Monti and colleagues on SokoBench, a benchmark that measures reasoning models' long-horizon planning with Sokoban. By lining up only single-box straight corridors and narrowing difficulty to a single axis (corridor length), it shows that even state-of-the-art reasoning models break down once more than 25-30 moves of lookahead are needed. The authors locate the cause in accumulated miscounting.
Luo 等人：AI 智能体能否在真实引擎中制作出可以游玩的完整游戏？——Fukai 的解读
Luo、Wang 等人提出的评估基准论文 GameCraft-Bench，衡量编程智能体是否能端到端生成游戏。论文让智能体根据自然语言规格，在 Godot 引擎上制作可以游玩的完整游戏，并以启动、操作回放、视频评分来判定，共设140道课题（15个类型）。最强配置的全体得分也仅为41.46%，表明智能体虽能制作出机制框架，但距离具备内容厚度、界面易读性和精加工的完成品仍有差距。
Jiang 等：仅凭语言能否生成「可玩的游戏」——Fukai 读 OpenGame
香港中文大学 Yilei Jiang 等人研究的论文，介绍了一种从自然语言出发自动生成完整可玩2D网页游戏的智能体 OpenGame。通过可复用骨架与「活的调试手册」抑制集成错误，在150个课题中达到最高水准。然而谜题类游戏依然是最难处理的类型。
Li 等人：LLM 能「玩并通关」2D游戏吗 — Fukai 解读 GVGAI-LLM
Li 等人（NYU等）提出的 GVGAI-LLM 论文。该基准测试让语言模型游玩118款2D游戏，以测量推理能力与空间感知。将盘面翻译为ASCII地图后以零样本方式求解，GPT-4o-mini 在540关中的477关胜率为0%，整体胜率仅10.27%，未能达到经典搜索算法的水平。本文按「问题·方法·发现·应用场景·局限」的顺序逐一解析。
