TAG
#benchmark
0 篇评论 · 8 篇随笔
相关随笔
Ye et al.: Measuring Image-Capable AI (MLLMs) with Children’s Intelligence Tests — Fukai Reads
A paper (arXiv preprint) by Hengwei Ye and colleagues at ShanghaiTech University on KidGym, an MLLM evaluation benchmark inspired by children’s intelligence tests (the Wechsler scales). It measures five abilities — Execution, Perception Reasoning, Memory, Learning, Planning — across 12 tasks on a 2D grid at three difficulty levels, evaluating nine models. Even top models reached only 0.30 on abstract-shape puzzles and 0.72 on counting against a human 1.00.
Waugh：用数独与 Slitherlink 测量 AI 的推理能力——Fukai 解读
Approximate Labs 的 Justin Waugh 撰写的论文(arXiv 预印本)，介绍了以铅笔解谜衡量 LLM 推理能力的基准 Pencil Puzzle Bench。从 62,231 道题、94 种类型中挑选 300 题，核心是机器可以逐步核算每一手是否违反规则，并据此评测了 51 个模型。即使最强的 GPT-5.2，在能动式解法下也只有 56.0%，约一半题目未能解出。
Ying et al.: Measuring AI's General Intelligence Through Every 'Human Game' — Fukai Reads
A preprint from a team at MIT, Harvard and others that measures AI's general intelligence through games humans made. Rebuilding 100 popular App Store and Steam titles with an LLM and having seven frontier vision-language models play them, the best reached only 8.5 against a human median of 100, falling far short on memory, planning and inferring rules.
Triebel et al.: Does AI Have Both a Head and a Hand on a Classic Physics Puzzle? — Fukai Reads
A paper by Triebel et al. evaluating VLMs on the classic physics puzzle The Incredible Machine 2. Using VLATIM, a five-stage benchmark, it asks whether screen-operating AI can solve problems like humans; the cleverer large models can plan but cannot click precisely, and no model solved even one puzzle to completion.
Monti et al.: Measuring AI's Planning Power on a Single-Corridor Sokoban — Fukai Reads
A paper by Monti and colleagues on SokoBench, a benchmark that measures reasoning models' long-horizon planning with Sokoban. By lining up only single-box straight corridors and narrowing difficulty to a single axis (corridor length), it shows that even state-of-the-art reasoning models break down once more than 25-30 moves of lookahead are needed. The authors locate the cause in accumulated miscounting.
Luo 等人：AI 智能体能否在真实引擎中制作出可以游玩的完整游戏？——Fukai 的解读
Luo、Wang 等人提出的评估基准论文 GameCraft-Bench，衡量编程智能体是否能端到端生成游戏。论文让智能体根据自然语言规格，在 Godot 引擎上制作可以游玩的完整游戏，并以启动、操作回放、视频评分来判定，共设140道课题（15个类型）。最强配置的全体得分也仅为41.46%，表明智能体虽能制作出机制框架，但距离具备内容厚度、界面易读性和精加工的完成品仍有差距。
Jiang 等：仅凭语言能否生成「可玩的游戏」——Fukai 读 OpenGame
香港中文大学 Yilei Jiang 等人研究的论文，介绍了一种从自然语言出发自动生成完整可玩2D网页游戏的智能体 OpenGame。通过可复用骨架与「活的调试手册」抑制集成错误，在150个课题中达到最高水准。然而谜题类游戏依然是最难处理的类型。
Li 等人：LLM 能「玩并通关」2D游戏吗 — Fukai 解读 GVGAI-LLM
Li 等人（NYU等）提出的 GVGAI-LLM 论文。该基准测试让语言模型游玩118款2D游戏，以测量推理能力与空间感知。将盘面翻译为ASCII地图后以零样本方式求解，GPT-4o-mini 在540关中的477关胜率为0%，整体胜率仅10.27%，未能达到经典搜索算法的水平。本文按「问题·方法·发现·应用场景·局限」的顺序逐一解析。
