Triebel et al.: Does AI Have Both a Head and a Hand on a Classic Physics Puzzle? — Fukai Reads

TL;DR

Today I cover a paper that turns the beloved physics puzzle The Incredible Machine 2 (1994, where you build Rube Goldberg-style contraptions to solve tasks) into a testbed for how human-like modern vision-language AI can be at solving problems. The authors build VLATIM, a benchmark with five progressive stages, and find a large gap between thinking and doing: capable large models can plan how to solve a puzzle, but cannot point precisely to where on the screen to click.

No model could solve a full puzzle end to end. It is an arXiv preprint (posted May 2026, not yet peer-reviewed), but for people who make puzzles and games it concretely marks the limits of the idea of "let AI solve your levels to gauge difficulty", which is why I picked it.

Introduction

The authors are Maximilian Triebel, Marco Menner, and Dominik Helfenstein. It is an arXiv preprint (arXiv:2605.11223), first posted on 11 May 2026 and revised (v2) on 17 May 2026, classified under cs.AI. It has not passed peer review, so I note up front that its claims are still at a pre-review stage, and citations are yet to accumulate.

Why choose it today? Scanning the arXiv new listings, this was a rare study that takes a real, and much-loved, puzzle game head-on as its material. The Incredible Machine lets you place balls, mice, ropes and fans to build a chaining contraption from start to goal — the archetypal Rube Goldberg machine (an overcomplicated chain that achieves a simple aim). What happens when you throw a modern AI at it? I felt there was a lot here for game makers too.

Background

Research on AI agents that see a screen and operate it has surged. Beyond VLMs (models that see images and reason in words), VLAs (Vision-Language-Action models that see, think, and act) have appeared, with many benchmarks for automating web and phone UIs. But the authors point out that most existing benchmarks do not sufficiently test physical, causal reasoning — "if I move this object, what happens next?"

Physics puzzles are hard precisely there. It is not enough to state the right answer in words; you must move a mouse to a precise location in continuous space, place a part, rotate it, stretch it. Both understanding in the head and acting correctly with the hand are tested. The authors frame this gap — between high-level logical reasoning and a continuous action space requiring precise mouse interaction — as the underexplored territory, and build a tool to measure it.

Approach

The authors build VLATIM (Vision-Language Against The Incredible Machine), a benchmark of five stages of increasing difficulty. Rather than immediately demanding solve the puzzle, it decomposes ability and measures one rung at a time — that is the core of the design.

Part 1 is visual grounding (can the model point, with bounding boxes, to what is where on screen), scored by overlap metrics like IoU (Intersection over Union, how much predicted and true boxes overlap) and by distances between boxes. Part 2 is domain understanding (does it grasp the game world's rules and the objects' properties and states). Part 3 is event reasoning (if I move this, what happens next?, asked in both text and images). Part 4 is manipulation (move, place, rotate, multi-step, remove, stretch — actually performed). Part 5 is solving a whole puzzle.

Scoring mixes box metrics, LLM-based automatic scoring and human evaluation depending on the task. Five models are evaluated: the open UI-TARS 1.5 7B (a GUI-operating agent), its base Qwen2.5 VL 7B, the large Qwen3 VL 235B, and the closed Gemini 2.5 Flash and GPT-5 Mini (both said to carry broad world knowledge). There are formulas in the paper, but I set them aside here: read it as a staged measure of can it see, understand, and touch the right place.

Findings

The overall score (paper Table 3, out of 100) is highest for Gemini 2.5 Flash at 39.54, followed by GPT-5 Mini 37.98, Qwen3 VL 32.52, Qwen2.5 VL 22.84, and UI-TARS 1.5 at 19.76. The gaps look large, but the breakdown is more interesting.

On domain understanding (Part 2), Gemini scores 78.19, GPT-5 Mini 75.48, Qwen3 VL 60.85 — the large models excel. On event reasoning (Part 3) the closed models again lead, Gemini 46.92 and GPT-5 Mini 43.92. So understanding rules and predicting what happens is the strength of large, closed models. But on manipulation (Part 4) every model collapses, and the ranking flips: the best, Qwen3 VL, reaches only 16.96, UI-TARS 14.83, Qwen2.5 12.9, while the supposedly clever Gemini manages just 5.83 and GPT-5 Mini 7.5. Visual grounding (Part 1) is also uniformly low, clustered in a narrow 18.89–28.46 band.

And in Part 5, the authors report that no model solved even a single puzzle to completion. From this they identify two failure archetypes. The closed large models can plan but cannot point precisely to the right screen position — the paper calls them "Blind Strategists". The open GUI-operating models are relatively better at clicking accurately, but their underlying reasoning is weak. In short, only the head or the hand is working, never both.

Use cases

First, as a warning for anyone considering AI-driven automated playtesting (having AI solve levels automatically to detect difficulty or dead-ends). If you make a physics-chain puzzle (an Incredible Machine-like, or a rolling-ball contraption puzzle) and plan to have an off-the-shelf VLM try to solve it to gauge difficulty, this paper says: pause. The model can narrate a solution yet cannot specify precise positions, and in practice trips on a single click. Before using it for difficulty estimation, verify on your own game whether the model can point accurately to a screen location.

Second, the staged-decomposition philosophy itself transfers. VLATIM's five stages (grounding, understanding, event reasoning, manipulation, full solve) map onto tutorials for AI agents and onboarding for human players alike. If you make a hyper-casual contraption puzzle, you can borrow the idea of splitting the learning curve into recognize objects, learn properties, predict causality, get used to manipulation, solve the whole, measuring stumbles at each rung.

Third, an implication for UI and interaction design. That the cleverer models stumbled on precise clicking suggests, conversely, that interfaces demanding continuous, precise pointing are costly for AI and humans alike. If you build a puzzle expecting AI assistance or automation, providing gridding/snapping (rounding positions to a grid or magnet points) or a UI that discretizes choices will be kinder to both. The paper does not say this directly; I read it out of the rout in the manipulation part.

Limitations

First, what the authors themselves acknowledge in the Outlook. The evaluation is largely zero-shot (solving on the spot, without extra fine-tuning or examples), and gives only the relevant necessary textual information. So the numbers assume no hand-holding practice; results could change if models were shown examples. Also the target is one specific title, The Incredible Machine 2, and these scores need not generalize to physics puzzles at large.

What I, Fukai, add here are two points. One is that only five models are evaluated, and the closed ones are opaque, so the cause of the manipulation collapse cannot be isolated. The other is that scoring mixes human evaluation and LLM-based scoring; a reasonable device for scoring physics puzzles, but one should read the numbers as movable depending on the rater or the prompt. More than the absolute overall scores, the information lies in the shape of the breakdown: high understanding, devastated manipulation.

Fukai's reading

I write this explicitly as my own reading. I want to read this study as a reconfirmation, in the cleanest possible laboratory of the puzzle, of a phenomenon observed everywhere lately: AI looks smart while reasoning in language, but turns infantile the moment you give it a body (a hand on the screen). In the vocabulary of design criticism, it visualizes that AI has grasped only one of the two layers The Incredible Machine always had — the joy of solving in the head and the joy of building with the hand. The satisfaction of a puzzle is born the instant thought and action mesh; AI's failure illuminates that from the other side, is how I read it.

Closing

For those who want to go deeper, reading GUI-benchmark studies of screen-operating agents alongside the linear-Sokoban planning evaluation (SokoBench) I covered earlier on this site will reveal the map of what are we actually measuring when we have AI solve puzzles. Evaluation on physics puzzles has only just begun, and replications and extensions will surely follow. Since the material is a classic game, this is one whose sequels I want to keep watching.

References

Papers and related material referenced in this article:

・Do Vision-Language-Models show human-like logical problem-solving capability in point and click puzzle games? (Triebel, Menner, Helfenstein, 2026, arXiv preprint arXiv:2605.11223)

・DOI: 10.48550/arXiv.2605.11223 (arXiv version, not peer-reviewed)

・Related (this site): Measuring AI planning with linear Sokoban (SokoBench, Monti et al., 2026)