Li et al.: Rereading Video World Models as Game Engines — the Unsolved Problem Called State — Fukai Reads

TL;DR

Research on building playable worlds with video generative models (AI that predicts and produces footage directly) has grown fast. This paper is a survey that reorganizes that scattered literature around a loop conventional game engines have always used: player input updates an internal game state under rules, and observations are rendered from that state. From this loop the authors extract four dimensions, player action control, game state dynamics, state-observation persistence, and real-time interactive generation, and group existing methods into families along each.

Their conclusion is clear. The capabilities that remain hard, determining outcomes from accumulated conditions, preserving consequences that happen off screen, and surfacing effects at rule-defined moments, all revolve around the game state, which most models keep implicit. The authors also build a data engine for Black Myth: Wukong and collect over 90 hours of boss-encounter gameplay with frame-aligned player actions, engine-exported game states, and visual observations, offered as a resource for state-aware game world modeling. It is an arXiv preprint dated 15 July 2026, so not yet peer-reviewed.

Introduction

Today I read "From Pixels to States: Rethinking Interactive World Models as Game Engines" (arXiv:2607.14076v1, submitted 15 July 2026, cs.CV). The authors are Zhen Li, Zian Meng, Shuwei Shi, Mingliang Zhai, Jiaming Tan, Chuanhao Li, and Kaipeng Zhang, listed under Alaya Lab, with Chuanhao Li and Kaipeng Zhang as corresponding authors. It exists only on arXiv as a preprint, and since it was posted very recently it should be read as work not yet widely discussed.

I did not pick it because it is novel. Quite the opposite: what this paper does is not propose a new method but re-shelve existing research using vocabulary that game makers already own. Papers on generating games with video models have poured out over the last two years. Individual papers are readable, but from outside it is hard to see what has actually advanced and what has not. A text that draws that map using the game engine loop, a diagram everyone knows, is valuable for non-specialist readers.

The other reason is that the concept at the centre of the paper, state, is the most familiar thing in the world to a puzzle maker. A puzzle has an explicit board state, input changes that state under rules, and the state is drawn on screen. Puzzle makers write by hand exactly the structure this paper says AI still finds difficult. So although this is an AI paper, it also works as a re-reading of the vocabulary of puzzle design.

Background

Some background. A conventional game engine takes player input, updates internal variables, the game state such as health, stamina, cooldowns, and equipment, according to rules, and then renders the screen from the updated state. The paper opens with a Black Myth: Wukong example: pressing attack does not directly produce pixels. The engine first consults stamina, cooldowns, and the boss's position and animation phase, applies the combat rules to commit an outcome (a miss, damage, an interrupted animation, a phase transition), and only then renders the coming frames.

Recent video generative models skip that whole procedure. Instead of hand-authoring geometry and behaviour rules, a single model learns the regularities of a world from footage and predicts the next observation directly, conditioned on user input. From this route came real-time neural game engines and world simulators built on video generation, increasingly described as potential next-generation game engines. That is the situation the paper describes.

The trouble is that this route has advanced in separate streams: camera control refining how viewpoint motion is specified, language- and entity-level interfaces broadening what can be expressed, memory mechanisms keeping long rollouts consistent, and distillation and streaming pushing generation toward real-time rates. The paper notes these lines address key aspects but their connections and combined implications remain less clear. What was missing was not any individual technique but a shared set of axes on which to line them up.

Approach

The authors' method is organisation, not experiment. Using the action-state-observation loop as axes, they sort existing methods into four dimensions: (1) player action control, how player intent is represented before it drives generation; (2) game state dynamics, how the world's internal condition is represented and evolved; (3) state-observation persistence, whether consequences stay consistent over long play; (4) real-time interactive generation, whether the loop returns at the latency of play. Each dimension follows the same template: state what an interactive game world requires, group approaches into families, then discuss strengths and trade-offs.

Action representation, for instance, splits into three families. Actions as geometric trajectories condition on camera rotation and translation; viewpoint control becomes precise, but it covers only navigation, and the precise trajectories required remain distant from natural user input. Actions as motor signals feed keyboard and mouse states directly; this stays faithful to how players operate, but the same input can mean different things depending on state, which makes composite operations such as combo skills harder to learn. Actions as semantic events state intent in natural language; this reaches interactions no fixed input scheme covers, but conditioning cost grows with grounding granularity.

State handling splits three ways in the same manner: states entangled in observations (no separate state at all, dynamics living entirely in pixels), states as learned latents (a compact internal representation evolving recurrently under actions), and states as explicit descriptions (written out symbolically or textually, turning state transition into reasoning). For the third the authors grant that such states are readable and verifiable, while listing the weaknesses plainly: large-scale state-annotated gameplay data remains scarce, discrete records capture continuous visual dynamics only partially, and integration into the generation loop is largely unexplored.

The paper's other pillar is a data engine. The authors target Black Myth: Wukong, an Unreal Engine action RPG, and collect boss encounters from crowdsourced players of diverse skill levels and play styles. They instrument the engine to export per-tick actions and states as JSON, while a custom ReShade shader splits the display so RGB frames and depth maps are captured together and OBS Studio logs the system timestamp of each frame. Stamping both streams with the same clock gives a common basis for later frame-level alignment. The in-game UI is removed by disabling the corresponding rendering passes.

Findings

This is not a paper that reports numerical results, so its findings lie mostly in the structure the organisation reveals. The sharpest is the discussion of real-time behaviour. The authors point out that two latencies of opposite character bound the interactive loop. Control latency runs from an input to its visible response, such as the viewpoint turning; it should be minimised, as it governs game feel. Consequence latency runs from an action to its rule-governed outcome, such as a strike landing; this one should instead be accurate, because an attack unfolds through startup, active, and recovery, and an outcome surfacing instantly reads as wrong.

Against that, the authors state that existing works treat every delay as a quantity to shrink: few-step distillation (training a small fast model to reproduce a large one's behaviour), streaming generation that denoises a rolling window in a pipeline, and hardware-algorithm co-design. All of these shorten the path from input to frame, and, in the authors' assessment, none concerns when an outcome should appear.

The persistence discussion yields a concrete failure picture too. Methods that keep memory as stored past observations are dependable in largely static scenes. But in an interactive game world things change continually, so a faithful copy of the past may not serve as the reference for the present. The authors' example is easy to picture: a skill demolishes a nearby building, yet on revisit the memory may restore the intact one. Some methods exclude dynamic entities from memory, but that does not help when the environment itself changes.

The verifiable numbers for the dataset are these: over 90 hours of gameplay, 1280x720 resolution, 30 FPS. Each frame carries three kinds of ground truth: actions as raw mouse and keyboard input, states describing the underlying evolution of the game world, and observations comprising RGB frames and depth maps. States cover the poses of the camera, the player character, and the boss, plus current animations, active skills, and attributes such as health, stamina, attack/defense, and equipment. Two annotation types are added: slot captions that textualize fixed-length time windows into structured action and state slots, and semantic captions generated by Qwen3-VL-235B-A22B-Instruct.

Where This Is Useful

First: separating control latency from consequence latency is vocabulary you can carry into a puzzle that uses no AI at all. If I were building a daily puzzle, selection highlights and cursor movement should always return instantly (minimise control latency), while the correct/incorrect verdict may deliberately take a beat (consequence latency is about being right, not fast). Optimising both as one undifferentiated notion of responsiveness produces a strange feel: input is snappy but the verdict looks flippant. I intend to use this distinction as a UI review lens from now on.

Second: the three-way split between holding state explicitly and burying it in appearance is exactly the line of responsibility when using generative AI for content. If I were adding generation to a Sokoban-like puzzle, the board state (boxes, walls, goals) and the solver that guarantees a solution exists must live on the symbolic side. What a generative model may own is the look, the theme, the decoration. The paper's remark that rules captured as pixel correlations make spatial and logical consistency hard to guarantee is a description of what happens when you cross that line.

Third: decide by design how the world changes off screen. If I were building a puzzle with a scrolling board or multiple rooms, the treatment of objects outside the viewport must not stay implicit. The paper's failure case, a demolished building returning intact, reads not only as an implementation bug but as a design accident that breaks the player's causal understanding. In puzzles where only part of the board is visible, the presentation has to assure the player that the unseen region advances under the same rules.

Fourth: the data engine design reads as a template for telemetry in a web puzzle game. What the paper does, in essence, is stamp the triple of action, state, and observation with the same clock so they line up. If I wanted to analyse the difficulty of a daily puzzle after the fact, logging only which cell was tapped at what time is not enough. Only when the board state and what the player actually saw (what was highlighted) are aligned with it can the moment of getting stuck be reconstructed. The slot caption idea, folding a time window into a structured summary, also transfers as a way to make logs human-readable.

Limitations

Start with what the authors themselves concede. For the explicit-state route, they note that maintaining such states presupposes large-scale state-annotated gameplay data which remains scarce, that discrete records capture continuous visual dynamics only partially, and that integration into the generation loop is largely unexplored. For the newer family that updates memory as an estimate of the present, they write that high-frequency interactions induce small changes accumulating into substantial divergence, demanding update decisions that are efficient and reliable at every step, and that a principled grounding for this remains an open question. For semantic-event control they note a trade-off between control fidelity and interactive efficiency as grounding granularity rises.

From here on, what Fukai noticed on reading. First, the paper proposes no method of its own and runs no comparative experiment. The four dimensions and the family groupings are the authors' organisation, not a scheme validated as the only one. The right way to receive it is that these axes are useful, not that these axes have been shown correct. Within what I could read, the strengths and costs attributed to each family are also assessments based on reading the literature, not measured values.

What Fukai would point out next is that the dataset has not yet been evaluated. The scale and specification, over 90 hours at 1280x720 and 30 FPS, are stated plainly in the text, but I found no experiment training anything on it or showing how it compares with existing datasets. Nor could I confirm the access route or licence from the text. Whether it proves useful is something that will only become clear once somebody uses it.

One last point. Because the target is a single game and a single situation, boss encounters in Black Myth: Wukong, the conclusions available here are tightly bound to combat in a 3D action RPG. The authors explain the choice reasonably: boss encounters feature frequent interactions such as attacks and dodges, continuous state transitions, and last several minutes, providing long rollouts for training and evaluating long-horizon generation. But the paper does not claim the same insights transfer to puzzles, to 2D, or to genres where time moves slowly. Neither will I.

Fukai's Reading

Here is my own reading. I would place this paper at the point where the story of generative AI swallowing games comes full circle and the evaluative axis returns to the old design philosophy of the game engine. Over two years, progress in video generation has repeatedly achieved the feat of making things look like they move. What lining up these four dimensions brings into view, though, is that the remaining hard parts, determining outcomes, preserving what happens off screen, and getting timing right, are all problems engine designers solved nearly half a century ago with the single instruction: keep a state. In the vocabulary of design criticism, this reads as the automation of representation reaching a certain level, after which the separate job of writing rules becomes visible again. From a puzzle maker's seat the conclusion is almost gratifying, because it reads as saying that our long habit of hand-writing a small board state was not a detour.

Closing

Since this paper is a map, the good way to read it is to move on from the landmarks it marks. If you want to see the "video generation as game engine" route in the flesh, GameNGen (Valevski et al.), Oasis (Decart), and the Genie line (Bruce et al., Parker-Holder et al.), all cited repeatedly here, are the starting points. If instead the explicit-state route interests you, WildWorld (Li et al.) with its per-frame state annotations and AnimeGamer (Cheng et al.), which predicts character attributes at every step, sit at the entrance to the unexplored ground the paper describes.

To build a bridge from the puzzle side, read it against work that generates content while holding an explicit board state, such as constraint- or solver-based level generation that guarantees playability. Generative freedom and rule guarantees are not an either/or choice but a question of where you draw the line. This paper tells you roughly where that line currently sits. References are collected below.

References

Papers and related material referenced in this article:

・From Pixels to States: Rethinking Interactive World Models as Game Engines (Zhen Li, Zian Meng, Shuwei Shi, Mingliang Zhai, Jiaming Tan, Chuanhao Li, Kaipeng Zhang, 2026, arXiv preprint 2607.14076)

・Full text (HTML version)

・Related: Diffusion Models Are Real-Time Game Engines / GameNGen (Valevski et al., 2024)

・Related: Genie: Generative Interactive Environments (Bruce et al., 2024)

・Related: World Models (Ha & Schmidhuber, 2018)

・Game referenced in the text: Black Myth: Wukong (Game Science, 2024)