Wermann et al.: How In-Game AI 'Words' vs 'Demonstration' Change Learning and Cognitive Load — Fukai Reads

The short version

A research team in Munich, Germany, had an in-game AI character (an NPC, or non-player character, one the player does not control) play two kinds of helper — one that assists with words, one that actually demonstrates the next move — and compared how learning outcomes and cognitive load (the mental processing burden) changed. Using Qookies, a serious game that teaches the basics of quantum technology without any math, they split 152 people into three groups: no AI support, words-only support, and words-plus-demonstration support.

The result was a little surprising. Every group improved significantly from pre- to post-test, but there was no difference in learning gains between support types. The difference showed up in cognitive load: the words-plus-demonstration group reported significantly lower intrinsic cognitive load (the burden that comes from the inherent difficulty of the task) than the words-only group (effect size d=0.60). That this was a pre-registered study — hypotheses and analysis fixed in advance — is one reason I picked it today.

Introduction: who, and where

The authors are Caroline Wermann, Karina E. Avila, corresponding author Stefan Küchemann, and eight others — eleven in total. Affiliations center on the Faculty of Physics at Ludwig-Maximilians-University Munich (LMU), and also include the Munich Center for Quantum Science and Technology, the Technical University of Munich, Munich Quantum Valley, plus game-industry firms Studio Merkas and Quantum Gaming GmbH. It is a lineup that pairs physics-education and learning-science researchers with game developers.

This is a preprint posted to arXiv in February 2026 (arXiv:2602.08893 — a pre-review stage that may not yet have passed peer review). That said, the study was pre-registered (the hypotheses and analysis plan were published and logged before the experiment, which guards against reshaping the interpretation after seeing the results), with the plan filed on AsPredicted. It was funded by the German Federal Ministry of Research, Technology and Space (BMFTR).

I chose it today because the question of 'how to let an AI help inside a game' now speaks directly not just to educational games but to ordinary puzzle and game design — the hint-giving NPC, the guide that sits beside a stuck player. Every designer has wrestled with this, and here it is tackled head-on in a pre-registered comparative experiment.

Background: what was known, and what wasn't

Serious games (games built not for entertainment but for a specific purpose such as learning, while still working as games) have long been seen as a promising way to support learning. But the field agrees that merely using a game does not automatically produce learning; gains depend heavily on how the game is designed and how support is embedded. It is a tightrope walk: leave the difficulty intact while still helping.

The framework for that help is scaffolding (temporary support that lowers a task the learner cannot yet solve alone into a reachable range; it rests on Vygotsky's 'zone of proximal development'). Support connects to cognitive load theory (which assumes that working memory — the amount of information one can hold at once — is limited). The paper splits this load into three kinds: intrinsic load (ICL) from the inherent difficulty of the task, extraneous load (ECL) from poor presentation of the material, and germane load (GCL), the useful load spent on building understanding.

Recently, large language models (LLMs — AI trained on huge amounts of text that generate context-sensitive language) can be dropped into NPCs to give explanations and hints tailored to each learner. But it is not all upside. Researchers have flagged the risk of the AI inventing wrong information ('hallucination') and the worry of 'cognitive offloading' (outsourcing the act of thinking to the AI) — learners swallowing the AI's output whole and no longer thinking for themselves. That is this study's starting point.

Approach: Qookies and three kinds of support

The stage is Qookies, a serious game that lets players feel out the basics of quantum technology without any math. Players advance through escape-room-style levels by solving interactive puzzles — for example, manipulating a qubit's state as an arrow on a sphere called the 'Bloch sphere' to match a target state. Short explanatory units are tied to each puzzle, introducing the concepts in plain language.

They compared three levels of AI support. (1) No support. (2) Verbal support — an NPC you can chat with (a Llama 3.1 70B model underneath). It is primed in the background only with that level's concepts, so it avoids hallucination while being kept 'unaware of the actual solution.' (3) Verbal plus demonstration — in addition to chat, the AI actually performs actions in the game. This demonstrating role is based on one-shot reinforcement learning (a way to run, from very few examples, the framework of learning high-reward actions through trial and error); it starts without knowing the solution and learns by watching play. It is a 'co-learning' design where player and AI learn together.

The procedure had three phases: pre-test, about 25 minutes of play, then post-test (roughly 50 minutes total). Understanding was measured with a 16-item multiple-choice test; cognitive load was self-reported on a 0–6 scale using nine items adapted from Leppink et al. Participants were 152 people drawn from school pupils, university students, and the general public; the a-priori power analysis had targeted 160. Notably, some people never used the AI even in their assigned group, so the authors regrouped participants by 'actual usage' — I return to this under Limitations.

Findings: learning rose, but the difference landed on load

First, learning. Scores rose significantly from pre- to post-test in all groups (main effect of time F(1,149)=58.85, p<.001). Medians went 9→11 in the no-support group, 10→12 in the words-only group, and 11→13 in the words-plus-demonstration group (out of 16). Within-group gain effect sizes were d=0.42–0.52. But the 'group × time' interaction was not significant (F(2,149)=0.54, p=.583), so no difference in learning gains by support type was found. The game itself worked, but the type of AI support made no difference — that is how I read it.

The difference landed on cognitive load. Of the three kinds, only intrinsic load (ICL) differed by group (F(2,148)=4, p=.020), and post-hoc it was significant between the words-only and words-plus-demonstration groups (t(148)=2.66, p=.026, d=0.60) — load was lower with words-plus-demonstration. Meanwhile the gap between no-support and words-plus-demonstration did not reach significance (p=.081), and no-support vs words-only showed no difference. Extraneous (ECL) and germane (GCL) load did not differ by group. So 'adding demonstration lowers load' holds only 'compared with words-only,' not compared with no AI. The authors themselves write that the hypothesis was 'only partially confirmed.'

They also analyzed the content of the AI dialogues. 71 participants sent 246 queries. Most were level-specific questions (157), and in substance they mostly asked for advice on the next action (124). Zero utterances offered the player's own hypothesis or idea to the AI, and only 12 counted as discussion. And the regrouping mentioned earlier: some in the verbal-support group never used the AI at all, and some said they 'wanted to solve it themselves; using the AI felt like cheating.' The authors read this as a sign of 'desirable difficulty.'

Where to use it: for makers

If you are building a puzzle game with a hint feature, this study is material suggesting that 'a demonstrated hint is lighter than a verbal one.' A text hint requires reading it, finding the object yourself, and translating it into an action. By contrast, 'the AI performs the move once' makes it immediately clear which object to use and how, potentially lowering intrinsic load. If your tutorials or hints are over-written in words, it may be worth replacing them with demonstrations.

If you are thinking about dynamic difficulty adjustment for a hyper-casual or mobile puzzle, the result that 'providing support' does not by itself guarantee improvement is useful. All groups gained points, yet the presence or type of AI support made no difference — so support may help 'clear a jam' without directly 'deepening understanding.' A practical split is to design support as churn-prevention at hard spots, while leaving growth itself to level design.

If you are designing an NPC guide, the observation that players tend to expect the AI to be an 'omniscient being that knows the answer' is worth noting. This study's demonstrating AI did not know the solution at first and was designed to learn alongside the player, so it could not help in new situations and got branded 'useless.' You need UI that conveys what the AI can and cannot do — not only at introduction, but right at the moment the player is stuck.

Also, the voice saying 'using the AI feels like cheating' speaks to optional hint features in general. Rather than making hints smarter, a design that lets players clear things without needing a hint — leaving a sense of getting over it on their own — may reconcile satisfaction with learning. Low hint-usage is not necessarily a design failure.

Limitations: the authors' account, and what I noticed

What I, Fukai, lay out here is first the weaknesses the authors themselves admit. First, there was no difference in learning gains between groups. Second, many participants barely used the AI at all, forcing a regrouping by actual usage. Third, the demonstrating AI cannot act until it has learned from the player, so it could not help in first-encounter situations, which created a 'useless' first impression. Fourth, the game design itself may not have given enough incentive to use the AI.

What I noticed on reading is this. Although assignment was pre-registered and randomized, the groups were ultimately reshuffled by 'actual usage,' and I want to view that cautiously. The regrouped no-support group swelled from 47 to 75, which means giving up part of the benefit of randomization (people who did not use the AI — possibly a non-random subset — pool into the no-support group). Also, the drop in intrinsic load was for 'words-only vs words-plus-demonstration'; it did not reach significance for 'no AI vs words-plus-demonstration' (p=.081). The headline 'demonstration lowers load' is best received with that caveat. The sample is self-selected, and I would also discount for this being a pre-review preprint.

Fukai's reading

I flag from here that this is my own reading. I would place this study not in the race over 'how smart to make the AI' but in a design lineage of 'how much to keep answers out of the AI's hands.' The demonstrating AI does not know the answer and learns alongside the player — this 'deliberately kept ignorant' design reads as the very key to curbing cognitive offloading (outsourcing the act of thinking). In the vocabulary of design criticism, this is closer to automating a 'partner who gets stuck with you' than automating an 'omniscient walkthrough AI.' I took it as a paper that quietly checks the naive intuition that smarter hints are always better.

Closing

For those who want to dig deeper: read Bjork & Bjork for the origin of 'desirable difficulty,' and Chi & Wylie's ICAP theory for the framework capturing levels of learner engagement — together they give you the map. On the downsides of over-relying on AI, the studies this paper cites — Krupp et al. (harms of LLM support in physics education) and Kosmyna et al. — form the counterweight. And if you are interested in the 'measuring' side of cognitive load, pairing this with the Wang et al. paper we covered earlier (estimating cognitive load from gaze) shows both wheels: designing to 'lower' load and technology to 'measure' it.

References

Papers and related material referenced in this article:

・AI-based Verbal and Visual Scaffolding in a Serious Game: Effects on Learning and Cognitive Load (Wermann, Avila, Küchemann, et al., 2026, arXiv preprint arXiv:2602.08893)

・Pre-registration: AsPredicted pre-registration

・Related: The ICAP Framework: Linking Cognitive Engagement to Active Learning Outcomes (Chi & Wylie, 2014)

・Related: Desirable Difficulties in Theory and Practice (Bjork & Bjork, 2020)