Wang et al.: An LLM Agent That Reads Mental Busyness From Gaze — Fukai Reads

TL;DR

AI assistants on smart glasses have no idea how hard the user is currently thinking -- that is, they cannot sense cognitive load, the mental effort a person spends on a task. This paper proposes GazeMind, a framework that estimates that load on a three-level scale (low / moderate / high) from eye gaze alone (how the eyes move and fixate). Its distinctive move is that, rather than retraining a dedicated model, it reshapes gaze into a table and has a large language model (LLM, a model trained on huge amounts of text) read and reason over it.

To evaluate it, the authors built a new gaze dataset, CogLoad-Bench, with 152 participants and over 40 hours of data. GazeMind reached 62.73% accuracy on three-way classification, beating supervised methods (33-38%) and feeding gaze straight to GPT-4o (39.62%) by more than 20 points (paper Table 1). For those of us who make games and puzzles, it reads as one more instrument for measuring difficulty from a player's internal state. Note this is an arXiv preprint (submitted May 2026, possibly not yet peer-reviewed). This article aims to convey the key points without your having to open the paper.

Introduction

The authors are a joint team led by Bin Wang, with Yue Liu, Benjamin Newman, Michael J. Proulx and others, from Meta Reality Labs Research, Northwestern University, and HarmonEyes. The paper is arXiv:2605.05790, posted to cs.HC (Human-Computer Interaction, the field studying how people and computers interact) on 7 May 2026 as a preprint. A footnote in the original states 'Preprint.', so it has not passed peer review at any particular venue, and I will not treat it as peer-reviewed here.

Why did I pick it today? I browse arXiv's cs.HC and cs.AI every morning, and this paper takes 'cognitive load' -- a concept sitting right at the center of puzzle and game design -- and actually tries to measure and predict it. How to set difficulty, and where players get stuck, ultimately come down to how busy the mind is. The setting is smart glasses, but the idea of reading load from gaze carries straight over to game UX. I read it with a mug of strong hot drip coffee, marking up a printout with colored pens.

Background

Cognitive load is the mental effort a person invests while performing a task. When it is low there is spare capacity to take in more information; when it is high the person is maxed out and would rather not be interrupted. If an AI assistant understood this, it could do considerate things like defer notifications during busy moments. Until now, though, measurement relied either on self-report -- asking the person how hard it feels -- or on dedicated sensors like EEG or fMRI. The former interrupts the work; the latter cannot be fitted into lightweight eyewear.

That is where gaze comes in. Fixations (holding the eyes on one point), saccades (the quick jumps of the gaze), and pupil size are known to be cues to cognitive load. But the authors note three weaknesses in prior gaze-based methods. First, they cannot explain why a given gaze pattern signals high load (poor interpretability). Second, the model must be retrained for each new task. Third, gaze habits differ greatly between people, so a model does not transfer to others (poor generalization). Solving all three at once was the open problem in this area.

Approach

The authors' idea is to convert gaze into 'a table an LLM can read' and let the language model's reasoning judge cognitive load. Because LLMs have absorbed a great deal of the cognitive-science literature, they start out knowing things like 'a dilating pupil tends to mean higher load.' With well-crafted instructions they can handle a new task without retraining, and by being given past examples as context they can adapt to individual differences. GazeMind assembles this into four modules. Some formulas appear in the original, but the flow reduces to: translate gaze into words and tables, add context, individual traits, and worked examples, and have the language model read it.

The first module is Temporal Gaze Encoding: it turns raw gaze into features such as fixation duration, saccade amplitude, blink count, and pupil size, and lays the past few seconds out as a row-and-column table in markdown (the experiments use the past 5 seconds). The second is Task-Guidance Reasoning: since the same gaze can mean low load while reading but high load while gaming, it prepares per-task interpretation rules ('when this feature moves this way, load is high') and hands them to the LLM.

The third is Adaptive User Profile Calibration: it sorts people by their gaze habits -- for example a 'High-Reactor' whose pupil is normally large and variable, a 'Low-Reactor' with little variation, or a 'Restless' user who blinks a lot -- computes each person's resting baseline, and judges by deviation from it. The fourth is Cognitive Retrieval-Augmented Generation (CogRAG): it pulls up past samples similar to the current gaze, with their correct labels, and shows them to the LLM as worked examples. The four are combined into a single query asking the LLM for low / moderate / high. The key point is that it does no training (no parameter updates) at all -- it works purely by supplying information as context.

Findings

For evaluation the authors built a new dataset, CogLoad-Bench. Using a glasses-type device called Project Aria, they synchronously recorded gaze (90 Hz), egocentric video, and audio from 152 participants, totaling 456 recordings and over 40 hours. Participants verbally reported their current load on a 7-point scale every 15-30 seconds, later collapsed into three levels (paper Sec.4). Training and evaluation use disjoint users (106 for training / database construction, 46 for testing), so the design measures how well the system holds up on people it has never seen.

The main results are as follows (paper Table 1). GazeMind reached 62.73% accuracy and a 62.11% F1 score on three-way classification. By contrast, supervised methods such as decision trees, SVM, and LSTM stayed at 33-38% accuracy, and feeding gaze straight to GPT-4o gave 39.62%. The authors state it 'outperforms existing methods by more than 20 points across all metrics.' By task (paper Table 2), reading reached 64.98% accuracy and the gaming task (a socially oriented game with many environmental distractions) 60.63%, with reading slightly higher -- which the authors attribute to gaze being more stable during reading.

The paper also includes an ablation study (removing components one at a time to see which part of the design drives the result), reported in Table 3. Starting from plain GPT-4o at 39.62% accuracy, adding task guidance raised it to 45.34%, adding user-profile calibration to 49.10%, and adding CogRAG (showing similar past examples) pushed it up to 62.73%. So the final 'show worked examples' step was the single largest lift. On individual differences, most users reach above 60% accuracy with GazeMind, whereas with plain GPT-4o nearly half fall below 40% (paper Figure 6).

Use cases

So how can game and puzzle makers use this? First, difficulty calibration. If you build a puzzle game in a setting where gaze can be measured -- a PC eye tracker or a VR headset -- you can estimate how busy the player's mind is from their gaze and feed it into dynamic difficulty adjustment (DDA, tuning difficulty automatically during play): offer a hint when load stays high, add challenge when it is too low. The design lesson the paper offers is that shaping gaze into features and adding context and individual traits worked better than hammering raw gaze with machine learning.

Second, playtest analysis. If you are making a Sokoban-style box-pushing puzzle and want to know which boards freeze players, you could visualize high-load stretches from testers' gaze logs and quantitatively flag 'this board is taxing minds more than intended.' Such analysis has traditionally leaned on self-report or clear rates, but per-second load estimation may let you catch the very moment someone gets stuck.

Third, the takeaway is the stance toward individual differences itself. The heart of this paper is confronting 'the same gaze means different things for different people.' Translated to game design, it is close to correcting, per player, the obvious fact that the same input mistake means something different for a beginner than for an expert. If you do hyper-casual procedural level generation (PCG, Procedural Content Generation), you can apply this by tuning against each player's deviation from their own baseline rather than a single difficulty curve for everyone. Fourth, in educational or serious games (games with practical goals such as learning or healthcare), one could detect the moment a learner is overloaded and ease the pace.

Limitations

The authors themselves acknowledge three limitations (paper Sec.6). First, although no model retraining is needed, the per-task 'interpretation rules' must still be prepared in advance by human analysis (it is not fully automatic). Second, because the ground-truth labels are self-reported, subjectivity creeps in, and accuracy is capped by the ceiling of how much humans agree with each other. Third, it relies on gaze alone, and the authors note that adding video or audio could improve contextual understanding.

What I (Fukai) would add starts with how to read the absolute number. 62.73% is accuracy on three-way classification -- clearly above chance (about 33%) and well above prior methods, but not yet at a level that 'nearly nails' how busy the mind is. In real use you must design around misjudgments and avoid over-serving hints. Next, this study's 'gaming' task is a socially oriented game full of environmental stimuli, which loads the mind differently from a puzzle you sit down and solve; to bring it into puzzle games, read it on the assumption that you will rebuild the interpretation rules for that domain. And since this is a preprint, keep in mind these figures have not been through peer review.

Fukai's reading

From here it is my (Fukai's) reading. I would place this study in the lineage of 'extending player modeling (estimating a player's internal state) one step toward physiological signals.' In the vocabulary of design criticism, it is close to an attempt to automate the judgment 'is this player maxed out right now?' -- long left to the designer's intuition -- by translating it into an externally visible signal, gaze. What I find striking is that the biggest lift came not from a cleverer classifier but from the plain trick of 'showing similar past examples.' Difficulty is not an absolute quantity but a deviation from that person's ordinary running state -- something game designers have known by experience, to which this paper, as I read it, lends one piece of data-backed support.

Closing

For those who want to go deeper: to follow the theoretical background of cognitive load itself, starting from a review of the gaze-load relationship (the related work this paper cites) will give you a map. If you are drawn to 'difficulty' and 'insight' on the game side, reading it alongside the insight-search study we covered earlier on this site and the difficulty-adjustment and PCG papers will surface both sides of the question 'how do we measure difficulty, and how do we build it?' Gaze measurement still needs special hardware, but as eye-tracking-capable headsets spread, today's story may become a much more everyday design option a few years from now.

References

Papers and materials referenced in this article:

・GazeMind: A Gaze-Guided LLM Agent for Personalized Cognitive Load Assessment (Wang et al., 2026, arXiv preprint 2605.05790, cs.HC)

・Full-text PDF of the same paper

・All figures cited here are based on Table 1, Table 2, Table 3, Figures 4-6 and the body text of the above preprint (not final values, as it is pre-peer-review)