Li et al.: the AI that showed up uninvited was closed by five players out of ten — Fukai Reads

In short — it was the delivery that got refused, not the advice

"An AI that helps you inside the game." It sounds useful. But when ten people actually played with one, at least five minimized the window or stopped opening it at all.

Today's paper evaluates PEARL, an AI support agent built into a puzzle game. It is peer-reviewed, accepted to the 2026 IEEE Conference on Games (CoG). What makes it unusual is that the people who built it write, plainly, that it did not work.

And what got refused was not the content of the advice. The same explanations were called useful when players went and asked for them. What players disliked was the fact that it arrived unasked. The authors turn the question on their own design: are we building Clippy?

Who wrote it, and what kind of paper it is

The authors are Jiahong Li, Sai Siddartha Maram, Atieh Kashani, Ulia Zaman, Zhiyu Lin, Cameron Marano, Roger Azevedo, Jichen Zhu and Magy Seif El-Nasr — nine in all. It was posted to arXiv on 12 September 2026 (arXiv:2609.13718).

What matters is that this is not an arXiv-only manuscript. What is posted is the author-accepted version, and it states that the paper was accepted to the 2026 IEEE Conference on Games (CoG). It has been through peer review. Eight pages, three figures, one table. The primary subject class is cs.HC (human-computer interaction), with cs.AI secondary.

The system is called PEARL (Parallel Education Agent for Reflection and Learning). Under the hood it is RAG (Retrieval-Augmented Generation — a method that first searches for outside material and then writes an answer using it as source). It exists to give a plain language model access to facts from inside the game.

I chose this paper today because it is a record of something that did not work. A success report only tells you to build the same thing again. A failure report tells you where the floor gives way. For anyone designing a hint system, the second is far more useful.

The game called Parallel, and the support it already had

Parallel is a puzzle game for learning parallel programming — the practice of running several processes at the same time. On screen the processes move as arrows. Players place signals and semaphores (a semaphore limits how many things may occupy the same place at once) and link them together to direct the traffic.

Two levels were used in the study. Level 3 is an eight-zone board that introduces the basics of synchronization. Level 13 has sixteen zones and requires coordinating multiple threads using both signals and semaphores.

The game already had a support feature called "Community". It is a visualization of an OPM (Open Player Model — a system that shows players the record of their own play). It offers heatmaps of where other people did what, node-and-edge graphs of solutions, and aggregate performance per level.

But seeing is not the same as understanding. Earlier work had already noted that even with these visualizations, "learners must still independently interpret what they observe, which is a non-trivial cognitive task". Across roughly fifty play sessions the same struggles kept appearing: trial and error without reflection, difficulty diagnosing why something failed, and getting stuck on one strategy.

Adding a language model is the obvious next move. But a plain language model has neither other players' traces nor an expert's annotations saying which move was good or bad. And in a logic puzzle, a single misplaced edge separates a solution from a deadlock. Searching on loose semantic similarity alone will get that one edge wrong.

How PEARL works, and how they tested it

PEARL searches along two tracks: one by the meaning of words, one by the shape of the board. Both are needed because, as above, semantic closeness alone is not enough for a logic puzzle.

The semantic track queries an expert-annotated knowledge graph. For the two levels used in the study it holds 133 knowledge units connected by 391 directed recommendation edges. Each unit records whether a move is good or bad, the concept it demonstrates, the common misconception behind it, and what to do instead. When the player makes a correct move, it returns four alternative good moves; when the move is wrong, three similar mistakes.

The structural track treats the board as a graph outright. Zones are nodes, the synchronization links the player drew are edges, and the adjacency matrix is flattened into a string of ones and zeros — 64 digits on Level 3 with its eight zones, 256 on Level 13 with sixteen. Then it simply counts how many positions differ from another player's board (the paper calls this the Hamming distance).

There are two modes. To answer "am I on the right track?", it surfaces the board that differs least, reinforcing the current approach. To answer "what is wrong?", it surfaces a board that was similar up to this point but diverged on the decisive move and reached a solution. It keeps at most five.

The two tracks are chained. The semantic search first produces "this kind of move would be better", and only boards containing those recommended moves are eligible for the structural search. It is a well-considered design.

The problem was a default setting. PEARL shipped with proactive delivery on: it speaks without being asked. And it fires every time the player creates a link. There is deduplication within a session, but no cooldown and no throttle on the number of moves.(Diagram) PEARL's two retrieval tracks. Two inputs — the move or question you typed, and the shape of your current board — are looked up against expert notes and other players' boards, then merged into a single reply.

Ten participants, all with at least an undergraduate degree in computer science. Each person used both forms of support, in counterbalanced order: half played Level 3 with Community and Level 13 with PEARL, the other half the reverse. Each session lasted twenty minutes, with a tutorial on the tools beforehand.

They measured a lot. TAM (the Technology Acceptance Model, a standard instrument asking whether people find a tool useful and easy to use), NASA-TLX (a standard workload measure), PENS (which measures how far a game satisfies players' psychological needs), and an eight-item test of parallel programming knowledge. On top of that, think-aloud during play and semi-structured interviews afterwards.

The analysis is codebook thematic analysis. Fourteen codes were fixed in advance from the research questions, and two researchers independently coded the same transcript (fourteen segments) first. Agreement was 92.9% with a Cohen's κ (kappa, an agreement measure that discounts chance) of .92. The codebook was then locked and the remaining transcripts divided up. As procedure goes, this is careful work.

What happened

Start with the numbers. On TAM perceived usefulness (7-point scale), Community scored 4.98 and PEARL 4.57. On ease of use, 5.02 against 4.75. The existing visualization came out ahead on both.

Overall NASA-TLX workload was 46.9 against 49.1 — barely different. But the frustration subscale alone opened up to 43.2 against 56.5. Players were not more tired. They were more annoyed.(Diagram) The NASA-TLX comparison. Overall workload is almost identical, yet frustration differs by thirteen points.

Behaviour was blunter still. At least five of the ten minimized or abandoned the AI window during play. The paper attributes this to three things: proactive delivery, generic responses, and a deficit of trust.

It was not a total rejection. Four of the ten (P4, P6, P7, P10) saw the two tools as serving distinct roles. P10 said Community "is the same for everyone… the AI is based on exactly where your stuff is right now." P6 noted that "it already knows your current solution state, you don't have to give it information."

When the semantic search landed, it did the work of naming things. P4 described an "aha" on realising that the arrangement he had reached by intuition was about mutexes. For three participants, though, the replies read like the same script each time. What P8 received amounted to "interesting choice you made… critically think about your thing" — nothing to act on.

On peer boards the results split. Seven of ten said seeing other people's struggles was reassuring — "at least I'm not the only one struggling", as P6 put it. Five used peer boards to find approaches structurally different from their own; P10 compared it to reading other people's solutions after solving a problem yourself. But four found the structural matching confusing even when it was conceptually aligned, and P3 found the peer boards discouraging rather than collaborative.

The most consistent thread was discomfort with interruption. Four of ten (P2, P3, P7, P10) named unsolicited intervention as a concrete negative. P3 called it "jarring" — "when I'm putting an element on the track… it interrupts and tries to give advice." P10: "telling me the answer before I asked it." Even P7, who valued PEARL overall, complained that it "gives you feedback every single time you make updates, even though you haven't finished your updates."

The authors' summary is the sharp part: "The objection was rarely to the content of the help. Several of the same participants endorsed PEARL's grounded explanations when they sought them on their own terms. What participants rejected was the unsolicited delivery."

For completeness: the eight-item knowledge test averaged 5.50 correct (68.8%, SD 0.94, range 4.0–7.0). All ten got the definitions of "semaphore" and "critical section" right; the hardest item was answered correctly by four. The authors state explicitly that these scores reflect aggregate post-session understanding and cannot be attributed to either condition.

What a puzzle designer can take from this

1. Make silence the default. Suppose you are building a Sokoban-like and want to add AI advice. The failure this paper documents is firing on every single link the player creates, with no cooldown and no move throttle. Do the opposite. Say nothing by default, and when the player stalls, let a button glow quietly. Whether to press it is theirs to decide.

2. Let players pick the grain. P7's sentence is already a spec: "If I'm not even close… I might need a full explanation. But today I was really close, I just needed a hint." Offer three levels — nudge, direction, answer — and let the player choose each time. The same person wants a different level on a different day.

3. If you show peer boards, show the fork. Merely similar boards do not help. PEARL's correction mode is the idea worth stealing: surface a record that matched yours up to this point, differed on one move, and got to the solution from there. For a daily puzzle, that can be a single other player's history that matched yours through the same move count today.

4. Show that it is not just them. Seven of ten found it reassuring to see other people stuck. That is not even advice. Simply displaying how many people went wrong on which move of today's board does the work. But some, like P3, find it discouraging, so make it something players can turn off.

5. Bolting on a general language model is not enough. PEARL could say why a move was bad because an expert had written 133 annotations and 391 recommendation edges. The flip side: wherever the annotation is thin, you get an AI that speaks only in generalities. Three of ten received exactly that. Do not stop at wiring up the model.

6. Find out what triggers help-seeking first. Most participants opened a tool only after repeated failure, and the threshold varied enormously. P8 needed to hit a wall "consistent"-ly before opening anything, P6 opened tools out of curiosity, and P2 held out until time pressure forced it. Decide who you are designing for before you design the trigger.

What remains unknown

The authors are explicit about the limits. First, there were ten participants, all holding a computer science degree. The findings do not transfer as-is to novices or to non-technical domains.

Second, because each person used both tools, order and difficulty effects remain. Counterbalancing reduces them; it does not remove them.

Third, and most important: the PEARL/Community comparison cannot isolate the effect of the retrieval architecture. The two differ in modality, timing, interface and interaction style all at once. The authors state that this "characterizes two contrasting support paradigms rather than providing an architectural comparison".

They also concede that some of the retrieval quality problems may reflect gaps in the knowledge graph's coverage. A systematic comparison of semantic-only, structural-only and combined retrieval against expert ground truth is left to future work.

What I would add is threefold. First, twenty minutes is short. An unfamiliar tool's interruptions are being judged at the moment they are most conspicuous. Whether a week of use would breed tolerance or deeper irritation, this paper cannot say.

Second, the study does not answer whether any of this helped people learn. The mean of 5.50 on the knowledge test cannot be split by condition, as the authors say themselves. What has been answered is whether the tool got used. Reading it as "AI support does not aid learning" goes well beyond the evidence.

Third, Community is a tool this community had already been using. As far as I can see, the limitations section does not name familiarity as a possible advantage for it. That said, the central finding — that interruption was resented — is hard to explain by familiarity alone.

How Fukai reads it

I would read this study as a marker that the question has moved from what an AI says to when it opens its mouth. The retrieval engineering here is substantial — 133 annotations, 391 recommendation edges, 256-digit board matching — and it was undone by a single default: fire on every link the player creates. In the vocabulary of design criticism, this is not a paper about automating the hint button. It is a paper about designing the act of asking for a hint. Help that was not wanted does not help, however correct it is.

What to read next to see the map

If you want to follow the literature: turning a puzzle solver into a move-by-move tutor is covered in Wang et al.; how much people buy hints when hints have a price is in McCaughey et al.; and how players grow their own library of parts while solving is in Zhao et al.. Those three plus today's paper sketch most of the map of "designing the teaching side".

If you would rather check by playing: 7 Billion Humans and Human Resource Machine sit closest to Parallel, turning concurrency itself into a puzzle. SpaceChem, which charts the distribution of everyone's solutions, is the Community idea in commercial form. And The Witness holds the line of offering no hints at all. Play those three side by side and today's axes stop being abstract.

References

Papers and related material referenced in this article:

・Oops, Not Now: PEARL, a RAG-Based Support Agent for Gameplay and What Players Want from AI Help (Li, Maram, Kashani, Zaman, Lin, Marano, Azevedo, Zhu & Seif El-Nasr, 2026, arXiv:2609.13718)

・Author-accepted manuscript, accepted to the 2026 IEEE Conference on Games (CoG) (peer-reviewed). Eight pages, three figures, one table

・Related articles mentioned above: Wang et al.: turning a puzzle solver into a move-by-move tutor / McCaughey et al.: people change how many hints they buy only when told the price changed / Zhao et al.: people build their own library of parts while solving

・Games mentioned above: 7 Billion Humans / Human Resource Machine / SpaceChem / The Witness