Johnson et al.: What Changes in a Game When You Build an LLM Into It — Fukai Reads

TL;DR

Johnson and colleagues at the University of Calgary built two games in which a large language model (LLM: an AI trained on large amounts of text that can generate text and answer questions) was embedded not as a decorative add-on but as part of the game's structure, and they recorded and analyzed the experience of building them. The question is simple: when you build an LLM in, how do gameplay, playability, and player experience change?

In one sentence: the LLM produces 'different content every time' and 'personalized experiences,' but it also brings new burdens of correctness, difficulty calibration, and overall coherence. Because the generated content is wired directly into rule progression, a wrong output turns straight into unfairness or a broken mood. The developers kept the games barely playable by forcing the LLM's output into fixed formats and inserting validation steps.

Up front: this is a preprint (a freshly posted manuscript that has not yet passed expert peer review), and no player-facing testing has been done yet. Please read this as a study that systematically examines the builders' own reflections.

Introduction

The paper is arXiv preprint 2603.27896, posted on 29 March 2026, in cs.SE (software engineering). The authors — Keeryn Johnson, Muhammad Ahmed, Charlie Lang, Sahib Thethi, Wilson Zheng, and Ronnie de Souza Santos — are all at the University of Calgary. The manuscript is written in an ACM conference format, but I want to be clear that it is not a peer-reviewed paper accepted at a specific venue; at present it is a preprint (not yet peer-reviewed).

Here is why I chose it today. There is no shortage of demos showing 'what LLMs can do' in games. But calm, reflective records of 'what actually changes in development and in play when you build an LLM into the skeleton of a game' are surprisingly rare. This paper is not a flashy new technique; it carefully reads the reflections of students who actually finished building two games, and it sits very close to the reality a maker will hit tomorrow. That is why I judged it worth translating and introducing.

Background

Game research has treated three concepts as central: gameplay, playability, and player experience. Borrowing the paper's framing: gameplay is 'the set of activities the player and other entities can perform within the virtual world'; playability is 'the instantiation of usability in games — the quality with which those activities can be performed'; and player experience concerns 'the immersion, enjoyment, and responses that arise during play.'

Traditional software is measured by 'does it work correctly (functional correctness)' and 'is it usable,' but games aim mainly at fun and entertainment, so the paper notes that general software measures alone are insufficient. Into this space LLMs have recently arrived. They have been used for procedural content generation (PCG: making levels, items, and so on by program), dialogue systems, and 'mixed-initiative' development in which humans and AI proceed together — with reported benefits of variability and personalization and reported challenges of unpredictability, coherence, and evaluation.

But the gap the authors point to is this: how 'embedding an LLM during development' relates to the established constructs of gameplay, playability, and player experience has rarely been examined head-on. The catalog of applications grows, but how integration reshapes structure remains almost untouched. The paper's aim is to fill that gap from the makers' first-hand experience.

Approach / Method

The method is collaborative autoethnography. Rather than a researcher observing from outside, the participants themselves record and systematically analyze their own practice — a qualitative approach (autoethnography = writing your own experience as data and reading out common patterns). Here, student developers write about their experience of embedding an LLM in a game and interpret it through the lens of the three constructs.

The subjects are two games. Across an undergraduate course (three months) and a research project (six months), seven students developed them, and five are co-authors of this paper. Integrating an LLM was a mandatory condition; it was used for dialogue generation, narrative branching, NPC (non-player character) behavior, and PCG, using Google Gemini and OpenAI models. The data are five reflective narratives (about two pages each) plus design documents and code. Analysis combines thematic reading (reading the narratives to pick out recurring themes) and content analysis.

The first game, Wizdom Run, is a 'study-while-you-play' RPG built in Unity. When the player imports their study notes as a PDF, the LLM (OpenAI) reads them and generates multiple-choice questions across three difficulty levels. The questions are shaped into a fixed JSON format and stored in a PostgreSQL database. In normal combat, correct answers restore the mana needed to cast spells while wrong answers restrict actions; boss fights use a turn-based card system in which correct answers grant bonuses or abilities. The second game, Sena, teaches sustainability in software engineering; the LLM handles clarifying dialogue, scenario interpretation, and feedback, with a mode whose state changes according to the player's decisions. In both, the LLM is embedded as an architectural component, not an auxiliary tool.

For the record: no equations or statistical models appear in this paper. It is not a numerical experiment but a qualitative study whose primary data are the makers' own words. So below I, too, will introduce it through concepts and concrete examples rather than numbers.

Findings

Impact on gameplay. Generated content became not 'optional narrative decoration' but a part wired directly into rule execution and progression. One participant (P1) says question generation 'directly drives skill-point replenishment, boss encounters, and level progression — a core part of the gameplay.' At the same time, the content varies with the input each time: P1 writes 'each campaign becomes a vastly different experience even though the underlying code stays basic,' and P2 notes 'each playthrough differs in its questions and answers.' But this variability is bounded — P5 says 'if you reuse the same document, you generally get the same kinds of questions,' showing that generative diversity is constrained by design.

Impact on playability. In exchange for flexibility, 'containment' was needed to keep the system from breaking. P1 explains that 'formatting the LLM's responses exactly as expected let the backend design stay cohesive,' and P4 says that specifying the return format 'allowed me to create objects on the C# side of the game.' Difficulty calibration became a tension: P3 reports that 'some questions generated for medium or hard felt like the easy set,' disrupting the balance of progression. Correctness tied directly to fairness. P2 recalls that 'a simple math question appeared, but none of the answer options were correct,' and P5 says of wrong outputs that 'the player isn't actually learning; they're just trying to figure out how the AI thinks, which tarnishes the experience.' P3 further points out that 'the correct option would show up in the same multiple-choice slot more than once,' warning of an unintended exploit.

Impact on player experience. Personalization did increase. P2 says 'the range of topics players can test themselves on is endless,' and P4 values that in the dialogue feature 'the player could ask clarifying questions with added context.' But experiential quality depended on balancing variability with predictability. P5 says 'some variability adds variety, but a game needs some predictability to stay fair and playable.' When grounding was insufficient, the system would 'simply answer like any publicly available LLM and not even judge whether the question was on topic' — P4's line captures well the moment coherence and trust collapse.

I want to stress again that these are all developer reflections; numerical metrics such as accuracy rates or session length do not appear in the paper. So we cannot speak in numbers about 'how much' it worked. What the authors show is not the size of the effect but where and in what form the effect appears.

How Makers Can Use It

1. If you are building a Sokoban-like (a box-pushing puzzle) or any puzzle where a unique solution is essential, and you auto-generate levels with an LLM: treat 'correctness' and 'solvability' as gameplay-critical properties. What the paper repeatedly cites as the key to playability was schema enforcement — forcing the LLM's output into a fixed format and validating it. In the same spirit, always insert a solver (a program that solves and checks automatically) between the LLM's output and the player, and never ship the 'question with no correct answer' that P2 describes.

2. If you build a quiz or trivia hypercasual (a mass-produced, easy-to-play mobile game): do not take the LLM's self-assigned difficulty labels at face value. In this paper there was a mismatch where 'hard' questions 'felt like easy.' It is safer to calibrate difficulty by an external measure such as the actual solve rate, not the label attached at generation time. Not generate-then-trust-the-label, but generate, measure, then relabel.

3. If you build a narrative- or dialogue-centered game: ground the LLM (bind it 'inside your fictional world' with retrieval or constraints) to prevent the failure P4 reported, where it 'answers off-topic like a generic LLM.' Also fix the output to a schema such as JSON so downstream code (for example, generating C# objects) can parse it reliably. This is precisely the technique the paper says actually worked.

4. Common to every genre: inspect the generated content for patterns (for example, a bias where the correct answer always lands in the same slot) and close the exploits. But do not over-close them. As P5 says, for fairness and playability it is healthy to keep 'some predictability.' Variability and predictability are not something to max out in one direction but to balance on purpose.

Limitations

Start with the weaknesses the authors themselves acknowledge. This is a qualitative study that does not aim for statistical generalization, and its data are small — five reflections written in two undergraduate settings. Because it is an insider view of participants describing themselves, and relies on self-report, interpretive bias and selective, convenient recall are hard to avoid; the authors say they mitigated this by triangulating across multiple reflections, consulting design and code artifacts, and peer-reviewing among co-authors. The biggest caveat: this is the developers' perspective, not independent player testing. The authors list user studies as future work.

Now the points I (Fukai) noticed on reading. First, both games are learning/education games, a kind where 'correctness' is especially fatal. So I would be cautious about generalizing these lessons directly to pure entertainment puzzles that prize feel and surprise over correctness. Second, the models used were Gemini and OpenAI at one point in time; models change behavior with updates, so reading this as 'a property of LLMs in general' is risky. Third, it has almost no citations yet (natural for a freshly posted preprint), so it is not a widely debated finding. And again, with no numerical metrics, this paper cannot tell us the size of the effect in numbers.

Fukai's Reading

I write this flagged as my own interpretation. I would place this study in the current of 'treating generation as structure, not decoration.' In the vocabulary of design criticism, building an LLM into a game reads less like 'adding content' and more like redrawing the boundary between the deterministic rules the author wrote by hand and the parts that are generated probabilistically. What this paper shows again and again is that the real source of fun lives in the seemingly dull plumbing of prompt design, schemas, and validation. Behind the flashy generation, what the maker is truly designing is that boundary itself — how much to delegate to the machine and where hand-guaranteed correctness begins. That is how I took it.

Closing

For those who want a wider map: the big picture is charted by Gallotta et al. 2024, 'Large language models and games: A survey and roadmap' (IEEE Transactions on Games). For the breadth of applications, Yang et al. 2024, 'GPT for games: A scoping review,' and Sweetser 2024's scoping review are easy entry points. This paper finds its place on that map as a record of 'first-hand experience from actually building two games and reflecting.' Next, when the player-facing validation the authors promise appears, I want to follow it. The relationship between generative AI and games is shifting from a season of demos to a season of designing the unglamorous plumbing.

References

Papers and related materials referenced in this article:

・Large Language Models in Game Development: Implications for Gameplay, Playability, and Player Experience (Johnson, Ahmed, Lang, Thethi, Zheng, de Souza Santos, 2026, arXiv preprint 2603.27896, cs.SE)

・PDF of the paper

・DOI: 10.48550/arXiv.2603.27896 (arXiv-issued, pre-peer-review preprint)

・Sena (one of the games studied; public site)

・Related work: Gallotta et al., 'Large language models and games: A survey and roadmap' (IEEE Transactions on Games, 2024); Yang et al., 'GPT for games: A scoping review' (IEEE CoG, 2024)