Sestini et al.: Making AAA Game NPCs Feel Authentic with Reinforcement Learning — Fukai Reads

TL;DR

Today I am translating a vision paper from the research team at Electronic Arts (EA). It reports on whether the NPCs (non-player characters, the characters other than the player) in a game can be made 'more believable' with reinforcement learning (a framework in which an agent gradually learns, through trial and error, the actions that earn higher reward), tested inside actual AAA titles.

The authors put into words seven 'production requirements' that academic papers often overlook: short training time, limited compute, and control by designers. They then show the difficulty and the solution through two concrete cases — the goalkeeper in EA SPORTS FC 25 and the infantry soldier in Battlefield 6. The core of their conclusion is that RL is a tool to augment, not replace, existing game AI.

Introduction

The paper is 'Augmenting Game AI with Deep Reinforcement Learning.' The six authors — Alessandro Sestini, Joakim Bergdahl, Amir Baghi, Jean-Philippe Barrette-LaPierre, Florian Fuchs, and Linus Gisslén — are all at Electronic Arts (Stockholm, Sweden). It is a preprint posted to arXiv as 2606.20210 (a manuscript before peer review; it is written in an IEEE-style format as a conference-oriented vision paper, but at the time of writing I cannot confirm it has passed peer review, so I treat it as a preprint).

I chose it today because it is, unusually, a paper written by people from the trenches. It is not about competing for the top benchmark score; it faces head-on the problems any maker meets when trying to put RL into a shipping blockbuster game — training that never finishes, a model too heavy to run on the target hardware, behavior that is not what the designer wanted. For readers who make games, few subjects are this grounded.

To be fair: this is a report based on two titles from one company, not a proof of any general law. The authors themselves frame it as a 'vision paper' pointing to future research directions. I read it in that spirit, and translate only what they actually wrote.

Background

NPC behavior in games has long been built by hand-coding. The paper lists three representative methods. FSM (a finite state machine — a classic method that prepares states like 'idle' or 'attack' in advance and switches between them on conditions), the Behavior Tree (a design that arranges actions in a tree and picks which to run from the top down), and GOAP (Goal-Oriented Action Planning, which builds an action sequence by working backward from a goal). All are readable and cheap to run, but the authors note a shared weakness: as games grow in scale they become hard to design and maintain, and the behavior turns rigid.

RL, meanwhile, has shown it can play games at superhuman levels, as in AlphaStar for StarCraft II or GT Sophy for Gran Turismo. Yet its main production use so far has been automated game testing (the authors mention using RL agents to test releases of the Battlefield series). Building the very NPCs a player faces with RL is a stage the field has not broadly reached.

Why it matters. In the authors' words, the quality of NPCs shapes immersion, and crude AI breaks 'the illusion of realism.' But a superhuman opponent is often frustrating for the player. So what game AI needs is not strength but authenticity — and that gap between research and production is the heart of the background.

Approach / Method

The authors first lay out seven requirements that RL must satisfy to be usable in production. Short training time (the game changes daily, so retraining should fit in one night), controllability (designers can qualitatively shape the final behavior), modularity (it can be dropped into an existing AI system as a part), maintainability (it survives future updates), bug detection and fixing (problems can be found and fixed), authenticity (behavior that is 'believable' rather than superhuman), and runtime inference constraints (inference — feeding input to a trained model to get an action — must fit within a fixed time budget even on low-end hardware).

They test these requirements in two real titles. The first is the goalkeeper 'positioning' system in EA SPORTS FC 25. It was built as a huge FSM whose state switches looked unnatural and were hard to maintain. They replace it with a reward function (a numeric definition of what counts as good movement, which an expert such as a professional goalkeeper can design without writing code). Training uses SAC (Soft Actor-Critic, a reinforcement-learning algorithm known for learning from relatively few trials).

The second is infantry 'locomotion' (movement) in Battlefield 6. It relied on a behavior tree plus pathfinding and a navmesh (a map of walkable surfaces), and moved rigidly. Here they use PPO (Proximal Policy Optimization, an algorithm that trains stably). What stands out is how they show the surroundings to the agent. Many studies use raycasts (shooting rays out from the agent to measure obstacles), but that is compute-heavy. Instead the authors reuse height data already in the engine to build an occupancy map (a grid representation that color-codes which cells are passable or not).

The idea running through all of it is 'augment, not replace.' Rather than an end-to-end approach (one model from input to output doing everything), they swap only some leaves (the terminal nodes) of an existing FSM or behavior tree for an RL part. This way, they argue, you keep the designer's controllability and modularity while smoothing only the parts that were rigid.

Findings

On the goalkeeper side, training time was the first big wall. Plain SAC initially took two to four days per run, but by combining a high update-to-data ratio with network resets, the use of pre-collected offline data, and scenario-based training, the authors say they cut it to about 12 hours (Figure 2, left). That met their goal of a run that fits in one night.

Model lightness was also a requirement. The budget for one inference (from fetching observations, through the model's forward pass, to returning an action) was a tight 200 microseconds, and the authors fit within 170 microseconds using a small 5-layer network (256 hidden units, about 300,000 parameters). As a result the new positioning was perceived by playtesters as 'more believable and human-like,' and — 'as a side note' — showed about a 10% higher save ratio than the previous hand-coded version (as the authors state it). Fixing exploits discovered after release could also be handled with roughly two-to-four-hour targeted fine-tuning.

On the infantry side, the comparison of how to show the surroundings is key. Twenty-four raycasts and the occupancy map performed about the same, but the occupancy map was cheaper to compute — the authors report about a 2.0x speedup (raycasts about 27 microseconds, occupancy map about 14 microseconds). Training used PPO for about 2 hours, running up to 240 agents in parallel. Further, integrating the RL movement part into an existing behavior tree and pitting it 1-on-1 against a hand-coded opponent over 20 matches, the RL side won 11. Comparable performance but more natural movement — that is the authors' reading.

Where you can use this

From a maker's view, here are concrete uses. If you build the CPU for a sports or competitive-action game and its state switches look jittery — rather than rewriting a giant FSM wholesale, you can replace only the single most unnatural state (say, defensive positioning) with a reward function. A practical hint from this paper is that you can have pros or advanced players help design that reward.

If you must run large-scale multiplayer enemy AI across a wide range of hardware — instead of heavy raycasts or first-person vision for perception, it is worth reusing terrain data already in the engine as a grid map (an occupancy map). The authors show it can be made lighter to compute while keeping performance.

If you run a live service where the game changes daily and players discover exploits — rather than rebuilding everything, it is realistic to split behavior into small parts and fix only the problem part with a few hours of fine-tuning. But the authors warn that repeated fine-tuning raises the risk of catastrophic forgetting (losing previously learned behavior through additional training).

If you are a small studio with limited compute — it helps to give up chasing superhuman strength and aim at authenticity, running a small network that fits in one night with sample-efficient SAC. And overall, design RL as 'a part that shores up the weaknesses of existing AI,' not as a replacement — this is the most reproducible attitude to take away from the paper.

Limitations

Starting with the limitations the authors acknowledge. The occupancy map that represents surroundings as a grid is light but incomplete; they state plainly that it cannot capture multi-layered environments, vertical structures, irregular terrain, dynamic obstacles, or destruction. They also say behavioral correction via fine-tuning carries the risk of catastrophic forgetting and that a systematic method is not yet established. They argue that end-to-end RL for the whole system is impractical in production — though this is more the authors' stance than a limitation. In addition, the paper focuses on RL and leaves integration of other machine-learning methods, such as large language models, to future work.

What I, Fukai, point out here is that this is a vision paper, and its quantitative comparisons are limited. Even the goalkeeper's 'about 10% higher save ratio' is written by the authors themselves as 'a side note,' and the infantry evaluation rests on a small trial — 11 wins in 20 one-on-one matches. It is more honest to read it as a report of lessons gained from implementation than as a controlled experiment.

One more point. The statement that playtesters perceived the behavior as 'more believable' is appealing, but the text does not detail how many people or what procedure produced that judgment. 'Authenticity' is a subjective measure, and I read this as something to defer to the systematic evaluation the authors themselves list as future work (Behavior Evaluation). Moreover, the targets are two specific EA titles, so generalizing to other genres should be done with care.

Fukai's reading

From here I make clear this is Fukai's personal interpretation. I want to place this study in the shift from an era of 'making AI solve games' to one of 'making AI perform in games.' Strong AI that wins benchmarks is already in hand. But the real hard part for game production is not strength; it is the 'good manners' that fit the designer's intent, the constraints of real hardware, and the daily changes. In the vocabulary of design criticism, I read this paper not as superhuman optimization but as a philosophy of 'grafting' (augmentation) — respecting existing craft such as FSMs and behavior trees while making only parts of it smarter. It is not flashy, but I would frame it as valuable for putting into words a philosophy that actually works on the floor.

Closing

For those who want to go deeper. To grasp the whole map of game AI, the textbook 'AI methods for games' by Yannakakis and Togelius, which the authors cite, is a good starting point. To learn the intersection of large language models and games, the survey by Gallotta et al. (2024) serves as a map. And the technical detail of this paper's goalkeeper is deferred to the authors' prior work (Sestini et al., Reinforcement Learning Conference 2026), so anyone who wants to reach the implementation should read that alongside it.

My after-reading impression in one line. Papers that build strong AI are many, but a paper that seriously discusses 'AI that may be weak, yet is believable, fixable, and light' is rare. Next, I would like to read a sequel that tackles how to measure authenticity — the evaluation methodology the authors list as future work.

Sources

Papers and related material referenced in this article:

・Augmenting Game AI with Deep Reinforcement Learning (Sestini, Bergdahl, Baghi, Barrette-LaPierre, Fuchs, Gisslén, 2026, arXiv preprint 2606.20210)

・Related: DeepCrawl: Deep Reinforcement Learning for Turn-Based Strategy Games (Sestini, Kuhnle, Bagdanov, 2020) (the authors' early RL game-AI work)

・Related: Large Language Models and Games: A Survey and Roadmap (Gallotta et al., 2024, IEEE Transactions on Games) (a map of LLMs and games)

・Background textbook: AI methods for games (Yannakakis & Togelius, Springer, 2025) — the big picture of game AI