Zeng et al.: Automating Game Balancing with LLM-vs-LLM Self-Play — Fukai Reads

TL;DR

Today I am translating a paper called "RuleSmith." The authors frame it as the first framework that automates game balancing — the work of tuning a game so no faction wins or loses too often — by handing the job to multiple LLMs (Large Language Models, prediction models trained on huge amounts of text). Traditionally this work has been done by humans playtesting over and over and nudging numbers by hand and intuition. RuleSmith tries to replace that with a trio of a game engine, LLM-versus-LLM self-play, and Bayesian optimization (a method for efficiently searching for good settings with few trials).

The validation was done on CivMini, a small strategy game the authors built themselves. By tuning twelve parameters that govern two asymmetric factions, they report that the system could find configurations where the win-rate gap drops to nearly 0%, even starting from deliberately imbalanced initial states. But I want to flag two things up front: this is a proof of concept (a first step showing the idea works), and it is a pre-review arXiv manuscript.

Introduction

The paper is titled "RuleSmith: Multi-Agent LLMs for Automated Game Balancing." The eight authors are Ziyao Zeng, Chen Liu, Tianyu Liu, Hao Wang, Xiatao Sun, Fengyu Yang, Xiaofeng Liu, and Zhiwen Fan. It was posted to arXiv as 2602.06232 in February 2026 as a preprint (a pre-review manuscript); since I cannot confirm it has passed peer review as of this writing, I treat it as a preprint. The code and a project page are public (Adonis-galaxy/RuleSmith on GitHub), which is a welcome sign for reproducibility.

I chose it today because balancing is a perpetual headache for anyone who makes games. Asymmetric games in particular — where each faction differs in abilities, economy, and mobility — have the nasty property that tweaking one number breaks something elsewhere. RuleSmith shows, implementation and all, a shift in perspective: making the game itself (its rules) the object of optimization. It is a young paper with few citations and little discussion yet, but I judged that it offers a concrete skeleton a maker can take home.

Background

What was already known. AI-based playtesting is not new: simulated agents have been shown to assess balance in board and card games and to reduce reliance on expensive human playtests (de Mesentier Silva et al., 2017). Other work uses deep reinforcement learning (a framework for learning, by trial and error, actions that raise a reward) or scripted agents to surface balance holes and edge cases in commercial games (Zhao et al., 2020; Bergdahl et al., 2020). There have also been attempts to tune continuous parameters with Bayesian optimization (Celemin, 2024).

Meanwhile, self-play reinforcement learning produced superhuman play in AlphaGo, AlphaZero, OpenAI Five, and AlphaStar — but those are about learning a strong policy under a fixed ruleset. PCG (Procedural Content Generation, the automatic creation of content) has grown as a toolset for making levels, maps, and mechanics.

So what was not known? According to the authors, using LLMs as agents that evaluate and optimize the rules themselves is almost entirely unexplored. LLMs can read a natural-language rulebook, follow role descriptions, reason over a game state in words, and make coherent decisions. That opens the possibility of using them as "zero-shot simulators" (stand-in players that run as-is, without extra training), with no hard-coded heuristics and no expensive RL training. This matters because modern games carry combinatorially exploding action spaces and many parameters, making hand tuning increasingly impractical in both time and cost.

Approach

RuleSmith has three parts. The first is the stage: a game called CivMini, a miniaturized Civilization-style 4X game (a strategy genre built on explore, expand, exploit, exterminate) played by two factions on a 7x7 grid. The factions are asymmetric. The Empire is specialized: its Farmers only gather resources and cannot fight, and its Soldiers only fight and cannot gather. The Nomads, by contrast, field a single versatile Cavalry unit that moves faster (two cells per turn versus the Empire's one), but cannot gather passively and gain resources only by killing enemies — a design that forces aggressive play.

There are six action types: gather, move, battle, produce-resource, produce-unit, and pass. Cities are attackable entities with hit points; destroying the enemy city is an instant win, and if the turn limit is reached the game is decided by score (a weighted sum of remaining resources, battles won, and surviving units). The second part is using this LLM self-play as the evaluator. Two LLM agents take the Empire and Nomads roles, read the natural-language rulebook and a description of the board, and output actions for all units at once as JSON (a structured data format). It runs zero-shot, with no extra training. Two tricks improve reliability: a lightweight RAG (Retrieval-Augmented Generation, a scheme that retrieves relevant information and feeds it in) that uses TF-IDF (a classic method for matching documents by word importance) to pull out only the rules relevant to the current situation, and constraining the output to JSON to reduce illegal moves, falling back to pass when parsing fails.

The third part is searching the rule space with Bayesian optimization. There are twelve parameters to tune (three economy, four combat, two production, three scoring). Brute-forcing them as discrete values would mean tens to hundreds of millions of combinations, so the method treats them as continuous, and uses a Gaussian process (a probabilistic model that estimates the value and uncertainty of unobserved points from observations) to predict how balanced a given setting is likely to be. Put without equations, the goal is to minimize an "imbalance size": how far each faction's win rate sits from 50%, plus a penalty for too many draws. Continuous proposals are rounded to integers or to steps of 0.1 to map back to valid rules.

The crux is adaptive sampling driven by an acquisition function (a measure of how worthwhile it is to try a given point next). Promising candidates get more games (up to 64) so their win rates are measured accurately, while low-prospect exploratory candidates get fewer (as few as 16). In other words, limited compute is concentrated where it matters. For the record, the evaluator was InternVL3.5, a language model that can also handle images, in its 2B and 8B variants; running 100 iterations of 8B versus 8B took about 40 hours on eight A100 GPUs.

Findings

The main result: in the introduction the authors state that RuleSmith could converge, even from deliberately imbalanced initial settings, to configurations where the win-rate gap drops to nearly 0%. Here "balanced" means win rates land within 50% +/- 5% (i.e. 45-55%), and they verify this by running 100 games using the best checkpoint whose imbalance metric fell to 0.1 or below.

What I found interesting is the caveat attached to that balance. According to the paper's Table 2, when a model of the same size trains and evaluates, the table's diagonal is balanced (for example 48-52 for 2B vs 2B, 51-49 for 8B vs 8B). But swap in a larger model on just one side and the now-smarter side wins more. In an extreme case, a "balanced setting" found with Empire-8B and Nomads-2B, re-evaluated with Empire-2B and Nomads-8B, collapsed to 06-94 (all figures from Table 2). In other words, the "balance" here can be read as presupposing that both players (here, the LLMs) are of matched strength.

On the qualitative side, the replay in Figure 3 is illustrative. The Nomads exploit cavalry mobility to skirt the Empire's central formation; losing two units along the way, one breaks through, and at turn 12 destroys the Empire city to win. Turning a costly, mobility-driven surprise into a strategic victory — an asymmetric-game maneuver the LLM actually pulls off. The paper also includes an ablation (an experiment that removes design elements one at a time to see which are doing the work) comparing optimization methods, but the body text I could retrieve cut off just before that numeric table, so I will not quote its specific numbers here.

Use Cases

Let me be concrete about how a game maker could use this. If you are designing an asymmetric board game (one where factions win in different ways, like Root or Vast): parameterize each faction's hit points, costs, and scoring weights, and use LLM self-play as a first-pass screening to push the setting roughly toward fairness before you bring it to a human playtest session. The value is catching obviously broken configurations early.

If you run a hypercasual or mobile title: auto-search the economy and combat numbers, tune toward a target win rate or draw rate, and only then send candidates to A/B testing (comparing two versions on real users). You can narrow down to promising candidates before trying everything by hand. If you are building a roguelike or a competitive game: use LLM matches to surface the "clearly too strong / too weak" cases when adding a new card or unit before release, so your human playtesting budget goes to the genuinely subtle final tuning.

The authors also point beyond games. They argue it can stress-test other rule-driven competitive systems: auction rules, supply-chain conflicts, medical triage (prioritizing who gets treated) protocols. And what should please designers is that the Bayesian optimization surfaces, in an interpretable form, which parameters — HP scaling, resource efficiency, production tempo — drive fairness. It does not just hand you a "balanced setting"; it leaves clues about why that spot works.

Limitations

Let me split the limitations into what the authors acknowledge and what struck me on reading. In the authors' own words, the body text states plainly that "LLM-driven gameplay evaluation is computationally expensive and noisy" — which is itself the motivation for the adaptive sampling. The study is also a proof of concept on the authors' own small game, CivMini, and the paper closes with a Limitations appendix (Appendix C). Honestly, the body text I could retrieve cut off before that appendix, so I have not read its contents. Summarizing what I have not read runs against my principles, so I will only report that the authors do document limitations separately.

What I, Fukai, want to point out is what the word "balance" contains. What this study showed is a setting where win rates even out between InternVL3.5 agents of the same size — not a setting that is fair and fun for humans to play. Balance depends on how players play, and Table 2 showed that merely changing a model's capacity breaks the balance. Since humans play differently than LLMs, a setting balanced for an LLM is not guaranteed to be balanced for humans — a place I think deserves careful reading.

One more thing I noticed is the narrowness of the metric. The balance measure is only win rate and draws; it does not measure the quality of experience — tension, the learning curve, fun. Twelve parameters, one game, and a single model family (InternVL3.5) also make the validation scope still narrow. The compute weight — about 40 hours for 100 iterations of 8B vs 8B — leaves a hurdle to bringing it straight into a studio's iteration loop. None of this negates the paper's value; they are conditions to read alongside its proof-of-concept stage.

Fukai's Reading

From here on, let me mark this as my (Fukai's) interpretation. I want to place this study in the long arc of "automating playtesting." In the vocabulary of design criticism, it is an attempt to move part of the playtester's role from human intuition to LLM self-play. Where conventional RL-based auto-playtesting learns a strong policy under one fixed ruleset, RuleSmith fixes the players and moves the rules instead. It flips the object of optimization from the players to the rulebook — and in that one move I feel the paper's greatest novelty. Yet it is a conditional novelty: that flip only works insofar as the LLM is a sufficiently valid stand-in for a human player.

Closing

For those who want to go deeper. Reading the classics of self-play reinforcement learning (the AlphaZero line), the early work on AI-based playtesting (de Mesentier Silva et al., 2017; Bergdahl et al., 2020), and Celemin (2024) on tuning games with Bayesian optimization together will give you a map of where this paper stands. Since RuleSmith publishes its code and a project page, actually running CivMini is probably the fastest route to understanding. As always, I read this today with a hot, strong drip coffee in hand, drawing colored-pen lines on a printed PDF.

References

Papers and related material referenced in this article:

・RuleSmith: Multi-Agent LLMs for Automated Game Balancing (Zeng, Liu, Liu, Wang, Sun, Yang, Liu, Fan, 2026, arXiv preprint 2602.06232)

・Code: GitHub Adonis-galaxy/RuleSmith (project page: https://adonis-galaxy.github.io/RuleSmith-website/)

・Related: A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play / AlphaZero (Silver et al., 2018)

・Related: Augmenting Automated Game Testing with Deep Reinforcement Learning (Bergdahl et al., 2020)