Li et al.: Making Geometry Problem Solving Verifiable with a Solver as Referee — Fukai Reads

TL;DR

Today I read a study about making an AI solve geometry problems - the kind written in a mix of diagrams and text - in a way where we can actually be sure the answer is correct. It is a preprint (a manuscript posted before peer review) submitted to arXiv in June 2026, by Can Li, Ting Zhang, Junbo Zhao, and Hua Huang. Their method is called SD-GPS.

The core idea is to use a symbolic solver (a program that reasons step by step under fixed rules) as the referee throughout the whole pipeline, from translation to proof. The problem is translated not into equations but into a formal language the solver can execute; when the reasoning gets stuck, the AI proposes a small helper theorem (a lemma), like drawing an auxiliary line, and the solver itself checks whether that proposal is sound. According to the abstract, on two benchmarks, Geometry3K and PGPS9K, the method consistently outperformed existing approaches.

One caveat: this piece, as a paper digest for the site, is built from the publicly posted abstract and bibliographic information. I have not followed every detailed number in the body, so please consult the original paper for exact figures.

Introduction - why I picked this one today

Every morning I browse the arXiv new listings with a coffee in hand, and today a title in the cs.AI (artificial intelligence) feed stopped me: 'Verifiable Geometry Problem Solving.' The authors are Can Li and three colleagues, and it is cross-listed under cs.AI, cs.CL (natural language processing), and cs.CV (computer vision). I could not confirm the authors' institutions from the abstract, so I will not guess at them here.

This is not a peer-reviewed conference paper but an arXiv preprint (submitted June 2026, and possibly not yet through peer review). It was posted recently and its citation count is still to accumulate, so the right stance is to read it as a study that has not yet been widely discussed.

The reason I wanted to introduce it is that the technique speaks directly to what puzzle makers care about. 'The AI translates a problem into a solvable form, has a referee confirm whether it is solvable, and adds a step when it gets stuck' - this is about geometry, yet to my ear it is exactly the old-yet-new problem of automatically generating puzzles that are guaranteed to be solvable.

Background - what is hard about an AI that solves geometry

Geometry problems play to human intuition about diagrams, but they are awkward for computers. In recent years a framework called neuro-symbolic (combining a neural network's intuitive guessing with rigorous rule-based symbolic reasoning) has been widely used. Roughly, the AI hazards a guess at 'this is probably how it is solved,' and a symbolic solver checks 'is that move valid under the rules.'

This division of labour has two sticking points. In the abstract the authors name, first, that autoformalization (the step of mechanically translating a diagram-and-text problem into a formal language the solver can read) has been treated as a stand-alone task, decoupled from whether it fits the downstream solver. A translation is useless if the solver cannot actually execute it.

Second, because the list of rules the solver can use (the rule library) is fixed, reasoning often hits a deductive impasse (a state where the next step cannot be derived from the available rules). Where a human would think 'let me draw an auxiliary line here,' the machine stalls for lack of rules. As I read it, this paper tries to loosen both knots at once.

Approach - using the solver as a referee all the way through

The backbone of the authors' method, SD-GPS, is - to borrow the abstract's words - to use the symbolic solver as an execution oracle (something that does not hand over the answer but judges whether a move is executable and sound) consistently, in both the formalization stage and the reasoning stage. Put without any equations: both the quality of the translation and each step of the proof are ultimately graded by 'could the solver actually run it.'

The first component is Solver-Driven Autoformalization. Here the training for translating into a formal language folds together supervised adaptation and reinforcement learning (a framework that learns by trial and error, strengthening actions that lead to good outcomes) guided by solvability into a single module. According to the abstract, this is built on QwenVL3-2B, a relatively small vision-and-language model, and it makes executability the central learning signal.

The second component is Verified Theorem Proposing. When reasoning stalls, the AI looks at the current state of the proof and proposes a locally useful helper lemma (a small intermediate proposition - in geometry, something like an auxiliary line or an intermediate equality). But it does not just propose and move on: every proposal is put through symbolic verification, and only the sound ones pass. As I understand it, this keeps the system in a state where it can be resourceful without ever lying.

Findings - what the abstract reports

On results, the abstract states the following. On two geometry benchmarks, Geometry3K and PGPS9K (shared problem sets for comparing performance), SD-GPS 'consistently outperforms' existing MLLM (Multimodal Large Language Model, one that handles both images and text) methods, purely neural methods, and neuro-symbolic methods - and does so across several evaluation formats: standard completion, multiple choice, and cross-modal reference between diagram and text.

To be honest, the abstract I could access does not carry specific numbers (by how many percentage points it wins, and so on). So I will not put a figure on 'by how much' here. The places where the authors speak in numbers should be in the tables of the original paper, so please verify those at the source. Inventing numbers is, to my mind, the single worst thing one can do in this series.

The conclusion the authors advance is the claim that 'closing the loop between multimodal perception (diagram and text) and symbolic execution genuinely improves geometric reasoning.' In other words: not separating guessing from rigorous verification, but wiring them into one loop, is what paid off.

Use cases - how puzzle and game makers might apply this

From here on, these are not the paper's claims but application ideas I (Fukai) came up with as 'how a maker could use this.' First: automatic generation of puzzles guaranteed to be solvable. The paper's 'put the solver in the referee's seat' idea transfers straight to puzzle generation. When auto-generating, say, Sudoku, geometry puzzles, or number-placement puzzles, have the solver attempt each candidate the generator emits and keep only those with a unique solution. Not 'build then check,' but 'build by checking.'

Second: use the 'number of lemmas' to tune difficulty. This paper inserts a helper step (a lemma) each time it stalls. Seen from the maker's side, the number of auxiliary steps needed to reach the answer is itself a proxy for difficulty. If you are building geometry or proof puzzles, you can sort problems by the number of auxiliary steps the solver needed and automatically assemble easy-to-hard courses.

Third: hints that do not lie. The paper's mechanism of 'always run proposals through symbolic verification' is useful for hint features. If you are having an LLM write hints for a puzzle app, have it produce hint candidates and then verify with a solver whether each is genuinely a step toward the solution before showing it. A wall of verification can stop the 'plausible but false hint' that leads players into dead ends.

Fourth, a bit further afield: if you run PCG (Procedural Content Generation) for hyper-casual games (light games playable in short bursts), the very design philosophy of a two-stage 'generator plus verifier solver' is worth borrowing. Rather than a single flashy generator, pairing a decent generator with a strict referee tends to yield unbroken levels more reliably.

Limitations - what can and cannot be claimed

Let me split limitations into two layers. First, frankly, the limits of this piece itself. What I (Fukai) must point out is that what I could read was mainly the abstract; I have not followed the method's details or the numerical results. So I cannot, in this article, discuss the contents of the ablation study (an experiment that verifies which part of the design matters by removing elements one at a time) that would separate out how much each component helped. Please check the original paper.

Second, the limits of the subject itself. Even within the abstract, evaluation is confined to geometry problems (Geometry3K, PGPS9K). The abstract does not claim it works as-is outside geometry - say, on combinatorial or logic puzzles. The base model, too, is the relatively small QwenVL3-2B, which can be an advantage in lightness and a limit in ceiling. I want to withhold judgement here until I read the paper's discussion.

One more thing I (Fukai) want to flag: the footing of the word 'verifiable.' The soundness of this framework rides on the soundness of the referee - the symbolic solver. If the solver's rules or implementation have holes, the reassurance of 'verified' is diminished by exactly those holes. When applying this as a maker, how to guarantee the referee's reliability is, as I read it, a separate matter that needs its own thought.

Fukai's reading

I write this with the caveat that it is my interpretation. I want to place this study in the lineage of a design philosophy that 'wires generation and verification into a single loop.' In the vocabulary of design criticism, it is close to automating the flash of insight that is an auxiliary line. But what makes this paper interesting is that it centers not on producing more flashes of insight, but on a mechanism that keeps insight from lying - always routing proposals past the referee. The 'guarantee of solvability' I always treasure in puzzle-making and this paper's idea of 'making executability the learning signal' are, as I read them, two sides of the same coin.

Closing

Translate geometry into a 'solvable form,' add a step when stuck, and have a referee confirm that step. The technical detail is confined to geometry, but the philosophy running through it - 'close off generation with verification' - is contiguous with the work of everyone who builds puzzles and levels. If you want to go deeper, first check the tables (numbers) in the original paper, and then read it alongside the lineage of automated geometry proving and of PCG that guarantees solvability; the study's place on the map should come into view.

To repeat: this piece is an introduction based on the abstract, and it deals with a recently posted preprint. Please verify the numbers and the details of the method at the source.

References

Papers and related materials referenced in this article:

・Verifiable Geometry Problem Solving: Solver-Driven Autoformalization and Theorem Proposing (Can Li, Ting Zhang, Junbo Zhao, Hua Huang, 2026, arXiv preprint arXiv:2606.27926)

・This piece is an introduction based on the paper's abstract and bibliographic information. Consult the original paper (PDF/HTML) for numbers and method details.

・For related reading, it helps to pair this with the lineage of automated geometry proving and of solvability-guaranteed puzzle generation (constructive PCG).