Gould & Ward et al.: Measuring Puzzle Difficulty in Units of Human Solve Time — Fukai Reads

TL;DR

Putting a number on "how hard is this puzzle" is harder than it sounds. Do you count the deductive steps, measure the success rate, or just award three stars? The answer this paper adopts is a plain one: the time a human needs to solve it. A team of more than twenty researchers led by Dewi Gould and Francis Rhys Ward (Redwood Research, the Astra Fellows Program and others) attached a human solve time to every question in a suite of 43 benchmarks and over 30,000 problems, and lined up AI model performance along that time axis. The attached times span from 0.5 seconds to over 15 hours.

The paper's actual purpose is not puzzles but AI safety. What the authors measure is how much reasoning a model can do without writing that reasoning out in words. Their result: the human solve time of tasks a model can complete at a 50% success rate without externalising its reasoning has doubled roughly every 373 days over six years (95% confidence interval 167 to 691 days), reaching about three minutes for the newest model, GPT-5.5. But I chose this paper today less for that conclusion than for the apparatus they built along the way — a way of measuring difficulty in units of human time that puzzle makers can borrow almost unchanged.

Introduction

The paper is "Think Fast: Estimating No-CoT Task-Completion Time Horizons of Frontier AI Models." Dewi Gould and Francis Rhys Ward are equal first authors, followed by Anders Cairns Woodruff, Rauno Arike and some twenty more. Affiliations span Redwood Research, the Astra Fellows Program, Aether Research, MATS Research, Polytechnic University of Catalonia, Imperial College London, the University of Cambridge, the University of Chicago, Durham University, MIT, the University of Oxford, the University of Glasgow and Constellation. It is arXiv:2606.07157v1, filed under cs.AI, submitted on 5 June 2026. The paper labels itself "Preprint" on its first page, so it has not been through peer review, and being this new it has not yet been widely discussed either. Both points are worth holding in mind while reading.

Let me be honest about why I picked this paper today. I normally scan the cs.HC and cs.AI new listings each morning for anything puzzle-adjacent, and today that shelf was thin. What caught my eye instead was the benchmark table: New York Times crosswords, Sudoku, Lichess chess puzzles, KenKen, Puzzle Baron logic puzzles, and the Tower of London task (a classic psychological test of planning in which you rearrange discs into a target configuration). This is research that cannot function unless somebody takes the quantification of puzzle difficulty entirely seriously, and that portion alone repays reading. One caveat: below I write only what I could confirm in the body text. The per-benchmark details in Appendix D were not in the text I was able to retrieve, so I do not venture into them.

Background

Why time? Benchmark accuracy is convenient for measuring AI capability but fragile as a comparison axis. You cannot line up benchmarks of differing difficulty against each other, and "scored 85%" tells you nothing about what that means relative to a person. Kwa et al. (METR) therefore proposed the 50% task-completion time horizon (TH): how long a human takes on tasks that a given model completes with 50% probability, expressed as a single number. Because the unit is minutes, the number means something to a non-specialist.

This idea has in fact been used tacitly in the puzzle world for a long time. The "easy / medium / hard" labels on a Sudoku app are, in the end, names attached to distributions of solve time. Crossword enthusiasts record their times to measure themselves and compare with others. Human solve time is arguably the oldest and plainest unit of difficulty in puzzle culture. This paper carries it over to the evaluation of AI.

What was not settled is how rigorously that plain unit can be handled. Human solve time varies enormously per problem. It changes with who is solving. And for many problems no human record exists at all. Earlier TH work handled this by picking a single representative value. The methodological contribution here is that the authors refuse to erase the uncertainty, and instead carry it as measurement error all the way through to the final conclusions — something they explicitly describe as one of their core contributions.

Approach

First the shape of the work. 43 benchmarks, over 30,000 questions, run inside Inspect, the evaluation framework from the UK AI Security Institute. The benchmarks fall into three classes: short-answer tasks where the model emits only an answer, generative tasks where it produces code or text, and multi-turn agentic tasks where it emits tool calls. The main analysis restricts to the 32 short-answer benchmarks. Fourteen models are evaluated, from GPT-2 (2019) to GPT-5.5 (2026), with eight attempts per problem at temperature 0.7.

A word on what no-CoT means: chain-of-thought (CoT, the technique of having a model write its reasoning out in words) is blocked. Prompts and few-shot examples push the model to give an answer without reasoning, and where more than 5% of outputs still leak intermediate reasoning, the output format is structurally constrained. What is being measured, then, is how much a model can reason purely internally — latently, in the paper's word. The safety motivation is direct: if a model can carry out complex reasoning without writing it down, then methods that monitor a model by reading its stated reasoning stop working.

Now, how were human solve times assigned? This is the heart of the matter for us. Twenty of the 43 benchmarks come with existing human solve-time data. For twelve more, the authors recruited 17 expert participants and collected 733 problem attempts. They then showed Claude Opus 4.7 up to 50 in-domain (problem, time) pairs and had it estimate solve times for the remaining problems. The accuracy of those estimates was checked on the 20 benchmarks with real data: the best prompting regime achieved a MALR (mean absolute log ratio — the average absolute log of the ratio between real and estimated time, which lets tasks on wildly different timescales be scored on one footing) of 0.55, which the paper restates as estimates being off by a factor of about 1.7 on average. The remaining eleven benchmarks rely on expert-guided heuristics plus model estimates.

The way the error is carried forward is equally concrete. The relationship between estimated and real time is fitted as a single Gaussian in log space (1,075 paired estimates). A hierarchical bootstrap — resampling the data at several levels with replacement to see how much the estimate wobbles — is then run 10,000 times, resampling four things: benchmarks, problems within each benchmark, model runs per problem, and the human solve times themselves. That last item, treating the human-time estimate as itself uncertain, is what differs from prior work. Without writing any formulas: success rate is fitted as a smooth S-shaped curve against the logarithm of solve time, and the point where that curve crosses 50% is called the time horizon.

The second difficulty axis is also worth noting. As a yardstick independent of human timing, the authors introduce a "reasoning token horizon": the minimum number of thinking tokens a reference reasoning model needs to solve a problem. Twelve candidate models were tried and o3-mini was selected as the anchor. Each problem is attempted eight times and the smallest token count among correct attempts is taken — the minimum, because the paper wants a lower bound on the reasoning effort sufficient for the task. Unlike human time, uncertainty on this axis is not modelled.

Findings

The headline result: the 50% time horizon without externalised reasoning has doubled every 373 days over six years (95% CI 167 to 691 days, R² = 0.84). Individual values: GPT-4 about 0.9 minutes, Sonnet 3.7 about 1.1 minutes, Opus 4.7 about 2.8 minutes, GPT-5.5 about 3.0 minutes (paper's Figure 5). The reasoning token horizon doubles every 437 days (CI 341 to 571, R² = 0.92), with GPT-4 at 290 tokens, Sonnet 3.7 at 584, and Opus 4.7 and GPT-5.5 both around 1,500. The two axes are tightly but imperfectly correlated in log space (slope about 0.85, R² about 0.95).

The comparison against reasoning-out-loud is reported too. Kwa et al.'s with-CoT time horizon doubles every 182 days, so the no-CoT trend is roughly twice as slow; the paper says this divergence begins with the release of GPT-4. Extrapolating the median slope gives about 7 minutes by 2028 (range 4 to 30) and about 28 minutes by 2030 (range 9 to 615). As the width of those ranges shows, the authors themselves heavily qualify the extrapolation.

The number that matters most to puzzle makers is actually in the appendix. The authors measure, per benchmark, how much the odds of failure increase when a task's solve time doubles (Figure 13). For most (model, benchmark) pairs the odds ratio sits around 2 — doubling the length roughly halves the chance of success. But Sudoku and the N-hop lookup task stand out with markedly higher values, sitting at the top of the ordering, while chess puzzles and crosswords sit lower down. The median slope is 1.03 on the time axis versus 0.64 on the token axis, on the basis of which the paper states that human solve time is a stronger per-question discriminator of difficulty than reasoning tokens. In other words, "twice as long" means a very different cliff depending on the genre.

The robustness work is careful. Refitting the error model as two separate regimes gives a doubling time of 368 days (169 to 720), essentially unchanged. Adding generative tasks gives 397 days; adding multi-turn tool-use tasks as well gives 319. By domain, maths and science come out at 246 days, language and reasoning at 387, control and monitoring at 247, and software engineering and cyber at 324 — variation the authors call suggestive but noisy. In a separate ablation over 35 open-weight models (1B to 1T), doubling the time horizon required a 4.2x increase in total parameters, 2.1x in active parameters, 1.3x in layer count, or 3.1x in pretraining compute. Layer count gives essentially the same slope for dense models and for Mixture-of-Experts models (an architecture that activates only some of its parts per question), whereas on total parameters the two diverge sharply: 2.2x for dense, 8.1x for MoE.

One aside I enjoyed. Inserting meaningless filler tokens such as "...." barely changed no-CoT accuracy. But simply repeating the question once in the input improved some benchmarks (N-hop lookup, Sally-Anne, numeric scheming) by 10 to 20 percentage points. Giving a model room to think and showing it the same input twice are apparently not the same thing.

Where you can use this

One: replace the unit of your difficulty label with median solve time. If you are making a daily puzzle, defining difficulty by the median solve time of real players rather than by an internal solver's step count communicates far better to readers. Then, copying the paper's construction, fit an S-shaped curve of completion rate against the logarithm of solve time, and you get a single number: the longest problem that 50% of your readers can finish. Instead of naming difficulty tiers by instinct, you can manage them with that number.

Two: measure the steepness of the difficulty curve in your own genre. The paper's "how much do the odds change per doubling of solve time" analysis works directly as a personality test for a puzzle genre. With a Sudoku-like cliff, players fall off abruptly unless you slice the levels finely. With a crossword-like gentle slope, a coarse slicing still yields a continuous experience. If you are building something Sokoban-like, measuring this slope on your own logs first, and matching your level granularity to it, seems the right order of operations. It also explains why the same "five difficulty tiers" is too coarse in one genre and too fine in another.

Three: show a language model a handful of real logged times, have it estimate times for new puzzles, and use those estimates on the explicit assumption that they are off by about 1.7x. Having built a hundred new puzzles with no play data yet — the cold-start situation — is common. The paper's procedure is simply to pick around fifty (problem, measured time) pairs from your own logs and pass them as few-shot examples. But since what comes out carries a factor-1.7 error, I would use it for triage — filtering out the obviously too-hard, drafting a provisional level order — rather than for the difficulty label you actually publish.

Four: revisit the assumptions behind your leaderboard. A model that writes no reasoning at all solves, with 50% probability, tasks that take a human about three minutes. That length covers a great many hyper-casual daily puzzles. If you run a ranking premised on players not using AI, the fact that this horizon has been roughly doubling each year is not something to ignore. The other design response is to provide an explicitly AI-assisted division from the start.

Five: use reasoning token count as a second difficulty axis during cold start. Even with zero human data, having a reasoning model attempt each puzzle eight times and recording the smallest token count among correct attempts gives you a usable proxy for difficulty. The paper's finding that human time is the stronger discriminator should temper how much you trust it — but with no human data at all, it is far better than nothing, and plenty good enough as a provisional ordering for generated levels.

Limitations

First the weaknesses the authors acknowledge. One: the no-CoT measurement depends on how the model is coaxed into not writing its reasoning. Demanding short or single-token answers is out-of-distribution for recent reasoning models, and the authors write that this may systematically under-estimate latent reasoning capability and the doubling rate. Two: the measurement does not cleanly separate latent reasoning from memorisation or heuristic pattern-matching. The horizons should therefore not be read as a direct measurement of internal deliberative depth — the paper says this explicitly. They quantify the difficulty of tasks solvable without externalising reasoning, and no more.

What I would add is that "human solve time" is not one thing. The New York Times crossword times come from a population of enthusiasts who log their results on sites like xwstats. The Lichess chess puzzles come from rated players. The Sudoku data is Wang (2024, IEEE Access), a dataset with difficulty metrics as experienced by human players. Each is respectable data, but the human distributions behind them are entirely different. For the paper's purpose — a relative comparison of AI against humans — that mixture is tolerable. For a puzzle maker borrowing the same numbers, it needs care. Your median reader is not a crossword enthusiast who times themselves daily.

Two more points of mine. First, the 50% time horizon is a single scalar, and as the appendix slope analysis shows, it compresses away a lot of per-genre structure. The urge to put difficulty on one line is understandable, but inside the number "three minutes" a Sudoku-style cliff and a crossword-style slope are living together. Second, this research measures solve time, not enjoyment. Time is a decent proxy for difficulty and a poor proxy for quality of experience; nowhere does the paper claim a longer puzzle is a better one. On top of that, the datasets are not publicly released, in order to keep them out of future training corpora (available on request from the authors), which raises the bar for independent replication. Taken together with its preprint status and its still-negligible citation count, the right way to read these numbers is as current estimates.

How Fukai reads it

This section is my own reading. I would place this work not among AI safety papers but in the lineage of papers that bring a new discipline to the old problem of quantifying difficulty. In the vocabulary of design criticism, it is less the automation of difficulty labelling than the making-honest of the act of measuring difficulty. What the authors did was to keep the plain unit of human solve time, give it a structure of error, and carry that error through to the final conclusion. Not "this problem takes about four minutes" but "about four minutes, off by a factor of roughly 1.7 on average." I know of few places in puzzle production where this is done. Difficulty is displayed as a number of stars, and how much those stars can be trusted is not displayed at all. The most exportable part of this paper, I read, is neither the three minutes nor the 373 days, but the habit of always writing an interval next to them.

Closing

For those who want to go deeper: the time-horizon framework itself originates with Kwa et al., "Measuring AI Ability to Complete Long Software Tasks" (arXiv:2503.14499); reading that first makes clear what this paper adds. How time horizons vary across domains is summarised briefly in a METR blog post. And Inspect, the framework used to actually run the evaluations, is published openly by the UK AI Security Institute.

Some puzzle-side material as well. The dataset measuring Sudoku difficulty as experienced by human players is Wang (2024, IEEE Access) — the very source this paper draws on. If you are curious about the distribution of crossword solve times, xwstats is a starting point; for chess puzzles, the public Lichess database. Browsing those three alone shows how deeply the idea of measuring difficulty in units of time is rooted in puzzle culture. That a paper from the AI side has now reached down to that root strikes me as pleasantly back-to-front.

Sources

Papers and resources referenced in this article:

・Think Fast: Estimating No-CoT Task-Completion Time Horizons of Frontier AI Models (Dewi Gould, Francis Rhys Ward et al., 2026, arXiv preprint arXiv:2606.07157)

・Related work: Measuring AI Ability to Complete Long Software Tasks (Kwa et al., 2026) — origin of the time-horizon framework

・Related work: A Dataset of Sudoku Puzzles With Difficulty Metrics Experienced by Human Players (Sheng-Wei Wang, 2024, IEEE Access 12:104254-104262)

・How does time horizon vary across domains? (METR, 2025)

・Inspect AI (UK AI Security Institute) — the framework used to run the evaluations

・Data sources: XW Stats (NYT crossword solve-time statistics) / Lichess puzzle database