Test-Time Compute Scaling

The core observation

As models become more capable, benchmark performance is increasingly a function of test-time, or inference, compute rather than of the underlying model alone, which means a single scalar score is misleading and "we likely don't know what the capability ceiling is for modern LLMs because it's too expensive to measure," in AI researcher Noam Brown's framing.¹ A newer model's launch benchmark grid can look only marginally better than its predecessor, while real-world use within hours reveals a much larger step change, and the reconciliation is that the two models were never evaluated at the same token or dollar budget. Plotted with tokens on the horizontal axis instead of collapsed into one number, the newer model's lead becomes substantially clearer, a distinction a single per-model score simply hides.

Why the obvious fix fails

The obvious fix, pushing test-time compute until performance plateaus, does not work in practice, because the plateau sits very far out and sometimes fails to appear at all within any practical budget. Autonomous research agents keep improving after hundreds of experiments rather than saturating, and cybersecurity evaluations show frontier models still improving rapidly past a hundred million tokens of inference. The pattern that emerges is that stronger models improve faster over longer horizons, pushing the plateau further out as capability increases and possibly removing it altogether, an inference-time twin of ordinary training-time scaling laws: capability now scales with compute applied per task at run time, not only with compute spent once during training.

The proposed fix

The proposed remedy is to evaluate with a performance-versus-test-time-compute curve, tokens, dollars, or wall-clock time on the horizontal axis, or to set an explicit compute budget and disclose it, the way a standardized test bounds how long a human test-taker has. Every choice of horizontal axis carries tradeoffs: token counts are not comparable across different tokenizers, dollar costs depend on batching and hardware, and wall-clock time can be gamed by running many attempts in parallel. Any curve is still a meaningful improvement over a single number.

Implications for safety evaluation

This has a direct consequence for how AI systems get evaluated for risk before release. Most release evaluations largely ignore inference budget, yet a scaffold using a hundred times the inference of a carded model can cross a capability threshold the original model never demonstrated. A separate controversy over a high-scoring model released without an accompanying model card illustrated the deeper issue: the high score likely reflected a reproducible scaffold built on top of already-carded models, obtainable by anyone willing to pay for enough inference, and the real gap was that the underlying system cards never reported performance as a function of test-time compute in the first place. A further complication is that an agent's real operating horizon can exceed a model's entire development cycle, so confidently evaluating misalignment risk at a one-year horizon may require actually running the agent for a year, a bind current release cycles are not built to accommodate. The general recommendation is that labs publish performance-versus-compute curves, that benchmarks report the inference budget behind every score, and that safety frameworks explicitly account for inference compute rather than treating a model as a single fixed point of capability.

The reasoning-model connection

This dynamic connects directly to Alexandr Wang's account of the shift from pretraining-driven gains toward reasoning and reinforcement learning as the frontier's primary lever, since reasoning models are exactly the models whose performance is most sensitive to how much they are allowed to think at inference time. The train-time power law scaling laws describe now has an inference-time counterpart: capability can be bought at run time, per task, not only once during pretraining.

Open questions

If capability scales with inference and the ceiling is effectively unmeasured, it is unclear how to set a meaningful safety threshold at all, since any stated threshold is relative to a compute budget that is often left unstated. And if a well-resourced actor can apply an unusually large amount of inference to a single high-stakes task, public model cards may systematically understate the real frontier risk a model poses in the hands of whoever can afford to run it the longest.