Evals & Benchmarks

Two uses of eval data

Evaluation benchmarks are the unit tests of large language model development: without them there is no way to know if a model improved or degraded. The same rubric-and-unit-test infrastructure serves two distinct purposes. As a benchmark, it measures model-versus-model or model-versus-time performance, the numbers labs publish at launch and enterprises use to make buying decisions. As a reward function, the same infrastructure feeds reinforcement learning: a model attempts a task dozens or hundreds of times, scores accumulate, and the model hill-climbs toward the rubric. This dual use is why building evaluations across every domain is treated as a literal path to deploying AI throughout the economy, since the same infrastructure that measures a model also trains it.¹

Why hard evals matter

Alexandr Wang argues the AI industry suffers from a persistent shortage of genuinely hard evaluations that reveal the frontier of model capability.² Once a benchmark becomes popular it becomes the north star researchers optimize toward, so a good, hard evaluation drives research in productive directions while a bad or easy one misdirects it. Building a canonical benchmark, in his framing, is deeply gratifying precisely because it shapes what an entire field aims for.

Evals as enterprise IP

Wang's further argument is that evaluations are core intellectual property in an AI-driven economy. When a frontier lab asked a set of startups to hand over their evaluations for training, the answer was no, on the reasoning that an evaluation set is a company's moat in the same way its codebase or database is. Just as it took years for the software industry to establish norms against giving away code or data, Wang expects the AI economy to learn over time that evaluations, data sets, and environments are the next generation of AI intellectual property and should be protected accordingly.

The benchmark saturation cycle

Every benchmark follows roughly the same arc: it launches with genuine difficulty and low scores, the research community races to optimize against it, scores rise faster than expected, the benchmark saturates as models approach its ceiling and further progress on it becomes uninformative, and a new, harder benchmark becomes necessary. Humanity's Last Exam illustrates the pattern well: scores moved from roughly seven or eight percent to over twenty percent within months of its 2025 launch, and labs are now requesting thinking budgets of up to a day per problem. Scale AI built the exam in partnership with the Center for AI Safety around problems that are novel, contributed directly from researchers' own recent work so they cannot simply be searched online, require genuine field expertise, and allow long reasoning time. Wang notes the naming irony: there will unfortunately be yet another exam beyond this one, so the name was aspirational rather than literally true.

The academic-to-economic transition

The first generation of frontier benchmarks was academic: Olympiad-level mathematics, PhD-level science questions, and competitive coding. These are well-defined, verifiable, and press-friendly, but structurally disconnected from outcomes that consumers and enterprises actually care about, in Brendan Foody's framing. The second generation is economically valuable: measuring what professionals actually do in their jobs, across investment banking, law, medicine, software engineering, and consulting. This distinction maps onto a broader real-to-sim gap in AI evaluation: academic benchmarks often measure performance that does not transfer to professional workflows, so a model that exceeds human performance on an Olympiad may still fail to draft a usable email.

APEX-SWE: the real-world generation

Mercor's benchmark suite targets economically valuable software-engineering work directly, covering integration, meaning end-to-end construction across heterogeneous services, and observability, meaning debugging with production-style telemetry, using roughly two hundred cases graded by human-authored rubrics combined with unit tests.³ Even the best models sit near or below fifty percent on these tasks, GPT-5.5 scoring 52.7 percent on Integration and Opus 4.8 scoring 43.3 percent on Observability.³ That ceiling is a useful check against launch-week claims that a new model is a one-shot wonder at real engineering work.⁴

Evals as the deployment bottleneck

Foody reframes evaluations from a measurement instrument into the production input for economy-wide AI deployment: reinforcement learning is effective enough that a model can be trained to solve almost any evaluation that exists, so the actual barrier to applying AI throughout the economy is simply creating evaluations for everything, which inherently requires expert humans in the loop.⁵ Where Wang treats a benchmark as a yardstick for the frontier, Foody treats per-task evaluations as assembly-line tooling for automation: to automate a consultant's job, someone has to author evaluations for everything a consultant does. Both converge on the same conclusion, that evaluations and environments are simultaneously the new intellectual property and the new bottleneck, and that producing them is fundamentally a human-data problem.

The real-to-sim gap and eval as PRD

Foody's second major contribution names the real-to-sim gap directly: bridging the divide between the tasks a model is trained and measured against and the actual distribution of capabilities people care about.⁶ The failure mode is optimizing for academic benchmarks that are crisp and gradeable but disconnected from what enterprises need, building a financial model, researching an investment, completing a multi-tool workflow. One study he cites found failure rates above ninety five percent on practical enterprise tasks despite models holding academic gold medals.

His companion formulation is that if a model is the product, the evaluation is the product requirements document: many organizations have been vibe-spending on AI without ever writing down what they wanted implemented or how they would measure success. Read alongside Wang's framing that evaluations are a strategic asset worth defending, Foody's point is that failing to write one at all is the more common enterprise failure mode; the two claims are complementary rather than contradictory.

Benchmark gaming: the LM Arena problem

A widely used head-to-head leaderboard where users vote on which of two model responses they prefer has a systematic flaw: participants tend to vote on formatting and length rather than content quality. One tester described a case where a model response was a complete hallucination but, because it used an emoji and a few bolded words, was rated as good.⁷ The concrete consequence observed was a top-ranked model on such a leaderboard getting a basic factual question wrong, misattributing a public figure's death to misinformation, while a model tuned to win on the leaderboard produces longer, more heavily formatted responses rather than more accurate ones. Labs optimizing for this kind of signal can spend six to twelve months making zero real progress while their leaderboard metrics climb, training their models, in effect, to produce better clickbait. The question worth asking of any benchmark, in this framing, is whether a lab is actually improving a model's raw intelligence or simply hacking the benchmark. Today's benchmark gaming is treated as a mild precursor to a much larger risk: as models become more powerful and the systems they control grow more consequential, accidentally optimizing toward the wrong objective carries far higher stakes.

The reasoning chain as the most valuable output

Wang draws a lesson from comparing two frontier models, one of which hid its reasoning chain to prevent theft while the other showed it openly. The reasoning chain, in his view, is more valuable than the final answer: the goal is not the answer itself but seeing how a model got there. That the hidden reasoning chain was reportedly stolen anyway is, in his telling, a parable about the limits of closed evaluation and reasoning in a world where industrial espionage on AI systems is real.

A second failure: a scalar can mislead at a single point in time

The saturation cycle describes benchmarks becoming too easy over time. A separate and orthogonal failure does not require any time to pass at all: a single scalar score is uninformative if it does not control for inference compute. A model's launch benchmarks can look marginal against its predecessor until plotted against tokens spent, at which point a real capability gain becomes visible. The fix is either a performance-versus-compute curve, tracking tokens, cost, or wall-clock time, or an explicit, communicated compute budget attached to every reported score, since a benchmark number is otherwise a lossy projection of a two-dimensional surface: capability and compute together.

A non-code instance: the Kryptos capture-the-flag

Almost every evaluation in this space measures software or knowledge-work tasks. An unusual non-code instance of the same logic came when Paradigm took stewardship of the solution to Kryptos, the unsolved fourth passage of a 1990 sculpture at CIA headquarters, and paired it with ten new Kryptos-style puzzles designed explicitly to give AI-assisted cryptanalysis the one thing it lacked: a feedback loop.⁸ Despite an explosion of AI-assisted attempts on the puzzle, none had succeeded, because there was no feedback loop solvers could use to test whether their tools were actually improving at attacking this class of problem. A graded ladder of increasing difficulty with an instant automated verifier supplies exactly that, an evaluation harness built for a domain that has no natural unit test.

Publishing a number that admits an eighty seven percent failure rate

A different use of a benchmark score appears in the launch of an early coding agent that scored thirteen percent on a widely used software-engineering benchmark against a prior state of the art of three to four percent.⁹ The team was direct that this still meant the system failed eighty seven percent of the time, and shipped anyway, because the claim being made was about the derivative, the rate of improvement, not the absolute level. This is a case of a benchmark number used rhetorically and correctly: a jump from four percent to thirteen percent was legible to the field in a way a demo video was not, and it dated the claim precisely. It is also a clean illustration of the saturation cycle's front end, since a benchmark whose state of the art sits at four percent is measuring something almost nobody can do, and the same benchmark becomes uninformative a short time later as the field catches up.

Tensions and open questions

If the next generation of evaluations is fuzzier, real-world tasks, it is unclear how to maintain objectivity and comparability across models; one answer is human rubrics paired with unit tests per task, another is reporting against an explicit compute budget. The saturation cycle itself suggests the field is on a permanent evaluation treadmill, raising the question of what happens once models are capable enough that any finite evaluation saturates within months. Treating evaluations as durable intellectual property assumes firms can keep them proprietary, but in practice evaluations leak, so the advantage may be shorter-lived than claimed. There is also an unraised conflict of interest worth naming directly: the loudest voices arguing that evaluations are the bottleneck and evaluations are IP both run companies that sell evaluation and human-data production as a service. The underlying claim may still be correct, but it currently rests on the word of parties with a direct commercial stake in it being believed.