Real-to-Sim Gap

The problem

The term inverts a robotics concept, sim-to-real transfer, where a robot trained in simulation fails when deployed in the real world. Here the problem runs the other direction: AI models are evaluated on academic simulations that bear little relationship to the actual tasks people need them to perform. AI research has optimized for benchmarks that are academically crisp, Olympiad-level mathematics, PhD-level reasoning, closed-form and easy to grade, and high-status, since publishing a state-of-the-art result on one of them marks career progress for researchers. But enterprises and consumers care about different questions entirely: can a model build a financial model the way a Goldman analyst would, can it research an investment opportunity by cross-referencing data sources and testing a product before synthesizing a memo, can it draft an email, schedule a meeting, and complete a ten-hour workflow using real tools.

Brendan Foody's assessment is blunt: "The evals that people have been going on of humanity's last exam and PhD level reasoning or Olympiad math are wholly disconnected from the outcomes that consumers and enterprises actually care about."¹ The failure is not abstract; the MIT study he cites found that even advanced models fail on practical enterprise tasks roughly 95 percent of the time despite holding academic gold medals.

The fix: rubric-based real-task evaluation

Foody's proposed solution is to grade a model the way a professor grades an essay, on a rubric that reflects the actual process a human expert would follow. His own example: imagine building a rubric, similar to how a professor grades an essay, that scores how well a model conducts the online research and uses the tools associated with researching an investment opportunity, cross-referencing Pitchbook data and testing the product itself. This requires three things: decomposing a task into what a real professional actually does at each step, authoring a rubric that defines what good looks like at each of those steps, which is the hard part because it requires genuine expertise, and running the evaluation inside a live environment with real tools rather than a sandbox. The rubric approach does not require a single correct answer, which is what rules out most academic benchmarks in the first place; it requires expert judgment on process quality, which is why a reliable supply of vetted human experts is the bottleneck rather than a nice-to-have.

Eval as PRD

Foody collapses the real-to-sim gap into a product-management framing: "If we think about the model is the product, then the eval is the PRD. So many people have been just like vibe spending on AI without actually writing the PRD of what do they want to implement and how do they measure that it's going to be successful." Vibe spending on AI, running pilots without ever defining success criteria, is the failure mode this exposes: organizations deploy AI, watch pilots fail in the large majority of cases, and conclude AI does not work, when the actual failure was never defining what working would even look like. The evaluation, in this framing, is the specification.

The parallel in physical AI

The robotics term this page inverts is load-bearing in its own right. Simulation and synthetic data are described as physical AI's own scaling primitive, a universal data engine that shifts the bottleneck from collecting real-world data to designing sufficiently diverse virtual environments, with world models supplying the bridge back to reality.² The evaluation gap in language models and the sim-to-real gap in robotics are the same underlying distribution-mismatch problem showing up in two different substrates.

Why it matters

Closing the gap gives enterprises an actual way to specify what they want AI to do, gives researchers the signal needed to improve capabilities in economically valuable directions rather than academically prestigious ones, and shifts the conversation from what a model can do on a benchmark to what it can do on a job, which is the only measure that ultimately matters.

Tensions and open questions

Real-task evaluations are harder and more expensive to build than academic ones, and the cost of authoring rubrics for genuinely expert work may throttle how quickly the gap can actually close. Some real-world tasks are inherently ambiguous, so grading them requires judgment calls that may not be replicable across different evaluators. There is also a straightforward commercial self-interest worth naming: the company arguing that academic evaluations are inadequate is also the company offering to sell the replacement evaluations.