Verifiability in AI

The mechanism

Frontier language models are trained using large reinforcement-learning environments, and the training signal in those environments is fundamentally a verification reward: a model gets signal when it produces an output that can be checked. The consequence is that models become much better at tasks where correctness can be verified and stay comparatively rough at tasks where it cannot. As Andrej Karpathy puts it, "Traditional computers can easily automate what you can specify in code. This latest round of LLMs can easily automate what you can verify."¹ This is the underlying logic behind why math, code, and adjacent domains became the earliest peak areas for these models: they are the easiest domains to build verification environments for, and commercially valuable enough that labs invested in building them.

The jagged capability profile

The result is a set of "jagged" capabilities: models that peak sharply in verifiable domains and plateau or fail in others, in a pattern that is not random but follows the shape of the reinforcement-learning training landscape. Karpathy's example is the car wash problem: asked whether to drive or walk fifty meters to a car wash, a state-of-the-art model may confidently recommend walking, while the same model can simultaneously refactor a hundred-thousand-line codebase or find zero-day security vulnerabilities. An earlier version of the same complaint was a model's inability to correctly count the letter R in "strawberry"; newer models now patch specific cases like it without resolving the underlying jaggedness.¹

The jaggedness has a second explanation beyond training distribution alone: labs choose what to invest in. Chess capability, for instance, spiked far beyond the general trend between GPT-3.5 and GPT-4 specifically because OpenAI added large volumes of chess data to pre-training, not purely from general capability growth.¹ Capability, in this reading, is partly a deliberate choice rather than only a gradient of general intelligence.

A practical heuristic

Karpathy offers a working test for whether to trust a model in a given domain: "If you're in the circuits that were part of the RL, you fly. If you're in the circuits that are out of the data distribution, you're going to struggle."¹ The practical questions that follow are whether a domain is represented in pre-training data at scale, whether labs have explicitly built reinforcement-learning environments for it, and whether its output is verifiable enough that reinforcement learning could plausibly have targeted it. His broader claim is that almost everything can eventually be made verifiable to some extent, even domains like writing, through mechanisms like a panel of model judges, which makes the hard-versus-easy dimension the real variable rather than a strict line between automatable and non-automatable work.

Verifiability as an underwriting rule

Marc Rowan of Apollo Global Management arrives at the same split independently, from outside AI research entirely, and converts it directly into a capital-allocation rule: "Change is taking place faster in places there are a right answer. So why do we see coding in software? Because at the end of the day, the AI can check whether the AI is right. So the rate of change is a vertical line. On the other hand, if you want an answer to what is the best Shakespeare essay, we're seeing improvement, but someone has to inform. There's no right answer."²

His resulting business mapping is direct: domains with a right answer, such as accounting and trade operations, are candidates for replacement by AI systems, while domains requiring judgment and know-how are candidates for augmentation rather than replacement. Rowan adds an explicit caveat: "Is this a permanent state of events? No. Because none of us know how this ends or how good the judgment will eventually be."² For a lender, this split is not an abstraction, since it determines which cash flows remain reliably underwritable as AI capability advances unevenly across domains.

Why it matters

The framework explains one of the more durable human remainders in this transition: taste and simplification sit largely outside current reinforcement-learning training, since there is no clean verification signal for whether a piece of code or writing is more elegant, only whether it works. That absence is not necessarily permanent, since it reflects what labs have chosen to build rather than a fundamental limit, which means today's verifiable-but-unbuilt domains are exactly where any human remainder is likely to keep shrinking over time.