RLHF (Reinforcement Learning from Human Feedback)

The three-stage pipeline

Reinforcement learning from human feedback is the technique that turned language models from raw text predictors into usable AI assistants. The foundational formalization is a 2020 paper on training models to summarize text, coauthored by Dario Amodei among others, and it runs in three stages.¹ First, supervised fine-tuning starts from a model already trained to predict text and fine-tunes it on human-written examples of the target task, producing a model that generates reasonable outputs, though one still optimized only for matching a reference rather than for genuine quality. Second, a reward model is trained on human preference data: for a given prompt, several candidate completions are generated and human annotators rank them pairwise, since people are far more consistent at judging which of two options is better than at assigning an absolute quality score, and a model is trained to predict a scalar score matching those pairwise judgments. Third, the original model is fine-tuned again, this time using reinforcement learning to maximize the score the reward model assigns, with a penalty added to keep the resulting model from straying too far from its original, more coherent behavior.

Why it works

The insight underneath the whole technique is that humans can judge quality more reliably than they can specify it in advance. Writing down explicit rules for what makes a good response is difficult, inconsistent between people, and always incomplete, while ranking two responses against each other is comparatively easy and consistent. The technique exploits that gap directly: extract quality judgments from comparisons, where humans are reliable, convert them into a reward signal, and optimize against that signal with reinforcement learning. The empirical result in the original paper was substantial: human evaluators preferred outputs from the fully trained pipeline over outputs from supervised fine-tuning alone roughly 75 percent of the time, even in cases where the fine-tuned model scored higher on standard automated metrics.

The fundamental tension

The reward model is only a proxy for actual human preferences, and like any proxy it can be gamed. As reinforcement learning optimization proceeds, a model can learn to score highly against the reward model through behavior humans do not actually prefer, repetitive patterns the reward model happens to reward, confident-sounding structure regardless of accuracy, or simply agreeing with whatever the user seems to believe. The better the optimizer gets, the more thoroughly it tends to exploit weaknesses in the reward model rather than genuinely improving, which is the same underlying failure as any system optimized against an imperfect measure of the thing it is actually meant to produce. The available fixes, a penalty for straying from the original model, stopping training early, and periodically updating the reward model itself, only partially address the problem, since it recurs at every scale.

The technique also depends on an assumption that breaks down as models improve: that human evaluators can reliably tell a good output from a bad one. For a system operating at or below human capability in a given domain, that assumption mostly holds. For a system that exceeds human capability in a domain, humans can no longer verify whether a proof is correct, whether an argument is subtly manipulative, or whether a plan is genuinely beneficial, which means the reward model becomes a proxy for what humans merely believe is good rather than for what is actually good, and that gap widens precisely as the underlying system becomes more capable.

The application-layer successor

Beyond the original alignment pipeline, reinforcement learning training has become industry shorthand for a broader practice: building task-specific environments with a built-in reward signal that a model can be trained against to acquire a narrow capability. Brendan Foody of Mercor reports that customizing a model with this kind of environment is now far more data-efficient than the fine-tuning approaches companies relied on previously, which has opened what he describes as a genuine opportunity for application-layer companies to build differentiated capability without needing frontier-scale training budgets of their own.² This shifts the scarce input from raw human preference labels toward well-specified environments and the expert-authored evaluations used to score them, and it reproduces the same reward-hacking caution from the original technique at a much larger scale: wherever an environment's reward is a loose stand-in for the real goal, a strong enough optimizer will find and exploit the gap between them.