Human Data Market

From crowdsourcing to vetting

In the earliest era of large language model training, data was fundamentally a crowdsourcing problem: get large numbers of low-skill workers to produce simple labeled text.¹ As models improved, the binding constraint flipped. Brendan Foody describes the shift precisely: what used to be a crowdsourcing problem that was very low-skilled moved toward a vetting problem of finding the most exceptional people in the world, in volume, who want to work directly with researchers to push model capability forward. The market today needs expert labor across fields such as software, finance, medicine, law, and consulting working on data that sits beyond the current frontier of model capability, which is closer to high-skill, high-wage expert work than to simple piecework.

Three stages

A fuller account from Foody describes the market's evolution in three stages.² The first stage is behavior cloning, the crowdsourcing era of supervised fine-tuning and preference ranking, low-skill and high-volume, workers paid on the order of thirty dollars an hour or less. The second stage is agentic data, in which humans structure the success criteria for a task rather than simply rating outputs, through rubrics or unit tests that can be used either as benchmarks or as reward signals for reinforcement learning, and which requires real domain expertise to build correctly. The third and current dominant stage is reinforcement learning environments: complex, multi-tool, long-horizon task environments in which a model attempts a task fifty to a hundred times, every attempt is scored, and the model learns to climb the resulting reward signal. Foody reports his company holding roughly half to sixty percent share of this category, with average pay around ninety five dollars an hour against roughly thirty dollars an hour for earlier-generation labeling work, a wage gap he treats as the sharpest available evidence of the underlying skill shift.

The competing poles

The market has several complementary and competing centers of gravity. Alexandr Wang's Scale AI positions itself as an API for human labor spanning labeling, preference data, and environments.³ Mercor positions itself around vetting and matching, using models to predict job performance and route the most exceptional experts to the labs that need them. Each converges on the same downstream claim, that evaluations, data sets, and environments are becoming the next generation of meaningful AI intellectual property, worth protecting the way a codebase is protected, and that producing them, not model architecture, is the real bottleneck on deploying AI more broadly.

The power law in contributor quality

Foody describes the market as dominated by a power law rather than simply shifting upmarket: on a given project, a large share of the resulting model improvement typically comes from a small share of the contributors, similar to how a company's value often traces back to a small share of its people.⁴ This has two structural consequences. Proprietary access to a network of the highest-quality contributors becomes a durable moat, since it is a relationship investment competitors cannot simply outspend their way into. And matching the right person to the right task matters as much as sourcing them in the first place, since even a pool of genuine experts only creates value if routed correctly, which is the specific proprietary claim companies in this space tend to make.

Why it matters

Demand for this kind of data grows with the breadth of AI deployment across the economy rather than only with model scale, since applying a model to any new role, whether a consultant or an assistant, requires its own evaluation built by people with real expertise in that role. That growth also has a geopolitical dimension, since data supply is increasingly a contest between the United States and China, with China running government-funded labeling centers, subsidies, and dedicated college programs. Open questions include how much demand is genuinely durable given that it is tied to discrete training runs, whether frontier labs eventually build the vetting and matching capability in-house once the playbook is understood, and how large a share of the underlying need, particularly in robotics, remains effectively unserved by either pole of the market today.