Mercor estimates that 90% of American working hours are spent on tasks with little or no representation in its initial index of 29 economically relevant AI benchmarks. That number exposes a denominator problem in job-loss forecasts: a benchmark score measures performance on selected tasks, while a job is a weighted bundle of covered, uncovered, physical, social, and organization-specific work. Before translating a leaderboard into a workforce prediction, build a task-coverage map and keep coverage, capability, and deployment evidence separate.
What the 90% figure actually measures
Mercor's Benchmark Map combines three layers:
- The US Department of Labor's O*NET 30.0 taxonomy, containing 923 occupations and 18,797 task statements in this analysis.
- An initial index of 29 public or inspectable benchmarks selected for agentic behavior, economic realism, diversity, and unsaturated difficulty.
- Estimates of the share of working time attached to each O*NET task.
A task counts as covered when an LLM mapper links it to at least five prompts from the benchmark index. On that definition, 924 of 18,797 O*NET tasks and 208 of 923 occupations have sufficient representation. After applying task-time estimates, Mercor concludes that 90% of US working hours fall on tasks with little or no coverage.
This is an important result, but its scope is narrower than the headline can sound. It is an estimate based on one curated list of 29 benchmarks. Mercor explicitly calls the result provisional and says an incomplete list biases coverage downward. The team also capped the number of benchmarks per job family to increase breadth, which reduces measured depth in heavily benchmarked areas such as software engineering.
The correct claim is therefore: this initial, economically oriented benchmark index leaves most estimated working time weakly measured. It does not prove that every benchmark ever published misses exactly 90% of work, and it does not prove that AI cannot perform the missing tasks.
Coverage, capability, and job exposure are three different tables
Many labor-market arguments collapse three questions into one.
| Layer | Question | Minimum evidence |
|---|---|---|
| Task coverage | Is this work represented in the evaluation? | Mapping from benchmark items to a work taxonomy |
| Model capability | How well does a frozen system perform the represented task? | Valid grader, repeated trials, environment and model version |
| Job exposure | How much of the occupation could be changed in practice? | Time weights, complementarity, human oversight, physical and institutional constraints |
A high score cannot compensate for missing coverage. An uncovered task has no score, not a score of zero. Conversely, low benchmark coverage does not establish low model capability; it establishes uncertainty.
Job exposure adds another transformation. A task may be technically automatable yet rare, low-impact, legally restricted, dependent on tacit context, or inseparable from physical execution. A model may also augment a worker without replacing the task. Mercor notes this limitation directly: most real-world adoption still involves human use, while current benchmarks mainly evaluate automation.
This distinction extends the site's earlier analysis of task movement before job replacement. Usage data shows what people delegate today. A coverage map shows what evaluations can currently observe. Neither alone forecasts employment.
The map itself has measurement error
The methodology page makes the mapping pipeline unusually visible. GPT-5.6 Luna received each benchmark prompt and the full O*NET catalog, returned all reasonable task paths, and selected one primary mapping. Domain experts then reviewed the primary mapping for 1,013 of 1,022 prompts from three Mercor APEX benchmarks.
Across 1,763 expert judgments, reviewers accepted the proposed first choice 76.7% of the time. When the reviewer's answer could match another task already listed by the model, agreement rose to 85.4%. The map counts every task on the model's candidate list, even though experts reviewed only the first choice. APEX-SWE was mapped after the review and has no expert judgments in this validation set.
Those facts do not invalidate the map. They define its uncertainty:
- the benchmark sample is selected rather than exhaustive;
- task coverage depends on an LLM semantic mapping;
- expert validation covers a subset and primarily checks the first choice;
- the five-prompt threshold is a policy choice, not a natural law;
- task-time shares are themselves model-derived estimates;
- public digital work is easier to benchmark than physical or institution-specific work.
The related paper, How Well Does Agent Development Reflect Real-World Work?, independently finds a programming-centric mismatch by mapping 43 benchmarks and 72,342 tasks across 1,016 occupations. It supports the direction of the result, while using a different benchmark set and taxonomy snapshot. That is cross-validation of the blind-spot hypothesis, not a replication of Mercor's 90% estimate.
Why job forecasts break when the denominator is missing
Suppose a model reaches 80% on a finance benchmark. Without a coverage map, four incompatible statements can masquerade as the same conclusion:
- it solves 80% of benchmark items;
- it solves 80% of the O*NET tasks represented by those items;
- it can perform 80% of a financial analyst's working time;
- it can replace 80% of the job.
Only the first statement follows directly from the score. Moving to the second requires task weighting and valid mappings. Moving to the third requires time allocation and realistic task fidelity. Moving to the fourth requires workflow integration, permissions, liability, exception handling, demand effects, and organizational redesign.
Mercor tests one correction by weighting APEX prompts equally within ONET tasks before averaging them into four occupation scores. Weighted and unweighted scores remain similar, but fewer than one-third of the relevant ONET tasks have enough APEX prompts to enter the calculation. Reweighting cannot repair missing cells. It can only rebalance the covered cells.
This is the key measurement lesson: precision inside a narrow sample does not create representativeness outside it.
Build a task-coverage map for your own organization
An enterprise does not need to reproduce the entire US economy to make better decisions. It needs a local denominator.
1. Inventory work at the task level
Start with actual workflows, not job titles. Record inputs, outputs, tools, permissions, frequency, time share, failure cost, and whether execution requires physical presence or tacit coordination.
2. Link every evaluation item to the inventory
For each benchmark or internal test, declare which task it represents and how faithful the environment is. Preserve unmapped items and uncovered tasks. A forced mapping hides the most valuable information.
3. Separate breadth from depth
Report at least two coverage thresholds. One prompt may indicate discovery; five or more may support a weak task estimate. High-risk tasks need more varied instances, adversarial cases, and repeated runs.
4. Weight by business reality
Combine task performance with time share, value, error cost, human review time, and exception frequency. A frequent low-risk task and a rare irreversible task should not share one average.
5. Add deployment evidence
Offline capability must survive tool permissions, live data, latency, cost, policy, recovery, and human handoff. The site's agent evaluation framework explains why the harness belongs inside the measured system.
6. Keep unknowns visible
Use three states: measured, partially measured, and unknown. Do not silently convert unknown tasks to zero capability or full safety. The unknown share is a planning variable and a research backlog.
A safer decision rule
Public benchmarks are useful for comparing systems under a declared test contract. They become dangerous when a narrow score is treated as a labor-market model.
For workforce planning, every claim should preserve the chain:
benchmark item -> work task -> task time/value -> system performance -> deployable workflow -> job-level effect
If a link is missing, stop at the last supported conclusion. Mercor's map matters because it quantifies how often the chain currently breaks near the beginning. The practical response is not to discard benchmarks. It is to treat coverage as a first-class metric alongside score.
Frequently asked questions
Does the study prove that AI benchmarks cover only 10% of all work?
It estimates that 90% of US working hours have little or no coverage in Mercor's initial set of 29 selected benchmarks. The authors describe the estimate as provisional and expect it to change as the index expands.
Does an uncovered task mean AI cannot do it?
No. It means the benchmark set provides insufficient evidence about that task. Capability may be high, low, or unknown.
Why use O*NET tasks instead of job titles?
Jobs contain heterogeneous tasks. Task-level mapping exposes which parts of an occupation are measured and prevents one heavily benchmarked activity from standing in for the whole job.
Can task weighting fix benchmark bias?
Weighting can reduce overrepresentation among covered tasks. It cannot estimate performance on tasks with too few or no valid benchmark items.
What should companies measure beyond benchmark accuracy?
Coverage, task fidelity, repeated reliability, human review, permissions, exception recovery, latency, cost, and real business outcomes should be measured separately.