Agent-workdays are useful as a capacity signal. They are not a unit of research progress.
That boundary matters more after METR's 20 November 2026 capability update. The update reports progress on bounded tasks and also documents the difficulty of evaluating open-ended research work, including the need for expert judgment, hard-to-verify tasks, and protection against reward hacking. Those observations strengthen the measurement case made here: capacity, task performance, validity, human control, and durable outcomes need separate ledgers.
Five ledgers for one research program
| Ledger | Example fields | Question |
|---|---|---|
| Capacity | Agent sessions, task assignments, compute allocation | How much automated effort was available? |
| Throughput | Completed runs, code changes, analysis artifacts | How much work moved through the pipeline? |
| Validity | Reproduction, held-out performance, artifact integrity | Which outputs survived verification? |
| Human control | Interventions, rejection, correction, safety pauses | Where did human judgment remain necessary? |
| Research outcome | Accepted result, durable capability, publication or deployment decision | What changed in the scientific program? |
Do not collapse these into one productivity number. A capacity entry with no valid result remains capacity. A benchmark score with no code, data, and environment remains a claim waiting for reproduction.
Keep the denominator frozen
A useful report can include four ratios:
- capacity ratio: automated effort divided by defined human effort;
- validated throughput: accepted or reproducible task units per research period;
- intervention burden: human interventions per completed task, split by task horizon;
- durable outcome rate: accepted outcomes divided by scoped research tasks.
Each ratio needs a frozen population and an explicit treatment of rejected and unresolved work. This is the difference between a measurement ledger and a success story.
What the new capability evidence changes
METR's update separates verifiable tasks from hard-to-verify and judgment-heavy research tasks. It describes task families with automated objectives, tasks requiring human or rubric-based judgment, and the work required to harden evaluations against reward hacking. The practical implication is narrow: a model's score on a verifiable task cannot stand in for research acceleration when the target work depends on judgment, evidence quality, or expert review.
The earlier OpenAI disclosure remains useful as an internal, self-reported example of why capacity metrics attract attention. Its agent-workday figure and task taxonomy are informative, but compute, tooling, organization mix, and human intervention make causal interpretation difficult. The figure should enter the capacity ledger, not the outcome ledger.
A state machine for research tasks
Every task should have an auditable state:
- requested and scoped;
- executable with frozen inputs;
- run completed;
- result checked against a declared criterion;
- independently reproduced or challenged;
- accepted, rejected, or unresolved.
This state machine preserves failed and inconclusive experiments. It also lets a team locate the bottleneck. If capacity rises while unresolved work accumulates, the limiting resource is validation. If validated throughput rises while durable outcomes do not, inspect task choice and the transfer from experiment to a research decision.
Benchmarks are instruments, not organizational outcomes
MLR-Bench, WhatWorkedBench, WorkflowArena, and METR's task suites measure different bounded objects. They can reveal weaknesses in research execution, experimental understanding, workflow orchestration, time horizons, or judgment. They cannot, by themselves, measure the durable progress of a particular laboratory.
The right practice is to connect benchmark evidence to the ledger it informs. A benchmark score may update the validity ledger. It does not automatically update the outcome ledger.
The evidence packet
Every acceleration claim should link to:
- the task definition and human baseline;
- the agent identity, harness, and tool version;
- the code, data, environment, and result artifact;
- the evaluation rule and declared success criterion;
- interventions, rejections, safety pauses, and recovery events;
- independent reproduction or an explicit unresolved status;
- the decision made from the result.
This packet prevents internal usage from being written as scientific capability and prevents a bounded benchmark from being written as a causal productivity estimate.
Limits
OpenAI's figures are an internal disclosure. METR's results are based on its own task suites, human estimates, rubrics, and coverage choices. Benchmarks have selection effects and can be hardened or gamed in ways that change what they measure. This article adds an engineering synthesis; it does not claim a randomized productivity effect or a universal ranking.
The operational conclusion is narrow: report agent capacity as capacity, retain validity and intervention evidence, and reserve research progress for outcomes that survive the validity and decision ledgers.