Harness² shows that a frozen model can improve by changing the system around it: prompts, guardrails, memory, tool guidance, subagents, skills, scripts, and plugins. The research result is credible enough to test. It is not sufficient evidence to let an Agent persist its own changes in production. That decision needs four separate gates: generalization, benchmark isolation, economic value, and recovery.
The central engineering question is not whether a self-improving Agent can raise an evaluation score. It is whether the system can detect a bad improvement, identify how it entered the harness, contain its state, and return to a known-good version.
Evidence reviewed on October 19, 2026. Harness² is an October 2026 arXiv preprint with an Apache-2.0 repository. Its reported benchmark results are author evaluations. This article distinguishes those results from controls that production adopters still need to implement.
What Harness² actually changes
The Harness² paper and official repository connect two recursive loops while keeping model weights fixed.
At the task level, Propose, Probe, Compose generates candidate harness edits, executes them on the current task, compares behavioral differences, and composes a refined harness. It does this without ground-truth feedback during the refinement step.
At the harnessing level, the system retains adopted harnesses and evidence-supported lessons in a persistent improvement library. Those artifacts then influence later tasks.
Harness² also provides two ways to spend a larger recursion budget:
- Parallel recursion probes several general and domain-specific candidates, then composes several successors. It searches wider.
- Sequential recursion carries the refined harness and accumulated evidence into the next step. It searches deeper.
Across JobBench, LAB, and WorkBuddy-Bench, and across four model-harness configurations, one Harness² step improved GLM-5 and Gemini 3.5 Flash over their base harnesses. The largest reported gains were 11.5 points on WorkBuddy-Bench, 4.5 points on JobBench, and 11.8 percentage points on LAB partial pass rate. Larger recursion budgets produced further gains in the tested range.
Those numbers establish a useful result: harness adaptation can move capability without weight updates. They do not establish that every retained edit generalizes, that evaluation inputs remained uncontaminated, or that the extra inference cost has positive production ROI.
Gate 1: prove generalization outside the improvement loop
A benchmark becomes training data as soon as the Agent repeatedly observes outcomes and changes its harness in response. Calling the procedure inference-time improvement does not remove the statistical problem.
The minimum evaluation design needs three task pools:
- Probe tasks are visible to the improvement loop. They provide execution evidence for proposing and composing changes.
- Selection tasks choose among candidate harnesses. They must be separate from the probe executions whenever selection uses outcome signals.
- Sealed tasks remain unavailable to editors, executors, memory, retrieval, and tool access until final evaluation.
For a continually improving harness, random splits are still weak. Near-duplicate tasks, shared templates, public rubrics, and previously indexed solutions can cross the boundary. Add time-based holdouts and cluster-level deduplication. Evaluate on tasks collected after the last improvement cycle, and keep entire task families on one side of the split.
Track both average quality and regression tails. A harness that gains five points overall while failing one critical workflow has not passed a production gate. Report per-domain deltas, confidence intervals, worst-case regressions, and repeated-run variance.
Harness² itself acknowledges that execution-based feedback remains sensitive to execution variability and shared model biases. Behavioral contrast supports search. It does not guarantee correctness or transfer to a different task.
The paper does provide partial transfer evidence. Removing the current task's own library entry still produced gains on WorkBuddy-Bench and JobBench, and a JobBench library transferred positively to WorkBuddy-Bench. These results weaken a pure task-memorization explanation. They still leave a gap between online adaptation across a test stream and acceptance on a frozen, independently collected production distribution.
Gate 2: close benchmark leakage paths
The repository documents a particularly important operational risk: executor network access is unrestricted, and public benchmark rubrics may be reachable online. It explicitly tells evaluators to configure network restrictions before evaluation.
The experiment guide goes further. In one release-validation run, Codex independently downloaded the public JobBench rubric. The paper states that rubrics were not provided to the executor, editor, or selector, but it does not document host-level network isolation for the formal experiments. The defensible conclusion is narrow: the released setup has a demonstrated leakage path. The available evidence does not establish that the paper's reported scores were contaminated.
That warning should become a default test policy, not an optional setup note.
Freeze and log the complete evaluation environment:
- network allowlist and DNS policy;
- mounted files, package caches, and benchmark artifacts;
- search tools, browser tools, and retrieval indexes;
- model and harness versions;
- task identifier, seed, recursion mode, and budget;
- judge configuration and grading rubric version.
Run canary probes that contain unique strings in inaccessible answer files. Any appearance of those strings in the Agent trace is a hard contamination signal. Search logs for task titles, rubric phrases, and answer fragments. A final score without an access ledger is weak evidence because the same score can result from reasoning, retrieval, or leakage.
Leakage control matters more for harness optimization than for a one-shot model benchmark. The harness can persist a retrieved rubric as a skill, memory, script, or tool instruction, turning one accidental exposure into cross-task contamination.
Gate 3: promote persistent state in stages
Harness² keeps adopted harnesses and lessons in an indexed library. The paper lists library consolidation, outdated-experience revision, and retrieval refinement as future work. That makes state governance an adoption prerequisite.
Use four state tiers:
| Tier | Lifetime | Allowed use | Promotion rule |
|---|---|---|---|
| Probe state | One execution | Explore candidate edits | Deleted after comparison |
| Task state | One task | Reuse evidence within a task | No cross-task retrieval |
| Candidate memory | Evaluation cycle | Test transfer across sealed tasks | Requires provenance and regression checks |
| Production harness | Versioned release | Serve real workloads | Requires approval, rollback artifact, and monitoring |
Every retained component should carry its source task, supporting traces, affected surfaces, evaluation results, cost impact, owner, creation time, expiry condition, and compatible harness versions. Retrieval must be observable. Otherwise a later regression cannot be attributed to a specific persistent lesson.
The safe default is append-only candidate state followed by explicit promotion. Directly rewriting the production harness removes the boundary between learning and serving.
Gate 4: make rollback a merge condition
Version control can restore text files, but a harness change may also alter memory, scripts, plugin registrations, tool schemas, external resources, and cached state. Rollback therefore needs a semantic contract.
For each proposed change, preserve:
- the exact pre-change harness and dependency lock;
- the proposed diff and its claimed effect scope;
- independently observed file, process, tool, and external side effects;
- a rollback or compensation procedure;
- a round-trip test showing that rollback restores relevant behavior.
Our earlier analysis of EvoUndo focuses on counterfactual recovery for individual harness mutations. Harness² adds a different problem: lessons and adopted components accumulate across tasks. The combined control is a versioned release train. Promote a cohort, run sealed regressions, deploy to a canary, monitor, and retain a one-command path to the previous cohort.
Rollback tests should include later legitimate changes. Restoring a full old snapshot can erase work that occurred after the faulty mutation. Prefer effect-scoped reversal, transactional state, or compensation when exact reversal is impossible.
Gate 5: budget the whole improvement cycle
Recursive refinement spends additional executor and editor calls. Parallel execution can reduce wall-clock time when capacity is available, but it does not remove compute cost. The paper evaluates bounded budgets and does not claim a universal optimum.
Measure improvement as net task economics:
net value = downstream savings from the new harness - refinement cost - evaluation cost - regression cost - operational risk
Record at least:
- total input and output tokens for proposals, probes, composition, judging, and reporting;
- wall-clock latency and peak concurrency;
- number of candidate executions per adopted change;
- quality gain on sealed tasks;
- regression rate and rollback frequency;
- useful lifetime of the adopted component.
A two-point gain that costs ten extra executions on every task may be rational for legal research and irrational for high-volume support. The budget belongs to the workflow, not to the benchmark headline.
The measured tool-call effect is also workload-dependent. The paper reports that one-step Harness² increased tool calls by 19.7% on JobBench and 21.2% on LAB, while reducing them by 11.0% on WorkBuddy-Bench. A quality gain is therefore neither automatically faster nor automatically cheaper.
Add stopping rules. Stop when new probes add little behavioral contrast, confidence intervals overlap, candidate edits repeat, the cost ceiling is reached, or sealed-task performance declines. More recursion is an experimental variable, not a default virtue.
A production audit contract
Before enabling persistent self-improvement, require this evidence package:
| Control | Required evidence | Failure action |
|---|---|---|
| Generalization | Sealed, deduplicated, time-shifted evaluation | Reject or narrow the change |
| Leakage | Network, retrieval, file, and rubric access ledger | Invalidate the run |
| Attribution | Component diff linked to tasks and traces | Keep in candidate state |
| Safety | Critical-task regression suite and policy tests | Block promotion |
| Cost | End-to-end token, latency, and concurrency ledger | Reduce budget or reject |
| Recovery | Effect-scoped rollback and round-trip test | Block promotion |
| Operations | Canary metrics, alert thresholds, previous good release | Auto-pause and revert |
The governing rule is simple: a score increase creates a candidate. Evidence earns promotion.
Frequently asked questions
What is an Agent harness?
It is the system around a model that enables agent behavior, including prompts, tools, memory, skills, orchestration, guardrails, scripts, and plugins.
Does Harness² update model weights?
No. The published framework keeps weights fixed and improves the surrounding execution and improvement harnesses.
What is the difference between parallel and sequential recursion?
Parallel recursion explores multiple candidates and compositions at the same depth. Sequential recursion carries one refined harness through successive improvement steps.
How can self-improving Agents overfit without model training?
Repeated task feedback changes persistent prompts, memory, tools, or policies. Those changes are parameters of the full Agent system, so optimizing them against a fixed benchmark can overfit.
Why is benchmark leakage especially dangerous here?
A leaked answer or rubric can be stored in persistent harness state and influence future tasks. One contaminated run can therefore affect later evaluations.
Is Git rollback enough for a self-modifying harness?
Only when every relevant effect is captured in versioned files. Memory, caches, plugins, resources, and external side effects need effect-scoped rollback or compensation.