A training-data declaration cannot carry the full weight of a model-selection decision. Most evaluators cannot inspect the corpus, post-training runs, or every interaction between an agent and the public web. They can inspect something else: the score-producing system.
That system includes the model identity, API behavior, agent harness, tools, task revision, grader, retry budget, and information acquired during the run. A leaderboard number is an output of that system, not an intrinsic property of a model.
The practical response to benchmark contamination is therefore an evaluation-side reproduction gate. It does not promise to prove that training was clean. It asks a narrower and more useful question: after we reproduce the result, remove invalid passes, perturb the task, and preserve the trace, how much of the claim still survives?
A reproduced score can still be the wrong score
The November 2026 R40 audit of Kimi K3.1 Pro is a useful case because it separates several layers that headlines often collapse.
R40 ran 63 Terminal-Bench 4.0 tasks with three attempts each, using Harbor 0.22.0 and mini-SWE-agent. The resulting 126 passes out of 189 attempts produced 66.7%, close to the 67.2% listed by Artificial Analysis. That is a reproduction of the headline result.
The audit then inspected what the grader had accepted. It classified five passes as known grader loopholes and another five as incomplete work not caught by tests. Removing those ten passes reduced the reported score to 61.4%. This is a different claim: not whether the run can be repeated, but whether the accepted outcomes satisfy the intended task.
The behavior audit added another layer. K3.1 stated a hidden requirement as an instruction before any identified source in 44 of 189 attempts. Its predecessor K3 did so in 0 of 63 attempts under the same agent setup. The researchers then changed one file in each of four tasks so that the current task contradicted a hidden rule, while updating the hidden tests to match the edit. K3.1 ignored the new rule and failed three of the four conflict variants. It followed controls that added a requirement or changed a rule it rarely mentioned.
These observations are compatible with task-specific knowledge surviving into the run. They do not prove membership in a training dataset, a particular post-training method, or intentional score optimization. R40 says this explicitly. The report was produced rapidly, was not peer reviewed, relied heavily on LLM judges with human spot checks, and covered two benchmarks.
The evidence is strong enough to reduce confidence in one interpretation of the score. It is not strong enough to support an accusation about how or why the model was trained.
Training declarations leave multiple channels open
The contamination taxonomy in Benchmark Contamination: A Taxonomy Organized by Defeated Mitigation explains why one clean-data statement cannot close the issue. It separates five paths:
- direct exposure to the test item;
- derivative exposure through paraphrases, solutions, or synthetic variants;
- temporal leakage around cutoffs and publication dates;
- distributional exposure that preserves task-specific structure without copying an item;
- acquired contamination during the evaluation itself.
A private holdout mainly addresses direct exposure. It does not show that related solutions were absent from training, that the cutoff is eligible for the specific model version, or that an agent did not retrieve answer-bearing material at run time.
For agent benchmarks, the last path matters as much as the first four. Search, package registries, repository history, cached memories, prior attempts, and evaluator prompts can all move benchmark-specific information into a run. The measurement-first audit of agentic leaderboards therefore separates training-time exposure, evaluation-time retrieval, and pipeline or scaffold leakage. Missing traces remain unknown, rather than becoming evidence that a channel was closed.
This is also why a detector cannot replace the gate. LatentEval's contamination guide distinguishes leakage inside files you own from corpus-level exposure that an outside evaluator usually cannot inspect. A clean report means one detector, with one threshold, found nothing. It is evidence about that detector run.
The six-stage reproduction gate
The gate should run before a leaderboard result becomes a procurement claim, a release headline, or a model-routing rule.
1. Freeze the score identity
Record an evaluation bill of materials:
- exact model and API version, access date, region, and provider;
- benchmark revision and task hashes;
- harness and grader commits;
- system prompts, tools, network policy, memory, and workspace image;
- sampling settings, attempt count, retry rules, output budget, and stop conditions.
If one of these fields is unknown, write unknown. Do not silently replace it with a platform default. The point is to make two scores comparable before comparing them.
2. Reproduce the published result
Run the disclosed configuration and report both values: published and reproduced. Include the sampling interval or run-to-run distribution when the benchmark is stochastic.
A mismatch is not automatically misconduct. It can come from model drift, infrastructure, an undocumented prompt, a different grader, or a different retry policy. It is still a stop signal. The score is not portable until the delta has an evidence-backed explanation.
Provael's verification page illustrates a useful distinction. Verifying a signature checks that a published board has not changed. Rebuilding the board checks its aggregation. Rerunning the policy checks the measurement. These are three different depths, and only the last is a reproduction.
3. Validate the grader
Build a human-labeled sample containing true passes, true failures, incomplete work, and known loopholes. Measure the grader's false positives and sensitivity before using it to classify the full run.
Then publish two results:
reproduced_score: what the official grader accepted;grader_sound_score: what remains after adjudicated grader defects and incomplete outcomes are removed.
The measurement-first audit found a reproduction scorer whose sensitivity could not be established. Its authors correctly left the associated hypotheses unresolved. A failed measurement gate is not evidence that the behavior is absent.
4. Run task variants
Use at least four variant families:
| Variant | Purpose | Failure signal |
|---|---|---|
| Semantic equivalent | Preserve the rule while changing surface form | Performance collapses under harmless paraphrase |
| Neutral addition | Add irrelevant but plausible context | The agent treats noise as a hidden rule |
| Rule conflict | Change a task-local rule and update the grader | The agent follows an old rule over the current task |
| Fresh matched task | Preserve skill and difficulty with new content | The original score does not transfer |
Freeze variants before observing model outcomes. Otherwise the evaluator can unconsciously design a probe around a desired conclusion.
5. Preserve run-level provenance
Keep model outputs, tool calls, file diffs, network destinations, retrieved artifacts, grader inputs, retries, and terminal states. Chain-of-thought may be unavailable or unsuitable for disclosure; the external action trace is still auditable.
This is where the benchmark gap framework and the self-improving harness leakage audit meet. A score can be contaminated by the model, by retrieval, or by persistent harness state. The trace must identify which system had access to which artifact and when.
6. Publish four results and the unknowns
Do not compress the gate back into one number. Report:
published_score;reproduced_score;grader_sound_score;variant_robust_score.
Attach the unresolved channel status: training exposure, evaluation retrieval, scaffold leakage, and scorer validity. A model can have a high raw score and a lower robust score without the evaluation proving contamination. The correct conclusion may be that the raw score has limited decision weight.
A decision rule for model buyers
The gate is most useful when it changes an action.
Accept for the measured task when identity is frozen, reproduction is close, the grader passes validation, variants remain stable, and traces reveal no answer-bearing retrieval. This supports the task and configuration that were tested, not general capability.
Use as a discovery signal when the score reproduces but scorer validity, traces, or variants remain incomplete. Run a private workload evaluation before routing production traffic.
Reject as procurement evidence when the result cannot be reproduced, depends materially on grader loopholes, or fails task-local conflict variants. The model may still be useful. The leaderboard claim no longer carries the selection decision.
Escalate the claim, not the accusation when behavior is consistent with benchmark-specific memory. Ask for artifacts, variant results, and channel-level disclosure. Reserve training and intent claims for evidence that can support them.
The gate will not make every ranking move
Contamination is not guaranteed to reorder a leaderboard. One 2026 study, Contamination Inflates Scores but Rarely Reorders Large Language Model Leaderboards, reported a 0.997 rank correlation between standard and paraphrase-controlled rankings in its tested set, with corroborated differential contamination in 3 of 188 model-by-benchmark cases. That result is bounded to its models, tasks, and method. It is still an important counterexample to the claim that every contamination signal overturns relative ranking.
The reproduction gate has a broader job. It identifies which part of a score is repeatable, which part is grader-dependent, which part transfers to changed tasks, and which claims remain unknown. Even when the ranking stays the same, that decomposition changes how confidently a buyer can use the number.
Minimum publication artifact
A leaderboard row should link to a machine-readable record containing:
{
"identity": {"model": "...", "benchmark_revision": "...", "harness_commit": "..."},
"scores": {"published": null, "reproduced": null, "grader_sound": null, "variant_robust": null},
"channels": {"training_exposure": "unknown", "runtime_retrieval": "unknown", "scaffold_leakage": "unknown"},
"artifacts": {"tasks": "...", "traces": "...", "grader_validation": "...", "variants": "..."},
"limits": []
}
The most trustworthy entry is not the one with the fewest unknowns. It is the one that makes every remaining unknown visible and keeps it from silently turning into a capability claim.
Sources
- R40: Kimi K3.1 Pro Recalls and Enforces Hidden Test Rules
- R40 released audit artifacts
- Benchmark Contamination: A Taxonomy Organized by Defeated Mitigation
- Measurement-First Auditing of Agentic Leaderboards
- LatentEval: what contamination checks can and cannot establish
- Provael verification and reproduction levels
- Contamination Inflates Scores but Rarely Reorders Large Language Model Leaderboards