Third-party AI assessment needs more than a reputable evaluator. It needs enforceable contracts for independence, access, and disclosure. This article turns recent OpenAI and Anthropic proposals into an audit protocol that shows what an assessment can actually prove.
Reading time: 7 minutes | Words: about 1,450
TL;DR
- Independence is a conflict-control system, not an organizational label.
- Access determines the ceiling of every conclusion. Restricted inputs require restricted claims.
- Disclosure rights decide whether the public receives findings or a company-approved summary.
- A credible assessment leaves an audit trail for scope, redactions, disagreements, remediation, and later corrections.
Third-party does not automatically mean independent
A model developer can hire an outside organization, provide a narrow test interface, review every sentence before publication, and still call the result a third-party assessment. The label tells us who performed the work. It does not tell us who controlled the question, evidence, or final report.
That is the central governance problem. Trust comes from interfaces that make limitations visible. For AI assessments, those interfaces are three contracts.
OpenAI's September 2026 principles provide useful building blocks: preregistered claims, proportionate access, conflict disclosure, transparent methods, editorial independence, and explicit redaction policies. Anthropic's October update provides a concrete implementation: six METR employees received access roughly comparable to Anthropic's own risk-assessment teams, while specified categories such as customer data and privileged information remained excluded.
These documents are company statements, not independent proof that the arrangements work. They are still valuable because they expose terms that can be audited.
Contract 1: independence controls incentives
Independence is not binary. An evaluator may be legally separate while remaining financially dependent on one lab, operationally embedded in its systems, or reliant on future contracts.
A useful independence contract should disclose at least:
| Control | Evidence to publish |
|---|---|
| Funding and compensation | Payer, contract type, and whether payment depends on findings or delivery |
| Financial interests | Equity, investments, grants, and material business relationships |
| Prior involvement | Whether assessors helped build the system or safeguards under review |
| Recusal | Which conflicts trigger removal from a claim or workstream |
| Renewal pressure | Whether the assessed company controls future access or contract renewal |
| Method ownership | Who chooses questions, methods, stopping rules, and severity labels |
Anthropic states that independent evaluators should not hold equity in the company or depend on it financially or operationally. OpenAI similarly calls for disclosing organizational and individual conflicts and protecting findings from commercial pressure.
Those principles become testable only when readers can see the controls. A declaration of independence without a conflict register is an assertion, not evidence.
Contract 2: access sets the claim ceiling
Every assessment conclusion inherits the limits of its inputs. Black-box API access can test observable behavior. It cannot validate internal monitoring coverage, training controls, incident handling, or whether safeguards were disabled in another deployment path.
Anthropic describes three access tiers. Its first tier allows narrowly scoped investigations with company-selected access and publication approval. The second provides more information and publication without approval, although defined information can still be removed. The third adds employee-like access, currently represented by the METR arrangement.
That hierarchy is useful, but a tier name is not enough. Each report should attach an access manifest:
- systems, model versions, checkpoints, logs, and documents available;
- data or environments excluded, with reasons;
- whether the evaluator chose samples or received company-selected samples;
- whether tests ran in evaluator-controlled or company-controlled infrastructure;
- whether results can be reproduced after model, policy, or safeguard changes.
The report should then bind every conclusion to that manifest. If architecture information was excluded, the evaluator can assess observed safeguards but should not imply a full architecture review. If customer data was unavailable, incident conclusions must identify that blind spot.
Contract 3: disclosure preserves the finding
Access without publication freedom can produce private learning without public accountability. Publication freedom without security controls can expose customer data, attack methods, or commercially sensitive information. The disclosure contract must handle both risks without giving either side an invisible veto.
Anthropic says it cannot veto METR's publication, but it can require redaction of narrow, predefined categories. METR may publicly characterize the redactions and say whether it considers them justified without revealing the protected information. OpenAI proposes a similar structure: editorial independence, defined redaction policies, a remediation window where appropriate, and visible notice when substantive redactions affect the report.
A disclosure contract should record:
- who may delay publication and for how long;
- the closed list of permissible redaction categories;
- whether the evaluator can disclose the existence and impact of a redaction;
- how factual corrections differ from disagreements over interpretation;
- whether the company may publish a dissenting response;
- when a report must be updated after remediation or model changes.
The critical distinction is between a right to protect specific sensitive information and a right to control the conclusion. Only the first is compatible with editorial independence.
A minimum assessment protocol
The three contracts fit into one evidence trail:
| Stage | Required artifact | Failure signal |
|---|---|---|
| Commission | Conflict register and funding terms | Undisclosed dependency |
| Scope | Preregistered claims and out-of-scope process | Company can remove inconvenient questions silently |
| Access | Versioned access manifest | Report implies evidence it never saw |
| Execution | Methods, test logs, uncertainty, and failed runs | Only favorable results survive |
| Disagreement | Factual correction and interpretive dissent log | Company review becomes editorial control |
| Publication | Redaction ledger and release rights | Findings require company approval |
| Remediation | Finding owner, deadline, retest, and residual risk | A promised fix is treated as a closed issue |
| Change | Model and safeguard version binding | Old findings are presented as current assurance |
This protocol also prevents a common mistake: treating access as the whole solution. Deep access can improve evidence while increasing evaluator dependence, security obligations, and the risk of becoming culturally aligned with the organization being assessed. Independence must be continuously maintained, not granted once at onboarding.
What current announcements prove
The OpenAI and Anthropic documents show that frontier labs are converging on several useful mechanisms: claim-scoped assessment, deeper evaluator access, conflict controls, publication rights, and structured redaction. Anthropic's METR arrangement adds concrete details about staffing, tooling access, excluded categories, and veto limits.
They do not yet prove that these mechanisms consistently surface serious findings, resist commercial pressure, or change deployment decisions. That evidence arrives only when reports, redaction records, remediation outcomes, and repeated assessments become public.
This distinction follows an evidence ladder:
- a company announces a governance mechanism;
- contracts and access terms become inspectable;
- evaluators publish findings with visible limits;
- findings are independently reviewed or reproduced;
- remediation or deployment decisions can be traced to the assessment.
Claims should stop at the highest completed rung.
Frequently asked questions
What makes an AI evaluation independent?
Separate legal identity helps, but operational independence also requires disclosed funding, conflict controls, method ownership, access sufficient for the claims, and freedom to publish conclusions.
Can an evaluator remain independent while embedded inside a lab?
Yes, if the arrangement protects editorial authority and exposes conflicts. Embedding increases access but also increases dependency risk, so recusal, renewal, information-security, and publication terms matter more.
Should companies be allowed to redact reports?
Narrow redactions for customer data, exploitable details, privilege, or legitimate intellectual property can be justified. The categories should be defined in advance, and the evaluator should be able to disclose that substantive redactions occurred and how they affected confidence.
Is black-box testing enough?
It is enough for some behavioral claims. It is insufficient for claims about training controls, internal deployment, monitor coverage, or incident response unless the evaluator receives evidence from those layers.
Who is responsible when an assessment misses a risk?
Responsibility is shared but not interchangeable. The lab remains responsible for safe development and deployment. The evaluator is responsible for accurate methods and bounded claims. Governance bodies are responsible for deciding what evidence is sufficient for the decision at hand.
Next action
Before accepting the next independent AI safety report, look for three appendices: a conflict register, an access manifest, and a disclosure or redaction ledger. If any one is missing, the report may still contain useful findings, but its assurance boundary is incomplete.
References
- OpenAI: Priorities and principles for effective third-party assessments
- Anthropic: An update on independent evaluators
- METR: Our approach to risk assessment
- AI Evaluator Forum: Minimum operating conditions for independent evaluations
- NTIA: Independent evaluations
- Related: Model misalignment reporting needs reproducible evidence
- Related: Frontier AI training needs a revocable safety case