GPT-6 Astra scored 62.7% or 99.9% on the same ARC-AGI-3 Semi-Private benchmark, depending on the surrounding harness. The gap is real, but the headline is easy to misread. The two best scores also use different reasoning levels, ARC-AGI-3 measures completion and action efficiency rather than ordinary accuracy, and the reported speed and token gains cover only the 167 game-reasoning pairs that both harnesses solved. The result is best understood as a measurement of two complete inference systems.
The headline compares two configurations
ARC Prize reports Astra's best Standard harness result as 62.7% at max reasoning for $26,098. Its best Provider Adapter result is 99.9% at high reasoning for $18,817. The fixed Astra results page shows the underlying displayed values as 62.71% and 99.95%.
That makes the best-to-best difference 37.24 percentage points, while reported cost falls by $7,281, or 27.90%. It does not isolate a harness effect because reasoning changes from max to high.
The paired results are more informative:
| Reasoning | Standard harness | Provider Adapter | Difference | Standard cost | Adapter cost |
|---|---|---|---|---|---|
| max | 62.71% | 98.55% | +35.84 pp | $26,098 | $17,332 |
| xhigh | 59.34% | 98.44% | +39.10 pp | $37,317 | $18,147 |
| high | 54.82% | 99.95% | +45.13 pp | $40,705 | $18,817 |
| medium | 38.59% | 98.44% | +59.85 pp | $48,090 | $19,285 |
| low | 17.45% | 98.03% | +80.58 pp | $38,166 | $21,298 |
| none | 35.18% | 96.72% | +61.54 pp | $49,791 | $23,457 |
At the same max setting, the score moves from 62.71% to 98.55%, a 35.84 point increase, while cost falls 33.59%. Every matched reasoning level shows a large gap. The evidence strongly supports a configuration-level effect even though it does not separate retained reasoning, compaction, API behavior, and their interactions into individual causal contributions.
This distinction extends the framework in The AI Agent Benchmark Gap: a leaderboard row belongs to a model-harness configuration, not to model weights alone. The Astra result supplies an unusually vivid case study because the model and task set stay fixed while the state contract changes.
A 99.9% ARC-AGI-3 score is not 99.9% accuracy
ARC-AGI-3 places agents in unfamiliar interactive environments. They must explore, infer goals, model causal rules, plan, and adjust after new observations. Its score is Relative Human Action Efficiency, or RHAE. The official benchmark page says a 100% score means an agent can beat every game as efficiently as humans.
RHAE combines progress through each environment with the number of actions used relative to a first-exposure human baseline. Later levels receive greater weight, and inefficient exploration lowers the score. A near-ceiling score therefore means near-ceiling aggregate completion and action efficiency under this protocol. It does not mean 99.9% of games were solved, 99.9% of answers were correct, or 99.9% of real-world tasks will succeed.
That metric choice matters. An agent can spend millions of tokens between environment actions while still appearing action-efficient. RHAE measures how much external experience the agent needs, while dollar cost and token use measure computational effort. A serious system evaluation needs both.
ARC Prize also states that saturating ARC-AGI-3 does not prove AGI. The environments are deterministic, closed-ended, and tightly bounded. They test a valuable slice of agentic adaptation, not open-world reliability.
What the Provider Adapter changes
The ARC-AGI-3 benchmarking repository documents two state strategies with the same games, legal actions, limits, and scoring.
The Standard harness uses provider-neutral text history. Each response must include any notes the model wants to preserve. When the client estimates that history has exceeded the context limit, it removes older turns. The model is responsible for converting past interaction into compact, visible state.
The Provider Adapter follows OpenAI's native conversation contract. Its implementation carries encrypted reasoning items between requests and accepts provider-generated compaction items for long conversations. After compaction, the client retains the latest compacted state and subsequent items. The evaluator can audit the transport contract, but the reasoning state itself remains opaque.
This moves the bottleneck. Under the Standard harness, Astra must repeatedly decide what to write into visible notes before earlier context disappears. Under the Provider Adapter, provider-native state preservation and compaction let the model reuse more of its previous work. The weights remain one component; state representation, memory policy, API semantics, and compression become part of the measured capability.
The Standard score answers a cross-provider question: how does Astra perform through a shared minimal interface? The Provider Adapter score answers a product-system question: how well does Astra perform with the context machinery designed for it? ARC Prize now labels both because they serve different decisions.
Faster and fewer tokens need a denominator
ARC Prize reports that Provider Adapter runs were about 3.66 times faster by aggregate recorded elapsed time and used 49% fewer total tokens. Those are substantial operating differences. Converted into relative terms, Adapter elapsed time was about 27.32% of Standard elapsed time, a 72.68% reduction, while Adapter token use was about 51% of Standard, making Standard roughly 1.96 times larger.
The denominator defines the claim. ARC Prize calculated both figures across Public and Semi-Private, across all reasoning levels, and only across the 167 game-reasoning pairs that both harnesses solved. This matched-success subset supports an efficiency comparison among shared successes. It excludes pairs where one configuration failed.
That exclusion matters for deployment economics. A failed run still consumes time and tokens. Conditioning on shared success can hide the cost of failures, retries, and timeouts, especially when Standard scores are much lower. The public article also provides aggregate elapsed time rather than request-level P50, P95, or P99 latency.
The open-source code records per-step duration and normalized input, output, reasoning, and cache usage. Yet the public repository currently lacks the complete set of frozen Astra configurations and the raw aggregation table behind the 167 pairs. Readers can verify the method and inspect public replays, while the two aggregate efficiency claims remain attributable to ARC Prize rather than independently reproducible from the released artifacts.
For production selection, the next benchmark should report all attempted pairs, failure and retry costs, latency percentiles, tokens per qualified completion, and the exact model, adapter, prompt, state, compaction, and budget versions. This follows the same variable-freezing logic used in real-world inference benchmarking.
The practical lesson is two leaderboards
A single ranking forces two questions into one column. Cross-provider research needs a neutral harness that holds the interface constant. Product and procurement work needs each provider's best supported stack, because users buy the deployable system rather than isolated weights.
Both leaderboards should publish configuration identity and resource boundaries. A team can then use the neutral result to reason about portability and the native-stack result to estimate attainable performance. Local evaluation still decides whether either result transfers to the team's workload.
The 62.7% and 99.9% scores expose the real unit of agent performance: model plus memory plus interface plus runtime policy. Once state management becomes the bottleneck, changing that contract can matter more than changing the model.
Frequently asked questions
What does Astra's 99.9% ARC-AGI-3 score mean?
It is a near-ceiling RHAE score combining environment progress and action efficiency relative to humans. It is not a 99.9% game solve rate or a general reliability guarantee.
Did the harness alone raise Astra from 62.7% to 99.9%?
The best-to-best comparison also changes reasoning from max to high. At the same max level, Provider Adapter scores 98.55% versus Standard at 62.71%. The public results still do not isolate each adapter feature through separate ablations.
What is the difference between Standard and Provider Adapter?
Standard carries visible model-written notes in a provider-neutral text history. Provider Adapter preserves OpenAI-native opaque reasoning state and uses provider compaction for long conversations.
Can the 3.66x speed and 49% token claims be reproduced?
The scope is public: 167 game-reasoning pairs solved by both harnesses across Public and Semi-Private sets. The published artifacts do not include the complete raw aggregation inputs needed for an independent recalculation.
Does a near-perfect ARC-AGI-3 score prove AGI?
ARC Prize says it does not. The benchmark covers adaptation in deterministic, closed-ended environments and leaves open-world complexity outside its tested boundary.