The AI Agent Benchmark Gap: A Practical Evaluation Framework for Coding Agents

An AI agent benchmark score looks like a property of a model. In practice, it is an outcome produced by a whole system: model, agent harness, task, tests, environment, budget, and grader.

JetBrains' first Kotlin Benchmark makes the gap visible. Claude Code with Opus 4.7 xhigh resolved 90 of 105 tasks. Junie with Opus 4.7 max resolved 86. The same model family differed by four tasks, about 3.8 percentage points, after the surrounding configuration changed. This is not a controlled harness-only experiment because the reasoning configurations also differ. It is still enough to show why a leaderboard row should be read as a configuration result, not a model constant.

The practical question for a development team is therefore larger than which model ranks first. It is whether the benchmark can explain why a result occurred, reproduce it under controlled conditions, and predict performance on the team's own codebase.

A leaderboard row is a system measurement

The Kotlin Benchmark uses 105 tasks mined from eight active open-source Kotlin repositories. Each agent receives an issue or pull-request description and the repository state, then produces a patch. A task passes only when the patch satisfies hidden regression tests in a reproducible container.

That is stronger evidence than a vendor demo. It covers repository navigation, implementation, and validation. JetBrains also publishes the benchmark assets and states an important limit: the score is a signal, not a guarantee for every codebase. Architecture, internal APIs, coding standards, tools, and validation processes all affect real results.

The leaderboard's Opus comparison captures several variables at once:

The result supports a precise conclusion: the deployable configuration is the useful comparison unit. It does not isolate how many points came from the harness, reasoning budget, prompts, tools, or their interaction.

Controlled research points in the same direction. The preliminary Scaffold Effect study held tasks and models fixed while comparing three open-source harnesses. Across 50 Terminal-Bench Pro tasks and two models, within-model pass-rate differences ranged from 0 to 8 percentage points. Most paired confidence intervals included zero at that sample size. The operational difference was much larger: token use per solved task varied by roughly 40 times, and each harness produced a recurring failure fingerprint.

The choice that looks like an accuracy tie can therefore be a cost, latency, or supervision landslide.

The benchmark itself can be the failing component

Even a perfectly reproduced score can measure the wrong thing.

OpenAI audited 138 SWE-bench Verified tasks that a strong model did not solve consistently. It found material problems in 59.4% of that audited hard subset. Some tests required a particular implementation even when another patch was functionally correct. Other tests checked behavior that the task description never requested. OpenAI also found evidence that frontier models had encountered task-specific information during training.

OpenAI initially recommended SWE-Bench Pro as a replacement. A later audit found problems there too. On its 731-task public split, an agent-assisted review pipeline flagged 200 tasks, or 27.4%, as broken. A separate campaign with experienced engineers identified 249, or 34.1%. OpenAI estimated that about 30% of the public tasks were broken and withdrew its earlier recommendation.

These audits expose four independent failure modes:

• the prompt can omit requirements enforced by hidden tests;
• tests can reject valid solutions because they encode one implementation;
• tests can be too weak, allowing incomplete fixes to pass;
• public tasks and gold patches can leak into training data.

A score cannot tell you which failure mode it contains. Task and grader validity must be evaluated separately from agent capability.

Use a five-layer AI agent evaluation framework

A useful evaluation stack makes every source of variation visible and gives each layer its own evidence.

1. Configuration identity

Record the system that actually ran:

• exact model and provider route;
• reasoning, sampling, and context settings;
• agent harness version or commit;
• system prompt, skills, and tool schemas;
• retry, turn, token, time, and network budgets;
• evaluator and judge versions;
• container image, dependency lockfile, and initial workspace hash.

Without this manifest, a result is a screenshot. With it, the result becomes an experiment someone can attempt to reproduce.

eval_run:
  suite: payments-agent-v3
  model: provider/model-snapshot
  harness: coding-agent@8f31c2a
  prompt_sha256: 6b7d...
  tool_schema_sha256: 91e4...
  environment: payments-eval@sha256:27ac...
  task_set_sha256: d063...
  grader_sha256: 46b1...
  trials_per_task: 5
  budgets:
    wall_seconds: 900
    max_tokens: 120000
    max_cost_usd: 4.00

2. Task and grader validity

Start with 20 to 50 tasks drawn from real manual checks, bugs, support incidents, and common workflows. Anthropic recommends this range as a practical starting point for early agent evaluations.

Each task should have:

• a clean initial state;
• an unambiguous request;
• a known working reference solution;
• tests that accept valid alternative implementations;
• positive, negative, and boundary cases;
• a named owner who can retire or revise the task.

Ask two domain experts whether they would independently reach the same pass or fail decision. Run the reference solution against every grader. Inspect tasks that remain at 0% across many attempts. A permanently impossible task often indicates a broken specification or grader.

For public benchmarks, also check task age, repository exposure, gold-patch exposure, and saturation. Public leaderboards remain useful for discovery, but their evidence should not be promoted directly into a deployment decision.

3. Isolated and repeated execution

Every trial should start from the same frozen state. Remove leftover files, caches, history, shared credentials, and resource contention that can couple one run to another.

Compare the current and candidate configurations on the same tasks, environments, graders, and budgets. Run repeated trials because agent behavior is non-deterministic. Report task-level paired results rather than only two aggregate averages.

Use the reliability metric that matches the product:

• pass@1 matters when the first attempt must work;
• pass@k matters when several attempts are allowed and one success is enough;
• pass^k matters when repeated consistency is the requirement.

There is no universal trial count. Increase repetitions when failures are costly, variance is high, or the expected improvement is small. Report uncertainty instead of converting one successful run into certainty.

4. Outcome, safety, process, and operations

One pass rate collapses distinct questions. Grade at least four dimensions separately.

Outcome and safety determine acceptance. Traces primarily diagnose failure. Requiring one exact tool sequence can reject a better valid solution, so path constraints should be reserved for steps with real security or compliance meaning.

The Harness-Bench preprint illustrates the value of this separation. Its 106 sandboxed tasks record final artifacts, execution traces, usage statistics, and validator outputs across 5,194 trajectories. Among failed trajectories, the authors identify contract and format errors, failed recovery, weak grounding, missing artifacts, and lost continuation state. A final pass rate alone cannot expose those repair targets.

5. Decision gates and evidence levels

State the evidence level attached to every conclusion:

1. Leaderboard signal: useful for forming a shortlist.
2. Audited external evidence: tasks, tests, contamination, and configuration limits have been inspected.
3. Local reproducible evidence: the configuration has been run on frozen, representative tasks with repeated trials.
4. Production evidence: shadow, canary, and monitored use confirm the offline result under real traffic.

Each level answers a different question. A public benchmark can show that a configuration is worth testing. It cannot prove that the same configuration is safe, economical, or reliable in your repository.

Build a small selection benchmark before buying a large platform

A team can build a useful coding-agent selection eval without reproducing SWE-bench.

1. Select 20 to 50 recent tasks from your issue tracker and manual release checklist.
2. Remove secrets, freeze the repository state, and preserve a working reference patch.
3. Write outcome tests that describe behavior rather than one implementation.
4. Add hard checks for unauthorized files, network access, secrets, and irreversible actions.
5. Freeze the model, harness, prompt, tools, environment, grader, and budgets in a run manifest.
6. Run the current and candidate configurations against identical task states.
7. Repeat trials according to risk and variance.
8. Report per-task regressions, cost per solved task, latency, and failure categories.
9. Manually inspect disagreements between tests, traces, and apparently valid patches.
10. Promote production failures into the suite, then deploy the winner behind a bounded canary.

Define the decision rule before looking at results. For example, every security check may require a hard pass; no critical regression task may move from reliable pass to repeated failure; cost and p95 latency may have explicit ceilings. Precommitted gates prevent an attractive average from hiding a dangerous failure.

If the immediate goal is release regression rather than product selection, the same assets can feed a versioned agent regression suite. The distinction is purpose: selection evals compare configurations, while regression evals protect behavior that the chosen system already provides.

What a trustworthy report should publish

The minimum report is more than a score:

• model and complete harness identity;
• task-set and grader versions;
• environment and dependency hashes;
• budgets, stop reasons, and failed runs;
• number of trials and uncertainty;
• outcome, safety, cost, latency, and supervision metrics;
• known broken or disputed tasks;
• raw or reviewable traces where privacy permits;
• a clear statement of what the evidence does and does not support.

AI agent evaluation is not a ranking problem with more columns. It is a verification system. The highest-value benchmark is the one that makes failures explainable, results reproducible, and deployment claims appropriately narrow.

Frequently asked questions

What is an AI agent evaluation framework?

It is the infrastructure and protocol that define tasks, initialize environments, run model-harness configurations, record traces, apply graders, and aggregate evidence. It should measure the complete executable system rather than the base model alone.

Is SWE-bench still useful?

Yes, as a research and discovery signal when its dataset version, contamination risk, task validity, harness, and grader limits are disclosed. It should not be treated as a production certificate.

How many tasks are needed to evaluate a coding agent?

Anthropic recommends starting with 20 to 50 representative tasks. The required size grows as the system matures, the expected improvement shrinks, or the consequences of failure increase.

Should teams compare models or coding-agent products?

Compare the deployable configuration: model, harness, settings, tools, and budget. A model-only comparison is valid only when the rest of the execution stack is truly held constant.

Why are passing tests insufficient?

Tests may be incomplete, overly strict, misaligned with the prompt, or exposed during training. They also omit cost, latency, permission violations, maintainability, and supervision burden unless those dimensions are measured separately.

References

• JetBrains: Introducing the Kotlin Benchmark for AI Coding Agents
• Kotlin Benchmark methodology
• OpenAI: Why SWE-bench Verified no longer measures frontier coding capabilities
• OpenAI: Separating signal from noise in coding evaluations
• Harness-Bench: Measuring Harness Effects across Models in Realistic Agent Workflows
• The Scaffold Effect in Coding Agents
• Anthropic: Demystifying evals for AI agents