Administrator
Published on 2026-11-21 / 5 Visits
0
0

AppSci-Bench: Building an AI Agent Is Harder Than Running One

Most agent benchmarks ask whether an already-built agent can complete a task. AppSci-Bench asks a harder question: can a model build the agent that a business actually needs, then test whether its own improvements survive new data and scenarios?

The distinction matters. Running an agent is an execution problem. Building one requires requirement elicitation, infrastructure discovery, implementation, evaluation design, iteration, and a decision about whether the artifact is safe and reliable enough to hand over.

What AppSci-Bench measures

The benchmark places a builder in a simulated business setting. The builder must interview the user, explore the available infrastructure, implement an artifact agent, deploy it, create an internal benchmark, and improve the artifact. The final artifact is evaluated on held-out examples that were not used to write its internal evaluation.

The seven tasks cover slides generation, compliance review, customer-call insights, order-to-cash, airline rebooking, CRM analysis, and banking customer support. The first five domains are newly introduced in the paper; the airline and banking tasks are adapted from the τ-bench family. Each task has its own success metric, rather than forcing every business problem into one generic answer score.

The paper evaluates five closed models and two open models, then also compares selected native coding harnesses. Its headline result is bounded: Claude Opus 5 with Claude Code reaches 59.1% on the paper's mean-minimum score across three independent builds. The number is a result for this task portfolio, harness, and evaluation protocol. It is not a production-readiness score or a universal model leaderboard.

Why mean-minimum score is useful

AppSci-Bench uses MMS@3. For each held-out example, it takes the minimum score across three independently built artifacts and then averages those minima across the task. A single unusable build can therefore expose reliability that an average score would hide.

This changes the reader's question from:

Can the model produce one impressive agent?

to:

Can the model repeatedly produce an agent that survives the same external task boundary?

That is a better fit for engineering decisions where a failed build is a real delivery failure. It also makes the metric conservative. A high typical score does not cancel a fragile build path.

The four gates before the final score

The paper's process measures are as important as its final score.

1. Requirement elicitation

The builder has to uncover rules that the simulated business user knows but has not placed in the initial prompt. Claude Opus 5 elicited 65% of the rubric requirements in the reported evaluation. The human baseline was 83%.

This is a specification problem. If the builder misses escalation rules, exception handling, or the evidence a reviewer needs, a technically functional artifact can still solve the wrong task. A production review should therefore preserve the elicitation transcript or a structured requirements record, not only the final prompt.

2. Infrastructure and artifact construction

The builder must use the customer environment and produce something deployable. The seven tasks span structured data, documents, calls, slides, and multi-turn support interactions. That variety is deliberate: a builder that succeeds only when the environment looks like a coding sandbox has not demonstrated general agent-building ability.

The paper also reports that native model-harness pairings can outperform the common AppSci harness. That is a reminder to record the harness as part of the system under test. A model score without the tool interface, handoff structure, and limits is incomplete evidence.

3. Internal evaluation fidelity

The builder is asked to create its own evaluation and use it to improve the artifact. This is where self-optimization can become self-confirmation. Across the reported builds, 86 of 132 self-authored evaluations reached a near-perfect internal score. The corresponding held-out score was often substantially lower.

The paper's analysis gives a concrete failure mode: an internal score can rise while the external score stays flat. An internal benchmark is useful only when it represents the business boundary. The acceptance packet should show its sample construction, scoring code, coverage of failure modes, and the mapping from internal cases to held-out cases.

4. Safety and answer validity

An artifact that returns a confident but fabricated answer is not a successful business agent. AppSci-Bench therefore includes safety-oriented measures alongside task quality. This is especially important for compliance, customer support, and financial workflow tasks, where an answer must be grounded in the available evidence and policy.

A practical translation for teams

Teams can turn the benchmark's structure into a local acceptance checklist:

Boundary Evidence to retain Failure that should block handoff
Requirements Elicitation rubric, transcript, resolved ambiguities Missing business rules or unowned exceptions
Environment Service map, data contracts, permission assumptions Artifact works only in a guessed environment
Artifact Versioned code, deployment result, interface contract Non-reproducible or unusable build
Internal evaluation Samples, scoring function, failure coverage Self-authored score saturates without external validation
Held-out behavior Frozen test set, per-case results, repeated builds One success hides an unstable build path
Safety Grounding checks, refusal and escalation cases Unsupported or fabricated customer-facing output

This is not a claim that every organization must reproduce AppSci-Bench. It is a way to make the hidden work of agent building visible. Each gate can be assigned to a different reviewer, and the final decision can distinguish incomplete evidence from a failed artifact.

What this benchmark does not show

AppSci-Bench does not measure the long-term economics or organizational outcomes of deploying an agent. It does not establish that one model is best for every business. The task portfolio is intentionally small, and some evaluation components use model-based judging. The paper's results also depend on the selected harness, prompt, environment, and held-out data.

The result is still useful because it narrows the claim. A model that can run an agent is not automatically a model that can discover the specification, build a reliable artifact, and construct a valid test of its own work. Those are separate capabilities and should be reported separately.

Conclusion

AppSci-Bench moves agent evaluation one layer upstream. The question is no longer only whether an agent can answer a task. It is whether a builder can turn an under-specified business need into a tested artifact whose performance survives an external task boundary.

For engineering and research leaders, the practical rule is simple: keep requirement elicitation, environment assumptions, internal evaluation, held-out reliability, and safety evidence next to the final score. The builder has passed only when the evidence chain passes with it.

References


Comment