Most agent benchmarks ask whether an already-built agent can complete a task. AppSci-Bench asks a harder question: can a model build the agent that a business actually needs, then test whether its own improvements survive new data and scenarios?
The distinction matters. Running an agent is an execution problem. Building one requires requirement elicitation, infrastructure discovery, implementation, evaluation design, iteration, and a decision about whether the artifact is safe and reliable enough to hand over.
What AppSci-Bench measures
The benchmark places a builder in a simulated business setting. The builder must interview the user, explore the available infrastructure, implement an artifact agent, deploy it, create an internal benchmark, and improve the artifact. The final artifact is evaluated on held-out examples that were not used to write its internal evaluation.
The seven tasks cover slides generation, compliance review, customer-call insights, order-to-cash, airline rebooking, CRM analysis, and banking customer support. The first five domains are newly introduced in the paper; the airline and banking tasks are adapted from the τ-bench family. Each task has its own success metric, rather than forcing every business problem into one generic answer score.
The paper evaluates five closed models and two open models, then also compares selected native coding harnesses. Its headline result is bounded: Claude Opus 5 with Claude Code reaches 59.1% on the paper's mean-minimum score across three independent builds. The number is a result for this task portfolio, harness, and evaluation protocol. It is not a production-readiness score or a universal model leaderboard.
Why mean-minimum score is useful
AppSci-Bench uses MMS@3. For each held-out example, it takes the minimum score across three independently built artifacts and then averages those minima across the task. A single unusable build can therefore expose reliability that an average score would hide.
This changes the reader's question from:
Can the model produce one impressive agent?
to:
Can the model repeatedly produce an agent that survives the same external task boundary?
That is a better fit for engineering decisions where a failed build is a real delivery failure. It also makes the metric conservative. A high typical score does not cancel a fragile build path.
The four gates before the final score
The paper's process measures are as important as its final score.
1. Requirement elicitation
The builder has to uncover rules that the simulated business user knows but has not placed in the initial prompt. Claude Opus 5 elicited 65% of the rubric requirements in the reported evaluation. The human baseline was 83%.
This is a specification problem. If the builder misses escalation rules, exception handling, or the evidence a reviewer needs, a technically functional artifact can still solve the wrong task. A production review should therefore preserve the elicitation transcript or a structured requirements record, not only the final prompt.
2. Infrastructure and artifact construction
The builder must use the customer environment and produce something deployable. The seven tasks span structured data, documents, calls, slides, and multi-turn support interactions. That variety is deliberate: a builder that succeeds only when the environment looks like a coding sandbox has not demonstrated general agent-building ability.
The paper also reports that native model-harness pairings can outperform the common AppSci harness. That is a reminder to record the harness as part of the system under test. A model score without the tool interface, handoff structure, and limits is incomplete evidence.
3. Internal evaluation fidelity
The builder is asked to create its own evaluation and use it to improve the artifact. This is where self-optimization can become self-confirmation. Across the reported builds, 86 of 132 self-authored evaluations reached a near-perfect internal score. The corresponding held-out score was often substantially lower.
The paper's analysis gives a concrete failure mode: an internal score can rise while the external score stays flat. An internal benchmark is useful only when it represents the business boundary. The acceptance packet should show its sample construction, scoring code, coverage of failure modes, and the mapping from internal cases to held-out cases.
4. Safety and answer validity
An artifact that returns a confident but fabricated answer is not a successful business agent. AppSci-Bench therefore includes safety-oriented measures alongside task quality. This is especially important for compliance, customer support, and financial workflow tasks, where an answer must be grounded in the available evidence and policy.
A practical translation for teams
Teams can turn the benchmark's structure into a local acceptance checklist:
| Boundary | Evidence to retain | Failure that should block handoff |
|---|---|---|
| Requirements | Elicitation rubric, transcript, resolved ambiguities | Missing business rules or unowned exceptions |
| Environment | Service map, data contracts, permission assumptions | Artifact works only in a guessed environment |
| Artifact | Versioned code, deployment result, interface contract | Non-reproducible or unusable build |
| Internal evaluation | Samples, scoring function, failure coverage | Self-authored score saturates without external validation |
| Held-out behavior | Frozen test set, per-case results, repeated builds | One success hides an unstable build path |
| Safety | Grounding checks, refusal and escalation cases | Unsupported or fabricated customer-facing output |
This is not a claim that every organization must reproduce AppSci-Bench. It is a way to make the hidden work of agent building visible. Each gate can be assigned to a different reviewer, and the final decision can distinguish incomplete evidence from a failed artifact.
What this benchmark does not show
AppSci-Bench does not measure the long-term economics or organizational outcomes of deploying an agent. It does not establish that one model is best for every business. The task portfolio is intentionally small, and some evaluation components use model-based judging. The paper's results also depend on the selected harness, prompt, environment, and held-out data.
The result is still useful because it narrows the claim. A model that can run an agent is not automatically a model that can discover the specification, build a reliable artifact, and construct a valid test of its own work. Those are separate capabilities and should be reported separately.
Conclusion
AppSci-Bench moves agent evaluation one layer upstream. The question is no longer only whether an agent can answer a task. It is whether a builder can turn an under-specified business need into a tested artifact whose performance survives an external task boundary.
For engineering and research leaders, the practical rule is simple: keep requirement elicitation, environment assumptions, internal evaluation, held-out reliability, and safety evidence next to the final score. The builder has passed only when the evidence chain passes with it.