An agent-readiness score can tell you that documentation, llms.txt, OpenAPI, or an MCP endpoint exists. It cannot tell you whether an AI agent can complete a user's task. Product AI usability requires a dynamic task eval: a clean starting state, a realistic goal, an observable product outcome, and a grader outside the agent's own claims.
Readiness has five layers
Teams often compress several different questions into one score. Separating them makes the measurement useful.
| Layer | Question | Minimum evidence |
|---|---|---|
| Discoverability | Can an agent find the product surface? | indexed docs, advertised tools, stable entry points |
| Readability | Can it parse the available material? | valid Markdown, schemas, examples, error documentation |
| Comprehension | Can it select the relevant instruction? | retrieval trace, correct version and scope |
| Executability | Can it invoke the interface correctly? | accepted tool call, command, or API request |
| Completion | Did the user's intended state result? | database, UI, API, or filesystem postcondition |
A static scanner can test the first two layers cheaply. It may also inspect whether an OpenAPI document or MCP card is present. Those are useful surface checks. They remain upstream indicators. Completion is the product outcome.
This distinction matters because a polished documentation site can coexist with stale examples, ambiguous permissions, silent partial failure, or a tool that returns success before the business state changes. The product is easy to describe and still hard to use.
What Vercel's 53%, 79%, and 100% results actually show
Vercel tested several ways of supplying version-matched Next.js knowledge. In its hardened Next.js 16 suite, the no-documentation baseline passed 53%. A Skill under default behavior also passed 53% because it was not invoked in 56% of cases. Explicit instructions raised Skill invocation above 95% and the pass rate to 79%. A compressed documentation index embedded in AGENTS.md, pointing to version-matched files in .next-docs, reached 100% in that suite.
The headline is tempting, but the reusable lesson is about system boundaries. The same documentation produced different outcomes depending on discovery, invocation, ordering, and version selection. Documentation quality alone was not the measured variable.
Vercel also explains that its first suite contained ambiguous prompts, implementation-specific assertions, and APIs already represented in model training data. It rebuilt the suite around observable behavior and Next.js 16 APIs outside that training window. The evaluator had to improve before the result became useful.
The 100% result belongs to one hardened suite under one disclosed comparison. It does not establish that AGENTS.md always beats Skills. Vercel describes the two mechanisms as complementary, and the public article does not fully specify every model, harness, task count, or trial detail needed for a universal claim.
Make the product the system under test
Most agent-evaluation guidance asks: how capable is the agent we built? A product team faces a different question: can unknown agents reliably use the product we expose?
That changes the experiment. Freeze the product version, task set, accounts, data, and graders. Then rotate the model, harness, documentation treatment, and tool surface. The task remains constant while the access path changes.
user goal
-> clean product state
-> agent plus docs/tools
-> action trace
-> product state
-> external grader
The agent's final message is evidence about its belief. The product's state is evidence about completion.
Supabase's public evaluation work illustrates this separation. It tests Skill loading, task completeness, and concrete tool use across agents and conditions. Its grader looks for actual calls, SQL, files, and valid outputs, not a promise that the work was done. Supabase also labels the published numbers as early results from only six scenarios per condition. That caveat is part of good evaluation practice.
Stripe's public AI repository goes further into realistic integration environments, with SDKs, tools, Skills, and benchmarks. Complex environments can provide strong evidence, but they are expensive to reproduce. A small product team needs a smaller starting point.
A minimum dynamic task eval for a product team
Start with 8 to 12 tasks drawn from support tickets, onboarding friction, common API workflows, and expensive mistakes. A useful first set should include:
- one happy-path setup task;
- one version-sensitive task;
- one permission or authentication failure;
- one ambiguous request that requires inspection before action;
- one partial-failure recovery;
- one destructive or high-impact action with a confirmation boundary;
- one task where several valid implementations should pass;
- one task that should be refused or escalated.
Each task needs a clean initial state, a user-visible goal, allowed resources, a time or cost budget, and a grader that the agent cannot edit. Prefer deterministic postconditions: a database row has the expected values, a deployed endpoint returns the required response, an access policy blocks the forbidden case, or a test suite passes without unrelated changes.
Use an LLM judge for qualities that resist simple assertions, such as explanation clarity or whether a recovery plan addressed the real failure. Keep those scores separate from deterministic completion.
Measure friction, not only pass rate
Pass rate answers whether the system eventually reached a target. Product design also needs to know how much friction it consumed.
Track at least:
| Metric | What it reveals |
|---|---|
| pass@1 | probability that the first attempt works |
| pass@k | whether retries can recover at least once |
| repeated consistency | whether a workflow is dependable, not occasionally lucky |
| correct-state rate | whether the final product state satisfies all invariants |
| recovery rate | whether errors and partial failures are diagnosable |
| tool and retrieval count | interface and documentation friction |
| latency and cost | practical operating budget |
| unsafe-action rate | whether speed hides control failures |
Avoid reporting only the best of several runs. Best-of-three may answer a capability question, while pass@1 answers a default-user question. Both can be useful when clearly labeled. Mixing them creates a false comparison.
Compare documentation, Skills, MCP, and CLI fairly
These surfaces solve different problems. Documentation distributes knowledge. AGENTS.md provides persistent project context. Skills package conditional procedures. MCP exposes structured live tools. CLI commands provide executable, inspectable operations.
Compare them with the same product version, tasks, initial states, graders, budgets, and model-harness pairs. Change one surface at a time. Record whether failure occurred at discovery, retrieval, invocation, execution, or state verification.
This failure taxonomy matters more than a single winner. If the agent never loads the Skill, improve triggering or persistent routing. If it loads current documentation but calls a nonexistent command, improve examples and interface discovery. If the call succeeds but the user goal remains incomplete, fix the product contract or grader. Each failure belongs to a different owner.
The same principle appears in cross-harness Skill compatibility and in coding-agent benchmark design: a format or leaderboard row is only one layer of a deployable system.
Turn capability tasks into release gates
A new task begins as a diagnostic. Once the product supports it reliably, promote it into a regression suite. Run the suite when any relevant input changes:
- product API, UI, or permission model;
- documentation or examples;
- SDK, CLI, Skill, plugin, or MCP server;
- model or agent harness;
- authentication and sandbox policy;
- grader or test fixture.
Store an evaluation manifest with exact versions, task and grader hashes, trial policy, budgets, and timestamps. Trend failures by layer rather than celebrating one aggregate score.
The goal is not a permanent 100%. The goal is a feedback loop that shows which product boundary broke, how costly the failure was, and whether the repair survives the next release.
FAQ
What does AI-usable mean for a product?
An AI-usable product lets an agent discover the right interface, understand the current contract, execute within permissions, recover from errors, and produce the intended product state. Documentation readability is one input to that outcome.
Is an agent-readiness score the same as task success?
No. A readiness score commonly measures static surfaces such as files, schemas, protocols, or crawlability. Task success requires running an agent in a controlled environment and verifying the resulting state.
How many trials should each task get?
There is no universal number. Use repeated trials when variance is high, failure is costly, or expected improvements are small. Always publish the trial policy and distinguish first-attempt success from best-of-k capability.
When should I use an LLM judge?
Use deterministic graders for product state, permissions, data integrity, and safety invariants. Add an LLM judge for subjective qualities that cannot be reduced to stable assertions, and report the two evidence types separately.
Does a 100% eval score prove the product is fully AI-usable?
It proves that the tested configurations passed the defined tasks under the recorded conditions. New tasks, versions, models, interfaces, and real users can expose unmeasured failures.