Turn the Claude Cookbook into an Agent Regression Suite

The Claude Cookbook is useful because its examples run. That is also why it is easy to overestimate what they prove. A notebook can demonstrate tool use, prompt caching, memory, or a managed-agent workflow under one model and one happy-path input. It does not tell you whether the same behavior survives a model upgrade, a tool-schema edit, a slower dependency, a permission change, or an adversarial request.

The right promotion path is not example to production. It is example to test fixture, fixture to regression suite, and regression suite to a release gate.

A cookbook recipe is an executable hypothesis

The Cookbook now spans responses, tools, agent patterns, Managed Agents, evals, observability, skills, and integrations. Some recipes already expose the pieces needed for testing. The building-evals recipe separates input, output, golden answer, and score. The tool-evaluation recipe records tool calls and durations. Managed-agent examples cover prompt versioning, rollback, and outcome evaluation.

Those pieces remain teaching material until a team defines the contract around them. Even an official example may contain deliberately simplified code. The tool-evaluation page, for example, warns that its dynamic eval() dispatch is for demonstration and recommends a safer production dispatch mechanism.

This does not mean the upstream repository is untested. It pins dependencies with uv.lock, runs linting and notebook-structure checks, and can execute changed notebooks in its maintainer workflow. Those controls protect recipe integrity. They do not know your refund policy, permitted data scope, service-level objective, or acceptable failure rate.

Treat every recipe as a hypothesis:

Under this model, prompt, tool contract, environment, and input, the agent should produce this outcome without violating these constraints.

A regression test makes every noun in that sentence explicit.

Start with a versioned test manifest

Copying a notebook into CI is not enough. Notebook state is implicit, cells can run out of order, and package or model aliases can move. Put each promoted scenario behind a machine-readable manifest.

id: invoice_lookup_unauthorized_account
suite: billing_agent_regression
model: claude-sonnet-5
api_version: 2023-06-01
prompt_version: billing-v17
tool_schema_sha256: 8d4e...
environment_image: billing-eval@sha256:31af...
trials: 8
max_tokens_per_trial: 12000
max_cost_usd_p95: 0.18
max_latency_ms_p95: 14000
required_graders:
  - no_unauthorized_tool_call
  - refusal_is_specific
  - no_secret_in_output
release_policy: all_hard_checks_and_7_of_8_quality_pass

Pin the model ID when the platform offers a snapshot. Also record the API version header, SDK and dependency lockfiles, system prompt, tool definitions, retrieval snapshot, environment image, feature flags, and grader versions. Anthropic preserves existing inputs and outputs within an API version, but may add optional inputs, output values, or error variants. Your parser still needs a regression test for those allowed changes.

Convert one expected answer into four contracts

An agent can reach an acceptable final answer through an unacceptable trajectory. Grade four layers separately.

1. Admissibility

These are fail-closed checks:

• only approved tools and network destinations were used;
• tool arguments satisfy authorization and data-boundary rules;
• no secret or protected record appears in the transcript or output;
• schemas, file types, and resource limits are valid;
• irreversible actions require the prescribed approval.

Do not average a permission violation with good prose. A hard failure blocks the candidate.

2. Outcome

Anthropic distinguishes the transcript from the outcome. A booking agent can claim success while no reservation exists. Verify the environment state: database row, generated file, ticket status, committed patch, or external-system event.

Use deterministic graders whenever possible. Exact comparisons, schema validation, unit tests, static analysis, database queries, and allowlists are faster, cheaper, and more reproducible than a model judge.

3. Behavior

Inspect the trace when the route matters:

• required tools were called;
• forbidden tools were not called;
• arguments and call order were valid;
• retries stopped at the limit;
• the agent escalated after the defined uncertainty threshold;
• intermediate files and state changes stayed inside the sandbox.

Avoid asserting one exact tool sequence unless the sequence is itself a safety requirement. Otherwise a better model may find a valid shorter path and fail a brittle test.

4. Operations

Track tokens, money, latency, retries, tool errors, and completion variance. A model change that preserves quality while doubling p95 latency is still a regression for a synchronous product. A cheaper average can hide expensive failure tails.

Separate capability tests from regression tests

Anthropic's agent-evaluation guidance makes a useful distinction. Capability tests ask what the agent can learn to do and should include difficult tasks with room to improve. Regression tests ask whether the agent still performs tasks it already handled and should run near a 100% pass rate.

Mixing them produces a misleading average. A gain on a new hard task can conceal a failure on password reset, refund authorization, or citation verification.

Organize suites by risk and purpose:

A capability case can graduate into regression after it becomes stable and important.

Repeat trials and compare pairs

One successful run is not evidence of stable behavior. Anthropic defines each attempt as a trial because model outputs vary. Run enough trials to reveal the failure rate that matters for the decision.

For upgrades, execute the current and candidate configurations against the same task, environment, and grader set. Preserve paired results:

• baseline passed, candidate failed;
• both passed, but cost or latency moved;
• candidate used a riskier trajectory;
• the judge disagreed with a deterministic outcome check.

Release decisions should focus on task-level regressions and confidence intervals, not only aggregate means. High-risk tasks may require every trial to pass; lower-risk quality criteria can use a threshold.

Use model judges last, then calibrate them

Open-ended tone, completeness, and explanation quality may need a model grader. Give it a specific rubric, a constrained output schema, and examples of boundary cases. Calibrate its decisions against human experts and recheck agreement after changing the judge model or rubric.

Never let the same vague judge decide permissions, factual environment state, and writing quality. Use deterministic checks for facts and hard constraints, then reserve model judgment for the dimensions code cannot express.

Trigger the smallest relevant suite

A regression suite becomes too slow if every edit runs everything. Map changes to affected surfaces:

Run a small deterministic set on each commit, broader repeated trials before merge, and the full migration and safety suites before a model switch.

Add the production loop

Offline fixtures age. Sample redacted production traces, cluster new failures, reproduce them in a controlled environment, and add representative cases to the suite. Keep a sealed set that prompt authors and optimizing agents cannot inspect.

After the offline gate, use shadow traffic or a bounded canary. Monitor the same outcome, safety, cost, and latency metrics used in evaluation. Pin the previous prompt and model so rollback is an operation, not a reconstruction project.

This is where GEPA's evaluator-as-interface lesson becomes practical: anything omitted from the evaluator remains an optimization opportunity. A regression suite is therefore both a quality system and a precise statement of what the agent is allowed to become.

A minimal adoption sequence

1. Choose one Cookbook recipe close to a real workflow.
2. Freeze its model, prompt, tools, environment, and dependencies.
3. Replace the demo input with representative positive, negative, and boundary cases.
4. Add hard authorization and side-effect assertions.
5. Verify outcome state, not only final text.
6. Record trace, tokens, cost, latency, and retries.
7. Run multiple trials and establish the current baseline.
8. Put the suite in CI with risk-based pass rules.
9. Replay production failures and maintain a sealed test set.
10. Deploy behind a canary with a pinned rollback target.

The Cookbook remains the source of executable patterns. The regression suite becomes the source of release evidence.

Frequently asked questions

Can I use the Cookbook notebook itself as a test?

Use it as a starting fixture, but move configuration, inputs, assertions, and dependencies into versioned files. A notebook's hidden state makes failures difficult to reproduce.

Should every result have a golden answer?

No. Some tasks need exact answers; others need environment-state checks, invariants, or rubric-based quality scores. The success condition should match the product outcome.

How many trials are enough?

It depends on the acceptable failure rate and consequence. Use more trials for non-deterministic or high-risk tasks, and report uncertainty rather than treating one pass as certainty.

Should an LLM grade its own agent?

It can grade subjective dimensions if the rubric is calibrated. Do not use it as the only judge for permissions, tool side effects, database state, or security boundaries.

References

• Claude Cookbook
• Building evals
• Tool evaluation
• Demystifying evals for AI agents
• Define success criteria and build evaluations
• Claude API versions