How to Write Agent Skills That Work: An Evidence Audit of 8,135 Runs

An Agent Skill should stabilize action, expose verification, and make recovery repeatable. Treating SKILL.md as a compact textbook spends context on facts the model may already know while leaving the execution path underspecified. A 2026 preprint analyzing 8,135 normalized trial records provides unusually concrete evidence for this distinction, but the useful conclusion is narrower than the headline: skills beat raw workflow traces reliably in matched comparisons; their advantage over an agent with no injected experience was smaller and statistically uncertain.

That distinction turns the paper from marketing for skills into a practical design guide.

What the 8,135 records actually represent

The researchers tested skills across Terminal-Bench 2.0, SkillsBench, and Terminal-Bench-Pro with Codex and Gemini CLI configurations. For each selected task, they reused the same prior trajectories in three forms:

The full manifest contained 8,135 normalized trial records. The mechanism taxonomy came from a much smaller sample: 240 open-coded trajectories produced 238 valid labels, and the final comparison covered 528 matched triples, or 1,584 arm-level assignments. Human review checked the grounding of those labels and reached 95.8% agreement with the LLM-assisted taxonomy mapping.

This matters because 8,135 records were normalized, while 8,135 records were not individually hand-classified. The headline number describes the evidence base, not the manual annotation count.

The strongest result is Skill versus Workflow Memory

Across the 528 matched triples, the outcome rates were:

Skill exceeded Workflow Memory by 6.06 percentage points, with a 95% bootstrap confidence interval of +0.76 to +11.36. Skill exceeded Raw by 2.84 points, but that interval crossed zero. The defensible claim is therefore precise: distilling the same experience into a reusable procedure worked better than injecting cleaned traces. The experiment offers weaker evidence for a universal claim that every skill improves every agent. The timeout column is a descriptive taxonomy share from one representative trajectory per task-setting arm, not a global timeout rate across all 8,135 records.

The failure pattern suggests why. Workflow Memory was already cleaned and structured, but it preserved more procedural flow, exploration, dead ends, task-specific details, and debugging residue than the distilled Skill. That residue can become an attention and timeout-budget tax. It does not imply higher average duration across every run.

Procedural anchoring dominates knowledge injection

Within the 528 Skill-arm mechanism labels, 347 were classified as procedural anchoring, or 65.7%, and 24 as explicit knowledge injection, or 4.5%. Another 87 were classified as counterproductive and 64 as failure warnings. These are LLM-assisted mechanism labels over representative trajectories, not causal shares of all 8,135 records. In this study, procedural anchoring meant specifying:

• setup steps and dependency order;
• tool and command sequences;
• intermediate checks;
• output and schema requirements;
• recurring pitfalls and recovery actions.

The descriptive differences concentrated in execution-layer failures. Environment and infrastructure failure labels fell from 5.3% in Raw to 0.2% with Skill. Output-format mismatch labels fell from 7.4% to 3.2%, and background-service lifecycle labels from 2.7% to 0.8%.

The effect was modest for problems requiring new reasoning. Algorithmic logic errors moved from 8.3% to 7.4%. Static verification without runtime checks moved from 12.5% to 11.7%. A skill can preserve a known routine. It still needs the agent or human to define the right objective, reformulate an unfamiliar problem, and decide what counts as correct.

This complements the blog's earlier guide to testing agent usability with completed tasks: a skill is valuable when it changes externally verified task outcomes, not when it merely looks comprehensive.

A six-part contract for an effective Agent Skill

The evidence suggests a concrete authoring contract.

1. Trigger on a task shape

Describe the situation that should activate the skill in terms of the job, inputs, and constraints. Also name adjacent cases that require another workflow. Clear routing reduces the 10.0% skill-group failure mode where guidance was misapplied or ignored.

Weak: Use this for deployment.

Better: Use this when publishing a static site from a repository with a frozen build command and an HTTP health check. Use the container-release workflow when an image or database migration is involved.

2. State observable preconditions

List the files, credentials, tool versions, permissions, and external state that must exist. Provide a check when possible. A precondition such as command -v jq is operational; a paragraph explaining what jq is usually adds little.

3. Encode ordered actions, not a narrative transcript

Keep commands, decision points, and dependencies. Remove the chronology of unsuccessful exploration unless a failure teaches a reusable boundary. Fabien Sanglard's agent.md is a useful field example: it captures repeated code-review corrections as direct constraints. It is an experience report rather than a controlled evaluation, so it supports plausibility rather than the paper's causal estimates.

4. Attach a signal to every risky action

After an action, specify what the agent should observe. A service start needs a readiness check. A file edit needs a diff or parser. A deployment needs the public URL and expected status. Verification is an interface, not a final ceremonial section.

5. Preserve outcome labels when mining traces

Success and failure labels give a skill creator evidence about which paths to preserve. In one Gemini Terminal-Bench-2 condition with three successful and two failed source trajectories, skill construction with outcome labels scored 0.7462, versus 0.4000 when the same outcomes were hidden. Results vary by benchmark and model, and several Gemini Terminal-Bench-Pro mixtures favored the no-hint condition. Outcome labels should therefore be preserved as provenance and tested downstream, rather than treated as an automatic guarantee of a better skill.

Store a trace record like this:

task_shape: publish-static-site
outcome: failed
verifier: public_url_http_200
failure_phase: dns_propagation
reusable_lesson: verify_origin_before_cdn

6. Include recovery and final verification

A reusable skill should say where to stop, what state remains safe, how to inspect the latest output, and which verification must run again. This turns a list of commands into a closed loop.

A larger skill library needs semantic hygiene

When candidate pools grew from 5 to 100, average execution-time precision for accessing the SkillsBench-annotated ground-truth skill fell from 29.6% to 3.3%. Downstream success stayed roughly flat, moving from 36.4% to 39.3%; the paper does not report a significance test for that change. Recall remained much higher because agents often inspected several skills. In this benchmark, exact ground-truth invocation was neither sufficient nor necessary because related skills could still contribute useful procedures, and a correctly retrieved skill could still be misapplied.

The practical response is semantic hygiene:

1. Give each skill one operational responsibility.
2. Put task shape and exclusions in the description.
3. Merge overlapping procedures.
4. retire obsolete environment assumptions;
5. evaluate retrieval and final task success as separate stages.

For format and runtime differences across coding tools, use the four-layer Agent Skills compatibility model. A portable Markdown file still depends on discovery, invocation, and runtime behavior.

Put each rule in the right control surface

Procedural anchoring does not imply that every checklist belongs in a Skill. Placement determines whether guidance is always present, loaded on demand, or enforced deterministically.

Fabien Sanglard's always-on agent.md mainly contains repository-wide coding preferences. Some belong in project instructions, and rules such as formatting or visibility policy may become linters or CI checks. A deployment or incident-recovery procedure fits an on-demand Skill. This placement rule avoids turning the Skill mechanism into a new name for every kind of context.

A ten-minute audit for an existing SKILL.md

Review every section and assign one label: action, decision, verification, recovery, fact, or anecdote. Then apply this filter:

• Keep facts that change a decision or prevent a verified failure.
• Convert vague advice into an observable action.
• Move reusable checks next to the action they verify.
• Remove chronology that only documents exploration.
• Add a negative trigger for the nearest confusable skill.
• Link every known pitfall to a real failed run or external source.
• Run the skill on a fixed task set and record success, timeout, manual intervention, and misapplication.

The goal is not the shortest file. The goal is the smallest operational contract that repeatedly closes the task.

Evidence boundaries

The paper is an arXiv preprint submitted on August 14, 2026. Its benchmarks emphasize terminal and tool use, debugging, and verification. They do not cover long-horizon web interaction or open-ended multi-agent collaboration. The controlled pool selected tasks whose Raw runs contained both successes and failures, so the results are conditioned on tasks where the tested agent already behaved inconsistently. The study uses a limited set of agent-model pairings, and the mechanism taxonomy samples about 3% of normalized records. The full matched-triple labeling remains LLM-assisted, although the taxonomy construction received independent human checks.

These limits support a practical next step: test the checklist on your own frozen tasks before standardizing it across an organization.

FAQ

What belongs in AGENTS.md versus SKILL.md?

Put broadly applicable repository rules in AGENTS.md. Put a bounded, task-triggered procedure with tools, decisions, verification, and recovery in a Skill. Move deterministic rules into linters or tests whenever possible.

Should an Agent Skill contain domain knowledge?

Yes, when the knowledge changes an action, threshold, tool choice, or verifier. Background material with no operational consequence belongs in reference documentation that the agent can retrieve when needed.

Are raw agent trajectories useful?

Yes, as evidence for debugging and skill mining. Directly injecting long traces can add process noise and timeout risk. Preserve the raw record outside the execution context, label its outcome, and distill only reusable procedures into the skill.

Does a checklist replace agent reasoning?

It stabilizes known execution. Novel decomposition, algorithm design, applicability judgment, and definition of correctness still require reasoning.

How should a team measure skill quality?

Use a fixed task set and compare final verifier success, timeout rate, manual intervention, token cost, and misapplication. Measure retrieval quality separately from completed-task quality.

How long should a SKILL.md be?

Length is a secondary metric. Keep the file long enough to express triggers, preconditions, ordered actions, checks, failure boundaries, and recovery. Move optional background into linked references.

References

• Jiang et al., Demystifying Agent Skills: Why They Work Until They Don't, arXiv preprint, August 14, 2026.
• Full experimental paper, arXiv HTML version.
• Fabien Sanglard, My agent.md to improve LLM-assisted code quality, August 21, 2026.
• Yage, Skill 不是教材，是检查单, a secondary interpretation used as a discovery and comparison source.