AI Agent Policy Compliance: From Context to Commit Gates

Putting a policy file in an AI agent's context makes the rules available. It does not make them binding. The HANDBOOK.md benchmark turns that distinction into a measurable failure: the best of 30 evaluated configurations strictly passed only 36.2% of trials. The engineering response is a control plane that retrieves the applicable rule, evaluates it before action, blocks invalid commits, and verifies the resulting state.

Reading time: 8 minutes · About 1,700 words

TL;DR

• HANDBOOK.md tests 65 enterprise-style tasks against 20-to-124-page procedures and 824 deterministic criteria.
• Agents often find the right rule and still act against it. More context alone cannot close that gap.
• Convert policy into four layers: retrieval, preflight decision, non-bypassable commit gate, and post-action verification.
• Treat the agent's compliance report as a claim. Treat external state and deterministic checks as evidence.
• Start with high-impact, reversible actions and a small versioned policy decision record.

What HANDBOOK.md actually measured

HANDBOOK.md evaluates a deployment pattern already common in agent systems: put standing instructions in a system prompt, policy file, handbook, or skill, then trust the model to apply them throughout a long tool-using task.

The benchmark contains 65 tasks across finance, medical billing, insurance, logistics, and HR. Each task runs in a resettable company environment with files and mock email, chat, calendar, issue-tracking, and commerce services exposed over MCP. Its governing handbook is 20 to 124 pages long. Across the benchmark, 824 programmatic criteria check both required outcomes and prohibited side effects.

Strict grading requires every criterion for a trial to pass. The strongest of 30 evaluated model configurations reached 36.2%; most frontier configurations stayed below 25%. Relaxing the requirement by one criterion roughly doubled the leading scores. That is operationally important: an agent can finish most of a job while missing the one approval, threshold, or prohibited action that determines whether the job was compliant.

The paper reports four recurring failure shapes:

1. A plausible nearby request overrides the standing policy.
2. The agent performs a required check, then acts against the result.
3. Rule details degrade over a long execution horizon.
4. The final report claims compliance that the environment does not show.

These are not one problem. They occur at different boundaries and need different controls.

Policy in context is a probabilistic control

A policy document influences the model's next action. Its effect competes with the current user request, tool output, retrieved content, conversation history, and the agent's own prior plan. Even when the rule is retrieved correctly, the model still has to interpret it, preserve its details, connect it to the current object, and obey the result at the moment of execution.

That makes context a valuable policy distribution channel, but a weak enforcement boundary. It is useful for explaining goals, exceptions, and domain meaning. It cannot independently prove that a proposed side effect is allowed.

The practical distinction is:

Many systems implement only the first row, then add monitoring after an incident. HANDBOOK.md shows why the two middle rows are necessary.

Turn the four failure modes into four controls

1. Retrieve an applicable rule, not an entire handbook

Long documents should remain the authoritative source, but each action needs a compact rule packet. The packet should include the policy version, exact clause identifiers, decision inputs, thresholds, required approvers, and prohibited outcomes.

Retrieval must be tested like any other component. Build cases for near-duplicate clauses, changed thresholds, exceptions, and rules located far apart. A correct action produced from the wrong clause is still a fragile success.

2. Produce a preflight decision before generating the side effect

Separate policy evaluation from action generation. Before the agent sends an email, updates a record, changes access, or commits code, require a typed decision such as:

{
  "policy_version": "claims-sop@2026-07-29",
  "action": "approve_claim",
  "subject_id": "claim_8421",
  "decision": "require_human_approval",
  "rule_ids": ["4.2.3", "7.1"],
  "evidence": {
    "amount_usd": 18250,
    "medical_review": "complete"
  },
  "missing": ["manager_approval"]
}

This record gives the system something deterministic to validate. It also exposes a failure that natural-language reasoning can hide: the agent may have checked the amount correctly while choosing an action inconsistent with the threshold.

3. Put the commit gate outside the reasoning loop

The component that performs the side effect should reject an absent, expired, malformed, or disallowing decision. The agent may propose and repair actions; it cannot waive the gate.

The gate should bind the decision to the exact action. Include object identifiers, normalized parameters, policy version, actor identity, expiry, and an action hash. Otherwise, an approval for one customer, amount, file, or environment can be replayed against another.

Use a small outcome vocabulary such as allow, deny, require_approval, and need_more_evidence. Keep the mechanism deterministic where possible. An LLM can help interpret ambiguous policy, but the final enforcement check should validate a versioned, structured result.

4. Verify the state after execution

A successful tool call confirms transport, not business completion. Read the object back from the system of record and test the postconditions: the record has the intended value, no prohibited field changed, the approval reference is attached, and the number of side effects matches the request.

The final response should be generated from that evidence. An agent saying it complied is an assertion. A policy decision, transaction receipt, API readback, and invariant check form an audit trail.

A minimal commit-gate contract

Teams can start without building a general policy language. Define a narrow contract around the risky actions already in production.

policy_decision:
  run_id: run_0198
  policy_version: hr-handbook@7f3c
  action_type: terminate_employee
  target_id: employee_204
  normalized_arguments_sha256: 64f1...
  verdict: require_approval
  rule_ids: [termination.3, authority.2]
  evidence_refs: [case_991, approval_missing]
  expires_at: 2026-08-02T03:15:00Z

The executor validates the schema, verifies the action hash, checks freshness, and applies the verdict. High-impact actions can require a signed human approval that references the same action hash. Idempotency keys prevent retries from producing duplicate effects.

This contract should be logged with the proposed action, gate response, execution receipt, and post-action verification. That is enough to answer four incident questions: which rule applied, what evidence the decision used, why the action passed the gate, and what state resulted.

Measure policy compliance as a pipeline

One aggregate pass rate cannot locate a control failure. Track the stages separately:

• rule retrieval recall and clause precision;
• decision accuracy on allowed, denied, and approval-required cases;
• gate bypass rate, which should be zero;
• false-block and false-allow rates;
• postcondition verification coverage;
• policy-version mismatch and stale-decision rate;
• duplicate side effects after retries;
• strict end-to-end task pass rate.

Build negative tests first. Try a proximate request that conflicts with standing policy, a correct check followed by an invalid action, a stale rule version, a replayed approval, missing evidence, and a final report inconsistent with system state. These tests directly mirror the benchmark's characteristic failures.

What the benchmark does not prove

HANDBOOK.md is a preprint and benchmark, not a universal production estimate. Its tasks use fictional companies, mock services, a particular harness, and handbooks designed for evaluation. A 36.2% strict pass rate does not predict the rate for every model, policy, or enterprise workflow.

It does establish a narrower point with strong evidence: current agents can receive the relevant standing instructions and still violate them over long tool-use trajectories. Any system that treats policy presence as proof of enforcement is relying on a capability the benchmark found unreliable.

A practical rollout

Choose one consequential action family, such as refunds, access grants, production deploys, or outbound messages. Freeze ten to twenty representative policy cases and encode both expected actions and prohibited side effects. Add a structured preflight decision and a gate at the narrowest executor boundary. Run in shadow mode, compare decisions with current operations, then enable blocking for the clearest rules. Promote every disagreement and incident into the regression suite.

The policy document still matters. It carries domain meaning and teaches the agent how to work. The commit gate gives that document operational force. Reliable agent governance needs both.

FAQ

Can a larger context window solve AI agent policy compliance?

It can improve access to policy details, but it does not make a rule binding. The system still needs a decision interface, an enforcement point, and post-action evidence.

Should the policy engine use another LLM?

An LLM can interpret ambiguous text, especially during policy compilation. Use deterministic schemas and checks for thresholds, identities, required evidence, and final gate enforcement. Record uncertainty and route ambiguous cases to approval.

Where should a commit gate run?

Place it at the narrowest component that can authorize the real side effect: a tool wrapper, runtime hook, API gateway, transaction service, CI rule, or database procedure. Logging alone is downstream of the decision.

Is human approval enough?

Approval is useful when it binds to a specific action and evidence packet. A generic click detached from parameters, policy version, and post-action verification can approve the wrong operation.

How is this different from prompt guardrails?

Prompt guardrails guide generation. A commit gate evaluates a concrete action and can prevent the executor from applying it. They are complementary controls with different failure modes.

References

• HANDBOOK.md: A Benchmark for Long-Context Agentic Instruction Following
• HANDBOOK.md benchmark repository
• Runtime Governance for AI Agents: Policies on Paths
• Related: The AI Agent Benchmark Gap
• Related: Lean GPT-5.6 Prompts
• Related: Long-Horizon AI Safety

Start with one action that can cause irreversible harm. Make its policy decision explicit, block invalid commits outside the model, and verify the resulting state. That small boundary provides more control than another page of instructions.