"Agent Security Hooks Need a Capability Matrix: Normalize Evidence Before Enforcing Policy"

Installing a security hook does not prove that an AI agent is controlled. The configuration may exist without being loaded, the hook may observe an action only after it runs, a rule match may be detection-only, and a recorded deny may not prove that the host honored the response.

These are not edge cases. They are different capabilities that are often compressed into one checkbox called “supported.”

Agent security needs a capability matrix before it needs more rules. The matrix should describe when evidence is captured, how faithfully the host exposes an action, whether the callback can make a synchronous decision, what happens on failure, and whether the resulting evidence is durable. Only then can a security team decide which policies are safe to enforce.

Numbat, an open-source endpoint visibility project from Perplexity, is a useful case because its documentation makes many of these boundaries explicit. It normalizes hooks, plugins, OTLP logs, and on-disk artifacts into one event vocabulary and evaluates CEL rules against that evidence. It also states that all shipped rules are monitor-only and that blocking is limited to supported synchronous pre-action hooks.

The broader architectural lesson extends beyond one tool: normalize evidence first, declare host capabilities second, and enforce only where the complete control path is testable.

Why per-agent rule sets do not scale

Coding agents expose different lifecycle names, payloads, trust controls, and response contracts. The same shell action may arrive as PreToolUse, BeforeTool, preToolUse, tool.call, or a plugin callback. One host supplies structured arguments and a call identifier. Another supplies only a generic tool name. A cloud execution path may bypass the local hook that covers the desktop path.

Writing a separate policy for every callback couples the policy layer to vendor-specific transport. Every host update can change both visibility and enforcement semantics. Rules drift, exceptions accumulate, and security teams lose the ability to answer a basic question: did these two findings describe the same type of action?

Numbat's event model takes the infrastructure-first approach. It maps supported evidence into a closed vocabulary such as:

• command.exec and command.result
• file.read, file.write, and file.delete
• network.indicator
• tool.call and tool.result
• permission, prompt, message, configuration, and session events

Classification is deliberately narrow. Unknown tools remain tool.call; free-form prose is not mined to invent structured actions. That preserves a critical distinction: a normalized event says what the source evidence supports, not what a detector hopes happened.

Normalization lets one rule express a stable policy, but it does not make every source equally capable. Provenance, confidence, timing, and missing fields still matter. The capability matrix carries those differences forward instead of erasing them.

This complements the production observability gap for AI agents: useful telemetry is the evidence the runtime actually emits, not reasoning a platform wishes it could inspect.

Seven columns every capability matrix needs

The row should be scoped to a host and execution path, not just a product name. “Codex supported” is too broad if local function tools enter a hook but hosted tools bypass it. “Cursor supported” is too broad if local, cloud, and Tab paths expose different callbacks.

A compact host matrix

The following examples are drawn from Numbat's current agent coverage documentation. They illustrate why visibility and control need separate columns.

This table is intentionally not a coverage leaderboard. A broad hook can still carry low-fidelity data. An at-rest parser can provide excellent forensic context while having zero preventive ability. A pre-action callback can request a deny while lacking post-action confirmation.

Application hooks and kernel enforcement are complementary

Application hooks have the richest semantic position. They can expose the tool name, proposed arguments, project, session, and sometimes the user's approval context before the action crosses into the operating system. Their weakness is coverage: a hosted tool, disabled plugin, untrusted project, or undocumented execution path may bypass the callback.

Kernel and runtime controls see a different boundary. Linux BPF LSM programs can return an error from a security hook, while systems such as Tetragon can attach monitoring or enforcement actions to supported resource events. This layer is closer to the file, process, socket, or workload side effect and is harder for an application path to omit. It usually cannot determine from a syscall alone whether a request came from prompt injection or legitimate user intent.

The two layers should feed one audit chain without pretending to be interchangeable. The application event explains what the agent proposed. The resource-boundary event shows what the process attempted or completed. Session, process, call, path, and time identifiers connect them. A sensitive policy may require both a semantic decision at the hook and an operating-system guardrail around the eventual effect.

Monitoring and enforcement are different proofs

Numbat's enforcement model separates five stages:

1. The model proposes an action.
2. The host invokes a pre-action hook.
3. Numbat validates and normalizes the payload, evaluates rules, and emits operator records.
4. On a clean enforced match, Numbat returns the host's native deny response.
5. The host interprets that response and decides whether to execute, prompt, or reject.

Numbat does not cancel the tool itself. It asks the host to deny. An enforcement record proves that Numbat computed deny or no_override; it does not prove response delivery or final host behavior.

That distinction creates three separate proof obligations:

• Sensor proof: the intended host path actually invoked the hook with the expected payload.
• Decision proof: the normalized event matched the intended policy and produced the expected response.
• Outcome proof: the host honored the response, or a post-action/result channel confirmed what happened.

A deployment is not enforcement-capable until all three have been exercised. A configuration-status command can help with the first setup step, but Numbat explicitly notes that hook status proves only that its configuration exists and is readable. It does not prove host activation, event delivery, or sink delivery.

Blocking should be narrower than detection

Detection can tolerate ambiguous evidence because a human can investigate a finding. Blocking interrupts work and therefore needs stronger guarantees.

Numbat's rule system illustrates this asymmetry. CEL rules can inspect raw command text or a parsed shell_commands view. Detection supports a broad range of shell structures, but enforcement permits only a smaller static subset. Dynamic substitutions, multiple statements, loops, conditionals, inline interpreters, ambiguous pipelines, parser diagnostics, and truncated projections remain detection-only.

This is a sound policy principle: do not convert an uncertain observation into a certain control action.

A clean deny also requires more than a match. The callback must be a supported synchronous pre-action event; the payload must validate; the rule must explicitly set enforce: true; the relevant projection must evaluate successfully; selected records must be accepted by the configured sink; and sequence state must be available when the rule depends on history. Malformed or oversized payloads, relevant evaluation errors, and output failures suppress the Numbat deny.

The shipped catalog remains monitor-only. Installing with --enforce does not silently turn high-severity findings into blocks. Severity and control effect are separate policy fields.

Failure semantics belong in the policy, not the footnotes

“Fail open” is often presented as one global setting. In a hook ecosystem, failures occur at several boundaries:

• the host fails to launch the hook;
• the hook times out;
• the payload cannot be normalized;
• a rule evaluation fails;
• the evidence sink is unavailable;
• the host rejects or ignores the returned control response.

Numbat usually withholds a deny when its own clean-decision requirements fail. That still does not guarantee execution, because the host may apply another rule or prompt the user. Some host contracts are exceptions. Its documentation notes, for example, that Copilot CLI may deny when a pre-command hook fails to launch, crashes, or exits non-zero, while a timeout allows. OpenClaw has different wrapper and host timeout boundaries.

A capability matrix should therefore record failure behavior per boundary. Otherwise “fail open” can conceal a mixed system in which one failure permits an action, another blocks the whole workflow, and neither produces durable evidence.

Durable records are part of enforcement quality

Security control without an audit trail is difficult to tune and difficult to trust. Numbat writes findings and applicable enforcement decisions to a local NDJSON file by default. A complete live event stream requires explicit selection. Direct HTTP delivery is an attempt, not a disk queue; when the receiver is unavailable, records are not automatically spooled for later delivery. At-rest reconstruction is limited to what the host persisted.

This produces another useful evidence ladder:

1. A rule matched an observed event.
2. An enforcement record says a deny was computed.
3. The local sink durably accepted the record.
4. The host response path was tested.
5. A result or independent endpoint signal confirms the outcome.

Even the final step should be worded carefully. A finding is a policy match, not proof of compromise. A missing exit code means the source omitted it, not that the command succeeded.

A safe rollout sequence

The correct deployment order is not “install, then enable blocking.” It is a calibration loop.

1. Inventory execution paths

List local CLI, IDE, desktop, remote, cloud, gateway, subagent, MCP, and hosted-tool paths. Give each a separate matrix row where its hook behavior differs.

2. Install in monitor mode

Capture benign known actions. Confirm source agent, event type, arguments, project, session, call identifier, and durable output. Compare the normalized record with the host's own log where possible.

3. Prove activation

Check configuration presence, then verify effective host configuration and trust. Trigger a known non-match and a known detection. Confirm that the local record and any forwarding path both receive them.

4. Measure false-positive cost

Start with high-signal rules for narrowly defined events. Complex raw command patterns should remain monitor-only until the parsed semantics and real workload distribution are understood.

5. Test the negative paths

Exercise malformed input, timeout, sink unavailability, disabled plugins, untrusted projects, and host approval behavior. Document which boundary fails open and which fails closed.

6. Enforce a small policy set

Enable only rules with explicit control intent, static evidence, a tested native deny transport, and a recovery path. Keep broad anomaly rules in monitoring.

7. Recalibrate after host changes

Agent versions, callback schemas, trust controls, and cloud execution paths change independently. Treat the capability matrix as a versioned operational artifact, not a one-time architecture diagram.

The infrastructure is the evidence contract

Security hooks are components. The durable system is the evidence contract that lets many components describe actions consistently while preserving their limits.

That contract should make it impossible to confuse:

• configuration with activation;
• pre-action intent with completed action;
• a finding with compromise;
• a computed deny with an honored deny;
• monitoring coverage with enforcement coverage;
• direct delivery with durable retention.

Once those distinctions are encoded in a capability matrix, policy becomes portable without becoming falsely uniform. The organization can use one normalized rule language while still making host-specific decisions about evidence quality, blocking, failure, and auditability.

FAQ

Can every AI agent hook block an action?

No. Blocking requires a synchronous pre-action callback and a host contract that accepts a deny response. Post-action hooks, OTLP logs, and at-rest artifacts are observation-only. Some integrations, such as OpenCode in Numbat's current matrix, are monitor-only.

Does a successful hook status check prove enforcement works?

No. It proves that the expected configuration is present and readable. You still need to verify host activation and trust, trigger known events, confirm durable delivery, and test whether the host honors a deny.

Why normalize events before writing security rules?

Normalization separates policy meaning from vendor callback names. A rule can target a stable action such as file.write or command.exec, while provenance and capability metadata preserve host-specific limits.

Should agent security hooks fail open or fail closed?

There is no safe universal answer. The choice depends on action risk, evidence confidence, availability requirements, recovery paths, and the host's own semantics. The important step is to document and test failure behavior at every boundary.

References

• Perplexity Numbat repository
• Numbat agent coverage matrix
• Numbat enforcement model
• Numbat deployment guide
• Numbat normalized event model
• Numbat rule authoring guide