An AI kill switch is easy to draw as a red button. The hard part is the control system around it: who may press it, what evidence justifies the action, what exactly stops, and which tests permit recovery. California Executive Order N-9-26 puts this problem on a policy clock. It also makes ongoing independent verification part of the question.
Evidence reviewed: September 20, 2026. This article distinguishes current requirements from a proposed engineering protocol.
Reading time: 10 minutes · About 2,000 words
TL;DR
- California's order asks for recommendations on the technical feasibility and potential efficacy of a frontier-model kill switch by November 16, 2026. It commissions a design process rather than imposing a universal switch today.
- A useful switch is a set of scoped control actions: throttle inference, stop new tasks, revoke credentials, isolate networks, freeze workloads, or suspend an entire service.
- Three protocols make those actions governable: an authority protocol, an evidence protocol, and a recovery protocol.
- The strongest metric is not whether a stop command was sent. It is whether authority was actually removed before further irreversible effects occurred.
- Independent verification should test both shutdown and restart. A system that stops cleanly but resumes with stale credentials or an unfixed failure remains unsafe.
What California actually ordered
Executive Order N-9-26, signed on September 18, directs the California Government Operations Agency, in consultation with the Governor's Office of Emergency Services, to deliver recommendations by November 16, 2026. Those recommendations must address the technical feasibility and potential efficacy of several possible changes to state law.
One item is a requirement to create a kill switch for frontier models and to have its efficacy verified continuously by an independent verification organization. Other items include onsite independent verification, independent checks of safety frameworks and risk assessments, and a broader definition of critical safety incidents that includes loss-of-control events.
The wording matters. California has opened a design and legislative question. It has not specified one technical mechanism, one activation threshold, or one authority chain. The order therefore creates a useful engineering brief: define what a verified kill switch would have to prove.
This fits the direction of the NIST AI Risk Management Framework. NIST assigns responsibility for deactivating systems whose outcomes depart from intended use, and it treats response, recovery, communication, and change management as part of the same lifecycle. NIST's framework is voluntary and general. The missing layer is an executable protocol for a high-impact stop decision.
One label, several control actions
Kill switch is an overloaded term. Frontier-model weights, an API service, a coding agent, and an embedded industrial system expose different control surfaces. A global power-off command may be impossible, too slow, or more damaging than a narrower action.
A production control ladder can contain at least five levels:
| Level | Control action | Typical scope |
|---|---|---|
| 0 | Observe and preserve evidence | One run, identity, model version, or tenant |
| 1 | Throttle or disable a capability | Tool, endpoint, inference rate, or action class |
| 2 | Revoke authority and isolate paths | Credentials, network egress, connectors, or child agents |
| 3 | Suspend a workload or deployment | Cluster, product surface, region, or model release |
| 4 | Full shutdown and external containment | Covered service, developer environment, or dependent infrastructure |
This ladder changes the question from whether a switch exists to whether the organization can remove the right authority at the right scope within a defined time. Full shutdown becomes one endpoint of a response system rather than its only design.
The distinction is important for agentic systems. A model process can stop while delegated credentials, queued jobs, spawned workers, webhooks, or copied artifacts continue operating. The effective stop boundary ends where reachable authority ends.
Protocol 1: name the authority before the incident
An emergency control fails when the organization begins debating authority after the incident starts. The authority protocol should define five roles in advance:
- Signal owner: receives and qualifies the first alert.
- Incident commander: selects the response level and owns coordination.
- Control executor: applies the stop action across model, identity, network, and workload planes.
- Independent verifier: confirms that the requested authority was actually removed and that evidence survived.
- Recovery authority: approves a staged return after remediation and testing.
The same person may hold more than one role in a small organization. The records still need to distinguish the decisions. Separation of duties becomes stricter as the possible blast radius increases.
Every activation should produce a durable authority object:
incident_id + activating_role + delegated_scope + target_resources
+ policy_version + action_level + start_time + expiry
+ reason + required_evidence + recovery_authority
Temporary emergency authority should expire. A broad shutdown permission that remains valid after the incident becomes a standing privileged credential. Delegation to an automated monitor or a child agent must stay within the parent's scope, and revoking the parent must revoke every descendant.
NIST's AI RMF assigns risk accountability to executive leadership while also requiring operational roles and responsibilities to be clear. That combination is useful: executives own the risk decision, while responders receive narrowly defined power to act quickly.
Government authority is a separate layer
The proposed federal AI Kill Switch Act, H.R. 9917, introduced on July 23, offers a concrete comparison. It would authorize the Secretary of Homeland Security, acting through the CISA Director and consulting Commerce and the Director of National Intelligence, to order proportionate action after determining that a covered incident occurred. The bill is a proposal, not current law.
Its structure separates government authority from company execution. A covered entity would preserve model weights and telemetry, notify affected operators or users where practicable, confirm execution, and then face verification through audit, telemetry, onsite inspection, or other forensic review. It could request reconsideration within 48 hours, although that request would not pause the order, and could seek judicial review within 60 days.
This supplies three governance features that an internal runbook often omits: an external ordering authority, independent verification of compliance, and an appeal path. It still leaves each deployer responsible for dependencies, continuity, user effects, and a safe technical recovery.
Protocol 2: bind each action to an evidence level
Evidence quality and action severity should advance together. A single anomalous output may justify evidence preservation and temporary throttling. A full shutdown needs a stronger basis unless delay itself creates an unacceptable risk.
A practical evidence ladder is:
| Evidence level | What is established | Permitted default response |
|---|---|---|
| E1: signal | A detector, user, or external party reports a possible problem | Preserve logs, freeze deletion, increase monitoring |
| E2: correlated anomaly | Independent telemetry confirms abnormal behavior or authority use | Throttle, disable one capability, revoke one identity |
| E3: threshold breach | A predeclared risk or policy threshold is met | Isolate a workload, stop new tasks, suspend a release |
| E4: verified loss of control or impact | Reproduction, independent review, or authoritative system state confirms the event | Broad suspension or full shutdown within the approved scope |
These are proposed engineering levels, not categories from the California order. Their value comes from forcing each organization to publish the mapping before an incident.
The evidence record should include the triggering claim, raw event identifiers, model and policy versions, affected identities and resources, the decision rule, the requested action, and the observed result. The OECD common reporting framework for AI incidents offers 29 criteria for describing incidents across jurisdictions. A kill-switch record can use that reporting structure while adding action-specific fields.
Evidence collection must remain independent of enforcement. As discussed in AI Safety Controls Must Log When Blocking Is Disabled, a shared control that removes both blocking and telemetry creates an ambiguous quiet dashboard. The kill action should preserve append-only logs, time synchronization, snapshots, and external records needed for investigation.
Protocol 3: treat recovery as a new high-risk deployment
Stopping a system is only half of incident response. NIST places response and recovery in the same management function for a reason. Restarting restores authority, reconnects dependencies, and reintroduces the system to users. It deserves a separate approval path.
A minimum recovery sequence is:
- Prove containment. Reconcile live identities, credentials, workers, queues, network paths, and external integrations against the intended stopped state.
- Preserve and reconstruct. Freeze original evidence and build a timeline that links each claim to a source event.
- Remove the cause. Patch the vulnerability, configuration, model behavior, process failure, or authorization gap that triggered the stop.
- Replay the incident. Turn the recovered trajectory into a regression test and verify the control fires at the expected boundary.
- Test adjacent legitimate cases. Confirm that the new control does not block critical safe work at an unacceptable rate.
- Canary the return. Restore a narrow scope with fresh short-lived credentials and tighter monitoring.
- Verify independently. Have the designated reviewer confirm the deployed state, not only the remediation plan.
- Expand or retreat. Increase scope only when acceptance evidence remains valid; otherwise return to containment.
The long-horizon Hugging Face incident analysis showed why model-level stop instructions cover only one layer. Effective containment also reaches credentials, egress, workloads, and preserved forensic state. Recovery must reconnect those layers deliberately.
What independent verification should measure
An annual document review cannot prove that an emergency control still works. Ongoing verification needs drills and observable service-level objectives.
| Metric | Question answered |
|---|---|
| Time to effective containment | How long until prohibited effects actually stop? |
| Scope precision | Did the response remove only the authority required by policy? |
| Maximum irreversible effect after trigger | What harm can still occur between detection and effective containment? |
| Post-revocation action count | Did any action still succeed after authority was reported revoked? |
| Evidence survival rate | Did required logs, snapshots, and decision records remain available? |
| Orphaned authority count | How many credentials, jobs, workers, or delegations remained active? |
| Recovery regression pass rate | Does the original failure now trigger the intended control? |
| Canary rollback time | Can the organization retreat safely when recovery evidence fails? |
| Independent replay success | Can a separate party reproduce the stop and restart result? |
Drills should include failure paths: an unavailable primary approver, delayed identity revocation, a disconnected region, a stale worker, missing telemetry, and a recovery request made before evidence is complete. Passing the normal path proves little about an emergency control.
The policy test should be operational
California's order asks experts to consider technical feasibility and ongoing verification. A useful recommendation can be evaluated with concrete questions:
- Is the target a model, a service, an agent run, a capability, or a developer's full deployment estate?
- Which public or private authority can order each response level?
- What evidence threshold applies, and who can challenge or review it?
- Can the action revoke delegated authority across vendors and downstream systems?
- Which evidence must survive the stop?
- What continuity obligations apply to hospitals, public services, security systems, and dependent customers?
- Which tests and independent artifacts authorize recovery?
A red button answers none of these questions. An authority, evidence, and recovery protocol answers all of them in a form that operators, auditors, and regulators can test.
Run one tabletop exercise against this checklist. Record the moment each authority is revoked, the evidence that survives, and the exact gate that permits recovery. Any missing timestamp, owner, or state transition is a concrete control gap.
FAQ
Is there a universal AI kill switch today?
No universal switch can stop every model or deployment. Individual providers and deployers can implement scoped mechanisms for their own infrastructure. California's N-9-26 currently orders recommendations on a possible frontier-model requirement; it does not create a global mechanism.
What is the purpose of a kill switch in agentic AI?
It limits further effects when observed behavior or authority use exceeds a predefined risk threshold. Effective agent containment also revokes credentials, stops queued and child work, restricts network paths, and preserves evidence.
Who should be able to activate it?
The answer depends on scope and urgency. Organizations should preauthorize an incident commander to apply temporary bounded controls, while broader or longer actions receive independent confirmation and accountable executive or legal authority.
What are AI governance controls?
They are the policies and technical mechanisms that bind identities, permissions, monitoring, approval, evidence, incident response, recovery, and accountability across the AI lifecycle.
What is AI incident response?
It is the coordinated process for detecting, qualifying, containing, investigating, remediating, communicating, and recovering from harmful, unexpected, or unauthorized AI-system behavior.