Web Bot Auth can verify a signing key, but that alone cannot decide whether an AI crawler should enter a site. Robot access is becoming a five-layer enforcement stack: site preference, request identity, declared purpose, directory classification, and local policy enforced at the edge. The important change is that these layers are starting to connect.
Cloudflare's 2026 controls make the transition visible. A site can treat Search, Agent, and Training traffic differently. BotBase records operators, behaviors, content uses, and verification methods. Bot Preference Sync aligns a site's published robots.txt preference with its dashboard policy. Edge controls then allow or block the request.
This is more useful than another list of crawler user agents. It also creates a new engineering problem: every layer proves something different. Collapsing them into a single trusted bot label creates a false sense of certainty.
The old interface expressed preference, not authorization
The Robots Exclusion Protocol has survived because it is simple. A site publishes groups of user-agent tokens and path rules in /robots.txt; cooperating crawlers fetch and follow them. RFC 9309 standardizes the protocol and states its security boundary clearly: these rules are not access authorization. Sensitive resources still need real application-layer controls.
That boundary matters more as one operator may run several crawlers with different jobs. A search crawler builds an index. A training crawler retains content to improve a model. A user-triggered agent fetches a page to complete a task now. One company name or one generic user agent no longer tells a site enough to make a policy decision.
Cloudflare's Bot Preference Sync addresses one operational failure: the policy enforced at the edge can diverge from the preference published in robots.txt. The service derives managed crawler directives from the site's Search, Agent, and Training settings while preserving existing directives. This reduces configuration drift, but it does not turn robots.txt into a security boundary. The published file remains a preference channel; the edge rule performs enforcement.
Layer 1: site preference
The first layer answers: what does the site owner want?
A useful preference model needs more resolution than allow all automation or block all automation. Cloudflare now separates three major behaviors:
- Search collects or indexes content for later answers.
- Agent performs real-time work on a person's behalf.
- Training absorbs content into model training or fine-tuning.
The classification can be multi-valued. A crawler used for Search and Training should carry both purposes instead of receiving the more convenient label. Cloudflare also extends its managed content signals with content-use levels such as reference or full reproduction. These signals describe the site's normative choice. They do not prove what a crawler will actually do.
Layer 2: request identity
The second layer answers a narrower question: did the holder of a particular key sign this request?
The current IETF Web Bot Auth working-group draft uses HTTP Message Signatures, a Signature-Agent identifier, and an HTTPS key directory. A verifier resolves the directory, obtains public keys, and checks the signed message components. This is stronger than trusting a self-reported User-Agent or treating a cloud IP range as one operator.
The draft also gives the correct trust statement. A resolved signature proves that a holder of a key published by the directory signed the covered message. It says nothing by itself about who operates the agent, whether its behavior is benign, or whether the request is authorized. Those conclusions belong to origin policy and other evidence.
Web Bot Auth is still work in progress. On September 1, 2026, the individual proposal became draft-ietf-webbotauth-httpsig-protocol-00, an active working-group document. It remains an Internet-Draft rather than an RFC.
The transition is already exposing interoperability risk. The working-group draft requires new signers to use the dictionary form of Signature-Agent, while allowing verifiers to accept a legacy string during migration. Google's experimental guide uses the dictionary form and warns that only a subset of participating requests are signed. Cloudflare's implementation guide currently documents the legacy string and says the later dictionary form will fail verification. A production control therefore needs to record protocol version and verifier behavior, not merely a Boolean signed flag.
Layer 3: declared purpose
Identity and intent are separate fields. A valid signature can prove continuity of a signing identity while the same operator runs Search, Agent, Training, SEO, monitoring, or transaction automation.
Cloudflare's 2026 taxonomy asks what the bot does and how it uses retrieved content. The distinction is practical because site owners often welcome discovery while rejecting model training or unrestricted real-time agents. Cloudflare encourages operators with mixed functions to separate crawlers, while also recording every applicable purpose when functions remain combined.
Purpose begins as a declaration. It becomes more credible when three evidence types agree:
- The operator's submitted description and public documentation.
- The authenticated request identity and stable technical identifiers.
- Observed behavior, including paths, rates, referrals, content reuse, and policy compliance.
This is why Web Bot Auth intentionally does not define a vocabulary for bot intent. The IETF working group is standardizing authentication, while platforms and sites retain responsibility for taxonomy and policy. A cryptographic signature can make an identity claim expensive to forge. It cannot make a purpose claim true.
Layer 4: directory classification
A directory connects technical identity to operational context. BotBase for Operators lets an operator submit who runs a bot, what it does, how it uses content, and how it proves identity. Operators can track review status and update records as user agents, IP lists, or signature methods change.
For site owners, BotBase provides searchable classifications and detection IDs that can feed security rules. For Cloudflare, it provides a place to compare a declared purpose with observed conduct and to revoke Verified status when behavior violates expectations.
The directory is a governance layer, not a root of absolute truth. Records combine operator declarations, platform review, technical verification, and continuing behavior. Each source should remain visible. Otherwise a site cannot tell whether a field was self-asserted, cryptographically verified, manually reviewed, or behaviorally inferred.
Directory availability also becomes a runtime dependency. The IETF draft distinguishes verified, invalid, and unverified outcomes. A failed key lookup produces insufficient evidence, which differs from a bad signature. Treating both as malicious will convert an infrastructure outage into a mass block.
Layer 5: local policy and edge enforcement
The final layer answers the decision question: given this identity, declared use, observed behavior, resource, and site preference, what action should the gateway take?
Cloudflare offers Allow, Block on pages with ads, and Block on all pages for its major AI behavior categories. Its bot control documentation says that on September 15, 2026, new domains will receive updated defaults: Training and Agent will be blocked on pages displaying ads, while Search will remain allowed. As of September 10, this is an announced future default, not an active universal rule. It applies to new domains, and customers can choose another policy.
That date illustrates why default policy needs versioning. Two otherwise identical requests may receive different outcomes because the domain joined on a different date, the owner opted out, the page contains ads, or the crawler has several classifications.
Other platforms can map the same identity evidence to another default. AWS WAF says verified Web Bot Auth traffic is automatically allowed by its managed Bot Control behavior, while custom labels support more specific rules. Identity therefore supplies an input. The platform and site supply the decision.
What each layer can prove
| Layer | Strongest supported conclusion | What still requires another layer |
|---|---|---|
| Site preference | The owner published or configured a desired treatment | Whether the crawler will comply and whether the edge acted |
| Web Bot Auth | A key published by a resolved identifier signed covered message components | Operator reputation, user authorization, purpose, and safety |
| Purpose declaration | The operator claims one or more uses | Whether observed conduct matches the claim |
| Directory classification | A platform linked identifiers, review, taxonomy, and behavior records | The site's final local decision |
| Edge enforcement | A particular policy produced an allow, block, challenge, meter, or rate limit | Whether the policy was economically and operationally correct |
This table is the core control model. Trust should accumulate across layers instead of jumping from one verified bit to full access.
A safer decision contract
A production policy can remain compact if it preserves evidence state.
First, define resource classes. Public articles, paid archives, account pages, checkout flows, and destructive actions should have different baselines.
Second, use a three-state identity result: verified, invalid, or unverified. Preserve unsigned traffic as a separate case during early adoption because Google says it does not yet sign every participating request.
Third, treat every applicable purpose as active. When a crawler combines Search and Training, use the more restrictive rule for content that must not enter training unless the platform provides verifiable separation.
Fourth, make local policy authoritative. A directory's Verified status improves provenance. It never grants universal permission.
Fifth, log the complete decision tuple:
timestamp
site-policy-version
resource-class
signature-protocol-version
identity-result
directory-record-version
declared-purposes
observed-behavior-signals
matched-rule
final-action
edge-status
origin-status
This turns a block or allow decision into evidence that can be audited, appealed, reproduced, and changed safely.
The log also enables a reconciliation loop. Periodically compare the published robots.txt, dashboard configuration, generated WAF rules, directory records, and actual responses from test crawlers. A policy that says allow while an upstream rule returns 403 is drift, even if every individual control panel looks healthy. Reconciliation should verify a positive path and a negative path, for example allowing a verified search crawler to a public page while preserving the training block on the same content.
Two migration checklists
For site owners:
- Separate preference publication from enforcement and keep them synchronized.
- Inventory policies by resource and purpose instead of maintaining one AI bot switch.
- Roll out Web Bot Auth in observation mode with verified, invalid, and unverified metrics.
- Test both current dictionary and legacy
Signature-Agentforms while implementations converge. - Review default changes by effective date, domain cohort, and exception state.
- Export the evidence behind allow and block actions, then reconcile configuration with real HTTP outcomes.
For bot operators:
- Publish a stable HTTPS identity and a highly available, cacheable key directory.
- Sign enough message components to bound replay scope, with short expiry and key rotation.
- Declare every behavior and content use; split incompatible purposes where possible.
- Keep directory records, user-agent patterns, and verification methods current.
- Test against each major verifier because draft support is currently version-sensitive.
- Provide a public contact and an appeal path for classification errors.
The infrastructure is the product
The web does not need one perfect bot label. It needs a small set of signals connected by an auditable decision path.
robots.txt remains a cheap preference interface. Web Bot Auth can add request provenance. A purpose taxonomy expresses why automation visits. A maintained directory joins declarations to verification and behavior. Edge policy converts that evidence into a local action.
The architecture becomes trustworthy when every layer keeps its own limits. A signed bot can still violate policy. An unsigned request can still be benign. A directory can be stale. A default can change. The enforcement stack works because those uncertainties stay visible long enough for policy to handle them.
For the complementary detection problem, see Detecting Browser Agents After Fingerprints Fail. That analysis focuses on behavior, signed identity, authorization, and false positives. This article focuses on how identity and declared use become inputs to platform policy and edge execution.
FAQ
What is Web Bot Auth?
Web Bot Auth is an emerging IETF protocol that uses HTTP Message Signatures and a public-key directory to authenticate automated HTTP clients. The current specification is a working-group Internet-Draft, not an RFC.
Can robots.txt block AI agents?
robots.txt publishes rules that cooperating crawlers are expected to follow. RFC 9309 explicitly separates it from access authorization. Reliable blocking requires an enforcement control such as a CDN, WAF, origin rule, or authenticated application policy.
Does a valid Web Bot Auth signature mean the request should be allowed?
No. It proves a defined signing identity for the covered message components. The site must still evaluate purpose, resource, behavior, user authorization where relevant, and local policy.
What is the difference between Search, Agent, and Training crawlers?
Search builds an index for later discovery, Agent acts in real time on behalf of a person, and Training uses content to train or fine-tune models. A crawler may have more than one purpose.
Why keep an unverified state separate from an invalid signature?
Verification can fail because a key directory is unavailable, a key is unknown, or a participating bot did not sign that request. Those conditions provide weaker evidence than a signature that was present and cryptographically invalid.
Are Google and Cloudflare Web Bot Auth implementations fully interoperable today?
The public documentation does not establish full interoperability. Google documents the current dictionary form of Signature-Agent; Cloudflare currently documents the legacy string form and says the dictionary form fails its verifier. The IETF draft provides migration guidance, so production compatibility should be tested rather than assumed.
Will Cloudflare block Agent and Training traffic by default?
Cloudflare has announced that beginning September 15, 2026, new domains will default to blocking Agent and Training categories on pages that display ads while allowing Search. This is a configurable Cloudflare default for new domains, not a universal web rule.
References
- RFC 9309: Robots Exclusion Protocol
- IETF Web Bot Auth Working Group
- IETF working-group draft: HTTP Message Signatures for automated traffic
- Cloudflare: New AI traffic options
- Cloudflare: Block AI Bots documentation
- Cloudflare: BotBase for Operators
- Cloudflare: Bot Preference Sync
- Cloudflare: Web Bot Auth implementation
- Google: Web Bot Auth experimental guide
- AWS WAF Web Bot Auth support