On July 1, 2026, Cloudflare replaced the single "block AI bots" switch with three separate controls, one per purpose. On September 15, 2026 the defaults for those controls changed. For a site on a newly onboarded domain, AI crawlers that train models or act as agents are now turned away from any page that shows ads, while crawlers that only build a search index still get in. Cloudflare sits in front of a large share of the web, so this is the biggest change to default AI access since robots.txt rules for GPTBot became common in 2023.
The Three Categories
| Category | Cloudflare's definition | Default on ad pages, new domains | Default elsewhere |
|---|---|---|---|
| Search | Collects or indexes content so it can answer questions about it later | Allowed | Allowed |
| Training | Takes content to train or fine-tune a model | Blocked | Allowed |
| Agent | Acts, usually in real time, on a person's behalf to get something done now | Blocked | Allowed |
Cloudflare's reasoning is short: an ad signals that the site owner meant a person to land on that page. A training crawler or an agent reading the page means no person sees the ad.
Who It Applies To
- New domains onboarding to Cloudflare from September 15, 2026 get the new defaults automatically.
- All existing customers, Free plan included, can already set the three controls themselves. The controls went live on July 1.
- Coverage of which existing zones were switched differs between reports. Some trade coverage says Free-plan zones were included and that existing customers had until September 15 to opt out. Check the Security settings of each zone yourself instead of assuming either way.
The Multi-Purpose Crawler Problem
The rule with the widest effect is about crawlers that do more than one job. Cloudflare applies the most restrictive rule that fits, and its announcement names Googlebot, Applebot and BingBot as multi-purpose crawlers that will be blocked where Training is blocked. The effect is that a page can lose its main search crawler because it chose not to allow training. TechCrunch reported this as a deadline that pushes AI companies to split their crawlers by purpose by September 15. A crawler that identifies only as search is allowed through.
For site owners, a well-meant "no training" setting on ad pages can now take those pages out of classic search and out of AI answers at the same time. How often that happens depends on how each crawler operator declares its purpose, and that is still changing.
What Else Shipped Alongside It
- Content Signals "use" field (testing): declares how content may be reused, at three levels: immediate (no storage or reuse), reference (index, excerpt, link back) and full (summarize and reproduce). Cloudflare describes these as a preference, not a block.
- BotBase: a searchable database of verified bots with their classifications, for Enterprise Bot Management customers.
- Pay Per Use: TechCrunch reports that the Pay Per Crawl marketplace is becoming Pay Per Use, which pays publishers when their content is used in a partner's AI answers rather than per fetch. Ceramic.ai and You.com are named as pilots. See the state of Pay Per Crawl for the marketplace itself.
Brand Visibility Implications
For a brand site the question is simple: do your commercial pages show ads? Most brand and product pages don't, so the new defaults leave them alone. Publishers, review sites, affiliate sites and comparison sites usually do, and those are the third-party sources AI answers cite most when recommending brands. If those sources become harder for agents and some multi-purpose crawlers to read, AI answers will rely more on whatever stays reachable: your own site, sources behind licensing deals, and pages without ads.
Agents are the bigger issue for commerce. A shopping or research agent blocked on an ad-supported review page goes looking elsewhere, and "elsewhere" is often the brand's own product page. That makes agent-readable product pages more valuable. The crawl-to-referral imbalance behind the change is covered in AI crawl-to-click ratios.
Methodology
Figures on this page are drawn from primary announcements and documentation wherever they exist, with established trade and legal press used only to corroborate or to fill gaps a primary source leaves open. Sources: Cloudflare blog, July 1 2026, Cloudflare AI Crawl Control docs, TechCrunch. Where outlets disagree on a date or number, the page gives the range or names the disagreement rather than choosing one. Status as of September 2026; this area changes monthly, so re-check before quoting.
How Presenc AI Helps
Presenc AI logs every AI crawler and agent request to a site, with purpose-level classification. Teams can see which bots stopped arriving after a Cloudflare setting changed, and whether their brand mentions in AI answers moved as a result.