Cloudflare splits AI bot controls into search, training, agent

Key Takeaways

• Cloudflare now offers three separate toggles for AI bots: Search, Training, and Agent crawlers
• Starting September 15, 2026, Training and Agent bots will be blocked by default on ad-supported pages
• Enterprise customers get BotBase, a searchable database of all known bots with classification details

Cloudflare is replacing its blanket AI bot blocker with a more nuanced system. Instead of a single on-off switch, site owners can now control three categories of AI crawlers independently: Search (for indexing), Training (for model development), and Agent (for bots acting on behalf of users, like ChatGPT). The update rolls out to all customers, including those on the free plan.

This shift matters because AI crawling has exploded. Cloudflare CEO Matthew Prince noted in June 2026 that bot traffic surpassed human traffic on the internet for the first time, hitting a milestone he originally expected in late 2027. The one-click block Cloudflare introduced in July 2024 was a blunt tool. It stopped everything. Now publishers can selectively welcome AI search crawlers while blocking training scrapers.

How the three-category system works

The new dashboard presents three toggles. Search covers crawlers that index pages for search engines. Training covers bots that collect data for AI model training. Agent covers autonomous bots that browse on behalf of users.

Multi-purpose crawlers pose an interesting problem. Googlebot, for example, both indexes pages for search and feeds data to Google's AI models. Cloudflare's solution: apply the strictest rule. If you allow Search but block Training, Googlebot gets blocked. This is a direct response to Prince's public criticism of Google for bundling its search and AI crawlers together.

Default blocks for ad-supported pages

Starting September 15, 2026, Cloudflare will block Training and Agent bots by default on pages that display ads. The logic: ads signal that a site is monetizing human visitors, so AI bots that scrape content without driving revenue work against the publisher's interests. Search crawlers will still be allowed by default, since they can send human traffic.

Publishers can override these defaults. But the change signals Cloudflare's view that content creators should be compensated for their work, or at least have the power to withhold it from AI training pipelines.

Enterprise customers get BotBase

For enterprise customers, Cloudflare is launching BotBase, a searchable database of known bots built into the dashboard. BotBase shows how each bot is classified and how it uses content, whether it simply links to pages or reproduces them in full. This gives security and product teams granular visibility into what's hitting their infrastructure.

Cloudflare is also tightening its verified bot program. Previously, verified bots passed through automatically. Now, a bot's category determines access. The "Verified" label alone won't open doors. Bot operators must prove they identify themselves honestly and don't abuse their access.

Why this matters for AI builders

If you're building AI products that crawl the web, Cloudflare just raised the bar. With roughly 20% of the internet behind Cloudflare's services, your crawler's behavior and transparency directly affect how much of the web you can reach. The days of a single undifferentiated bot are numbered.

Prince has pushed bot operators to split their crawlers by purpose. This isn't just a technical preference. It's a business requirement. If your training crawler gets lumped with your search crawler, strict publishers will block both.

The broader context

This update lands amid ongoing tension between AI companies and publishers. Major lawsuits, including the New York Times suit against OpenAI, have pushed the industry toward more explicit consent mechanisms. Cloudflare's granular controls give publishers a middle ground: they can participate in AI-powered search while opting out of training data collection.

When Cloudflare launched its one-click block in July 2024, 85.2% of site operators who used the feature chose to block AI training bots. That number suggests most publishers want control, not a blanket ban. The new three-category system gives them exactly that.

Frequently Asked Questions

When do the new Cloudflare AI bot controls take effect?

The controls are available now for all customers. The default blocking of Training and Agent bots on ad-supported pages begins September 15, 2026.

Does Cloudflare's free plan include the new AI bot controls?

Yes. The three-category bot controls are available to all Cloudflare customers, including those on the free plan.

How does Cloudflare handle crawlers that serve multiple purposes?

Multi-purpose crawlers like Googlebot are subject to the strictest rule. If you allow Search but block Training, the crawler gets blocked.

What is BotBase and who can access it?

BotBase is a searchable database of known bots showing their classification and content usage. It's available to enterprise customers through the Cloudflare dashboard.

What happens to verified bots under the new system?

Verification alone no longer grants automatic access. A bot's category determines whether it gets through, and operators must prove honest identification and appropriate use.

Source: The Decoder / Matthias Bastian