Cloudflare Launches Disallow AI Training Setting to Block AI Scraping While Preserving Search Rankings

Cloudflare Introduces Disallow AI Training Setting for Website Owner Control

Cloudflare has launched its Disallow AI Training setting alongside a new Accountable designation for AI crawling, addressing a critical challenge facing website owners in the age of artificial intelligence. The announcement marks a significant shift in how sites can manage AI training without sacrificing search visibility1

1

2

2

. The problem stems from mixed-use crawlers, which collect content for both search indexing and AI training simultaneously. Until now, blocking these crawlers meant losing search rankings entirely. Mixed-use crawlers now represent 36.6% of verified crawler traffic on Cloudflare's network, making them the single largest category2

2

.

Apple, Google, and Microsoft Meet Accountable Designation Criteria

Cloudflare established four clear requirements for its Accountable designation for AI crawling. Operators must provide site owners a way to opt out of AI training through robots.txt or comparable standards, allow opt-outs from AI-generated search summaries, deliver URL-level visibility into content usage, and publicly confirm that training opt-outs won't affect traditional search rankings2

2

. Apple, Google, and Microsoft have either met these criteria or committed to doing so within specified timelines. Google, for instance, offers Google-Extended, enabling sites to block AI scrapers from training while maintaining search presence2

2

. Other leading AI companies operate separate crawlers for search and training, allowing Cloudflare to block training crawlers without impacting search functionality.

New Controls Replace Block AI Bots Switch with Granular Options

Cloudflare is replacing its previous "Block AI Bots" switch with three independent controls covering search, AI training, and AI agents separately2

2

. New websites joining Cloudflare will receive recommended settings tailored to their business model. Sites carrying advertising will see search crawling enabled, AI training disallowed, and AI agents blocked on ad-carrying pages. All other sites default to allowing search, AI training, and AI agents, reflecting how most non-ad-supported websites currently operate2

2

. The company is also introducing Bot Preference Sync, replacing its "Managed Robots.txt" feature, which applies crawling preferences across all supported crawlers automatically.

Website Owners Gain Meaningful Control Over Content Usage

Matthew Prince, Cloudflare Co-Founder and CEO, emphasized the importance of preserving internet openness while giving website owners meaningful control. "This is how we make the internet better: preserving the openness that makes search valuable while giving the people and businesses behind the web meaningful control over how their work is used," Prince stated1

1

. The data reveals a clear preference among site owners: fewer than 1% block search crawlers, while 17% restrict AI training2

2

. Bryan Becker, director of product, appsec integrity and trust solutions at Cloudflare, noted that discussions with crawler operators since July have yielded encouraging responses, with almost all agreeing that site owners deserve control and transparency1

1

.

Bot Activity Management Challenges Persist Across Digital Ecosystem

Managing bot activity remains a significant challenge for organizations. According to PYMNTS Intelligence research, nearly 90% of organizations report difficulties managing bot activity1

1

. As AI-powered agents interact with platforms alongside human users, weak or outdated controls allow malicious bots to penetrate defenses while legitimate users face incorrect blocking. Every operator designated as Accountable must provide site owners a way to opt out of AI summaries, currently a yes-or-no choice set individually with each operator2

2

.

Future Developments Include AI Summary Controls and Open Standards

Cloudflare aims to enhance website owner control by early next year, planning to let site owners determine how much content gets included in AI summaries from a single interface2

2

. The company actively participates in open standards development at the Internet Engineering Task Force (IETF), including work on "ai-prefs," a specification that would enable any website to express AI access preferences in a standardized, portable format. These controls are designed to work alongside emerging standards, ensuring compatibility as the digital ecosystem evolves. Prince indicated Cloudflare looks forward to continued collaborative engagement with companies like Apple, Google and Microsoft to build a healthy ecosystem1

1

.