6 Sources
[1]
The biggest names in AI are backing a new way to opt out of scraping
OpenAI, Google, Microsoft, and Anthropic say they will adopt a new standard that publishers can use to opt out of being scraped by AI, as reported earlier by Search Engine Roundtable. With this control, publishers can apply their AI training preferences across multiple AI companies, instead of
[2]
Google joins OpenAI and Anthropic in promising new scraping controls
The new robots.txt directive will prevent the companies from scraping content for generative AI training. Google, Anthropic, OpenAI and Microsoft have committed to honoring a new robots.txt directive, which will allow publishers to specify their preferences when it comes to the scraping of their
[3]
AI training opt-out: publishers 'should not have to choose'
Anthropic proposed the shared AI training opt-out at the IETF and has started building support. Google and Microsoft start in 2027, and Microsoft says it creates no licensing or payment deals. Google, Microsoft, OpenAI and Anthropic will honour one shared AI training opt-out in robots.txt. It lets
[4]
'We believe this is a practical step toward an open web': Don't want AI companies training on your website? OpenAI, Google and Microsoft are backing a new way to opt out
OpenAI, Google, Microsoft and Anthropic have agreed to support a new way for website owners to stop their content being used to train AI models. The companies are backing a shared opt-out signal in a website's robots.txt file, which tells AI crawlers whether they're allowed to use its content for
[5]
Google, Anthropic and others to support new AI crawling opt-out standard
Google LLC, Microsoft Corp., Anthropic PBC and OpenAI Group PBC today announced plans to support an upcoming website crawling standard. The technology will enable website operators to block artificial intelligence developers from using their content to train their models. Many websites contain a
[6]
The 4 AI Giants Just Agreed to Let Websites Opt Out. They've Already Taken the Data
Agreeing to honor a single robots.txt signal doesn't solve the bigger problem of scraping the entire internet already. On Friday, four of the biggest AI companies in the world said they would support a common "policy token," to let websites determine whether their content is used to train AI
Share
Copy Link
Google, OpenAI, Microsoft, and Anthropic commit to honoring a shared opt-out standard for AI scraping. Publishers can now block AI training crawlers with a single robots.txt entry instead of managing separate rules for each company, simplifying content control.
Google, Microsoft, OpenAI, and Anthropic have committed to supporting a new website crawling standard that allows publishers to block AI scraping with a single robots.txt entry
1
2
. The shared opt-out signal addresses a critical pain point for publishers who currently must set AI training opt-out preferences separately for each AI company's crawlers3
. This unified approach represents a significant shift in how content control works across the AI industry.
Source: SiliconANGLE
The proposal comes from Kevin Kelley, Anthropic's technical lead for standards, submitted as part of the Internet Engineering Task Force (IETF) process
1
5
. "Publishers should not have to choose between being discoverable in Bing and deciding how generative models use their work," Krishna Madhavan, principal product manager for Microsoft Bing, stated3
.The new system introduces a policy token called "__usage_ai-training" that lives within a website's robots.txt file
2
3
. Publishers add a user-agent section specifically for AI training, which blocks AI crawlers from accessing content for generative AI training purposes2
. The implementation allows granular control—sites can block entire domains while still permitting access to specific sections like press kits2
3
.
Source: The Next Web
Crucially, this shared opt-out signal doesn't prevent websites from appearing in search results or being crawled for other purposes
1
2
. Microsoft specifically clarified that websites will still appear in Bing's search results regardless of their AI data collection preferences1
. Google will interpret the token as blocking its "Google-Extended" user-agent, which crawls websites for materials used to train Gemini models2
.Anthropoc has already started supporting the new directive, while implementation timelines vary across companies
2
3
. OpenAI pledged to implement the signal "in the coming months," with existing crawler controls like GPTBot continuing to apply1
3
. Google and Microsoft will start supporting it in early 2027, with Microsoft's specific date set for February 1, 20272
3
.Microsoft is introducing additional granular controls beyond the standard opt-out. Publishers can separately manage AI Grounding—the process where an AI model accesses a webpage to answer user questions rather than collect training data
1
5
. One option allows Microsoft's own products like Copilot while blocking grounding services Microsoft sells to other companies3
. Microsoft emphasized that blocking AI Grounding "will significantly reduce your content's visibility in AI-generated experiences"3
.Related Stories
The new directive cannot undo past AI scraping that companies have already completed
2
. If AI crawlers previously accessed content, those companies already possess that data—the opt-out standard only prevents future scraping2
. This limitation matters for publishers concerned about content already incorporated into existing AI models.Microsoft made clear that these new preferences "do not establish licensing or payment arrangements"
3
. The IETF draft specifies that the policy token states a preference about content use rather than issuing an order about crawling3
. Private agreements between publishers and AI companies can override the opt-out signal3
. Additionally, the draft remains an individual submission without formal IETF endorsement3
.This development arrives as publishers grapple with the continuing influx of AI crawlers scraping their sites
1
. Some organizations have attempted alternative solutions—Cloudflare implemented default blocking of AI crawlers and introduced the Really Simple Licensing (RSL) standard allowing publishers to charge crawlers1
. Cloudflare's chief executive warned that bot traffic could reach 1,000 times human web traffic3
.
Source: Inc.
Google noted the proposal aligns with the EU's Code of Practice for general-purpose AI under the AI Act, which recognizes robots.txt as a primary method for publishers to state machine-readable preferences
3
. The draft acknowledges input from major European publishers including the Financial Times, BBC, Bertelsmann, and the European Publishers Council3
. Meta communications head Andy Stone voiced support for a common signal, though Meta hasn't committed to implementing it3
5
.Watch for how this standard affects the broader debate over AI training data and whether it becomes a stepping stone toward licensing frameworks or remains purely voluntary. Publishers should also monitor whether other AI companies beyond the initial four adopt this shared signal, and how enforcement mechanisms develop as implementation proceeds through 2027.
Summarized by
Navi
[3]
09 Nov 2026•Policy and Regulation

09 Dec 2025•Policy and Regulation

16 Sept 2026•Technology
