Google, OpenAI, Microsoft Back Unified AI Scraping Opt-Out Standard Using Robots.Txt

Reviewed byNidhi Govil

6 Sources

Share

Google, OpenAI, Microsoft, and Anthropic commit to honoring a shared opt-out standard for AI scraping. Publishers can now block AI training crawlers with a single robots.txt entry instead of managing separate rules for each company, simplifying content control.

Major AI Companies Unite Behind Shared Opt-Out Standard

Google, Microsoft, OpenAI, and Anthropic have committed to supporting a new website crawling standard that allows publishers to block AI scraping with a single robots.txt entry

1

2

. The shared opt-out signal addresses a critical pain point for publishers who currently must set AI training opt-out preferences separately for each AI company's crawlers

3

. This unified approach represents a significant shift in how content control works across the AI industry.

Source: SiliconANGLE

Source: SiliconANGLE

The proposal comes from Kevin Kelley, Anthropic's technical lead for standards, submitted as part of the Internet Engineering Task Force (IETF) process

1

5

. "Publishers should not have to choose between being discoverable in Bing and deciding how generative models use their work," Krishna Madhavan, principal product manager for Microsoft Bing, stated

3

.

How the Policy Token Works in Robots.Txt

The new system introduces a policy token called "__usage_ai-training" that lives within a website's robots.txt file

2

3

. Publishers add a user-agent section specifically for AI training, which blocks AI crawlers from accessing content for generative AI training purposes

2

. The implementation allows granular control—sites can block entire domains while still permitting access to specific sections like press kits

2

3

.

Source: The Next Web

Source: The Next Web

Crucially, this shared opt-out signal doesn't prevent websites from appearing in search results or being crawled for other purposes

1

2

. Microsoft specifically clarified that websites will still appear in Bing's search results regardless of their AI data collection preferences

1

. Google will interpret the token as blocking its "Google-Extended" user-agent, which crawls websites for materials used to train Gemini models

2

.

Implementation Timeline and Microsoft's AI Grounding Controls

Anthropoc has already started supporting the new directive, while implementation timelines vary across companies

2

3

. OpenAI pledged to implement the signal "in the coming months," with existing crawler controls like GPTBot continuing to apply

1

3

. Google and Microsoft will start supporting it in early 2027, with Microsoft's specific date set for February 1, 2027

2

3

.

Microsoft is introducing additional granular controls beyond the standard opt-out. Publishers can separately manage AI Grounding—the process where an AI model accesses a webpage to answer user questions rather than collect training data

1

5

. One option allows Microsoft's own products like Copilot while blocking grounding services Microsoft sells to other companies

3

. Microsoft emphasized that blocking AI Grounding "will significantly reduce your content's visibility in AI-generated experiences"

3

.

What Publishers Need to Know About Limitations

The new directive cannot undo past AI scraping that companies have already completed

2

. If AI crawlers previously accessed content, those companies already possess that data—the opt-out standard only prevents future scraping

2

. This limitation matters for publishers concerned about content already incorporated into existing AI models.

Microsoft made clear that these new preferences "do not establish licensing or payment arrangements"

3

. The IETF draft specifies that the policy token states a preference about content use rather than issuing an order about crawling

3

. Private agreements between publishers and AI companies can override the opt-out signal

3

. Additionally, the draft remains an individual submission without formal IETF endorsement

3

.

Industry Context and Future Implications

This development arrives as publishers grapple with the continuing influx of AI crawlers scraping their sites

1

. Some organizations have attempted alternative solutions—Cloudflare implemented default blocking of AI crawlers and introduced the Really Simple Licensing (RSL) standard allowing publishers to charge crawlers

1

. Cloudflare's chief executive warned that bot traffic could reach 1,000 times human web traffic

3

.

Source: Inc.

Source: Inc.

Google noted the proposal aligns with the EU's Code of Practice for general-purpose AI under the AI Act, which recognizes robots.txt as a primary method for publishers to state machine-readable preferences

3

. The draft acknowledges input from major European publishers including the Financial Times, BBC, Bertelsmann, and the European Publishers Council

3

. Meta communications head Andy Stone voiced support for a common signal, though Meta hasn't committed to implementing it

3

5

.

Watch for how this standard affects the broader debate over AI training data and whether it becomes a stepping stone toward licensing frameworks or remains purely voluntary. Publishers should also monitor whether other AI companies beyond the initial four adopt this shared signal, and how enforcement mechanisms develop as implementation proceeds through 2027.

© 2026 TheOutpost.AI All rights reserved