Google, Microsoft, OpenAI Back Shared Robots.txt Standard for AI Training Opt-Out

Major AI Players Commit to Shared Robots.txt Signal

Google, Microsoft, OpenAI, and Anthropic have announced they will honor a single robots.txt entry that lets website operators opt out of AI model training1

1

2

2

. The shared robots.txt signal, designated as "__usage_ai-training," aims to simplify the process for webmasters who previously had to write separate rules for each company's crawler, and sometimes multiple crawlers operated by the same company1

1

. This represents genuine progress for overworked site administrators managing AI data scraping permissions.

Implementation Timeline Stretches to 2027

The rollout timeline reveals a fragmented approach to this website crawling standard. Anthropic is the only company that has already begun implementing the policy token2

2

. OpenAI promises implementation "in the coming months" without providing a specific date1

1

. Microsoft will begin honoring the signal from February 1, 2027, while Google is targeting sometime in 20272

2

. This staggered implementation means websites seeking immediate protection from AI training data collection will need to continue using existing per-bot controls.

The Mechanism Behind the AI Training Opt-Out

The mechanism originates from an Internet-Draft published by Kevin Kelley of Anthropic, which introduces the "__usage_ai-training" policy token in the User-Agent line of a robots.txt group2

2

. The Internet Engineering Task Force's AI Preferences working group defines training as the process of changing a generative model's learned parameters, specifically excluding models used purely for classification or ranking tasks1

1

. OpenAI describes what it will honor as "foundational generative AI training," which appears narrower than the draft's definition1

1

.

Critical Limitations of Opting-Out of Internet Scraping

The commitment carries significant limitations that temper its effectiveness. The draft explicitly states that "this policy token is not a directive for crawling behavior"1

1

. Bots can continue fetching every page, and the signal merely permits a crawler gathering content only for training to skip pages it cannot use2

2

. Microsoft states plainly: "These settings govern use of content, not crawl access"1

1

. The signal defaults to allowing training and does nothing about content already collected by AI developers1

1

.

Inconsistent Implementation Across Companies

Each company interprets the shared signal differently, creating confusion for website operators. Google plans to treat sites using the shared token to opt out without separately specifying Google-Extended as though they had disallowed Google-Extended2

2

. This creates an unintended consequence: because Google-Extended also governs grounding in Gemini Apps, a site wanting only to stop AI model training could lose citations and clicks in Gemini answers1

1

. Google's documentation describes no way to separate these two functions.

Microsoft Takes Different Approach to Content Usage

Microsoft adopts the shared line for training but adds a separate page-level AI Grounding preference1

1

. Its DisallowExtended option keeps content available to Microsoft's own products, including Copilot, while excluding grounding services it provides to other businesses2

2

. This dual-track approach offers more granular control but adds complexity for webmasters managing multiple platforms.

Meta's Uncommitted Position Raises Questions

Meta has expressed support for the common signal but has not committed to honoring it2

2

. Meta's VP of communications, Andy Stone, called it "an important step," yet the company's actions suggest hesitation1

1

. Meta pushed back to November 20 revised commercial terms that would allow it to crawl websites linked to business accounts, removing training from listed uses after backlash1

1

. Sam Quigley of the News/Media Alliance noted that with training removed but not expressly banned, "It's still potentially on the table"2

2

.

Questionable Legal Status of IETF Draft

Calling this a standard may be premature. The IETF's own tracker labels the document an individual draft that the IETF has not endorsed and that has no formal standing1

1

. The draft will lapse on May 3, 2027 unless revised2

2

. This uncertain status means the mechanism lacks the formal backing typically associated with internet standards, and compliance remains entirely voluntary.

Legal Battles Continue Despite New Signal

The announcement arrives amid ongoing legal disputes over AI data scraping. The New York Times lawsuit against OpenAI and Microsoft, filed in December 2023, continues as both companies invoke fair use defenses2

2

. Anthropic agreed to pay $1.5 billion to settle authors' claims over pirated books used for AI training2

2

. Penske Media sued Google over AI Overviews in 20252

2

. These cases highlight that the voluntary signal does not resolve fundamental legal questions about whether scraping content for AI model training constitutes fair use.

Alternative Protection Measures Emerge

Cloudflare has taken more aggressive action, blocking AI crawlers by default for new customers since July 2025 and allowing participating sites to charge crawlers per visit2

2

. This approach demonstrates that third-party services may offer more immediate and enforceable protection than voluntary industry commitments. Google noted the shared token aligns with the European Union's code of practice for general-purpose AI, which treats robots.txt as publishers' primary machine-readable opt-out2

2

.