OpenAI, Google, Microsoft & Anthropic Back New AI Scraping Opt Out Standard for Publishers

Major AI Companies Unite Behind Simplified Opt Out Standard

OpenAI, Google, Microsoft, and Anthropic have announced support for a new AI crawling opt-out standard that fundamentally changes how publishers control AI scraping of their content1

1

2

2

. The shared "policy token" signal, proposed by Kevin Kelley, a technical lead for standards at Anthropic, will allow website operators to block AI developers from using their content to train generative AI models with a single robots.txt entry1

1

. This marks a significant shift from the current system where publishers must set AI training preferences individually for each company, creating separate entries for every AI provider and each of their scrapers2

2

.

How the Policy Token Transforms AI Data Collection Controls

The new standard operates within a website's robots.txt file, which has historically signaled which web crawlers can and can't scrape a site for search indexing, archiving, or AI training1

1

. Currently, website operators face a cumbersome process requiring multiple edits to block unauthorized AI crawlers. The policy token simplifies this by enabling publishers to apply their AI training preferences across multiple AI companies simultaneously, rather than managing each relationship separately1

1

. The proposal has been submitted to the Internet Engineering Task Force (IETF), the nonprofit that maintains the Robots Exclusion Protocol, ensuring the standard follows established formatting conventions2

2

.

Microsoft Introduces Granular AI Grounding Controls

Microsoft plans to offer more sophisticated controls beyond basic opt-out functionality. The company will provide website operators the ability to distinguish between AI training and AI Grounding, the process of pointing an AI model to a verified source of information to reduce hallucinations1

1

. This AI Grounding setting addresses situations where an AI model seeks to access a webpage to answer a user question rather than to collect training data2

2

. Microsoft emphasized that websites will still appear in Bing search results regardless of their AI scraping preferences, separating search indexing from AI data collection1

1

. The feature will extend to third-party AI tools that use Microsoft technologies to process prompts2

2

.

Implementation Timeline and Industry Adoption

Anthropichas already adopted the signal, demonstrating immediate commitment to the standard1

1

. OpenAI plans to implement the opt out standard in the coming months, while both Google and Microsoft will add support next year1

1

2

2

. Meta Platforms expressed support for the concept but hasn't committed to implementing it yet2

2

. The setting specifically targets models that perform generative tasks, explicitly excluding those that "perform exclusively non-generative tasks"2

2

.

Why This Standard Matters for Publishers

The announcement addresses growing publisher concerns about the continuing influx of unauthorized AI crawlers scraping their sites and feeding content into AI models1

1

. Publishers have struggled with fragmented control mechanisms, leading some to adopt alternative solutions like the Really Simple Licensing (RSL) standard, which enables them to charge crawlers that visit their sites, or Cloudflare's default blocking of AI crawlers1

1

. This unified approach gives website operators clearer authority over how their content feeds generative AI models while reducing administrative burden. Watch for how smaller AI companies respond to this standard and whether enforcement mechanisms emerge for non-compliant crawlers as the technology landscape evolves.