Google, Microsoft, OpenAI Back Shared Robots.txt Standard for AI Training Opt-Out
Google, Microsoft, OpenAI, and Anthropic have pledged to honor a unified robots.txt signal that allows websites to opt out of AI training data collection. But the commitment comes with significant caveats: it doesn't stop crawlers from accessing content, applies only to future scraping, and won't be fully implemented until 2027.