Apple's AI Ambitions Face Resistance from Major Publishers

Major Sites Are Saying No to Apple's AI Scraping

This summer, Apple gave websites more control over whether the company could train its AI models on their data. Major publishers and platforms like The New York Times and Facebook have already opted out. Less than three months after Apple quietly debuted a tool for publishers to opt out of its AI training, a number of prominent news outlets and social platforms have taken the company up on it. WIRED can confirm that Facebook, Instagram, Craigslist, Tumblr, The New York Times, The Financial Times, The Atlantic, Vox Media, the USA Today network, and WIRED's parent company, Condé Nast, are among the many organizations opting to exclude their data from Apple's AI training. The cold reception reflects a significant shift in both the perception and use of the robotic crawlers that have trawled the web for decades. Now that these bots play a key role in collecting AI training data, they've become a conflict zone over intellectual property and the future of the web. This new tool, Applebot-Extended, is an extension to Apple's web-crawling bot that specifically lets website owners tell Apple not to use their data for AI training. (Apple calls this "controlling data usage" in a blog post explaining how it works.) The original Applebot, announced in 2015, initially crawled the internet to power Apple's search products like Siri and Spotlight. Recently, though, Applebot's purpose has expanded: The data it collects can also be used to train the foundational models Apple created for its AI efforts. Applebot-Extended is a way to respect publishers' rights, says Apple spokesperson Nadine Haija. It doesn't actually stop the original Applebot from crawling the website -- which would then impact how that website's content appeared in Apple search products -- but instead prevents that data from being used to train Apple's large language models and other generative AI projects. It is, in essence, a bot to customize how another bot works. Publishers can block Applebot-Extended by updating a text file on their websites known as the Robots Exclusion Protocol, or robots.txt. This file has governed how bots go about scraping the web for decades -- and like the bots themselves, it is now at the center of a larger fight over how AI gets trained. Many publishers have already updated their robots.txt files to block AI bots from OpenAI, Anthropic, and other major AI players. Robots.txt allows website owners to block or permit bots on a case-by-case basis. While there's no legal obligation for bots to adhere to what the text file says, compliance is a long-standing norm. (A norm that is sometimes ignored: Earlier this year, a WIRED investigation revealed that the AI startup Perplexity was ignoring robots.txt and surreptitiously scraping websites.) Applebot-Extended is so new that relatively few websites block it yet. Ontario, Canada-based AI-detection startup Originality AI analyzed a sampling of 1,000 high-traffic websites last week and found that approximately 7 percent -- predominantly news and media outlets -- were blocking Applebot-Extended. This week, the AI agent watchdog service Dark Visitors ran its own analysis of another sampling of 1,000 high-traffic websites, finding that approximately 6 percent had the bot blocked. Taken together, these efforts suggest that the vast majority of website owners either don't object to Apple's AI training practices are simply unaware of the option to block Applebot-Extended.

Facebook, New York Times, and more, refuse to let Apple Intelligence train on their data

Future expansions to Apple Intelligence may involve more AI partners, paid subscriptions Website owners have a simple mechanism to tell Apple Intelligence not to scrape the site for training purposes, and reportedly major platforms like Facebook and the New York Times are using it. Apple has been offering publishers millions of dollars for the right to scrape their sites, as opposed to Google which believes all data should be freely available to train AI large language modules. As part of this, Apple honors a system where a site can just say in a particular file that it does not want to be scraped. That file is a simple text one called robots.txt, and according to Wired, very many major publishers are choosing to use this to block Apple's AI training. This robots.txt file is no technical barrier to scraping, nor even really a legal one, and there are firms that are known to ignore being blocked. Reportedly, many news sites that are blocking Apple Intelligence. Significant ones include: In Apple's case, Wired says that two main studies in the last week have shown that around 6% to 7% of high-traffic websites are blocking Apple's search tool, called Applebot-Extended. Then a further study by Ben Welsh, also undertaken in the last week, says that just over a 25% of sites checked are blocking it. The discrepancy is down to which sets of high-traffic websites were researched. The Welsh study, for comparison, found that OpenAI's bot is blocked by 53% of news sites checked, and Google's equivalent Google-Extended is blocked by almost 43%. Wired concludes that while sites might not care whether Apple Intelligence is scraping them, the major reason for low blocking figures is that Apple's AI bot is too little known for firms to notice it. Yet Apple Intelligence is not exactly hiding in the dark, and AppleBot-Extended is a superset of AppleBot. That was first spotted by sites in November 2014, and officially revealed by Apple in May 2015. So for ten years, AppleBot has been searching and scraping websites, and doing so in order to power Siri and Spotlight searches. Consequently, it's less likely that websites owners haven't heard of Apple Intelligence, and more likely that they have head of Apple making deals worth millions. While negotiations are continuing, or just conceivably might start, some sites are consciously blocking Apple Intelligence. That includes The New York Times, which is also suing OpenAI over copyright infringement because of its AI scraping. "As the law and The Times' own terms of service make clear, scraping or using our content for commercial purposes is prohibited without our prior written permission" says the newspaper's Charlie Stadtlander. "Importantly, copyright law still applies whether or not technical blocking measures are in place."

Many of the biggest websites opted out of Apple Intelligence training

Generative AI systems are trained by letting them surf the web to scrape content. Apple allows publishers to opt out of its scraping, and a new report says that many of the biggest websites have specifically opted out of Apple Intelligence training. This includes both Facebook and Instagram, as well as many high-profile news and media sites like The New York Times and The Atlantic ... Large language models like ChatGPT are trained by giving them access to millions of words of source material, ranging from news stories to user comments. In Apple's case, the company has for years been using Applebot to train Siri and surface Spotlight suggestions. More recently, the company has also been using Applebot to train Apple Intelligence. The practice is controversial, as AIs are effectively using copyrighted material to generate their own versions of it. For more niche topics, where source material is scarce, they have even been found to regurgitate entire paragraphs with almost no changes made. But Apple does this in an ethical way, allowing publishers to opt out, and screening out personal data (though it did get caught out by one third-party source). We train our foundation models on licensed data, including data selected to enhance specific features, as well as publicly available data collected by our web-crawler, AppleBot. Web publishers have the option to opt out of the use of their web content for Apple Intelligence training with a data usage control [...] We apply filters to remove personally identifiable information like social security and credit card numbers that are publicly available on the Internet. Apple uses an Applebot-Extended tag to allow sites to opt out of AI training while still allowing search indexing - meaning that their pieces can still be included in Spotlight and Siri searches. Since opting out is done using a publicly-accessible robots.txt file, it's easy to see which sites have done this. Wired checked a number of the biggest news and social media sites. WIRED can confirm that Facebook, Instagram, Craigslist, Tumblr, The New York Times, The Financial Times, The Atlantic, Vox Media, the USA Today network, and WIRED's parent company, Condé Nast, are among the many organizations opting to exclude their data from Apple's AI training [...] In a separate analysis conducted this week, data journalist Ben Welsh found that just over a quarter of the news websites he surveyed (294 of 1,167 primarily English-language, US-based publications) are blocking Applebot-Extended. Applebot-Extended is a relatively new tag, so it's likely that more websites will also opt out once awareness increases. Apple is believed to have struck deals with some media companies, paying a fee in return for the right to use their content for training. It's likely this is the motivation for at least some sites currently blocking Apple - holding out for a payment offer. "A lot of the largest publishers in the world are clearly taking a strategic approach," says Originality AI founder Jon Gillham. "I think in some cases, there's a business strategy involved -- like, withholding the data until a partnership agreement is in place." iOS 18.1 beta 3 includes several new Apple Intelligence features, including Photo Clean Up and more notification summaries.