Cloudflare Unveils 'AI Labyrinth' to Combat Unauthorized AI Web Scraping

Cloudflare turns AI against itself with endless maze of irrelevant facts

On Wednesday, web infrastructure provider Cloudflare announced a new feature called "AI Labyrinth" that aims to combat unauthorized AI data scraping by serving fake AI-generated content to bots. The tool will attempt to thwart AI companies that crawl websites without permission to collect training data for large language models that power AI assistants like ChatGPT. Cloudflare, founded in 2009, is probably best known as a company that provides infrastructure and security services for websites, particularly protection against distributed denial-of-service (DDoS) attacks and other malicious traffic. Instead of simply blocking bots, Cloudflare's new system lures them into a "maze" of realistic-looking but irrelevant pages, wasting the crawler's computing resources. The approach is a notable shift from the standard block-and-defend strategy used by most website protection services. Cloudflare says blocking bots sometimes backfires because it alerts the crawler's operators that they've been detected. "When we detect unauthorized crawling, rather than blocking the request, we will link to a series of AI-generated pages that are convincing enough to entice a crawler to traverse them," writes Cloudflare. "But while real looking, this content is not actually the content of the site we are protecting, so the crawler wastes time and resources." The company says the content served to bots is deliberately irrelevant to the website being crawled, but it is carefully sourced or generated using real scientific facts -- such as neutral information about biology, physics, or mathematics -- to avoid spreading misinformation (whether this approach effectively prevents misinformation, however, remains unproven). Cloudflare creates this content using its Workers AI service, a commercial platform that runs AI tasks. Cloudflare designed the trap pages and links to remain invisible and inaccessible to regular visitors, so people browsing the web don't run into them by accident. A smarter honeypot AI Labyrinth functions as what Cloudflare calls a "next-generation honeypot." Traditional honeypots are invisible links that human visitors can't see but bots parsing HTML code might follow. But Cloudflare says modern bots have become adept at spotting these simple traps, necessitating more sophisticated deception. The false links contain appropriate meta directives to prevent search engine indexing while remaining attractive to data-scraping bots. "No real human would go four links deep into a maze of AI-generated nonsense," Cloudflare explains. "Any visitor that does is very likely to be a bot, so this gives us a brand-new tool to identify and fingerprint bad bots." This identification feeds into a machine learning feedback loop -- data gathered from AI Labyrinth is used to continuously enhance bot detection across Cloudflare's network, improving customer protection over time. Customers on any Cloudflare plan -- even the free tier -- can enable the feature with a single toggle in their dashboard settings. A growing problem Cloudflare's AI Labyrinth joins a growing field of tools designed to counter aggressive AI web crawling. In January, we reported on "Nepenthes," software that similarly lures AI crawlers into mazes of fake content. Both approaches share the core concept of wasting crawler resources rather than simply blocking them. However, while Nepenthes' anonymous creator described it as "aggressive malware" meant to trap bots for months, Cloudflare positions its tool as a legitimate security feature that can be enabled easily on its commercial service. The scale of AI crawling on the web appears substantial, according to Cloudflare's data that lines up with anecdotal reports we've heard from sources. The company says that AI crawlers generate more than 50 billion requests to their network daily, amounting to nearly 1 percent of all web traffic they process. Many of these crawlers collect website data to train large language models without permission from site owners, a practice that has sparked numerous lawsuits from content creators and publishers. The technique represents an interesting defensive application of AI, protecting website owners and creators rather than threatening their intellectual property. However, it's unclear how quickly AI crawlers might adapt to detect and avoid such traps, potentially forcing Cloudflare to increase the complexity of its deception tactics. Also, wasting AI company resources might not please people who are critical of the perceived energy and environmental costs of running AI models. Cloudflare describes this as just "the first iteration" of using AI defensively against bots. Future plans include making the fake content harder to detect and integrating the fake pages more seamlessly into website structures. The cat-and-mouse game between websites and data scrapers continues, with AI now being used on both sides of the battle.

Cloudflare is luring web-scraping bots into an 'AI Labyrinth'

Wes Davis is a weekend editor who covers the latest in tech and entertainment. He has written news, reviews, and more as a tech journalist since 2020. Cloudflare, one of the biggest network internet infrastructure companies in the world, has announced AI Labyrinth, a new tool to fight web-crawling bots that scrape sites for AI training data without permission. The company says in a blog post that when it detects "inappropriate bot behavior," the free, opt-in tool lures crawlers down a path of links to AI-generated decoy pages that "slow down, confuse, and waste the resources" of those acting in bad faith. Websites have long used the honor system approach of robots.txt, a text file that gives or denies permission to scrapers, but which AI companies, even well-known ones like Anthropic and Perplexity AI, have been accused of ignoring. Cloudflare writes that it sees over 50 billion web crawler requests per day, and although it has tools for spotting and blocking the malicious ones, this often prompts attackers to switch tactics in "a never-ending arms race." Cloudflare says rather than block bots, AI Labyrinth fights back by making them process data that has nothing to do with a given website's actual data. The company says it also functions as "a next-generation honeypot," drawing in AI crawlers that keep following links to fake pages deeper, whereas a regular human being wouldn't. It says this makes it easier to fingerprint malicious bots for Cloudflare's list of bad actors as well as identify "new bot patterns and signatures" it wouldn't have detected otherwise. According to the post, these links shouldn't be visible to human visitors. You can read more about how AI Labyrinth works on Cloudflare's blog, but here's a bit more detail from the post: We found that generating a diverse set of topics first, then creating content for each topic, produced more varied and convincing results. It is important to us that we don't generate inaccurate content that contributes to the spread of misinformation on the Internet, so the content we generate is real and related to scientific facts, just not relevant or proprietary to the site being crawled. Website administrators can opt into using AI Labyrinth by navigating to the Bot Management section of their site's Cloudflare dashboard's settings and toggling it on. The company says that this "is only the first iteration of using generative AI to thwart bots." It plans to create "whole networks of linked URLs" that bots that end up in will have a hard time clocking as fake. As Ars Technica notes, AI Labyrinth sounds similar to Nepenthes, a tool that's designed to sideline crawlers for "months" in a hell of AI-generated junk data.

AI bots scraping your data? This free tool gives those pesky crawlers the run-around

Cloudflare's AI Labyrinth has a message for bots: Get lost. Here's how to toggle on the tool. The rise of AI-generated content, also known as synthetic media, has mostly caused problems: It helps spread misinformation, steal from artists, and erode trust in what we see online. However, Cloudflare may have found a use case where artificial intelligence could help protect original content from the tentacles of AI companies. On Wednesday, the company released AI Labyrinth, a tool that uses AI-generated content to "slow down, confuse, and waste the resources" of unauthorized AI crawlers. Also: Chatbots are distorting news - even for paid users Multiple studies have found that AI chatbots -- including ChatGPT and Perplexity -- are still accessing content from sites that block their crawlers. Cloudflare noted in the announcement that crawlers "generate more than 50 billion requests to the Cloudflare network every day or just under 1% of all web requests we see" -- and how you block them matters. "While Cloudflare has several tools for identifying and blocking unauthorized AI crawling, we have found that blocking malicious bots can alert the attacker that you are on to them, leading to a shift in approach, and a never-ending arms race," the company explained. "We wanted to create a new way to thwart these unwanted bots, without letting them know they've been thwarted." When Cloudflare detects an unauthorized crawling request, AI Labyrinth -- rather than simply blocking the crawler -- links to several AI-generated web pages that look real enough to convince the crawler they're legitimate. This way, the crawler believes it's successfully scraped the content it was looking for, while the site's actual data remains protected from prying eyes. The crawler also squanders computational resources, which Cloudflare also sees as a win. Also: 10 Siri tips and tricks to make it less terrible (and more helpful) "Cloudflare will automatically deploy an AI-generated set of linked pages when we detect inappropriate bot activity, without the need for customers to create any custom rules," the announcement explains. The company used Workers AI and an open-source model to create unique, human-looking synthetic pages on various topics ahead of time, as creating them on demand could result in performance lags. This "pre-generation pipeline [...] sanitizes the content to prevent any XSS vulnerabilities and stores it in R2 for faster retrieval," the company said. AI Labyrinth only presents links to AI-generated content to AI scrapers; the content is otherwise hidden from human visitors on existing pages on the site and does not alter the site's structure, appearance, or SEO. Cloudflare also noted it did not want the tool to add more AI slop to the internet at large. "It is important to us that we don't generate inaccurate content that contributes to the spread of misinformation on the internet, so the content we generate is real and related to scientific facts, just not relevant or proprietary to the site being crawled," the announcement added. Also: 10 professional developers on the true promise and peril of vibe coding Additionally, Cloudflare believes the tool can act as a honeypot to help identify more illicit crawlers. The company noted that real human visitors are unlikely to "go four links deep into a maze of AI-generated nonsense," and that the tool will, therefore, know based on click activity where new bots are popping up. This will in turn help AI Labyrinth better identify bad actors. Bots have evolved to detect traditional honeypot techniques. To stay ahead, Cloudflare aims for AI Labyrinth AI to "eventually create whole networks of linked URLs that are much more realistic, and not trivial for automated programs to spot." AI Labyrinth could be a useful tool to try for publishers or individuals who don't want their work used to train AI (or misrepresented by chatbots in the process). Also: Google Maps yanks over 10,000 fake business listings - how to spot the scam All Cloudflare customers, including those on the Free tier, can opt in to AI Labyrinth today. Simply go to your Cloudflare dashboard, navigate to the bot management section, and switch the AI Labyrinth toggle on.

Cloudflare builds an AI to make life hell for other AIs

Slop-making machine will feed unauthorized scrapers what they so richly deserve, hopefully without poisoning the internet Cloudflare has created a bot-busting AI to make life hell for AI crawlers. The network-taming company built the tool after noticing that almost one percent of all requests to access web content that it can see now come from AI crawler bots. Those bots are probably scraping data that's gathered up to train AI models. Web site operators can in theory block AI crawlers using various means such as a robots.txt file or changing web server settings to disallow visits from bots. Some even use CAPTCHAs to test whether visitors to a site are human, or adopt software designed to stymie bots. In reality crawler operators ignore the instructions in robots.txt files, or work around CAPTCHAs and web server settings. The result is a lot of unwanted crawler traffic consuming resources, and info fed into training data without creators' permission - a contentious practice currently being tested in court amidst allegations of copyright abuse. No human would go four links deep into a maze of AI-generated nonsense Cloudflare's response is to let crawler bots in and use generative AI to create junk content for them to devour in what the company has termed an "AI Labyrinth". "When we detect unauthorized crawling, rather than blocking the request, we will link to a series of AI-generated pages that are convincing enough to entice a crawler to traverse them," explained Cloudflare's Reid Tatoris, Harsh Saxena, and Luis Miglietti. Cloudflare uses its own serverless Workers to create the content. The trio wrote that the content is "real looking" but "not actually the content of the site we are protecting, so the crawler wastes time and resources." The content is also "real and related to scientific facts" because Cloudflare doesn't want to inadvertently create misinformation. The AI slop is also designed not to mess with sites' reputations or search engine optimization efforts. It is, however, designed to act as a deterrent to crawler operators, by keeping their bots busy and thereby increasing the cost of operating content scrapers. Cloudflare thinks this stuff is also a useful tool to detect bot activity. "No real human would go four links deep into a maze of AI-generated nonsense," Cloudflare's trio wrote. "Any visitor that does is very likely to be a bot, so this gives us a brand-new tool to identify and fingerprint bad bots, which we add to our list of known bad actors." This sort of thing usually creates an arms race and Cloudflare is already thinking about what it will take to stay ahead. "In the future, we'll continue to work to make these links harder to spot and make them fit seamlessly into the existing structure of the website they're embedded in," its authors wrote. Cloudflare customers can enable the AI Labyrinth in their management consoles. ®

One company's devious plan to stop AI web scrapers from stealing your content

AI is stealing your content. We know this is how AI companies have built their highly-valued businesses - by scraping the web and using your data to train their chatbots. Web scraping isn't new. In the past, websites could rely on simple protocols like robots.txt to define what could, and could not, be used by web crawlers. Those guidelines were respected by the companies doing the scraping to, say, build results for search engines. AI companies, however, are not abiding by this social contract and are ignoring those instructions. Cloudflare, a global network service that helps some of the biggest websites in the world deliver content to users, has devised a new plan to deal with AI companies' web scrapers. And the idea is as positively devious as it is ingenious. In a new blog post, Cloudflare has shared how it's now "trapping misbehaving bots in an AI labyrinth." Basically, bots that don't follow the rules laid out for them via protocols such as robots.txt, a simple text file that lays out what web crawlers are allowed to do on a site, will be messed with in order to waste the time and resources of the company in charge of the bot. "AI-generated content has exploded...at the same time, we've also seen an explosion of new crawlers used by AI companies to scrape data for model training," Cloudflare said in its post. "AI Crawlers generate more than 50 billion requests to the Cloudflare network every day, or just under 1% of all web requests we see." Cloudflare says it previously just blocked AI web crawlers and scrapers. However, doing so alerted those behind the bots that their access had been denied, and as a result they would shift strategies in order to continue their scraping campaigns. So, Cloudflare came up with an idea to build a honeypot: a series of fake webpages created with AI-generated content. The fact that Cloudflare is utilizing AI-generated content to fight AI web scrapers isn't just for schadenfreude. When AI trains off of AI-generated content, it actually degrades the AI model itself. The industry even has a term for it: "model collapse." Cloudflare is essentially making sure that bots that break the rules are punished for doing so. Cloudflare's post gets into the technical details of building the AI labyrinth. But, the main gist of it is that Cloudflare devised things in a way where a human visitor shouldn't ever see these AI-generated honeypot pages. In addition, humans would notice the "AI-generated nonsense" on these pages. Bots, however, would fall down the rabbit hole, wasting computational resources as they go deeper and deeper through the multiple pages of AI-generated content. Cloudflare customers are able to opt-in to using the AI labyrinth right now to protect their content from web scrapers.

'No real human would go four links deep into a maze of AI-generated nonsense': Cloudflare's AI Labyrinth uses decoy pages to trap web-crawling bots and feed them slop 'as a defensive weapon'

The web is plagued by bots. That's nothing new of course, but now we're in the midst of our much-loved AI revolution (you do love it, right?) many websites are continually crawled by bots aiming to scrape them of their precious data to train AI content. Cloudflare thinks it might have the solution, however, as its newly-announced AI Labyrinth tool aims to take the fight to the nefarious bots by "using generative AI as a defensive weapon." Cloudflare says that AI crawlers generate more than 50 billion requests to its network every day -- and while tools exist to block them, these methods can alert attackers that they've been noticed, causing them to shift approach (via The Verge). AI Labyrinth, however, links detected bots to a series of AI-generated pages that are convincing enough to draw them in, but contain no useful information. Why? Well, because they were generated by AI, of course. Essentially this creates an ouroboros of AI slop in, AI slop out, to the point where the bot wastes precious time and resources churning through useless content instead of scraping something created by an actual human being. "As an added benefit, AI Labyrinth also acts as a next-generation honeypot. No real human would go four links deep into a maze of AI-generated nonsense," says Cloudflare. "Any visitor that does is very likely to be a bot, so this gives us a brand-new tool to identify and fingerprint bad bots, which we add to our list of known bad actors." It's bots, bots all the way down. The AI-generated "poisoned" content is integrated in the form of hidden links on existing pages, meaning a human is unlikely to find them but a web crawler will. To double down on the human-first angle, Cloudflare also says these links will only be added to pages viewed by suspected AI scrapers, so the rest of us shouldn't even notice it's working away in the background, fighting evil bots like some sort of Batman-esque caped crusader. Enabling the tool is a simple matter of ticking a checkbox in Cloudflare's settings page, and ta-da, off to work the AI Labyrinth goes. Cloudflare says this is merely the first iteration of this particular tech and encourages its users to opt in to the system so it can be refined in future. I do have a question, though. Given AI is now, let's face it, bloody everywhere, are we really sure that making its training process worse isn't going to have longer-term effects? Far be it from me to take the side of the nefarious crawlers, but I wonder if this will simply lead to a glut of even-more-terrible AI models in future if their training data is hamstrung from the start. Ah, screw it, I've talked myself out of my own counter argument. Something needs to be done about relentless permission-free data scraping from genuine human endeavour, and I salute the clever thinking behind this particular defensive tool. If I could make one suggestion, however, could we perhaps add a Minotaur? All good labyrinths need one, and then I can write something like "Cloudflare has grabbed the bull by the horns and..." Fill in your own headline there. Or, y'know, get an AI to do it for you. Kidding, kidding. I probably shouldn't be feeding the AI any more of my terrible jokes anyway.

This company has a cunning plan to stop AI bots from stealing content

"We wanted to create a new way to thwart these unwanted bots, without letting them know," Cloudfare said of its "honeypot" for web crawlers. How can we stop artificial intelligence (AI) from stealing our content? US-based web services provider Cloudflare says it has come up with a solution to web scraping - by setting up an "AI labyrinth" to trap bots. More specifically, this maze is aimed at detecting "AI crawlers," bots that systematically mine data from web pages' content and trap them there. The company said in a blog post published last week that it has seen "an explosion of new crawlers used by AI companies to scrape data for model training". Generative artificial intelligence (genAI) requires enormous databases for training its models. Several tech companies - such as OpenAI, Meta, or Stability AI - have been accused of extracting data that includes copyrighted content. To prevent the phenomenon, Cloudflare will "link to a series of AI-generated pages that are convincing enough to entice a crawler to traverse them" when detecting "inappropriate bot activity" to make them waste time and resources. "We wanted to create a new way to thwart these unwanted bots, without letting them know they've been thwarted," the company said, comparing the process to a "honeypot" while also helping it to catalogue nefarious actors. Cloudflare is used in around 20 per cent of all websites, according to the latest estimations. The decoy is made of "real and related to scientific facts" content but "just not relevant or proprietary to the site being crawled," the blog post added. It will also be invisible to human visitors and won't impact web referencing, the company said. An increasing number of voices are calling for stronger measures, including regulations, to protect content from being stolen by AI actors. Visual artists are now exploring how to "poison" models by adding a layer of data acting as a decoy for AI and therefore, preserving their artistic style by making it harder to mimic by genAI. Other different approaches have been explored, including, for example, several deals struck by news publishers with tech companies agreeing to allow AI to train on their content in exchange for undisclosed sums. Others, like the news agency Reuters and several artists, have decided to take the matter to court over the potential infringement of copyright laws.

Cloudflare fights AI scrapers with a maze of useless content

Cloudflare has developed a powerful AI tool designed to make life difficult for AI scraping bots. The network infrastructure company launched the bot-busting AI after observing that nearly one percent of all incoming web requests it monitors are generated by AI crawlers, likely harvesting data for AI model training. While website operators can block these bots using tools like robots.txt files or CAPTCHAs, most crawlers bypass these barriers, leading to wasted resources and unauthorized data collection. The practice of scraping data for training purposes without permission has sparked legal disputes over potential copyright violations. To combat this, Cloudflare is taking a unique approach: allowing these crawlers in but directing them to an "AI Labyrinth" -- a maze of AI-generated junk content. Rather than blocking scraping attempts, Cloudflare's AI creates a series of convincing, yet irrelevant, pages that lure bots deeper into a trap. The pages look real but are full of distractions, wasting the bots' time and resources. The content is also scientifically accurate to prevent spreading misinformation, and it's crafted to ensure websites' reputations and SEO are unaffected. The goal? To deter bot operators by increasing the cost of scraping. Cloudflare's AI Labyrinth keeps bots occupied, making scraping more resource-intensive. Additionally, Cloudflare views this tactic as a new way to detect bot activity. "No human would navigate four links deep into a maze of AI-generated nonsense," the company said. Anyone doing so is likely a bot, allowing Cloudflare to flag and fingerprint bad actors more effectively. While this solution could spark a back-and-forth battle between bots and defenders, Cloudflare is already looking ahead. The company plans to further refine the AI Labyrinth to make it even harder for crawlers to recognize and adapt to. For Cloudflare customers, the AI Labyrinth can be enabled directly from their management consoles.

AI vs AI: Cloudflare AI Labyrinth Thwarting AI Models' Training

Disclaimer: This content generated by AI & may have errors or hallucinations. Edit before use. Read our Terms of use Cloudflare is fighting AI with AI-generated content as part of its approach to tackling unauthorised AI web crawlers. This approach, called 'AI Labyrinth' seeks to punish AI web crawlers by redirecting them to a series of AI-generated pages "that are convincing enough to entice a crawler". Cloudflare explains that this AI-generated content does not have anything to do with the content of the site that the crawler was looking to scrape, and as such, ends up wasting time and resources. A web crawler is a bot that downloads and indexes content from across the internet. Typically search engines like Google and Bing operate such web crawlers, however, now AI companies like OpenAI also use web crawlers to collect data to train their models. Websites use robots.txt files to tell web crawlers which parts of their sites they can and cannot access. As MediaNama Editor Nikhil Pahwa pointed out in his 2023 editorial, while search engine web crawlers respect robots.txt, other bots do not necessarily do that. MediaNama, in fact, has previously been hit with bots looking to scrape our site content despite the site's robots.txt file restricting them from doing so. Explaining the rationale behind AI labyrinth, Cloudflare notes that while it provides its customers with tools to block crawlers, blocking ends up alerting the attackers. This causes them to change their tactics, getting the attacker and the site stuck in what Cloudflare calls a "never-ending arms race". Besides wasting the time of the bot, the AI labyrinth also acts as a honeypot, helping Cloudflare identify and fingerprint bad bots and add them to its list of known bad actors. Cloudflare used its AI model Workers AI to generate HTML pages on diverse topics. The company explains that this content is not misinformation. Instead, Cloudflare explains that "content we generate is real and related to scientific facts" but unrelated to the website that the bot is scraping. This makes it different from the tool "NightShade" that the researchers at the University of Chicago released in January 2024, which protects art from web scraping by transforming it into content that is unsuitable for AI model training. Unlike Nightshade, Cloudflare's approach doesn't poison the AI model or cause it to generate faulty outputs, but rather just feeds it unrelated information. "This pre-generated content is seamlessly integrated as hidden links on existing pages via our custom HTML transformation process, without disrupting the original structure or content of the page," the company explains. It adds that human visitors cannot see links to these AI-generated HTML pages. Further, adding these links to a website does not adversely impact the site's efforts towards search engine optimisation. This tool is relevant in the growing debate around unauthorised web scraping and the limited defences websites have to prevent their data from becoming a part of an AI model's dataset. This is especially concerning for news and publication businesses who have taken AI companies like Meta and OpenAI to court over unauthorised web scraping for model training. Cloudflare had previously released AI audit tools to give websites details of the different kinds of AI bots attempting to scrape their site. Its tools also help companies block all AI bots together, and also provides metrics on the kind of content the bots are trying to scrape to help them better understand the information that they need while negotiating licensing deals with AI companies. However, it is important to consider that AI companies will not be inclined to enter into licensing deals with all digital publications. As such, smaller digital publications need to have measures in place to prevent AI web crawlers from accessing their content. For instance MediaNama has changed its terms of service to restrict AI companies from training models on its content. But this begs the question: how effective are such policies from preventing scraping? "These terms cannot technologically block automated scraping because many scrapers do not typically respect robots.txt exclusions or read terms and conditions. What these terms do is give us legal basis for action. For example, based on the IT Act, 2000 and what the IT Ministry said in Parliament, AI scrapers accessing our website would be in violation of our terms and conditions that restrict such access," Pahwa explained. When asked how the company will ensure enforcement, Pahwa mentioned that for now, it will be a manual process. This means the company will have to query various AI models to see if they scraped the publication's latest content/content from after the publication updated its terms of service. "If we see verbatim text or clear summaries of our content, that's a red flag. In future, if automated tools are available, we will use those to detect usage. We can also detect AI tools on the basis of server logs," Pahwa mentioned. He added that the company can use its terms to issue notice to AI companies to get them to stop scraping its work and also delete said work from its dataset. "If they fail to act, then we may have to consider going to court," he suggested. Based on an initial search, when you query ChatGPT with a MediaNama headline like "Parliament Panel Bats for Unified Media Council Across OTT, Print, TV; Industry Not in Sync", you get part of the story (second paragraph in the screenshot) as a response. The chatbot directly attributes this to MediaNama's site, with a link to the exact article. This suggests that at least OpenAI's search engine is accessing MediaNama's content for output generation.