Cloudflare Accuses Perplexity AI of Stealthy Web Scraping, Sparking Debate on AI Crawlers
29 Sources
[1]
AI site Perplexity uses "stealth tactics" to flout no-crawl edicts, Cloudflare says
AI search engine Perplexity is using stealth bots and other tactics to evade websites' no-crawl directives, an allegation that if true violates Internet norms that have been in place for more than three decades, network security and optimization service Cloudflare said Monday. In a blog post,
[2]
Perplexity accused of scraping websites that explicitly blocked AI scraping | TechCrunch
AI startup Perplexity is crawling and scraping content from websites that have explicitly indicated they don't want to be scraped, according to internet infrastructure provider Cloudflare. On Monday, Cloudflare published research saying it observed the AI startup ignore blocks and hide its
[3]
Some people are defending Perplexity after Cloudflare 'named and shamed' it | TechCrunch
When Cloudflare accused AI search engine Perplexity of stealthily scraping websites on Monday, while ignoring a site's specific methods to block it, this wasn't a clear-cut case of an AI web crawler gone wild. Many people came to Perplexity's defense. They argued that Perplexity accessing sites in
[4]
Perplexity says Cloudflare's accusations of 'stealth' AI scraping are based on embarrassing errors
Cloudflare now offers services to block aggressive AI crawlers. Cloudflare, a leading content delivery network (CDN) company, has accused the AI startup Perplexity of evading websites' "no crawl" directives by stealthily deploying web crawlers to scrape content from sites that have explicitly
[5]
Perplexity is sneaking onto websites to scrape blocked content, says Cloudflare
Cloudflare now offers services to block aggressive AI crawlers. Cloudflare, a leading content delivery network (CDN) company, has accused the AI startup Perplexity of evading websites' "no crawl" directives by stealthily deploying web crawlers to scrape content from sites that have explicitly
[6]
Cloudflare says Perplexity's AI bots are 'stealth crawling' blocked sites
The AI search startup Perplexity is allegedly skirting restrictions meant to stop its AI web crawlers from accessing certain websites, according to a report from Cloudflare. In the report, Cloudflare claims that when Perplexity encounters a block, the startup will conceal its crawling identity "in
[7]
Cloudflare: Perplexity AI Acts Like North Korean Hackers, Ignores Scraping Blocks
Search engine provider Perplexity AI is accused of acting like "North Korean hackers" after the company's bots were found crawling websites with anti-scraping rules in place. The accusation comes from Cloudflare, an internet infrastructure provider that's developed safeguards to prevent AI
[8]
Perplexity vexed by Cloudflare's claims its bots are bad
AI search biz insists its content capture and summarization is okay because someone asked for it AI search biz Perplexity claims that Cloudflare has mischaracterized its site crawlers as malicious bots and that the content delivery network made technical errors in its analysis of Perplexity's
[9]
Perplexity AI crawlers accused of stealth data scraping
Cloudflare finds AI search biz ignoring crawl prohibitions and trying to hide its spiders Perplexity, an AI search startup, has been spotted trying to disguise its content-scraping bots while flouting websites' no-crawl directives. According to Cloudflare, a network infrastructure company that
[10]
Perplexity is allegedly scraping websites it's not supposed to, again
Web crawlers deployed by Perplexity to scrape websites are allegedly skirting restrictions, according to a new report from Cloudflare. Specifically, the report claims that the company's bots appear to be "stealth crawling" sites by disguising their identity to get around robots.txt files and
[11]
The War for the Web Has Begun
One of the internet's biggest gatekeepers has accused a rising AI star of breaking the web's oldest rules. The explosive feud could change how we all get information online. A high-stakes war has just broken out over the future of the internet. In one corner is Cloudflare, a giant of web
[12]
Perplexity accused of scraping websites even when told not to -- here's their response
Perplexity is riding high in the AI world right now. After launching the company's Comet browser, leading the way in agentic browsing, they've ran into some controversy. Cloudflare, in an online blog, published research that showed Perplexity has been crawling and scraping content from websites
[13]
Perplexity hits back after Cloudflare slams its online scraping tools
Perplexity wants Cloudflare to engage in dialogue - not just to post accusations online Perplexity AI has accused Cloudflare of mischaracterizing its web crawlers as malicious bots after the latter claimed the AI company obfuscated its bot identity using deceptive strings and unexpected IP
[14]
Perplexity accused of breaking a major online AI scraping rule - but it says it has done nothing wrong
OpenAI adheres to responsible crawling, but Perplexity quiet for now Cloudflare has accused AI giant Perplexity of scraping websites which explicitly disallowed crawling via robots.txt and other network-level rules by hiding its identity and conducting obfuscated crawling activity. Researchers
[15]
Cloudflare Accuses Perplexity AI of Using Stealth Crawlers to Evade Website Blocks - Decrypt
Perplexity denies the claims, calling Cloudflare's evidence a "sales pitch" and disputing that any banned content was accessed. Perplexity's crawlers kept accessing content from tens of thousands of websites even after those sites explicitly blocked them, according to internet infrastructure
[16]
Cloudflare calls out Perplexity for hiding 'crawling activity' as AI bot scrapes websites that explicitly disallow it, Perplexity responds by calling them 'more flair than cloud'
It's AI versus the internet as Cloudflare and Perplexity have a public falling out over the 'stealth crawling' of restricted websites. The disagreement has spiralled to name calling, even, as Perplexity snaps back at Cloudflare calling it "more flair than cloud", which isn't quite the burn they
[17]
The Cloudflare-Perplexity Clash Over Web Crawling | AIM
Opinions are divided, discussions abound on where to draw the line on web scraping. The heated dispute between cloud infrastructure giant Cloudflare and Perplexity, the AI search application company, regarding the latter's web crawling practices, has brought forth concerns about AI applications
[18]
Inside the looming AI-agents war that will redefine the economics of the web
There's a war brewing in the world of AI agents. After declaring a month ago that it would block AI crawlers by default on its network, Cloudflare openly accused Perplexity of deliberately bypassing internet standards to scrape websites. It published a detailed blog post, explaining how, even if
[19]
Cloudflare de-lists Perplexity, alleges stealth scraping
Cloudflare has accused Perplexity, the AI search engine, of ignoring website rules and using stealth crawling to bypass robots.txt protocols. Leading internet security player Cloudflare said on Monday (5 August) in a post that it was delisting Perplexity's crawler as a verified bot, and would
[20]
Cloudflare vs. Perplexity: a web scraping war with big implications for AI
When the web was established several decades ago, it was built on a number of principles. Among them was a key, overarching standard dubbed "netiquette": Do unto others as you'd want done unto you. It's a principle that lived on through other companies, including Google, whose motto for a period
[21]
Cloudflare accuses Perplexity of evading anti-bot rules
Perplexity allegedly evaded detection by impersonating Chrome and rotating its network identifiers. Cloudflare observed AI startup Perplexity bypassing website content access restrictions, alleging the company obscured its bot identities to circumvent digital preferences. This activity involved
[22]
Perplexity Might Be Using Illegitimate Means to Scrape Websites' Data
When Perplexity cannot crawl a website, its response quality drops Perplexity is said to be illegitimately accessing content from websites despite being prohibited from doing so. Cloudflare, a global web security services company, conducted a test to confirm the stealth behaviour of the answer
[23]
Perplexity is Taking Website Data Even After Being Blocked
Perplexity, an AI (artificial intelligence) chat product available globally, has been accused of taking data from websites even after being blocked to do so. This was said by Cloudflare, a global web security services company. Cloudflare, in a blog post confirmed that Perplexity bots are crawling
[24]
'Name, Shame, and Hard Block Them': Cloudflare Blasts Perplexity Over AI Website Scraping
The AI startup was caught doing so across tens of thousands of domains, making millions of requests per day. Perplexity has been caught red-handed by Cloudflare, as the startup has been sneaking around websites that do not want to be scraped by AI crawlers. Typically, AI answer engines like
[25]
Cloudflare Exposes Perplexity For Secretly Crawling Blocked Websites, Igniting Major Backlash And Raising Alarms Over AI's Ethics, Transparency, And Content Scraping Practices
The AI search startup Perplexity has landed itself in the middle of controversy, facing allegations of bypassing restrictions designed to prevent its web crawlers from accessing certain websites. According to a report by Cloudflare, the company has been able to skip these protections by concealing
[26]
Cloudflare Accuses Perplexity of Skirting No-Crawl Rules | PYMNTS.com
By completing this form, you agree to receive marketing communications from PYMNTS and to the sharing of your information with our sponsor, if applicable, in accordance with our Privacy Policy and Terms and Conditions. Writing on its blog Monday (Aug. 4), the company said it had gotten complaints
[27]
Are AI Crawlers User-Driven Tools or Malicious Bots?
In response to Cloudflare blocking Perplexity's AI bots, the AI company has issued a statement saying that Cloudflare has got nearly everything wrong about how modern AI assistants function. The AI company argued that AI assistants differ from traditional bots that crawl websites' content and
[28]
Cloudflare Blocks Perplexity AI Bots Amid Scraping Allegations
MediaNama is using Cloudflare to block AI bots. Yet, we find that our articles have been scraped by them. Our terms and conditions state that scraping by AI bots is unauthorised access under the IT Act, but it seems like Perplexity is bent on violating both technical and legal blocks. Also, I want
[29]
Perplexity AI Accused of Bypassing Anti-Scraping Measures to Access Restricted Content
Perplexity Allegedly Used Disguised Crawlers to Bypass Robots.txt Across Millions of Requests AI startup has come under fire again, this time for allegedly scraping from websites that specifically blocked the startup's crawlers. The criticism comes from Cloudflare, a large internet infrastructure
Share
Copy Link
Cloudflare alleges that AI search engine Perplexity is using stealth tactics to bypass website crawling restrictions, leading to a broader discussion on the ethics and legality of AI web crawling practices.
Cloudflare's Allegations Against Perplexity AI
Cloudflare, a leading content delivery network (CDN) provider, has accused AI search engine Perplexity of employing "stealth tactics" to circumvent websites' no-crawl directives
1
. According to Cloudflare's research, Perplexity allegedly used undeclared crawlers, multiple IP addresses, and IP rotation techniques to access content from websites that had explicitly blocked its known bots2
.
Source: The Verge
The company claims to have observed this behavior across tens of thousands of domains, involving millions of requests per day. Cloudflare researchers reported that when Perplexity's declared crawlers encountered blocks from robots.txt files or firewall rules, the AI search engine would deploy a stealth bot that impersonated a generic browser, such as Google Chrome on macOS
1
.Perplexity's Response and Counterarguments
Perplexity has vehemently denied Cloudflare's accusations, dismissing them as a "sales pitch" and claiming that the bot identified in Cloudflare's blog post "isn't even ours"
2
. In a subsequent blog post, Perplexity attributed the behavior to a third-party service it occasionally uses and argued that Cloudflare's systems are "fundamentally inadequate for distinguishing between legitimate AI assistants and actual threats"3
.The Broader Debate on AI Web Crawling
Source: The Register
This incident has sparked a wider discussion about the ethics and legality of AI web crawling practices. Defenders of Perplexity argue that AI agents accessing websites on behalf of users should be treated similarly to human users making the same requests
3
. They contend that blocking such access could potentially harm the functionality of AI assistants and limit users' ability to retrieve information.Industry Impact and Responses
In response to the growing concerns about AI crawlers, Cloudflare has taken several actions:
- De-listing Perplexity's bots from its verified list
- Adding new techniques to block stealth crawling
- Launching a marketplace allowing website owners to charge AI scrapers for content access
- Offering services to block aggressive AI crawlers
4
Source: Ars Technica
Related Stories
The Changing Landscape of Internet Traffic
The controversy highlights the shifting dynamics of internet traffic. For the first time in the internet's history, bot activity is outpacing human activity online, with AI traffic accounting for over 50% of all traffic
3
. This trend is reshaping how websites manage their content and interact with AI-driven services.Legal and Ethical Implications
The incident raises important questions about the legal and ethical boundaries of AI web crawling. While some argue for unrestricted access to public web content for AI training purposes, others emphasize the need to respect website owners' rights and established internet norms
1
. The debate underscores the need for clearer regulations and industry standards governing AI's interaction with web content.As the AI industry continues to evolve, this controversy serves as a catalyst for ongoing discussions about balancing innovation with respect for established internet protocols and content creators' rights.
References
Summarized by
Navi
[2]
[3]
[4]
Related Stories
Recent Highlights
1
OpenAI Solves Navier-Stokes Problem With AI, Sparking Existential Crisis in Mathematics
Science and Research
2
OpenAI Agents Launched RubyGems Attack With 2,000+ Malicious Packages, Attempted API Key Theft
Technology
3
OpenAI Reveals 6 New Cases of AI Models Hiding Mistakes and Breaking Rules
Policy and Regulation
Recent Highlights
Today's Top Stories
Don’t drown in AI news. We cut through the noise - filtering, ranking and summarizing the most important AI news, breakthroughs and research daily. Follow topics that matter to you and stay ahead.
