Cloudflare Launches Pay-Per-Crawl: A New Era for AI Web Scraping
35 Sources
[1]
Pay up or stop scraping: Cloudflare program charges bots for each crawl
Cloudflare is now experimenting with tools that will allow content creators to charge a fee to AI crawlers to scrape their websites. In a blog Tuesday, Cloudflare explained that its "pay-per-crawl" feature is currently in a private beta. A small number of publishers -- including AdWeek, The
[2]
Cloudflare launches a marketplace that lets websites charge AI bots for scraping | TechCrunch
Cloudflare, a cloud infrastructure provider that serves 20% of the web, announced Tuesday the launch of a new marketplace that reimagines the relationship between website owners and AI companies -- ideally giving publishers greater control over their content. For the last year, Cloudflare has
[3]
Why Cloudflare wants AI companies to pay for content
Cloudflare wants AI companies to pay up. The cloud infrastructure provider, which powers around 20% of the web, is launching a new experiment that would let publishers charge AI firms every time their bots scrape a site. It's called Pay per Crawl, and it could reshape how content is accessed and
[4]
Cloudflare will now, by default, block AI bots from crawling its clients' websites
However, such systems don't provide the same opportunities for monetization and credit as search engines historically have. AI models draw from a great deal of data on the web to generate their outputs, but these data sources are often not credited, limiting the creators' ability to make money from
[5]
Cloudflare Is Blocking AI Crawlers by Default
The age of the AI scraping free-for-all may be coming to an end. At least if Cloudflare gets its way. Last year, internet infrastructure firm Cloudflare launched tools enabling its customers to block AI scrapers. Today the company has taken its fight against permissionless scraping several steps
[6]
Pay Up, AI Bot: That's the Message From a Key Company in How the Internet Works
Expertise Artificial intelligence, home energy, heating and cooling, home technology. AI companies might find it harder to access the entire web to train their large language models after the internet infrastructure provider Cloudflare said this week it would block AI data crawlers by
[7]
Cloudflare declares war on AI crawlers - and the stakes couldn't be higher
The major Internet Content Delivery Network (CDN), Cloudflare, has declared war on AI companies. Starting July 1, Cloudflare now blocks by default AI web crawlers accessing content from your websites without permission or compensation. The change addresses a real problem. My own small site, where
[8]
Cloudflare just changed the internet, and it's bad new for the AI giants
The major internet Content Delivery Network (CDN), Cloudflare, has declared war on AI companies. Starting July 1, Cloudflare now blocks by default AI web crawlers accessing content from your websites without permission or compensation. The change addresses a real problem. My own small site, where
[9]
Cloudflare to Block AI Crawlers From Scraping Websites Unless They Pay
(Credit: SOPA Images / Contributor / LightRocket via Getty Images) Cloudflare has introduced a new way for websites to stop AI crawlers from stealing their content without permission -- and to potentially get them to pay up. Cloudflare is an internet infrastructure service that helps websites
[10]
Cloudflare creates AI crawler tollbooth to pay publishers
The bargain between content makers and crawlers has broken down ai-pocalypse Cloudflare has started blocking AI web crawlers by default in a bid to become the internet's gatekeeper. The term "gatekeeper" has been applied in a pejorative sense to platform companies like Apple and Google that use
[11]
Cloudflare launches tool to help website owners monetize AI bot crawler access
NEW YORK, July 1 (Reuters) - Cloudflare has launched a tool that blocks bot crawlers from accessing content without permission or compensation to help websites make money from AI firms trying to access and train on their content, the software company said on Tuesday. The tool allows website owners
[12]
Cloudflare experiment will block AI bot scrapers unless they pay a fee
New customers will also be asked if they want to block AI crawlers by default. Cloudflare has rolled out a couple of new measures meant to keep AI bot crawlers at bay. To start with, every new domain customer that signs up with the company to manage their website traffic will now be asked if they
[13]
Web giant Cloudflare to block AI bots from scraping content by default
Internet firm Cloudflare will start blocking artificial intelligence crawlers from accessing content without website owners' permission or compensation by default, in a move that could significantly impact AI developers' ability to train their models. Starting Tuesday, every new web domain that
[14]
Cloudflare tests "pay-per-crawl" system to charge AI firms for scraping website content
Serving tech enthusiasts for over 25 years. TechSpot means tech analysis and advice you can trust. What just happened? Cloudflare is experimenting with a new way to prevent AI crawlers from scraping website content. The CDN/security company has announced that it will block them from accessing
[15]
Free Lunch Is Over for the AI That Broke the Web
The foundational deal of the modern web, a handshake agreement that powered two decades of search and content, is officially dead. Cloudflare just put a price on scraping the internet, and it’s coming for artificial intelligence’s free lunch. Almost 30 years ago, two Stanford grad students,
[16]
Cloudflare launches way to charge AI bots for crawling sites
Cloudflare announced a new tool on Tuesday that can allow publishers to charge AI bots for scraping websites. That could prove to be a major change to how the internet has functioned so far in the AI age -- models have, by default, scraped the internet with abandon and without permission, often to
[17]
Cloudflare will now block AI crawlers on your website - and even force them to pay you
Specific crawlers can be granted free access, charged, or blocked Online creators often have very little control over the types of crawlers that can access their content, but Cloudflare may have a solution. The company has revived HTTP response code 402 as a neat way to block or charge AI
[18]
Cloudflare will let content creators charge fees to AI chatbots
Cybersecurity company Cloudflare announced on Tuesday it is adding a new feature that allows users to entirely block AI chatbots from crawling their websites or charge a fee to grant bots access. "The web is being stripmined by AI crawlers with content creators seeing almost no traffic and
[19]
Cloudflare Hits the Kill Switch on AI Crawlers -- And an Entire Industry Cheers - Decrypt
Cloudflare's "pay-per-crawl" marketplace lets publishers set fees for AI companies seeking access, potentially creating a new revenue stream -- and a new internet business model. Cloudflare CEO Matthew Prince just made it official: Starting today, Cloudflare will block by default known AI bots
[20]
Cloudflare will block AI web crawlers by default, and introduces new Pay Per Crawl scheme that means AI companies will have to fork out for the privilege to scrape
As time goes on, just saying 'no' to AI feels more and more futile. However, Cloudflare is announcing a few more tools for your anti-AI arsenal that put some of the power back in your hands. Cloudflare offers a range of network services from domain registration and content delivery to DDOS
[21]
AI game changer? Cloudflare lets websites charge AI developers for content access - SiliconANGLE
AI game changer? Cloudflare lets websites charge AI developers for content access Cloudflare Inc. today debuted a tool that enables website operators to charge artificial intelligence developers for access to their content. The new pay per crawl feature, as it's called, is in private
[22]
Cloudflare gives creators more control over AI crawlers
The company wants to give users a new option when it comes to AI crawlers and that is monetisation. Cybersecurity and network services provider Cloudflare has announced an update that will give content creators, publishers and website operators more control over who can access their work via AI
[23]
Cloudflare Just Became an Enemy of All AI Companies | AIM
"Our goal is to put the power back in the hands of creators, while still helping AI companies innovate." Cloudflare might have just killed the web search functionality of AI chatbots. The company announced that it would start blocking AI crawlers by default, drawing a line in the open web where
[24]
New Cloudflare feature lets websites charge AI developers for content access - SiliconANGLE
New Cloudflare feature lets websites charge AI developers for content access Cloudflare Inc. today debuted a tool that enables website operators to charge artificial intelligence developers for access to their content. The new pay per crawl feature, as it's called, is in private beta. Cloudflare
[25]
New Cloudflare tool lets sites charge AI crawlers
Cloudflare announced the launch of "Pay per Crawl," a private beta marketplace enabling website owners to charge AI crawlers for content access, fulfilling a vision articulated by CEO Matthew Prince in 2024. Cloudflare has, over the past year, introduced various tools designed to manage AI crawler
[26]
Publishers, website owners to block AI 'crawlers' that don't pay
Gift 5 articles to anyone you choose each month when you subscribe. Huge artificial intelligence companies that train their models on data harvested without permission from commercial websites will face a new roadblock, with global internet infrastructure giant Cloudflare making it easier for
[27]
Cloudflare declares war on AI companies, first blood was drawn on July 1st
As an Amazon Associate, we earn from qualifying purchases. TweakTown may also earn commissions from other affiliate partners at no extra cost to you. One of the most popular Content Delivery Networks (CDN) available, Cloudflare, has recently announced it is drawing a line in the sand with AI
[28]
Cloudflare launches tool to help website owners monetise AI bot crawler access - The Economic Times
The tool allows website owners to choose whether artificial intelligence crawlers can access their material and set a price for access through a "pay per crawl" model, which will help them control how their work is used and compensated, Cloudflare said.Cloudflare has launched a tool that blocks bot
[29]
Cloudflare introduces default blocking of AI data scrapers - The Economic Times
Cloudflare has introduced a new setting allowing websites to block AI bots from scraping content without permission. The move aims to protect original digital content and addresses growing concerns over AI companies using data without consent. Cloudflare's update impacts how online data is accessed
[30]
Cloudflare Launches Pay Per Crawl to Help Publishers Monetise AI Access
Aims to give control and revenue options to all content publishers. In a significant move to reshape how online content is accessed and monetized in the age of artificial intelligence (AI), Cloudflare has announced the private beta of Pay Per Crawl, a system that allows publishers to charge AI
[31]
Cloudflare Debuts Bot Blocker to Help 'Internet Survive Age of AI' | PYMNTS.com
By completing this form, you agree to receive marketing communications from PYMNTS and to the sharing of your information with our sponsor, if applicable, in accordance with our Privacy Policy and Terms and Conditions. The new offering, announced Tuesday (July 1), lets website owners decide if
[32]
Cloudflare Rolls Out Permission-Based Web Scraping For AI Bots
Internet infrastructure company Cloudflare has announced a major update to how it handles AI web crawlers. The company will now block AI bots from accessing websites by default, unless website owners explicitly choose to allow them. This change applies to all new domains that sign up with
[33]
Cloudflare launches tool to help website owners monetize AI bot crawler access
NEW YORK (Reuters) -Cloudflare has launched a tool that blocks bot crawlers from accessing content without permission or compensation to help websites make money from AI firms trying to access and train on their content, the software company said on Tuesday. The tool allows website owners to
[34]
Cloudflare, Inc. Just Changes How AI Crawlers Scrape the Internet-At-Large; Permission-Based Approach Makes Way for A New Business Model
Cloudflare, Inc. announced it is now the first Internet infrastructure provider to block AI crawlers accessing content without permission or compensation, by default. Starting July 1, 2025, website owners can choose if they want AI crawlers to access their content, and decide how AI companies can
[35]
Cloudflare takes a stand: AI crawlers must now ask permission to access web content
In a landmark move that could reshape the relationship between content creators and artificial intelligence companies, Cloudflare has announced a sweeping policy update that effectively locks out AI crawlers unless they explicitly seek permission. The new default, live from July 1, 2025, makes
Share
Copy Link
Cloudflare introduces a marketplace allowing websites to charge AI bots for content scraping, potentially reshaping the relationship between content creators and AI companies.
Cloudflare's Pay-Per-Crawl: A New Approach to AI Web Scraping
Cloudflare, a leading cloud infrastructure provider serving 20% of the web, has introduced a groundbreaking marketplace called Pay-Per-Crawl. This new system allows website owners to charge AI companies for scraping their content, potentially reshaping the relationship between content creators and AI firms
1
.
Source: PC Gamer
The Pay-Per-Crawl System
The Pay-Per-Crawl feature, currently in private beta, enables publishers to set their own prices for AI bots to access their content. Participating publishers include AdWeek, The Associated Press, The Atlantic, BuzzFeed, Fortune, Gannett, and Condé Nast
2
.Matthew Prince, CEO of Cloudflare, emphasized the importance of this initiative:
"Original content is what makes the Internet one of the greatest inventions in the last century, and it's essential that creators continue making it. Our goal is to put the power back in the hands of creators, while still helping AI companies innovate."
2
Default Blocking of AI Crawlers
In a significant move, Cloudflare announced that all new customers, including those on free plans, will have their domains set to block all known AI crawlers by default
2
. This shift towards a permission-based model for AI scraping marks a departure from the previous opt-out approach.
Source: ZDNet
The Changing Landscape of Web Crawling
Cloudflare's data reveals a stark contrast between traditional search engine crawlers and AI bots. While Google's crawler scraped websites 14 times for every referral, OpenAI's crawler scraped 17,000 times and Anthropic's 73,000 times for each referral
1
. This disparity highlights the changing dynamics of web content usage in the AI era.Publisher Perspectives and Industry Support
Several large publishers, including Conde Nast, TIME, and The Associated Press, have expressed support for Cloudflare's initiative. Stack Overflow CEO Prashanth Chandrasekar stated:
"Community platforms that fuel LLMs should be compensated for their contributions so they can invest back in their communities."
4
Related Stories
Challenges and Future Prospects
While Cloudflare's marketplace presents a bold vision for the future, its success depends on widespread adoption by both publishers and AI companies. Some challenges include:
- Convincing AI firms to participate, given they currently scrape content for free
1
. - Ensuring fair pricing that satisfies both content creators and AI companies
2
. - Detecting and managing AI bots that attempt to evade the system
5
.
Source: PYMNTS
The Future of AI-Content Creator Relationships
Cloudflare envisions a future where intelligent agents can programmatically negotiate access to digital resources. This could lead to more sophisticated research tools and personalized content acquisition
2
.As the AI landscape evolves, Cloudflare's Pay-Per-Crawl system may play a crucial role in balancing the interests of content creators and AI companies, potentially providing a sustainable model for content monetization in the AI era.
References
Summarized by
Navi
[2]
[3]
[4]
Related Stories
Cloudflare Sets September Deadline for AI Crawlers to Pay Publishers or Face Default Blocks
01 Jul 2026•Technology
Cloudflare Unveils Tools to Combat AI Data Scraping, Empowering Website Owners
23 Sept 2024
Publishers Deploy New Licensing Standard to Make AI Crawlers Pay for Content Scraping
09 Dec 2025•Policy and Regulation
Recent Highlights
1
OpenAI Solves Navier-Stokes Problem With AI, Sparking Existential Crisis in Mathematics
Science and Research
2
OpenAI Agents Launched RubyGems Attack With 2,000+ Malicious Packages, Attempted API Key Theft
Technology
3
OpenAI Reveals 6 New Cases of AI Models Hiding Mistakes and Breaking Rules
Policy and Regulation
Recent Highlights
Today's Top Stories
Don’t drown in AI news. We cut through the noise - filtering, ranking and summarizing the most important AI news, breakthroughs and research daily. Follow topics that matter to you and stay ahead.
