Cloudflare CEO Warns of AI's Threat to Publishers as Traffic Referrals Plummet

Cloudflare CEO says people aren't checking AI chatbots' source links

He's encouraging publishers to make sure they're fairly compensated by AI companies taking their content. Companies that develop generative AI always make it a point to say that they include links to websites in the answers that their chatbots generate for users. But Cloudflare CEO Matthew Prince has revealed to Axios that search traffic referrals keep plummeting. Publishers are facing an existential threat, he said, because people aren't clicking through those chatbot links and are relying more and more on AI summaries without digging deeper. Prince told Axios that 10 years ago, Google sent a publisher one visitor for every two pages it had crawled. Six months ago, the ratio was one visitor for every six pages, and now it's one for every 18. OpenAI sent one visitor to a publisher for every 250 pages it crawled six months ago, while Anthropic sent one visitor for every 6,000 pages. These days, OpenAI sends one visitor to a publisher for every 1,500 pages, whereas Anthropic sends one visitor for every 60,000 pages. People have come to trust AI chatbots more over the past few months. The problem for publishers is that they don't earn from advertisements if people don't click through links leading to their websites, and that's why Prince is encouraging them to take action to make sure they're fairly compensated. Prince said Cloudflare is currently working on a tool to block bots that scrape content for large language models even if a web page already has a "no crawl" instruction. If you'll recall, several outlets had reported in 2024 that AI companies have been ignoring websites' Robots Exclusion Protocol, or robots.txt, files and taking their content anyway to train their technologies. Cloudflare has been looking for ways to block scrapers since last year. But it was only in March when Cloudflare officially introduced AI Labyrinth, which uses AI-generated content to "slow down, confuse, and waste the resources of AI Crawlers and other bots that don't respect 'no crawl' directives." It works by linking an unauthorized crawler a series of AI-generated pages that are convincing enough but don't actually have the contents of the site the tool it's protecting. That way, the crawler ends up wasting time and resources. "I go to war every single day with the Chinese government, the Russian government, the Iranians, the North Koreans, probably Americans, the Israelis, all of them who are trying to hack into our customer sites," Prince said. "And you're telling me, I can't stop some nerd with a C-corporation in Palo Alto?"

Cloudflare CEO warns AI crawlers and summaries are eroding the internet's business model

In context: Cloudflare CEO Matthew Prince recently reiterated his warning that generative AI crawlers and summaries threaten the foundations of the internet's business model. To protect publishers from a flood of artificial AI traffic that offers virtually no authentic site visits in return, the company is devising methods to combat AI scrapers. Speaking at an Axios event in Cannes last week, Prince explained that search engines and chatbots using generative AI to summarize web content have significantly reduced the number of human visitors to many websites. Even compared to six months ago, the problem has worsened considerably. Traditionally, for every six times Google crawled a website, one person might visit and potentially view ads. In contrast, the rate was about 250 to 1 with OpenAI's crawlers and 6,000 to 1 with Anthropic. Today, Cloudflare's CEO estimates that Google's crawl-to-visitor rate has declined to 18 to 1, OpenAI's has worsened to 1,500 to 1, and Anthropic's is approximately 60,000 to 1. This decline is likely because chatbots and search engines now save users the effort of visiting websites. Chatbots can retrieve and summarize information without the user needing to leave the chat interface, and AI overviews from major search engines now offer users answers before they click on search results. When Google first unveiled AI overviews, it claimed that the technology would boost traffic to the original sources of the summarized content. Similarly, large language models such as ChatGPT have recently started citing sources in their responses to help direct traffic back to content creators. However, Prince claims that users, by and large, aren't clicking on the footnotes. Instead, many many are accepting the AI's responses at face value, as trust in the technology has grown over the past six months. Aside from starving websites of traffic and revenue, the trend is potentially dangerous due to AI's known tendency to generate inaccurate or misleading information. In response, Cloudflare, which offers cybersecurity solutions for websites, has launched a new tool called the AI Labyrinth. This tool is designed to use generative AI against the crawlers themselves. Although websites can include instructions to block AI crawlers, many bots either bypass or ignore these directives. When the AI Labyrinth detects such behavior, it leads the bot through a maze of AI-generated links that no human would reasonably follow, causing the bot to waste time and computing resources. Despite how daunting it might seem to oppose AI giants like Google, Microsoft, and OpenAI, Prince emphasizes that Cloudflare has a strong track record of successfully defending its clients, even against attacks from powerful national governments.

Publishers facing existential threat from AI, Cloudflare CEO says

Publishers face an existential threat in the AI era and need to take action to make sure they are fairly compensated for their content, Cloudflare CEO Matthew Prince told Axios at an event in Cannes on Thursday. Why it matters: Search traffic referrals have plummeted as people increasingly rely on AI summaries to answer their queries, forcing many publishers to reevaluate their business models. Startling stat: Ten years ago, Google crawled two pages for every visitor it sent a publisher, per Prince. Between the lines: "People aren't following the footnotes," Prince said. What to watch: Prince said Cloudflare is working on a new tool that will stop content scraping. The bottom line: Prince is optimistic Cloudflare can pull this off.