AI Bots Surge to 1 in 31 Web Visits as Publishers Face Traffic Crisis and Arms Race

AI Bots Rapidly Transform Web Traffic Landscape

The internet is undergoing a fundamental shift as AI bots account for an increasingly significant portion of web traffic 1

1

2

2

. According to a new report from TollBit, a company that tracks web-scraping activity, AI bot web traffic reached one visit for every 31 human visits by the fourth quarter of 2025, a dramatic increase from just one bot visit for every 200 human visits in the first quarter of the same year 3

3

. This represents approximately 2 percent of all web traffic in Q4 2025, up from just 0.5 percent in Q1 5

5

. Data shared with WIRED by internet infrastructure company Akamai confirms that AI bots already account for a meaningful share of web traffic, with training-related bot activity rising steadily since July 1

1

.

Retrieval Augmented Generation Drives Increase in AI Bot Activity

The surge in bots scraping content is being driven less by model training and more by Retrieval Augmented Generation (RAG) systems that fetch real-time information for AI tools like ChatGPT and Gemini 3

3

. While training scrapes actually dropped by 15 percent between Q2 and Q4 of 2025, RAG bot traffic increased by 33 percent during the same period 3

3

4

4

. AI search indexers, which build indexes used by RAG bots, saw traffic jump by 59 percent 3

3

. This shift reflects changing user behavior, with 37 percent of active AI users now starting their searches in AI platforms like ChatGPT or Gemini rather than traditional search engines 3

3

. OpenAI leads the charge, with its ChatGPT-User RAG bot averaging five times as many scrapes per page as the second-highest scraper from Meta, and approximately 16 times higher than Perplexity's agent 3

3

4

4

.

Arms Race Intensifies as Bots Bypass Website Defenses

An increasingly sophisticated arms race is unfolding as AI bots deploy clever tactics to bypass website defenses 1

1

. More than 13 percent of bot requests were bypassing robots.txt in Q4 2025, a file that websites use to indicate which pages bots should avoid 1

1

2

2

. The share of AI bots disregarding robots.txt increased 400 percent from Q2 to Q4 of last year 1

1

2

2

. Some bots disguise themselves by making their traffic appear like it's coming from a normal web browser or send requests designed to mimic how humans interact with websites 1

1

. TollBit's study notes that the behavior of some AI agents is now almost indistinguishable from human web traffic 1

1

. In response, TollBit reported a 336 percent increase in the number of websites making attempts to block AI bots over the past year 1

1

2

2

.

Publishers Face Declining Human Web Traffic and Click-Through Rates

Publishers are being hit particularly hard by these shifts. Human web traffic declined by 5 percent between Q3 and Q4 of 2025 3

3

4

4

. Even more concerning, click-through rates from AI apps to source websites fell dramatically from 0.8 percent in Q2 2025 to just 0.27 percent in Q4, a nearly threefold decline 3

3

. Even websites with AI licensing deals aren't insulated from this trend, with their click-through rates dropping to just 1.33 percent in Q4, a 6.5-fold decrease from earlier in the year 3

3

. "Anyone who relies on human web traffic -- starting with publishers, but basically everyone -- is going to be impacted," says Toshit Pangrahi, cofounder and CEO of TollBit . WIRED's parent company, Condé Nast, along with other publishers, are currently suing several AI companies over alleged copyright infringement related to AI training .

Machine-to-Machine Interactions Shape Future of the Web

Robert Blumofe, Akamai's chief technology officer, warns that "AI is changing the web as we know it. The ensuing arms race will determine the future look, feel, and functionality of the web, as well as the basics of doing business" . Pangrahi emphasizes the need for new systems: "There needs to be a faster way to have that machine-to-machine, programmatic exchange of value" . TollBit markets tools that website owners can use to charge AI scrapers for accessing their content, with other firms including Cloudflare offering similar countermeasures . Olivia Joslin, TollBit's cofounder and COO, predicts that "AI traffic will continue to surge and replace direct human visitors to sites. Ultimately, AI will become the primary reader of the Internet" 3

3

. She estimates that 2026 could be the year when AI visitors become the dominant visitors to publisher sites 3

3

. TollBit calls for regulators to step in to establish acceptable AI patterns and protect intellectual property as robots.txt has been deemed effectively obsolete 4

4

. Web scraping firms like Bright Data, ScrapingBee, and Oxylabs maintain that their content scraping practices focus on publicly available information, though several have faced lawsuits from Meta and other platforms .