AI Bots Surge to 1 in 31 Web Visits as Publishers Face Traffic Crisis and Arms Race

Increase of AI bots on the Internet sparks arms race

The viral virtual assistant OpenClaw -- formerly known as Moltbot, and before that Clawdbot -- is a symbol of a broader revolution underway that could fundamentally alter how the Internet functions. Instead of a place primarily inhabited by humans, the web may very soon be dominated by autonomous AI bots. A new report measuring bot activity on the web, as well as related data shared with WIRED by the Internet infrastructure company Akamai, shows that AI bots already account for a meaningful share of web traffic. The findings also shed light on an increasingly sophisticated arms race unfolding as bots deploy clever tactics to bypass website defenses meant to keep them out. "The majority of the Internet is going to be bot traffic in the future," says Toshit Pangrahi, cofounder and CEO of TollBit, a company that tracks web-scraping activity and published the new report. "It's not just a copyright problem, there is a new visitor emerging on the Internet." Most big websites try to limit what content bots can scrape and feed to AI systems for training purposes. (WIRED's parent company, Condé Nast, as well as other publishers, are currently suing several AI companies over alleged copyright infringement related to AI training.) But another kind of AI-related website scraping is now on the rise as well. Many chatbots and other AI tools can now retrieve real-time information from the web and use it to augment and improve their outputs. This might include up-to-the-minute product prices, movie theater schedules, or summaries of the latest news. According to the data from Akamai, training-related bot traffic has been rising steadily since last July. Meanwhile, global activity from bots fetching web content for AI agents is also on the upswing. "AI is changing the web as we know it," Robert Blumofe, Akamai's chief technology officer, tells WIRED. "The ensuing arms race will determine the future look, feel, and functionality of the web, as well as the basics of doing business." In the fourth quarter of 2025, TollBit estimates that an average of one out of every 31 visits to its customers' websites was from an AI scraping bot. In the first quarter, that figure was only one out of every 200. The company says that in the fourth quarter, more than 13 percent of bot requests were bypassing robots.txt, a file that some websites use to indicate which pages bots are supposed to avoid. TollBit says the share of AI bots disregarding robots.txt increased 400 percent from the second quarter to the fourth quarter of last year. TollBit also reported a 336 percent increase in the number of websites making attempts to block AI bots over the past year. Pangrahi says that scraping techniques are getting more sophisticated as sites try to assert control over how bots access their content. Some bots disguise themselves by making their traffic appear like it's coming from a normal web browser or send requests designed to mimic how humans normally interact with websites. TollBit's study notes that the behavior of some AI agents is now almost indistinguishable from human web traffic. TollBit markets tools that website owners can use to charge AI scrapers for accessing their content. Other firms, including Cloudflare, offer similar tools. "Anyone who relies on human web traffic -- starting with publishers, but basically everyone -- is going to be impacted," Pangrahi says. "There needs to be a faster way to have that machine-to-machine, programmatic exchange of value." WIRED attempted to contact 15 AI scraping companies cited in the TollBit report for comment. The majority did not respond or could not be reached. Several said that their AI systems aim to respect technical boundaries that websites put in place to limit scraping, but they noted such guardrails can often be complex and difficult to follow. Or Lenchner, the CEO of Bright Data, one of the world's largest web-scraping firms, says that his company's bots do not collect nonpublic information. Bright Data was previously sued by Meta and X for allegedly improperly scraping content from their platforms. (Meta later dropped its suit, and a federal judge in California dismissed the case brought by X.) Karolis Stasiulevičiu, a spokesperson for another cited company, ScrapingBee, told WIRED: "ScrapingBee operates on one of the Internet's core principles: that the open web is meant to be accessible. Public web pages are, by design, readable by both humans and machines." Oxylabs, another scraping firm, said in an unsigned statement that its bots don't have "access to content behind logins, paywalls, or authentication. We require customers to use our services only for accessing publicly available information, and we enforce compliance standards throughout our platform." Oxylabs added that there are many legitimate reasons for firms to scrape web content, including for cybersecurity purposes and to conduct investigative journalism. The company also says that the countermeasures some websites use do not discriminate between different use cases. "The reality is that many modern anti-bot systems don't distinguish well between malicious traffic and legitimate automated access," Oxylabs says. In addition to causing headaches for publishers, the web-scraping wars are creating new business opportunities. TollBit's report found more than 40 companies that are now marketing bots that can collect web content for AI training or other purposes. The rise of AI-powered search engines, as well as tools like OpenClaw, are likely helping drive up demand for these services. Some firms promise to help companies surface content for AI agents rather than try to block them, a strategy known as generative engine optimization, or GEO. "We're essentially seeing the rise of a new marketing channel," says Uri Gafni, chief business officer of Brandlight, a company that optimizes content so that it appears prominently in AI tools. "This will only intensify in 2026, and we're going to see this rollout kind of as a full-on marketing channel, with search, ads, media, and commerce converging," Gafni says. This story originally appeared on wired.com.

AI Bots Are Now a Signifigant Source of Web Traffic

The viral virtual assistant OpenClaw -- formerly known as Moltbot, and before that Clawdbot -- is a symbol of a broader revolution underway that could fundamentally alter how the internet functions. Instead of a place primarily inhabited by humans, the web may very soon be dominated by autonomous AI bots. A new report measuring bot activity on the web, as well as related data shared with WIRED by the internet infrastructure company Akamai, shows that AI bots already account for a meaningful share of web traffic. The findings also shed light on an increasingly sophisticated arms race unfolding as bots deploy clever tactics to bypass website defenses meant to keep them out. "The majority of the internet is going to be bot traffic in the future," says Toshit Pangrahi, cofounder and CEO of TollBit, a company that tracks web-scraping activity and published the new report. "It's not just a copyright problem, there is a new visitor emerging on the internet." Most big websites try to limit what content bots can scrape and feed to AI systems for training purposes. (WIRED's parent company, Condé Nast, as well as other publishers, are currently suing several AI companies over alleged copyright infringement related to AI training.) But another kind of AI-related website scraping is now on the rise as well. Many chatbots and other AI tools can now retrieve real-time information from the web and use it to augment and improve their outputs. This might include up-to-the-minute product prices, movie theater schedules, or summaries of the latest news. According to the data from Akamai, training-related bot traffic has been rising steadily since last July. Meanwhile, global activity from bots fetching web content for AI agents is also on the upswing. "AI is changing the web as we know it," Robert Blumofe, Akamai's chief technology officer, tells WIRED "The ensuing arms race will determine the future look, feel, and functionality of the web, as well as the basics of doing business." In the fourth quarter of 2025, TollBit estimates that an average of one out of every 50 visits to its customers' websites was from an AI scraping bot. In the first three months of 2025, that figure was only one out of every 200. The company says that in the fourth quarter, more than 13 percent of bot requests were bypassing robots.txt, a file that some websites use to indicate which pages bots are supposed to avoid. TollBit says the share of AI bots disregarding robots.txt increased 400 percent from the second quarter to the fourth quarter of last year. TollBit also reported a 336 percent increase in the number of websites making attempts to block AI bots over the past year. Pangrahi says that scraping techniques are getting more sophisticated as sites try to assert control over how bots access their content. Some bots disguise themselves by making their traffic appear like it's coming from a normal web browser or send requests designed to mimic how humans normally interact with websites. TollBit's study notes that the behavior of some AI agents is now almost indistinguishable from human web traffic. TollBit markets tools that website owners can use to charge AI scrapers for accessing their content. Other firms, including Cloudflare, offer similar tools. "Anyone who relies on human web traffic -- starting with publishers, but basically everyone -- is going to be impacted," Pangrahi says. "There needs to be a faster way to have that machine-to-machine, programmatic exchange of value."

AI bot traffic closing in on human web visits, study finds

RAG bots could overtake human visitors on publisher sites this year, trackers tell us The AI bot takeover of the internet continues apace, and the latest data suggests the surge is being driven less by model-training scrapes and more by the growing use of AI tools as a stand-in for web search. Tollbit, an outfit that tracks AI bot traffic, said in its latest State of the Bots report that by Q4 2025, there was roughly one AI bot visit for every 31 human visits to a site, up from just one bot visit for every 200 human visits in Q1. That's likely a conservative estimate too, says Tollbit, as AI bots just keep getting better at seeming like humans when they navigate a website. "From the tests we ran ... many of these web scrapers are indistinguishable from human visitors on sites," Tollbit noted. "In light of this, the data below is conservative; it is likely worse than these numbers." To further drive home the fact that we're losing the internet to AI bots, human web traffic is also on the decline, Tollbit noted in its data. From Q3 to Q4 of 2025, the company noted, human visitors to websites declined by five percent. It seems like just yesterday that the biggest concern about AI taking over the web had to do with firms sucking up every piece of content they could find to train their bots with. That's still going on - as evidenced by all the copyright cases being fought in courts - but training scrapers are no longer the reason for most of the AI bot traffic. Training scrapes actually dropped by 15 percent between Q2 and Q4 of last year, Tollbit noted. Instead of scraping to develop models, bot traffic is being dominated by retrieval augmented generation (RAG) bots, which are what companies like OpenAI, Google, and others use to extract real-time information from the web to answer queries put to ChatGPT, Gemini, and the like. RAG bot traffic increased by 33 percent in the same period that training bot traffic declined. AI search indexers, which build indexes used by RAG bots, also saw traffic increase by 59 percent in the same period. Unsurprisingly, the heaviest scraping activity is attributed to OpenAI. Per Tollbit, ChatGPT-User, the company's RAG bot, averages five times as many scrapes per page as the second highest scraper, which comes from Meta. If it's not clear what that means, it's that humans are abandoning the internet and letting bots pull and collate information for them at an increasing rate. According to marketing firm Eight Oh Two, 37 percent of active AI users now start their searches in AI platforms like ChatGPT or Gemini rather than turning first to Google or other traditional search engines. Pew Research found that 62 percent of US adults use AI in some form at least several times a week, suggesting that, if it's not gaining traction in the enterprise, consumers are at least lapping it up. Tollbit said that B2B and professional websites, national news, and lifestyle content are the most AI-scraped sites, though the highest increase in scraping since Q2 of last year was seen in the tech and consumer electronics space, which jumped by 107 percent. The second-highest increase, B2B and professional, rose just 62 percent. "This growth is likely to be driven by a corresponding increase in relevant prompts on consumer AI applications that stems from more users turning to these tools for information retrieval tasks," Tollbit noted. AI users driving the RAG surge aren't exactly checking their references. The already abysmal referral traffic from AI apps to the sites they source is also on the decline, Tollbit said, and dramatically. From 0.8 percent in Q2 2025, clickthrough rates from AI apps fell to just 0.27 percent in Q4, a decline of nearly threefold. Tollbit noted that websites with AI licensing deals aren't being insulated from this either, with their clickthrough rates dropping to just 1.33 percent in Q4 - a 6.5x decrease from earlier in the year. What that means for the future of the internet, according to Tollbit cofounder and chief operating officer Olivia Joslin, is clear: The internet may soon not be a place for us meatbags. "AI traffic will continue to surge and replace direct human visitors to sites," Joslin told The Register in an email. "Ultimately, AI will become the primary reader of the Internet." Joslin further predicted that, at the rate human traffic is declining and bot traffic is increasing, the internet could become a bot-first operation in short order. "It could be this year that we see AI visitors being the dominant visitors to publisher sites," Joslin estimated. Unfortunately for publishers, Joslin sees the shift from human to bot dominance online as inevitable. "AI is terrific at answering our questions and enabling us to complete deep research; it's inescapable," she noted. "AI visitors read far more than human visitors; they don't get tired, and they can do far deeper research, while we get bored after looking at the third link." "The odds are very much stacked against publishers," Joslin told us. The future might not be great for AI users, either. Some studies suggest that AI use has a direct negative impact on critical thinking and skills. Young people are frequent users of it despite mental health warnings, and studies of students found that those relying on AI for help writing essays showed poorer knowledge retention than those doing the legwork on their own. Warnings of an AI-fueled intellectual crisis don't even touch on the fact that AI is yet another level of curation on top of search engine result algorithms that affect what sort of information gets delivered to users and in what format. So here we are, yet again, with more data suggesting the current iteration of the internet is dying. This time, however, it seems we need to point the finger at the internet users who are gladly sacrificing it on the altar of convenience that is the AI chatbot. ®

AI bot web traffic is closing in on human usage, experts warn

Publishers are battling with dwindling click-through rates New data has claimed AI bots are rapidly taking over web traffic, with Tollbit information showing there was a new AI bot visit for every 31 human visits in the final months of 2025, up from 1:200 at the start of 2025. At the same time, and with humans interacting more directly with AI, human visits fell by around 5% between Q3 and Q4 2025. And it's clear human AI usage is responsible for this shift - while training crawls fell by around 15% between Q2 and Q4 2025, RAG bots rose around 33% and AI search indexers rose around 59%. OpenAI leads the way in terms of scraping - its RAG bot 'ChatGPT-User' was around five times more active than the second-most active bot by Meta, and around 16x higher than Perplexity's agent. In terms of user behavior, a separate survey cited in the report found more than one-third (37%) of active AI users now start searches with artificial intelligence (like ChatGPT or Gemini) rather than traditional search. More alarmingly, the analysis found that robots.txt, a set of instructions to tell automated bots which parts of the page they can and cannot crawl, was ignored around 30% of the time on average, and up to 42% of the time by ChatGPT-User. With robots.txt unenforceable and totally reliant on goodwill, the instructions have been deemed effectively obsolete. And that's not the only bad news for publishers, who are struggling on the traffic front. Sites without direct AI licensing deals saw click-through rates (CTR) drop around 3x between Q2 and Q4 2025. Even those with AI licensing deals aren't immune, with CTR rates also dropping. With all of this at play, Tollbit calls for regulators to step in to establish acceptable AI patterns and protect intellectual property. But with AI going nowhere, it's clear that the internet is undergoing a major shift and website owners should prepare for AI bots to become their primary readers.

AI Bot Traffic to Websites is Rising Rapidly. It Could Change the Nature of the Web

Tollbit, which tracks web-scraping activity, found that AI bots made up 2 percent of all traffic on the web in the fourth quarter of last year. That's up from just half a percent in the first quarter, underlining just how quickly things are shifting. In the first quarter, on average, there was one AI bot visit for every 200 human visits to a website. By the end of the year, that ratio had shifted to one AI bot visit for every 31 visits made by humans. The new generation of bots is a lot more aggressive than early content scrapers, too. The report found that AI bots are more likely to ignore barriers like firewalls or robots.txt files, which are meant to block bots. Several publishers and content sites have already filed lawsuits against AI companies for scraping their content and archives without permission. But Tollbit said more needs to be done on the enforcement side to discourage this behavior.