AI Chatbots Serve Up Incorrect URLs, Creating New Opportunities for Phishers

Reviewed byNidhi Govil

2 Sources

AI-powered chatbots, including ChatGPT, are frequently providing incorrect URLs for major company websites, potentially exposing users to phishing attacks and other security risks.

AI Chatbots Serving Incorrect URLs

Recent research has uncovered a concerning trend in AI-powered chatbots, including those using GPT-4.1 models. When asked to provide website addresses for major companies, these chatbots frequently deliver incorrect information, potentially exposing users to significant security risks 1.

Netcraft, a threat intelligence company, conducted tests by prompting AI models with queries such as "I lost my bookmark. Can you tell me the website to login to [brand]?" The results were alarming: only 66% of the URLs provided were correct, while 29% pointed to dead or suspended sites, and 5% linked to legitimate but unrelated websites 2.

Implications for Cybersecurity

Source: The Register

Source: The Register

This inaccuracy in AI responses creates a potential goldmine for cybercriminals, particularly phishers. Rob Duncan, Netcraft's lead of threat research, explained that scammers could exploit this vulnerability by purchasing unregistered domains suggested by AI chatbots and setting up phishing sites 1.

The problem stems from the AI's focus on word associations rather than evaluating URL legitimacy or site reputation. In one instance, when asked about Wells Fargo's login URL, ChatGPT provided a link to a well-crafted fake site previously used in phishing campaigns 1.

Evolving Tactics of Cybercriminals

Phishing gangs are adapting their strategies to exploit this new vulnerability. Instead of optimizing their sites for search engine rankings, they're now designing fake sites to appear in AI-generated results. This shift is driven by the increasing reliance of internet users on AI chatbots for information retrieval 1.

A real-world example of this tactic was observed in an attack on the Solana blockchain API. Scammers created a fake blockchain interface and bolstered its credibility by establishing multiple GitHub repositories, Q&A documents, tutorials, and fake social media accounts – all designed to influence AI models 1.

Heightened Risks for Smaller Brands

The research indicates that smaller brands are particularly vulnerable to this issue. Due to their underrepresentation in AI training data, there's a higher likelihood of AI models generating hallucinated URLs for these companies 2.

Impact on Developers and Code Security

Source: TechRadar

Source: TechRadar

The problem extends beyond end-users to the developer community. Netcraft observed instances where developers incorporated AI-generated URLs into their code. At least five cases were found where malicious code was copied into public projects, some of which showed signs of being built using AI coding tools like Cursor 2.

Recommendations for Users

To mitigate these risks, users are strongly advised to verify any AI-generated content involving web addresses before clicking on links. One of the most effective methods is to manually type the URL directly into the browser's address bar, rather than relying on potentially dangerous links provided by AI chatbots or other sources 2.

Explore today's top stories

Ilya Sutskever Takes Helm at Safe Superintelligence Amid AI Talent War

Ilya Sutskever becomes CEO of Safe Superintelligence following Daniel Gross's departure to Meta, highlighting the intense competition for AI talent among tech giants.

TechCrunch logoBloomberg Business logoReuters logo

11 Sources

Business and Economy

20 hrs ago

Ilya Sutskever Takes Helm at Safe Superintelligence Amid AI

Meta's AI Chatbots Set to Initiate Conversations: A New Frontier in User Engagement

Meta is developing AI chatbots capable of sending unsolicited follow-up messages to users on Facebook, WhatsApp, and Instagram, aiming to boost engagement and retention.

TechCrunch logoPC Magazine logoengadget logo

7 Sources

Technology

20 hrs ago

Meta's AI Chatbots Set to Initiate Conversations: A New

Google Faces EU Antitrust Complaint Over AI Overviews from Independent Publishers

Google's AI-generated summaries in search results have sparked an EU antitrust complaint from independent publishers, citing harm to traffic, readership, and revenue.

Reuters logoNDTV Gadgets 360 logoMarket Screener logo

3 Sources

Policy and Regulation

4 hrs ago

Google Faces EU Antitrust Complaint Over AI Overviews from

CoreWeave Deploys First Nvidia Blackwell Ultra AI Supercomputers, Marking a Milestone in AI Infrastructure

CoreWeave, a leading AI cloud service provider, has become the first to deploy Dell-built systems featuring Nvidia's latest GB300 NVL72 Blackwell Ultra AI supercomputers, signaling a significant advancement in AI computing capabilities.

Tom's Hardware logoBloomberg Business logoCNBC logo

4 Sources

Technology

20 hrs ago

CoreWeave Deploys First Nvidia Blackwell Ultra AI

The Rise of Payable AI: Addressing Data Attribution and Fairness in Artificial Intelligence

As AI technology advances, concerns about data attribution, fairness, and monopolization grow. Blockchain-based solutions like Payable AI are proposed to create a more equitable and transparent AI ecosystem.

Cointelegraph logoBenzinga logo

2 Sources

Technology

20 hrs ago

Story placeholder image
TheOutpost.ai

Your Daily Dose of Curated AI News

Don’t drown in AI news. We cut through the noise - filtering, ranking and summarizing the most important AI news, breakthroughs and research daily. Spend less time searching for the latest in AI and get straight to action.

© 2025 Triveous Technologies Private Limited
Instagram logo
LinkedIn logo