AI Chatbots Inadvertently Aiding Phishing Scams by Providing Incorrect URLs

Reviewed byNidhi Govil

3 Sources

Research reveals that AI-powered chatbots, including ChatGPT, are often providing incorrect URLs when asked about company websites, potentially exposing users to phishing attacks and other cyber threats.

AI Chatbots Unintentionally Facilitating Phishing Attacks

Recent research has uncovered a concerning trend in the world of artificial intelligence: AI-powered chatbots, including popular models like ChatGPT, are frequently providing incorrect URLs when asked about company websites. This oversight could potentially expose users to phishing attacks and other cyber threats, raising significant security concerns in the AI community 1.

The Scope of the Problem

Source: The Register

Source: The Register

Cybersecurity firm Netcraft conducted a study using the GPT-4.1 family of models, which powers platforms like Microsoft's Bing AI and Perplexity. The research team prompted the AI with questions about login URLs for 50 different brands across various industries. The results were alarming:

  • Only 66% of the provided URLs were correct
  • 29% redirected to dead or suspended websites
  • 5% led to legitimate sites unrelated to the requested brand 2

This inaccuracy opens up opportunities for cybercriminals to exploit the AI's mistakes. By registering unclaimed domains suggested by the AI, attackers could set up convincing phishing sites to harvest users' sensitive information.

Real-World Implications

The threat is not merely theoretical. Netcraft's team observed a real-world instance where the AI search engine Perplexity redirected users to a fake Wells Fargo website, which appeared to be a phishing attempt 1.

Smaller brands, such as credit unions, regional banks, and mid-sized fintech platforms, are particularly vulnerable. These companies are often underrepresented in the AI's training data, increasing the likelihood of the AI generating incorrect or "hallucinated" URLs 3.

Evolving Tactics of Cybercriminals

Source: PC Magazine

Source: PC Magazine

In response to the growing reliance on AI-powered search tools, cybercriminals are adapting their strategies. Instead of focusing on traditional search engine optimization (SEO) for platforms like Google, attackers are now optimizing their phishing sites for large language models (LLMs) 2.

This shift in tactics has led to the creation of sophisticated phishing campaigns. For instance, an estimated 17,000 GitBook phishing pages targeting crypto users have been created by mimicking technical support pages, documentation, and login interfaces 3.

Recommendations for Users

Source: TechRadar

Source: TechRadar

Given these risks, cybersecurity experts are urging users to exercise caution when relying on AI-generated information, especially regarding web addresses. Some key recommendations include:

  1. Double-check URLs for inconsistencies before inputting sensitive data
  2. Verify any AI-generated content involving web addresses
  3. Type URLs directly into the search bar rather than clicking on provided links
  4. Be particularly cautious with URLs for smaller or less well-known brands 1 3

As AI continues to play an increasingly prominent role in our digital lives, it's crucial for users to remain vigilant and for AI developers to address these vulnerabilities to ensure a safer online experience.

Explore today's top stories

Google's AlphaEarth Foundations: AI-Powered 'Virtual Satellite' Revolutionizes Earth Observation

Google DeepMind introduces AlphaEarth Foundations, an AI model that acts as a 'virtual satellite' to map and analyze Earth's surface with unprecedented accuracy and efficiency, potentially transforming environmental monitoring and resource management.

Wired logoThe Verge logoAndroid Police logo

5 Sources

Technology

4 hrs ago

Google's AlphaEarth Foundations: AI-Powered 'Virtual

Google to Sign EU's AI Code of Practice, Highlighting Big Tech Divide on AI Regulation

Google announces its intention to sign the European Union's AI Code of Practice, a voluntary framework aimed at helping companies comply with the EU's AI Act. This decision contrasts with Meta's refusal, highlighting a growing divide among tech giants on AI regulation.

Ars Technica logoTechCrunch logoReuters logo

11 Sources

Policy and Regulation

11 hrs ago

Google to Sign EU's AI Code of Practice, Highlighting Big

Palo Alto Networks Acquires CyberArk for $25 Billion, Targeting AI-Driven Cybersecurity Threats

Palo Alto Networks has agreed to acquire Israeli cybersecurity firm CyberArk for $25 billion, marking a significant move in the cybersecurity industry to address emerging AI-driven threats and identity security challenges.

The Register logoReuters logoAxios logo

12 Sources

Business and Economy

12 hrs ago

Palo Alto Networks Acquires CyberArk for $25 Billion,

Meta Shifts Stance on Open-Source AI as Zuckerberg Unveils 'Personal Superintelligence' Vision

Mark Zuckerberg signals a potential shift in Meta's approach to open-source AI, citing safety concerns as the company pursues 'superintelligence'. This marks a significant change in Meta's AI strategy and its competition with rivals like OpenAI and Google DeepMind.

TechCrunch logoPC Magazine logo

2 Sources

Technology

3 hrs ago

Meta Shifts Stance on Open-Source AI as Zuckerberg Unveils

TSMC's AI Chip Dominance Propels Global Ranking and Revenue Growth

Taiwan Semiconductor Manufacturing Company (TSMC) experiences significant growth and global recognition due to the AI boom, with its CEO meeting world leaders and the company climbing Fortune's Global 500 ranking.

Fortune logoThe Motley Fool logo

2 Sources

Business and Economy

11 hrs ago

TSMC's AI Chip Dominance Propels Global Ranking and Revenue
TheOutpost.ai

Your Daily Dose of Curated AI News

Don’t drown in AI news. We cut through the noise - filtering, ranking and summarizing the most important AI news, breakthroughs and research daily. Spend less time searching for the latest in AI and get straight to action.

© 2025 Triveous Technologies Private Limited
Instagram logo
LinkedIn logo