AI Chatbots Inadvertently Aiding Phishing Scams by Providing Incorrect URLs

AI Chatbots Unintentionally Facilitating Phishing Attacks

Recent research has uncovered a concerning trend in the world of artificial intelligence: AI-powered chatbots, including popular models like ChatGPT, are frequently providing incorrect URLs when asked about company websites. This oversight could potentially expose users to phishing attacks and other cyber threats, raising significant security concerns in the AI community 1

1

The Scope of the Problem

Source: The Register

Cybersecurity firm Netcraft conducted a study using the GPT-4.1 family of models, which powers platforms like Microsoft's Bing AI and Perplexity. The research team prompted the AI with questions about login URLs for 50 different brands across various industries. The results were alarming:

• Only 66% of the provided URLs were correct
• 29% redirected to dead or suspended websites
• 5% led to legitimate sites unrelated to the requested brand 2

  2

This inaccuracy opens up opportunities for cybercriminals to exploit the AI's mistakes. By registering unclaimed domains suggested by the AI, attackers could set up convincing phishing sites to harvest users' sensitive information.

Real-World Implications

The threat is not merely theoretical. Netcraft's team observed a real-world instance where the AI search engine Perplexity redirected users to a fake Wells Fargo website, which appeared to be a phishing attempt 1

1

Smaller brands, such as credit unions, regional banks, and mid-sized fintech platforms, are particularly vulnerable. These companies are often underrepresented in the AI's training data, increasing the likelihood of the AI generating incorrect or "hallucinated" URLs 3

3

Related Stories

ChatGPT caught recommending scam websites that steal your credit card info

ChatGPT Crawler Vulnerability: Potential for DDoS Attacks and Prompt Injection

AI Chatbots: Unwitting Accomplices in Phishing Scams Targeting Seniors

OpenAI Confirms ChatGPT Abuse by Hackers for Malware and Election Interference

Evolving Tactics of Cybercriminals

Source: PC Magazine

In response to the growing reliance on AI-powered search tools, cybercriminals are adapting their strategies. Instead of focusing on traditional search engine optimization (SEO) for platforms like Google, attackers are now optimizing their phishing sites for large language models (LLMs) 2

2

This shift in tactics has led to the creation of sophisticated phishing campaigns. For instance, an estimated 17,000 GitBook phishing pages targeting crypto users have been created by mimicking technical support pages, documentation, and login interfaces 3

3

Recommendations for Users

Source: TechRadar

Given these risks, cybersecurity experts are urging users to exercise caution when relying on AI-generated information, especially regarding web addresses. Some key recommendations include:

1. Double-check URLs for inconsistencies before inputting sensitive data
2. Verify any AI-generated content involving web addresses
3. Type URLs directly into the search bar rather than clicking on provided links
4. Be particularly cautious with URLs for smaller or less well-known brands 1

   1

   3

   3

As AI continues to play an increasingly prominent role in our digital lives, it's crucial for users to remain vigilant and for AI developers to address these vulnerabilities to ensure a safer online experience.