OpenAI Disrupts Malicious AI Use by State-Sponsored and Cybercriminal Groups

Reviewed byNidhi Govil

11 Sources

Share

OpenAI's latest threat report reveals how state-sponsored actors and cybercriminals are misusing AI tools for surveillance, malware development, and influence campaigns. The company has banned over 40 malicious networks since February 2024.

OpenAI Uncovers Widespread AI Misuse by Malicious Actors

OpenAI has released a comprehensive threat report detailing how state-sponsored groups and cybercriminals are exploiting artificial intelligence (AI) tools for malicious purposes. Since February 2024, the company has disrupted over 40 networks that violated its usage policies, providing valuable insights into the evolving landscape of AI-related cybercrime

1

2

.

Source: The Hacker News

Source: The Hacker News

State-Sponsored Actors Leverage AI for Surveillance and Influence Campaigns

The report highlights concerning activities by state-linked entities:

  1. Chinese Government-Affiliated Accounts: OpenAI banned accounts suspected to be linked to Chinese government entities for attempting to use ChatGPT to design large-scale monitoring systems. These accounts requested help in creating tools to analyze social media networks, monitor the Uyghur minority group, and identify funding sources for government critics

    1

    2

    3

    .

  2. Russian Influence Operations: A cluster of Russian-linked accounts was found using ChatGPT to generate video prompts for influence campaigns. These accounts also explored using multiple AI models to produce content for social media platforms like YouTube and TikTok

    2

    4

    .

  3. North Korean Cyber Activities: OpenAI disrupted North Korean-linked accounts that were using ChatGPT for malware and command-and-control (C2) development. These actors explored techniques for developing macOS malware, configuring VPNs, and converting browser extensions

    3

    .

Source: TechRadar

Source: TechRadar

Cybercriminals Adapt AI into Existing Workflows

The report also sheds light on how cybercriminal groups are integrating AI into their operations:

  1. Malware Development: Russian-speaking criminal groups were found using ChatGPT to develop and refine malware, including remote access trojans (RATs) and credential stealers

    1

    2

    .

  2. Phishing Campaigns: Chinese-language accounts, possibly linked to the threat group UTA0388, used AI tools to craft phishing content and debug malicious code

    1

    3

    .

  3. Efficiency Improvements: An organized crime network, believed to be based in Cambodia, attempted to use ChatGPT to make their workflows more efficient and error-free

    1

    4

    .

Source: SiliconANGLE

Source: SiliconANGLE

AI Models Show Resilience Against Novel Attack Requests

Despite these concerning trends, OpenAI reports that there is little to no evidence of existing AI models being used to develop entirely new attack techniques. The company's AI systems have consistently refused direct requests for malicious content, forcing threat actors to work around these limitations

1

3

.

OpenAI's Approach to Threat Detection and Mitigation

To combat these threats, OpenAI employs a nuanced approach that focuses on patterns of threat actor behavior rather than isolated model interactions. This strategy allows the company to detect and disrupt malicious activities effectively while minimizing disruption to legitimate users

4

5

.

As the AI landscape continues to evolve, OpenAI remains committed to developing tools and models that benefit defenders across society. However, the company acknowledges the ongoing challenge of balancing innovation with security, as threat actors continue to adapt and innovate in their use of AI technologies

1

5

.

TheOutpost.ai

Your Daily Dose of Curated AI News

Don’t drown in AI news. We cut through the noise - filtering, ranking and summarizing the most important AI news, breakthroughs and research daily. Spend less time searching for the latest in AI and get straight to action.

© 2025 Triveous Technologies Private Limited
Instagram logo
LinkedIn logo