OpenAI Disrupts Malicious AI Use by State-Sponsored and Cybercriminal Groups

AI is making cybercriminal workflows more efficient too, OpenAI finds

OpenAI has disrupted over 40 networks involved in abuse to date. OpenAI has published research revealing how state-sponsored and cybercriminal groups are abusing artificial intelligence (AI) to spread malware and perform widespread surveillance. Also: Everything OpenAI announced at DevDay 2025: Agent Kit, Apps SDK, ChatGPT, and more (Disclosure: Ziff Davis, ZDNET's parent company, filed an April 2025 lawsuit against OpenAI, alleging it infringed Ziff Davis copyrights in training and operating its AI systems.) AI has benefits in the cybersecurity space; it can automate tedious and time-consuming tasks, freeing up human specialists to focus on complex projects and research, for example. However, as with any technology -- whether it is an AI system designed to triage cybercrime alerts or a penetration testing tool -- there is a capacity for malicious use. Also: 43% of workers say they've shared sensitive info with AI - including financial and client data With this in mind, since February 2024, OpenAI has issued public threat reports and has closely monitored the use of AI tools by threat actors. Since last year, OpenAI has disrupted over 40 malicious networks that have violated its usage policies, and an analysis of these networks is now complete, giving us a glimpse into the current trends of AI-related cybercrime. Published on Monday, OpenAI's report, "Disrupting malicious uses of AI: an update" (PDF), details four major trends, all of which expose how AI is being used to rapidly change the existing Tactics, Techniques, and Procedures (TTPs) of threat actors. The first trend is the increasing use of AI in existing workflows. Many of the accounts banned by the developer were repeatedly building AI into cybercriminal networks. For example, the OpenAI team found evidence of this abuse, believed to be located in Cambodia, when an organized crime network tried to use ChatGPT to "make their workflows more efficient and error-free." A number of accounts were also banned for attempting to generate Remote Access Trojans (RATs), credential stealers, obfuscation tools, as well as crypters and payload crafting code. The second significant area of concern is threat groups that use multiple AI tools and models for distinct malicious or abusive purposes. These include a likely Russian entity that used various AI tools to generate video prompts and fraudulent content designed to be spread over social media, news-style short videos, and propaganda. Want more stories about AI? Sign up for Innovation, our weekly newsletter. In another case, a number of Chinese-language accounts were banned for trying to use ChatGPT to craft phishing content and for debugging. It is believed that this group could be threat actors tracked as UTA0388, known for targeting Taiwan's semiconductor industry, think tanks, and US academia. OpenAI also described how cybercriminals are using AI for adaptation and obfuscation. A number of networks, thought to originate from Cambodia, Myanmar, and Nigeria, are aware that AI content and code are detectable, and so have asked AI models to remove markers such as em-dashes from output. "For months, em-dashes have been the focus of online discussion as a possible indicator of AI usage: this case suggests that the threat actors were aware of that discussion," the report notes. Also: Navigating AI-powered cyber threats in 2025: 4 expert security tips for businesses Concerningly, but perhaps not unsurprisingly, AI is also finding its way into the hands of state-sponsored groups. Recently, OpenAI disrupted networks thought to be linked to numerous People's Republic of China (PRC) government entities, with accounts asking ChatGPT to generate proposals for large systems designed to monitor social media networks. In addition, some accounts requested help to write a proposal for a tool that would analyze transport bookings and compare them with police records, thereby monitoring the movements of the Uyghur minority group, whereas another tried to use ChatGPT to identify funding streams related to an X account that criticized the Chinese government. While AI is being weaponized, it should be noted that there is little to no evidence of existing AI models being used to develop what OpenAI describes as "novel" attacks; in other words, AI models are refusing malicious requests that would give threat actors enhanced offensive capabilities using new tactics unknown to cybersecurity experts. "We continue to see threat actors bolt AI onto old playbooks to move faster, not gain novel offensive capability from our models," OpenAI said. "As the threatscape evolves, we expect to see further adversarial adaptations and innovations, but we will also continue to build tools and models that can be used to benefit the defenders -- not just within AI labs, but across society as a whole."

OpenAI bans some Chinese, Russian accounts using AI for evil

It also banned some suspected Russian accounts trying to create influence campaigns and malware OpenAI has banned ChatGPT accounts believed to be linked to Chinese government entities attempting to use AI models to surveil individuals and social media accounts. In its most recent threat report [PDF] published today, the GenAI giant said that these users usually asked ChatGPT to help design tools for large-scale monitoring and analysis - but stopped short of asking the model to perform the surveillance activities. "What we saw and banned in those cases was typically threat actors asking ChatGPT to help put together plans or documentation for AI-powered tools, but not then to implement them," Ben Nimmo, principal investigator on OpenAI's Intelligence and Investigations team, told reporters. One now-banned user, suspected to be using a VPN to access the AI service from China, asked ChatGPT to design promotional materials and project plans for a social media listening tool, described as a "probe," that could scan X, Facebook, Instagram, Reddit, TikTok, and YouTube for what the user described as extremist speech, and ethnic, religious, and political content. This user claimed a government client wanted this scanning tool, but stopped short of using the model to monitor social media. OpenAI said it's unable to verify if the Chinese government ended up using any such tool. In two other cases, the company banned one user who asked ChatGPT to identify funding sources for an X account that criticized the Chinese government and another one who asked ChatGPT to identify petition organizers in Mongolia. In both, we're told, OpenAI's models only provided publicly available information - not identities, funding sources, or other sensitive details. "Cases like these are limited snapshots, but they do give us important insights into how authoritarian regimes might abuse future AI capabilities," Nimmo said. "They point to something about the direction of travel, even if they also suggest that maybe the destination is somewhere away." Since the company started producing threat reports in February 2024, OpenAI said it has banned more than 40 networks that violated its usage policies. Also since that time, the threat groups and individuals attempting to use AI for evil have been employing the models to improve their existing tradecraft, not to develop entirely new cyberattacks or workflows. That still seems to be the case, according to OpenAI execs. More recently, however, some of the disrupted accounts appear to be using multiple AI models to achieve their nefarious goals. "One China-linked cluster that we investigated, for example, used ChatGPT to draft phishing lures and then explored another model, DeepSeek, to automate mass targeting," said Michael Flossman, who leads OpenAI's threat intelligence team. Similarly, a set of suspected and now-banned Russian accounts used ChatGPT to generate video prompts for an influence operation dubbed Stop News, but then attempted to use other companies' AI tools to produce the videos that were later posted on YouTube and TikTok. OpenAI could not independently confirm which other models this group used. "We're seeing adversaries routinely use multiple AI tools hopping between models for small gains in speed or automation," Flossman said. In another example of attempted model abuse originating from Russia, the company banned accounts asking ChatGPT to develop and refine malware, including a remote-access trojan, credential stealers, and features to help malware evade detection. The company wrote: These accounts appear to be linked with Russian-speaking criminal groups, as the threat intel team saw them posting about their activities in a Telegram channel connected to a specific criminal gang. OpenAI execs declined to attribute the malware-making endeavors to a particular cybercrime crew, but said they have "medium to high confidence on who is behind it." ®

OpenAI Disrupts Russian, North Korean, and Chinese Hackers Misusing ChatGPT for Cyberattacks

OpenAI on Tuesday said it disrupted three activity clusters for misusing its ChatGPT artificial intelligence (AI) tool to facilitate malware development. This includes a Russian‑language threat actor, who is said to have used the chatbot to help develop and refine a remote access trojan (RAT), a credential stealer with an aim to evade detection. The operator also used several ChatGPT accounts to prototype and troubleshoot technical components that enable post‑exploitation and credential theft. "These accounts appear to be affiliated with Russian-speaking criminal groups, as we observed them posting evidence of their activities in a Telegram channel dedicated to those actors," OpenAI said. The AI company said while its large language models (LLMs) refused the threat actor's direct requests to produce malicious content, they worked around the limitation by creating building-block code, which was then assembled to create the workflows. Some of the produced output involved code for obfuscation, clipboard monitoring, and basic utilities to exfiltrate data using a Telegram bot. It's worth pointing out that none of these outputs are inherently malicious on their own. "The threat actor made a mix of high‑ and lower‑sophistication requests: many prompts required deep Windows-platform knowledge and iterative debugging, while others automated commodity tasks (such as mass password generation and scripted job applications)," OpenAI added. "The operator used a small number of ChatGPT accounts and iterated on the same code across conversations, a pattern consistent with ongoing development rather than occasional testing." The second cluster of activity originated from North Korea and shared overlaps with a campaign detailed by Trellix in August 2025 that targeted diplomatic missions in South Korea using spear-phishing emails to deliver Xeno RAT. OpenAI said the cluster used ChatGPT for malware and command-and-control (C2) development, and that the actors engaged in specific efforts such as developing macOS Finder extensions, configuring Windows Server VPNs, or converting Chrome extensions to their Safari equivalents. In addition, the threat actors have been found to use the AI chatbot to draft phishing emails, experiment with cloud services and GitHub functions, and explore techniques to facilitate DLL loading, in-memory execution, Windows API hooking, and credential theft. The third set of banned accounts, OpenAI noted, shared overlaps with a cluster tracked by Proofpoint under the name UNK_DropPitch (aka UTA0388), a Chinese hacking group which has been attributed to phishing campaigns targeting major investment firms with a focus on the Taiwanese semiconductor industry, with a backdoor dubbed HealthKick (aka GOVERSHELL). The accounts used the tool to generate content for phishing campaigns in English, Chinese, and Japanese; assist with tooling to accelerate routine tasks such as remote execution and traffic protection using HTTPS; and search for information related to installing open-source tools like nuclei and fscan. OpenAI described the threat actor as "technically competent but unsophisticated." Outside of these three malicious cyber activities, the company also blocked accounts used for scam and influence operations - In two different cases, suspected Chinese accounts asked ChatGPT to identify organizers of a petition in Mongolia and funding sources for an X account that criticized the Chinese government. OpenAI said its models returned only publicly available information as responses and did not include any sensitive information. "A novel use for this [China-linked influence network was requests for advice on social media growth strategies, including how to start a TikTok challenge and get others to post content about the #MyImmigrantStory hashtag (a widely used hashtag of long standing whose popularity the operation likely strove to leverage)," OpenAI said. "They asked our model to ideate, then generate a transcript for a TikTok post, in addition to providing recommendations for background music and pictures to accompany the post." OpenAI reiterated that its tools provided the threat actors with novel capabilities that they could not otherwise have obtained from multiple publicly available resources online, and that they were used to provide incremental efficiency to their existing workflows. But one of the most interesting takeaways from the report is that threat actors are trying to adapt their tactics to remove possible signs that could indicate that the content was generated by an AI tool. "One of the scam networks [from Cambodia] we disrupted asked our model to remove the em-dashes (long dash, -) from their output, or appears to have removed the em-dashes manually before publication," the company said. "For months, em-dashes have been the focus of online discussion as a possible indicator of AI usage: this case suggests that the threat actors were aware of that discussion." The findings from OpenAI come as rival Anthropic released an open-source auditing tool called Petri (short for "Parallel Exploration Tool for Risky Interactions") to accelerate AI safety research and better understand model behavior across various categories like deception, sycophancy, encouragement of user delusion, cooperation with harmful requests, and self-perseveration. "Petri deploys an automated agent to test a target AI system through diverse multi-turn conversations involving simulated users and tools," Anthropic said. "Researchers give Petri a list of seed instructions targeting scenarios and behaviors they want to test. Petri then operates on each seed instruction in parallel. For each seed instruction, an auditor agent makes a plan and interacts with the target model in a tool use loop. At the end, a judge scores each of the resulting transcripts across multiple dimensions so researchers can quickly search and filter for the most interesting transcripts."

OpenAI Gives Us a Glimpse of How It Monitors for Misuse on ChatGPT

The AI company's latest report provides more info on how it detects misuse of its chatbots. OpenAI’s latest report on malicious AI use underscores the tightrope that AI companies are walking between preventing misuse of their chatbots and reassuring users that their privacy is respected. The report, which dropped today, highlights several cases where OpenAI investigated and disrupted harmful activity involving its models, focusing on scams, cyberattacks, and government-linked influence campaigns. However, it arrives amid growing scrutiny over another type of AI risk, the potential psychological harms of chatbots. This year alone has seen several reports of users committing acts of self-harm, suicide, and murder after interacting with AI models. This new report, along with previous company disclosures, provides some additional insight into how OpenAI moderates chats for different kinds of misuse. OpenAI said that since it began reporting public threats in February 2024, it has disrupted and reported more than 40 networks that violated their usage policies. In today’s report, the company shared new case studies from the past quarter and details on how it detects and disrupts malicious use of its models. For example, the company identified an organized crime network, reportedly based in Cambodia, that tried to use AI to streamline its workflows. Additionally, a Russian political influence operation reportedly used ChatGPT to generate video prompts for other AI models. OpenAI also flagged accounts linked to the Chinese government that violated its policies on national security use, including requests to generate proposals for large-scale systems designed to monitor social media conversations. The company has previously said, including in its privacy policy, that it uses personal data, such as user prompts, to â€~prevent fraud, illegal activity, or misuse’ of its services. OpenAI has also said it relies on both automated systems and human reviewers to monitor activity. But in today’s report, the company offered slightly more insight into its thought process for preventing misuse while still protecting users more broadly. “To detect and disrupt threats effectively without disrupting the work of everyday users, we employ a nuanced and informed approach that focuses on patterns of threat actor behavior rather than isolated model interactions,†the company wrote in the report. While monitoring for national security breaches is one thing, the company also recently outlined how it addresses harmful use of its models by users experiencing emotional or mental distress. Just over a month ago, the company published a blog post detailing how it handles these types of situations. The post came amid media coverage of violent incidents reportedly linked to ChatGPT interactions, including a murder-suicide in Connecticut. The company said that when users write that they want to hurt themselves, ChatGPT is trained not to comply and instead acknowledge the user's feelings and steer them toward help and real-world resources. When the AI detects someone is planning to harm others, the conversations are flagged for human review. If a human reviewer determines the person represents an imminent threat to others, they can report them to law enforcement. OpenAI also acknowledged that its model’s safety performance can degrade during longer user interactions and said it’s already working to improve its safeguards.

OpenAI bans Chinese, North Korean hacker accounts using ChatGPT to launch surveillance

North Korean actors tested phishing, credential theft, and macOS malware development using rephrased prompts OpenAI has banned Chinese, North Korean, and other accounts which were reportedly using ChatGPT to launch surveillance campaigns, develop phishing techniques and malware, and engage in other malicious practices. In a new report, OpenAI said it observed individuals reportedly affiliated with Chinese government entities, or state-linked organizations, using its Large Language Model (LLM) to help write proposals for surveillance systems and profiling technologies. These included tools for monitoring individuals and analyzing behavioral patterns. "Some of the accounts that we banned appeared to be attempting to use ChatGPT to develop tools for large-scale monitoring: analyzing datasets, often gathered from Western or Chinese social media platforms," the report reads. "These users typically asked ChatGPT to help design such tools or generate promotional materials about them, but not to implement the monitoring." The prompts were framed in a way that avoided triggering safety filters, and were often phrased as academic or technical inquiries. While the returned content did not directly enable surveillance, its outputs were used to refine documentation and planning for such systems, it was said. The North Koreans, on the other hand, used ChatGPT to explore phishing techniques, credential theft, and macOS malware development. OpenAI said it observed these accounts testing prompts related to social engineering, password harvesting, and debugging malicious code, especially targeting Apple systems. The model refused direct requests for malicious code, OpenAI said, but stressed that the threat actors still tried to bypass safeguards by rephrasing prompts, or asking for general technical help. Just like any other tool, LLMs are being used by both financially motivated, and state-sponsored threat actors, for all sorts of malicious activity. This AI misuse is evolving, with threat actors increasingly integrating AI into existing workflows to improve their efficiency. While developers such as OpenAI work hard on minimizing risk and making sure their products cannot be used like this, there are many prompts that fall between legitimate and malicious use. This gray zone activity, the report hints, requires nuanced detection strategies. Via The Register

Foreign adversaries are using multiple AI tools, OpenAI warns

Why it matters: In the cases OpenAI discovered, the adversaries typically turned to ChatGPT to help plan their schemes, then used other models to carry them out -- reflecting the range of applications for AI tools in such operations. Zoom in: OpenAI banned several accounts tied to nation-state campaigns that seemed to be using multiple AI models to improve their operations. * A Russian-based actor that was generating content for a covert influence operation used ChatGPT to write prompts seemingly for another AI video model. * A cluster of Chinese-language accounts used ChatGPT to research and refine phishing automation they wanted to run on China-based model DeepSeek. * OpenAI also confirmed that an actor the company previously disrupted was the same one Anthropic recently flagged in a threat report, suggesting they were using both tools. Between the lines: OpenAI mostly observed threat actors using ChatGPT to improve their existing tactics, rather than creating new ones, Ben Nimmo, principal investigator on OpenAI's intelligence and investigations team, told reporters in a call ahead of the report's release. * However, the multi-model approach means that investigators have "just a glimpse" at how threat actors are using a specific model, Nimmo said. The intrigue: Nation-state hackers and scammers are also learning to hide the telltale signs of AI usage, OpenAI's research team found. One scam network asked ChatGPT to remove em dashes from its writing, for example. The big picture: Much like the U.S. government, foreign adversaries have been exploring ways to use ChatGPT and similar tools for years. * In the latest report, OpenAI said it had banned accounts that appeared to be tied to both China-based entities and Russian-speaking criminal groups for using the model to help develop malware and write phishing emails. * The company also banned accounts linked to Chinese government entities, including some that were asking OpenAI's models to "generate work proposals for large-scale systems designed to monitor social media conversations," according to the report. What to watch: The campaigns OpenAI identified didn't seem to be very effective, per the report. But nation-state entities are still early in their AI experimentations.

OpenAI details expanding efforts to disrupt malicious use of AI in new report - SiliconANGLE

OpenAI details expanding efforts to disrupt malicious use of AI in new report OpenAI today released a new report highlighting the company's growing efforts to identify, expose and disable misuse of its models for cyberattacks, scams and state-linked influence operations. The "Disrupting Malicious Uses of AI" report, part of a monthly series of reports from OpenAI, details a growing trend of threat actors integrating artificial intelligence into their existing toolchains rather than developing AI-driven workflows from scratch and provides various examples. Russian-language threat groups were found to be attempting to refine malware components such as remote-access trojans and credential stealers, Korean-language operators were found to be developing command-and-control systems and alleged China-linked groups were found to be crafting phishing content and debugging malware targeting Taiwan's semiconductor sector, U.S. academia and political groups. In all cases, OpenAI's safeguards blocked direct malicious requests and findings were shared with industry partners to strengthen collective defenses. A major section of the report focuses on organized crime scams originating from Cambodia, Myanmar and Nigeria; both Cambodian and Myanmar scam operators have made headlines globally over the last year due to the size of their operations and some argue that Thailand's current conflict with Cambodia is linked to these operations, which account for as much as 60% of Cambodia's gross domestic product. The groups in the three countries were found to be using ChatGPT to translate messages, generate social media content and craft fraudulent investment personas. Some operations even asked the model to remove em dashes, a known indicator of AI text, in attempts to disguise their use of AI. OpenAI found that despite the attempts at misuse, its models were used to detect scams three times more often than to create them, as millions of users sought help identifying fraudulent activity. The report also details alleged authoritarian-linked abuses linked to Chinese actors. The users sought assistance designing social media monitoring tools, profiling dissidents and generating propaganda proposals, activities that violated OpenAI's national security policies. OpenAI banned these accounts and reiterated its commitment to building "democratic AI," emphasizing transparency, safety and protection against surveillance-state misuse. "As the threatscape evolves, we expect to see further adversarial adaptations and innovations, but we will also continue to build tools and models that can be used to benefit the defenders - not just within AI labs, but across society as a whole," the report notes. Discussing the report, Cory Kennedy, chief threat intelligence officer at security ratings firm SecurityScorecard Inc., told SiliconANGLE via email that "the report highlights how threat actors are increasingly combining multiple AI models to scale their operations." "While OpenAI banned the accounts involved, it noted that some attempts, such as proposals for large-scale monitoring of social media and movement, offer insight into how generative AI is being explored for population tracking and narrative control," added Kennedy. "These findings underscore the urgency of proactive disruption, vendor transparency and cross-platform threat intelligence where AI tools intersect with sensitive data and global influence efforts."

OpenAI report reveals it scans chats to stop spies and criminals

The company clarified that while users can opt out of having conversations train future AI models all chats are still monitored for safety and platform integrity. OpenAI released a report on a recent Tuesday detailing its disruption of over 40 malicious networks that violated its usage policies, a result of monitoring ChatGPT conversations with automated tools and human reviewers to prevent system misuse. While users of AI chatbots, including ChatGPT, can enable privacy settings to prevent their conversations from being used to train future AI models, this does not preclude all monitoring. AI firms like OpenAI scrutinize chats to maintain user safety and platform integrity. This process involves the deployment of automated systems and the use of human reviewers. The stated objective is to prevent the misuse of ChatGPT for activities that could cause harm, such as the creation of malware or the development of tools for mass surveillance and other security threats. Users have the option to adjust privacy configurations to keep personal or work-related data from being absorbed into the data pool used for training subsequent AI versions. The company's report detailed the disruption and reporting of more than 40 networks found to be in violation of its usage policies. The malicious actors identified as attempting to misuse ChatGPT included a range of entities. OpenAI categorized these as "authoritarian regimes to control populations or coerce other states, as well as abuses like scams, malicious cyber activity, and covert influence operations." The findings from OpenAI indicate that these threat actors are primarily using artificial intelligence to accelerate and refine existing tactics. The company noted that these actors are using AI to improve "old playbooks to move faster," but are not necessarily acquiring fundamentally new capabilities directly from their use of ChatGPT. In addition to countering malicious state and criminal actors, OpenAI also monitors conversations for indications of self-harm to assist users who may be in distress. This focus on individual user safety has been described as a key priority, an emphasis that was heightened following reports of a teen's death by suicide, which had been linked to interactions with ChatGPT. In connection with this safety focus, OpenAI has also introduced parental controls for the ChatGPT service in recent weeks. OpenAI's report does not offer a detailed explanation of the specific mechanisms involved in flagging potential abuse. The company acknowledged the existence of a gray area, where certain activities could be interpreted as either benign or malicious depending on the context. Examples of such ambiguous activities include "prompts and generations that could, depending on their context, indicate either innocuous activities or abuse, such as translating texts, modifying code, or creating a website." To navigate this complexity, the company stated it employs a "nuanced and informed approach that focuses on patterns of threat actor behavior rather than isolated model interactions." This method is designed to identify genuine threats without disrupting legitimate usage for the broader user base. Reporting from Gizmodo provided specific examples of high-level threats that OpenAI identified and disrupted. One such case involved an organized crime network, believed to be operating from Cambodia, which attempted to use ChatGPT to streamline its illicit operations. Another instance involved a Russian political influence operation that used the AI to generate prompts intended for use with third-party video-generation AI models. OpenAI also stopped accounts associated with the Chinese government that were seeking assistance with designing systems for monitoring conversations on social media platforms. Further details on disrupted activities were provided by Reuters. According to its reporting, OpenAI banned a set of Chinese-language accounts that were seeking assistance with phishing and malware campaigns. These accounts also sought help with automations that could be executed via DeepSeek, another AI tool. Separately, accounts linked to Russian criminal groups attempting to develop malware with ChatGPT's assistance were also stopped. In a similar action, Korean-speaking users who were found to be using the platform to facilitate phishing campaigns were banned from the service. The October report from OpenAI concentrated exclusively on malicious activities and covert operations, and did not include data or details regarding conversations about self-harm. However, the company has made separate announcements related to this issue. In a recent statement on the social media platform X, OpenAI said it had updated a model named GPT-5 Instant to "better recognize and support people in moments of distress." The company explained that sensitive portions of conversations are now routed to this specialized model, which is designed to provide more helpful responses. ChatGPT will now also inform users about which specific model is being used during their conversation. This update is part of a broader initiative to improve user safety. In late August, OpenAI announced that ChatGPT had been trained not to respond directly to prompts that mention intentions of self-harm. Instead of fulfilling such requests, the AI is programmed to respond with empathy and to direct users toward professional help, providing information for suicide prevention services and crisis hotlines. For situations where the AI detects a potential risk of physical harm to other individuals, a different protocol is activated. These conversations are routed to specialized systems that may involve human review and can lead to an escalation with law enforcement agencies if deemed necessary.

US foreign adversaries use ChatGPT with other AI models in cyber operations: Report

Malicious actors from U.S. foreign adversaries used ChatGPT jointly with other AI models to conduct various cyber operations, according to a new OpenAI report. Users linked to China and Russia relied on OpenAI's technology in conjunction with other models, such as China's DeepSeek, to conduct phishing campaigns and covert influence operations, the report found. "Increasingly, we have disrupted threat actors who appeared to be using multiple AI models to achieve their aims," OpenAI noted. A cluster of ChatGPT accounts that showed signs consistent with Chinese government intelligence efforts used the AI model to generate content for phishing campaigns in multiple languages, in addition to developing tools and malware. This group also looked at using DeepSeek to automate this process, such as analyzing online content to generate a list of email targets and produce content that would likely appeal to them. OpenAI banned the accounts but noted it could not confirm whether they ultimately used automation with other AI models. Another cluster of accounts based in Russia used ChatGPT to develop scripts, SEO-optimized descriptions and hashtags, translations and prompts for generating news-style videos with other AI models. The activity appears to be part of a Russian influence operation that OpenAI previously identified, which posted AI-generated content across websites and social media platforms, the report noted. Its latest content criticized France and the U.S. for their role in Africa while praising Russia. The accounts, now banned by OpenAI, also produced content critical of Ukraine and its supporters. However, the ChatGPT maker found that these efforts gained little traction. OpenAI separately noted in the report that it banned several accounts seemingly linked to the Chinese government that sought to use ChatGPT to develop proposals for large-scale monitoring, such as tracking social media or movements. "While these uses appear to have been individual rather than institutional, they provide a rare snapshot into the broader world of authoritarian abuses of AI," the company wrote.

OpenAI Bans ChatGPT Accounts Tied to China-Linked Surveillance

OpenAI found no evidence that the proposed tools were ever built OpenAI announced on Tuesday that it has banned several potential China-linked ChatGPT accounts that were attempting to utilise the chatbot to develop mass surveillance tools. The San Francisco-based artificial intelligence (AI) giant said in a published report that some accounts were also seeking suggestions to build social media listening tools and profiling systems to target specific groups of people. OpenAI highlighted that all such accounts detected to be involved in such practices have been banned. The report also mentioned other accounts from Russia that were trying to use AI to build phishing tools. OpenAI Says Multiple China-Based Users Were Trying to Build Monitoring Tools According to OpenAI's Disrupting Malicious Uses of AI: October 2025 report, a cluster of accounts, potentially linked with the Chinese government, was using ChatGPT to seek information and create autocratic tools. Calling it "a rare snapshot into the broader world of authoritarian abuses of AI," the AI company highlighted several instances of these accounts trying to develop specialised tools for mass surveillance, profiling, and online monitoring. Notably, these incidents did not take place all at once and were spread across 2025. One user asked ChatGPT to help draft project plans and promotional material for a "social media listening tool" allegedly intended for a government client, the report stated. The proposed system, referred to as a "social media probe," was described as capable of scanning platforms such as X (formerly known as Twitter), Facebook, Instagram, Reddit, TikTok, and YouTube for extremist or politically sensitive content. OpenAI said it found no evidence that the tool was ever developed or operated. Another banned account was linked to a user who sought assistance in drafting a proposal for what was described as a High-Risk Uyghur-Related Inflow Warning Model. The model was described as a system that would analyse transport bookings and cross-reference them with police records to track individuals labelled as "high-risk". As with the previous case, OpenAI clarified that the model was not used to build or run such a tool and that its existence could not be independently verified. Other accounts appeared to use ChatGPT for profiling and online research. In one instance, a user asked the AI model to identify funding sources for an X account critical of the Chinese government. Another user sought information on the organisers of a petition in Mongolia. In both cases, ChatGPT returned only publicly available information. Apart from this, OpenAI highlighted that some accounts also used ChatGPT as an open-source research tool, similar to a search engine. These accounts asked the chatbot to identify and summarise breaking news that would be relevant to China. It also sought information on sensitive topics such as the Tiananmen Square massacre in 1989 and the birthday of the Dalai Lama.

OpenAI bans suspected China-linked accounts for seeking surveillance proposals

In its latest public threat report, OpenAI said some individuals had asked its chatbot to outline social media "listening" tools and other monitoring concepts, violating the startup's national security policy. OpenAI said on Tuesday it has banned several ChatGPT accounts with suspected links to the Chinese government entities after the users asked for proposals to monitor social media conversations. In its latest public threat report, OpenAI said some individuals had asked its chatbot to outline social media "listening" tools and other monitoring concepts, violating the startup's national security policy. The San Francisco-based firm's report raises safety concerns over potential misuse of generative AI amid growing competition between the U.S. and China to shape the technology's development and rules. OpenAI said it also banned several Chinese‑language accounts that used ChatGPT to assist phishing and malware campaigns and asked the model to research additional automation that could be achieved through China's DeepSeek. The Chinese embassy in the U.S. did not immediately respond to a request for comment on the report. It also banned accounts tied to suspected Russian‑speaking criminal groups that used the chatbot to help develop certain malware, OpenAI said. The Microsoft-backed startup has disrupted and reported more than 40 networks since it began public threat reporting in February last year and its models refused overtly malicious prompts, the AI company added. "We found no evidence of new tactics or that our models provided threat actors with novel offensive capabilities," the company said in the report. OpenAI, which now has more than 800 million weekly ChatGPT users, became the world's most valuable startup at a $500 billion valuation after completing a secondary share sale last week.