Google Reveals State-Sponsored Hackers' Attempts to Exploit Gemini AI

AI is Not a Game Changer for Threat Actors Yet, Says Google

A new security report from the company states that threat actors haven't been able to use AI to develop novel capabilities to accelerate and amplify attacks. Google Threat Intelligence Group (GITG) recently published a report analysing various attempts to misuse Google's AI assistant Gemini. The report explored threats posed by individual and state-sponsored attackers. These attackers sought to exploit Gemini in two ways: to accelerate their malicious campaigns or instruct a model or AI agent to take a malicious action. The majority of the activity falls under the first category. State-sponsored cyber attacks were associated with threat actors from countries like China, North Korea, Iran, and Russia. These actors used Gemini for reconnaissance, vulnerability research, phishing campaigns, and defence-related intelligence. North Korean threat actors used AI to place covert IT workers in Western firms by creating fake CVs. However, Google concluded the report with positive findings. "While AI can be a useful tool for threat actors, it is not yet the game-changer it is sometimes portrayed to be," read the report. Google further said that it did not see any indications of the threat actors developing any novel capabilities. Moreover, the company added that threat actors unsuccessfully attempted to use Gemini to abuse Google's products, involving activities like phishing, data theft, and bypassing Google accounts in products like Chrome and Gmail. Google also observed a handful of unsuccessful attempts to use publicly available jailbreak prompts to bypass Gemini's safety controls. In one such attempt, a threat actor tried to get Gemini to perform coding tasks, including wiring Python code for a distributed denial-of-service (DDoS) tool. In the end, Google provided the code but with a safety-filtered response stating that it could not assist. Kent Walker, president of global affairs at Alphabet (Google), said, "In other words, the defenders are still ahead, for now." Beyond using a chat-focused AI model to accelerate malicious campaigns, an even greater threat lies in the direct exploitation of AI agents. Google highlighted this as the second kind of attack. Google's Secure AI Framework (SAIF) map outlines all the AI risks associated with the model creator, consumer, or both. "We did not observe any original or persistent attempts by threat actors to use prompt attacks or other machine learning-focused threats as outlined in the SAIF risk taxonomy," the report said. "Rather than engineering tailored prompts, threat actors used more basic measures or publicly available jailbreak prompts in unsuccessful attempts to bypass Gemini's safety controls," the report added. However, this alone should not create a sense of complacency. The capabilities of these AI agents are tempting startups, big organisations, and individual users alike. It is the need of the hour to safeguard the same. AIM spoke to Omer Yoachimik, a senior product manager at Cloudflare, one of the world's leading cybersecurity companies. Yoachimik particularly emphasised the criticality of DDoS protection, given that these systems increasingly depend upon real-time access to external services and data. "With the growing adoption of AI agents across industries, they become attractive targets for attackers aiming to create widespread disruption," Yoachimik said. He added that the approach towards AI and cybersecurity should be different from the traditional ones. "While traditional products often focus on static defenses, AI-driven systems demand adaptive, real-time security measures that evolve with emerging attack patterns to ensure resilience in a highly dynamic threat landscape," he said. A research study from the University of California, Davis, states that data inside an AI agent system faces risks similar to those concerning confidentiality. "Malicious applications might manipulate the system by injecting misleading prompts as part of the instruction or manual, altering data inappropriately," the study added. It isn't all about high-stakes cybersecurity threats. For instance, the research quotes an example of an AI agent booking a flight, where it could be misled to favour a less efficient option through false information. The research also offered a few defence mechanisms against these attacks. It proposes using techniques like sandboxing to restrict an AI agent's capabilities by limiting its consumption of CPU resources and access to file systems. Earlier, we covered a detailed story on reports of how prompt injection in Anthropic Claude's experimental autonomous Computer Use feature compromised its security. In an experiment conducted by Hidden Layer, Computer Use was exposed to prompt injection to delete all the system files via a command in the Unix/Linux environment. Another study from UC Berkeley introduced methods to mitigate prompt injection. In these methods, the LLM is trained to follow only instructions from the original prompts and ignore any other instructions. AIM also spoke to Sudipta Biswas, co-founder of Floworks, which has built an AI sales agent called Alisha. He outlined three aspects of focus for security in an AI agent: data held by the organisation building the agent, data accessed by the agent itself, and access authentication. However, Biswas admitted that providing an AI agent with privileges such as access to a password-protected email account, critical permissions, and access is an open problem and a big opportunity for companies, and developers in cybersecurity. "We are approaching this with a two-step process," he added. "When certain data needs to be entered into a system of records, we ask the users for another round of approval - 'Hey, is this what you really meant?'," he added, indicating that this process builds a sense of confidence among the users.

Google says hackers abuse Gemini AI to empower their attacks

Multiple state-sponsored groups are experimenting with the AI-powered Gemini assistant from Google to increase productivity and to conduct research on potential infrastructure for attacks or for reconnaissance on targets. Google's Threat Intelligence Group (GTIG) detected government-linked advanced persistent threat (APT) groups using Gemini primarily for productivity gains rather than to develop or conduct novel AI-enabled cyberattacks that can bypass traditional defenses. Threat actors have been trying to leverage AI tools for their attack purposes to various degrees of success as these utilities can at least shorten the preparation period. Google has identified Gemini activity associated with APT groups from more than 20 countries but the most prominent ones were from Iran and China. Among the most common cases were assistance with coding tasks for developing tools and scripts, research on publicly disclosed vulnerabilities, checking on technologies (explanations, translation), finding details on target organizations, and searching for methods to evade detection, escalate privileges, or run internal reconnaissance in a compromised network. Google says APTs from Iran, China, North Korea, and Russia, have all experimented with Gemini, exploring the tool's potential in helping them discover security gaps, evade detection, and plan their post-compromise activities. These are summarized as follows: Google also mentions having observed cases where the threat actors attempted to use public jailbreaks against Gemini or rephrasing their prompts to bypass the platform's security measures. These attempts were reportedly unsuccessful. OpenAI, the creator of the popular AI chatbot ChatGPT, made a similar disclosure in October 2024, so Google's latest report comes as a confirmation of the large-scale misuse of generative AI tools by threat actors of all levels. While jailbreaks and security bypasses are a concern in mainstream AI products, the AI market is gradually filling with AI models that lack proper the protections to prevent abuse. Unfortunately, some of them with restrictions that are trivial to bypass are also enjoying increased popularity. Cybersecurity intelligence firm KELA has recently published the details about the lax security measures for DeepSeek R1 and Alibaba's Qwen 2.5, which are vulnerable to prompt injection attacks that could streamline malicious use. Unit 42 researchers also demonstrated effective jailbreaking techniques against DeepSeek R1 and V3, showing that the models are easy to abuse for nefarious purposes.

Google exposes government-backed misuse of Gemini AI

While artificial intelligence advancements unlock opportunities in various industries, innovations may also become targets of hackers, highlighting a concerning potential for AI misuse. Google's threat intelligence department released a paper titled Adversarial Misuse of Generative AI, revealing how threat actors have approached their artificial intelligence chatbot Gemini. According to Google, threat actors attempted to jailbreak the AI using prompts. In addition, government-backed advanced persistent threat (APT) groups have tried using Gemini to assist them in malicious endeavors. Google reports unsuccessful attempts to jailbreak Gemini Google said while threat actors had attempted to jailbreak Gemini, the company saw no advanced attempts in this attack vector. According to Google, hackers only used basic measures like rephrasing or repeatedly sending the same prompt. Google said the attempts were unsuccessful. AI jailbreaks are prompt injection attacks that aim to get an AI model to perform tasks that it had been prohibited from doing. This includes leaking sensitive information or providing unsafe content. Google said that in one instance, an APT actor used publicly available prompts to trick Gemini into performing malicious coding tasks. However, Google said the attempt was unsuccessful as Gemini provided a safety-filtered response. Related: India to launch generative AI model in 2025 amid DeepSeek frenzy How government-backed threat actors used Gemini In addition to low-effort jailbreak attempts, Google reported how government-backed APTs have approached Gemini. Google said these attackers attempted to use Gemini to assist in their malicious activities. This included information gathering on their targets, researching publicly known vulnerabilities and coding and scripting tasks. In addition, Google said there have been attempts to enable post-compromise activities like defense evasion. Google reported that Iran-based APT actors focused on using AI in crafting phishing campaigns. They also used the AI model to conduct recon on defense experts and organizations. The APT actors in Iran also used AI to generate cybersecurity content. Meanwhile, China's APT actors have used Gemini to troubleshoot code, scripting and development. In addition, they used AI to research how to obtain deeper access to their target networks. APT actors in North Korea have also used Gemini for different phases of their attack lifecycle, from research to development. The report said: "They also used Gemini to research topics of strategic interest to the North Korean government, such as the South Korean military and cryptocurrency."

Google says Gemini is being misused to launch major cyberattacks

Hackers are experimenting, but haven't found "novel capabilities" just yet Dozens of cybercriminal organizations from all around the world are abusing Google's Artificial Intelligence (AI) solution Gemini in their attacks, the company has admitted. In an in-depth analysis discussing who the threat actors are, and what they're using the tools for, Google's Threat Intelligence Group highlighted how the platform has not yet been used to discover new attack methods, but is rather used to fine-tune existing ones. "Threat actors are experimenting with Gemini to enable their operations, finding productivity gains but not yet developing novel capabilities," the team said in its analysis. "At present, they primarily use AI for research, troubleshooting code, and creating and localizing content." The biggest Gemini users among cybercriminals are the Iranians, Russians, the Chinese, and North Koreans, who utilize the platform for reconnaissance, vulnerability research, scripting and development, translation and explanation, and deeper system access and post-compromise actions. In total, Google observed 57 groups, more than 20 of which were from China, and among the 10+ North Korean threat actors using Gemini, one group stands out - APT42. Over 30% of threat actor Gemini use from the country was linked to APT42, Google said. "APT42's Gemini activity reflected the group's focus on crafting successful phishing campaigns. We observed the group using Gemini to conduct reconnaissance into individual policy and defense experts, as well as organizations of interest for the group." APT42 also used text generation and editing capabilities to craft phishing messages, particularly those targeting US defense organizations. "APT42 also utilized Gemini for translation including localization, or tailoring content for a local audience. This includes content tailored to local culture and local language, such as asking for translations to be in fluent English." Ever since ChatGPT was first published, security researchers have been warning about the abuse in cybercrime. Before GenAI, the best way to spot phishing attacks was to look for spelling and grammar errors, and inconsistent wording. Now, with AI doing the writing and the editing, the method practically no longer works, and security pros are turning to new approaches.

Google: Over 57 Nation-State Threat Groups Using AI for Cyber Operations

Over 57 distinct threat actors with ties to China, Iran, North Korea, and Russia have been observed using artificial intelligence (AI) technology powered by Google to further enable their malicious cyber and information operations. "Threat actors are experimenting with Gemini to enable their operations, finding productivity gains but not yet developing novel capabilities," Google Threat Intelligence Group (GTIG) said in a new report. "At present, they primarily use AI for research, troubleshooting code, and creating and localizing content." Government-backed attackers, otherwise known as Advanced Persistent Threat (APT) groups, have sought to use its tools to bolster multiple phases of the attack cycle, including coding and scripting tasks, payload development, gathering information about potential targets, researching publicly known vulnerabilities, and enabling post-compromise activities, such as defense evasion. Describing Iranian APT actors as the "heaviest users of Gemini," GTIG said the hacking crew known as APT42, which accounted for more than 30% of Gemini use by hackers from the country, leveraged its tools for crafting phishing campaigns, conducting reconnaissance on defense experts and organizations, and generating content with cybersecurity themes. APT42, which overlaps with clusters tracked as Charming Kitten and Mint Sandstorm, has a history of orchestrating enhanced social engineering schemes to infiltrate target networks and cloud environments. Last May, Mandiant revealed the threat actor's targeting of Western and Middle Eastern NGOs, media organizations, academia, legal services and activists by posing as journalists and event organizers. The adversarial collective has also been found to research military and weapons systems, study strategic trends in China's defense industry, and gain a better understanding of U.S.-made aerospace systems. Chinese APT groups were found searching Gemini for ways to conduct reconnaissance, troubleshoot code, and methods to burrow deep into victim networks through techniques like lateral movement, privilege escalation, data exfiltration, and detection evasion. While Russian APT actors limited their use to Gemini to convert publicly available malware into another coding language and adding encryption layers to existing code, North Korean actors employed Google's AI service to research infrastructure and hosting providers. "Of note, North Korean actors also used Gemini to draft cover letters and research jobs -- activities that would likely support North Korea's efforts to place clandestine IT workers at Western companies," GTIG noted. "One North Korea-backed group utilized Gemini to draft cover letters and proposals for job descriptions, researched average salaries for specific jobs, and asked about jobs on LinkedIn. The group also used Gemini for information about overseas employee exchanges. Many of the topics would be common for anyone researching and applying for jobs." The tech giant further noted that it has seen underground forum posts advertising nefarious versions of large language models (LLMs) that are capable of generating responses sans any safety or ethical constraints. Examples of such tools include WormGPT, WolfGPT, EscapeGPT, FraudGPT, and GhostGPT, which are explicitly designed to craft personalized phishing emails, generate templates for business email compromise (BEC) attacks, and design fraudulent websites. Attempts to misuse Gemini have also revolved around research into topical events, and content creation, translation, and localization as part of influence operations mounted by Iran, China, and Russia. In all, APT groups from more than 20 countries used Gemini. Google, which said it's "actively deploying defenses" to counter prompt injection attacks, has further emphasized the need for heightened public-private collaboration to raise cyber defenses and disrupt threats, stating "American industry and government need to work together to support our national and economic security."

Google details nefarious Gemini use by Iranian spies

And you, China, Russia, North Korea ... Guardrails block malware generation Google says it's spotted Chinese, Russian, Iranian, and North Korean government agents using its Gemini AI for nefarious purposes, with Tehran by far the most frequent naughty user out of the four. The web giant has been tracking the use of Gemini by these nations, using not just simple things presumably like IP addresses to spot them but a combination of technical signals and behavioral patterns, we're told. And while these state-backed snoops have managed to use Gemini for translating and tailoring phishing lures for specific victims, looking up for information about surveillance targets, and writing some software scripts, Google admitted, the biz claims its guardrails at least stopped its AI from generating malware. Overall, the American internet goliath reckons Iran et al aren't doing anything too outrageous, and are mainly asking the LLM for info and guidance as it was designed for. In other words, foreign governments are using Google AI for bad things, but it's not too bad, or so we're told. "While AI can be a useful tool for threat actors, it is not yet the gamechanger it is sometimes portrayed to be," Google said in a Threat Intelligence Group (TIG) report [PDF] this week. "While we do see threat actors using generative AI to perform common tasks like troubleshooting, research, and content generation, we do not see indications of them developing novel capabilities." While AI can be a useful tool for threat actors, it is not yet the gamechanger it is sometimes portrayed to be Iranian spies accounted for 75 percent of all observed Gemini use by the aforementioned quartet's agents, the TIG report notes. The Google team identified over 10 Iran-backed cyber-crews using the AI service, with some particularly focused on researching Android-related security. More broadly, these groups used Gemini for reconnaissance, researching vulnerabilities, identifying free hosting providers, and crafting local personas and content for cyber operations. Notably, Iran's APT42 unit leveraged Gemini to craft phishing content, making up 30 percent of all Iranian APT, or advanced threat actors, activity on the platform. Chinese spies have also been using it for content creation and basic research, with 20 groups from the Middle Kingdom identified so far. Much of this activity focuses on researching US government institutions, while Beijing-backed snoops have also sought assistance with Microsoft-related systems and translation work, according to the report. Google also says it has spotted North Korean operatives using its LLM to write job applications for IT workers as part of the hermit nation's ongoing efforts to insert its workers into Western companies. Nine distinct groups of Norks also tried to find freelancer forums on Discord, and information related to South Korean military and nuclear technology, through Gemini. Russians are relatively light users of Gemini, it seems, with only three groups observed by the team. Google speculates that this could be down to them either using domestically generated LLMs or attempting to limit exposure to avoid being monitored. Or maybe they're just really good at hiding their usage of the LLM. Around 40 percent of Russian activity came from operators linked to "Russian state-sponsored entities formerly controlled by the late Russian oligarch Yevgeny Prigozhin," the cloud behemoth said. This presumably means the Wagner Group and its offshoots. Google notes a Russian operative used Gemini to generate and manipulate content, including rewriting articles with a pro-Kremlin slant for use in influence campaigns. This is exactly the sort of shenanigans Prigozhin's Internet Research Agency used to do. When it comes to breaking Gemini's guardrails and exploiting the engine to write malicious code or cough up personal information, Google claims the LLM is successfully blocking such attempts. It has noted an uptick in folks trying to use publicly known jailbreak prompts and then adapting them slightly in an attempt to get around the filters, but these appear ineffective. The ad giant reported one case involved a request to embed encoded text in an executable and a separate attempt to generate Python code for a denial-of-service attack. While Gemini processed a Base64-to-hex conversion request, it refused further malicious queries. Google has also detected attempts to use Gemini for researching methods to abuse its other services. The biz states its safety systems blocked these efforts, and that it is working on further improvements on these defenses. As well as this, its DeepMind wing is also mentioned in that the lab is apparently coming up with ways to protect AI services from attacks and prohibited queries. "Google DeepMind also develops threat models for generative AI to identify potential vulnerabilities, and creates new evaluation and training techniques to address misuse caused by them," the report added. "In conjunction with this research, DeepMind has shared how they're actively deploying defenses within AI systems along with measurement and monitoring tools, one of which is a robust evaluation framework used to automatically red team an AI system's vulnerability to indirect prompt injection attacks." ®

Foreign Hackers Are Using Google's Gemini in Attacks on the US

The rapid rise of DeepSeek, a Chinese generative AI platform, heightened concerns this week over the United States' AI dominance as Americans increasingly adopt Chinese-owned digital services. With ongoing criticism over alleged security issues posed by TikTok's relationship to China, DeepSeek's own privacy policy confirms that it stores user data on servers in the country. Meanwhile, security researchers at Wiz discovered that DeepSeek left a critical database exposed online, leaking over 1 million records, including user prompts, system logs, and API authentication tokens. As the platform promotes its cheaper R1 reasoning model, security researchers tested 50 well-known jailbreaks against DeepSeek's chatbot and found lagging safety protections as compared to Western competitors. Brandon Russell, the 29-year-old cofounder of the Atomwaffen Division, a neo-Nazi guerrilla organization, is on trial this week over an alleged plot to knock out Baltimore's power grid and trigger a race war. The trial provides a look into federal law enforcement's investigation into a disturbing propaganda network aiming to inspire mass casualty events in the US and beyond. An informal group of West African fraudsters calling themselves the Yahoo Boys are using AI-generated news anchors to extort victims, producing fabricated news reports falsely accusing them of crimes. A WIRED review of Telegram posts reveals that these scammers create highly convincing fake news broadcasts to pressure victims into paying ransoms by threatening public humiliation. That's not all. Each week, we round up the security and privacy news we didn't cover in depth ourselves. Click on the headlines to read the full stories. And stay safe out there. According to a report by The Wall Street Journal, hacking groups with known ties to China, Iran, Russia, and North Korea are leveraging AI chatbots like Google Gemini to assist with tasks such as writing malicious code and researching potential attack targets. While Western officials and security experts have long warned about AI's potential for malicious use, the Journal, citing a Wednesday report from Google, noted that the dozens of hacking groups across more than 20 countries are primarily using the platform as a research and productivity tool -- focusing on efficiency rather than developing sophisticated and novel hacking techniques. Iranian groups, for instance, used the chatbot to generate phishing content in English, Hebrew, and Farsi. China-linked groups used Gemini for tactical research into technical concepts like data exfiltration and privilege escalation. In North Korea, hackers used it to draft cover letters for remote technology jobs, reportedly in support of the regime's effort to place spies in tech roles to fund its nuclear program. This is not the first time foreign hacking groups have been found using chatbots. Last year, OpenAI disclosed that five such groups had used ChatGPT in similar ways. On Friday, WhatsApp disclosed that nearly 100 journalists and civil society members were targeted by spyware developed by the Israeli firm Paragon Solutions. The Meta-owned company alerted affected individuals, stating with "high confidence" that at least 90 users had been targeted and "possibly compromised," according to a statement to The Guardian. WhatsApp did not reveal where the victims were located, including whether any were in the United States. The attack appears to have used a "zero-click" exploit, meaning victims were infected without needing to open a malicious link or attachment. Once a phone is compromised, the spyware -- known as Graphite -- grants the operator full access, including the ability to read end-to-end encrypted messages sent via apps like WhatsApp and Signal.

Gemini chatbot is being exploited by hackers from Iran, China, and North Korea for cyber attacks, confirms Google

North Korean hackers leverage the chatbot to create fake cover letters and research remote IT jobs for infiltration. Hackers supported by state-sponsored organizations from nations like China, North Korea, and Iran are using the Gemini chatbot to enhance their cyberattack capabilities, according to Google's Threat Intelligence Group. The report claims that these attackers are reportedly becoming more productive, but the AI tool has not yet allowed them to create noticeably more advanced methods. The report claims that scammers are utilizing the Gemini for a wide range of novel tasks, such as creating new codes, investigating targets, or even figuring out system weaknesses. The report also stated that disinformation agents are using the chatbot to construct narratives, translate content, and establish virtual identities. The Iranian agents are among the most active scammers on Gemini, the report also stated. They are conducting reconnaissance on defense experts and organizations and using the chatbot to help with phishing campaigns. However, Gemini is being used by Chinese hacker groups to debug code and take advantage of holes in target networks. As they try to extract sensitive data, they concentrate on privilege escalation, lateral movement across systems, and avoiding detection. Also read: iPhone 15 available at Rs 9,901 discount on Flipkart, further savings possible According to reports, North Korean threat actors are using the chatbot to research remote IT job openings in Western nations and create fake cover letters, most likely as part of an infiltration strategy. The report also claimed that the chatbot has been used by Russian hackers less frequently. It is being used by the users to generate codes. These include incorporating encryption features into pre-existing code and translating publicly accessible malware into various programming languages. Also read: Pixel 8 Pro gets Rs 32,000 off on Flipkart, bank offers also available Notwithstanding these results, Google has pointed out that although the AI tool has increased the productivity of these attackers, it hasn't allowed them to create new methods or resources.

Google shares how all the world's baddies have been using Gemini for their nefarious deeds

The most common uses of Gemini appear to be for researching targets and helping at coding. From pretty much the moment that powerful AI language models debuted on the scene, bad guys have been looking to do bad stuff with them. The companies behind them make concerted efforts to protect their models with safeguards against abuse, but bad actors are always coming up with new ways to try and get around these barriers. This week Google shares what it's observed when it comes to Gemini and some well-connected international groups trying to use it for nasty business.