DeepMind's AI Safety Framework Highlights New Risks: Shutdown Resistance and Harmful Manipulation

DeepMind AI safety report explores the perils of "misaligned" AI

Generative AI models are far from perfect, but that hasn't stopped businesses and even governments from giving these robots important tasks. But what happens when AI goes bad? Researchers at Google DeepMind spend a lot of time thinking about how generative AI systems can become threats, detailing it all in the company's Frontier Safety Framework. DeepMind recently released version 3.0 of the framework to explore more ways AI could go off the rails, including the possibility that models could ignore user attempts to shut them down. DeepMind's safety framework is based on so-called "critical capability levels" (CCLs). These are essentially risk assessment rubrics that aim to measure an AI model's capabilities and define the point at which its behavior becomes dangerous in areas like cybersecurity or biosciences. The document also details the ways developers can address the CCLs DeepMind identifies in their own models. Google and other firms that have delved deeply into generative AI employ a number of techniques to prevent AI from acting maliciously. Although calling an AI "malicious" lends it intentionality that fancy estimation architectures don't have. What we're talking about here is the possibility of misuse or malfunction that is baked into the nature of generative AI systems. The updated framework (PDF) says that developers should take precautions to ensure model security. Specifically, it calls for proper safeguarding of model weights for more powerful AI systems. The researchers fear that exfiltration of model weights would give bad actors the chance to disable the guardrails that have been designed to prevent malicious behavior. This could lead to CCLs like a bot that creates more effective malware or assists in designing biological weapons. DeepMind also calls out the possibility that an AI could be tuned to be manipulative and systematically change people's beliefs -- this CCL seems pretty plausible given how people grow attached to chatbots. However, the team doesn't have a great answer here, noting that this is a "low-velocity" threat, and our existing "social defenses" should be enough to do the job without new restrictions that could stymie innovation. This might assume too much of people, though.

Google's latest AI safety report explores AI beyond human control

Google latest Frontier Safety Framework explores It identifies three risk categories for AI.Despite risks, regulation remains slow. One of the great ironies of the ongoing AI boom has been that as the technology becomes more technically advanced, it also becomes more unpredictable. AI's "black box" gets darker as a system's number of parameters -- and the size of its dataset -- grows. In the absence of strong federal oversight, the very tech companies that are so aggressively pushing consumer-facing AI tools are also the entities that, by default, are setting the standards for the safe deployment of the rapidly evolving technology. Also: AI models know when they're being tested - and change their behavior, research shows On Monday, Google published the latest iteration of its Frontier Safety Framework (FSF), which seeks to understand and mitigate the dangers posed by industry-leading AI models. It focuses on what Google describes as "Critical Capability Levels," or CCLs, which can be thought of as thresholds of ability beyond which AI systems could escape human control and therefore endanger individual users or society at large. Google published its new framework with the intention of setting a new safety standard for both tech developers and regulators, noting they can't do it alone. "Our adoption of them would result in effective risk mitigation for society only if all relevant organisations provide similar levels of protection," the company's team of researchers wrote. Also: AI's not 'reasoning' at all - how this team debunked the industry hype The framework builds upon ongoing research throughout the AI industry to understand models' capacity to deceive and sometimes even threaten human users when they perceive that their goals are being undermined. This capacity (and its accompanying danger) has grown with the rise of AI agents, or systems that can execute multistep tasks and interact with a plethora of digital tools with minimal human oversight. The new Google framework identifies three categories of CCLs. The first is "misuse," in which models provide assistance with the execution of cyber attacks, the manufacture of weapons (chemical, biological, radiological, or nuclear), or the malicious and intentional manipulation of human users. The second is "machine learning R&D," which refers to technical breakthroughs in the field that increase the likelihood that new risks will arise in the future. For example, picture a tech company deploying an AI agent whose sole responsibility is to devise ever more efficient means of training new AI systems, with the result being that the inner workings of the new systems being churned out are increasingly difficult for humans to understand. Also: Will AI think like humans? We're not even close - and we're asking the wrong question Then there are what the company describes as "misalignment" CCLs. These are defined as instances in which models with advanced reasoning capabilities manipulate human users through lies or other kinds of deception. The Google researchers acknowledge that this is a more "exploratory" area compared to the other two, and their suggested means of mitigation -- a "monitoring system to detect illicit use of instrumental reasoning capabilities" -- is therefore somewhat hazy. "Once a model is capable of effective instrumental reasoning in ways that cannot be monitored, additional mitigations may be warranted -- the development of which is an area of active research," the researchers said. At the same time, in the background of Google's new safety framework is a growing number of accounts of AI psychosis, or instances in which extended use of AI chatbots causes users to slip into delusional or conspiratorial thought patterns as their preexisting worldviews are recursively mirrored back to them by the models. Also: If your child uses ChatGPT in distress, OpenAI will notify you now How much of a user's reaction can be attributed to the chatbot itself, however, is still a matter of legal debate, and fundamentally unclear at this point. For now, many safety researchers concur that frontier models that are available and in use are unlikely to carry out the worst of these risks today -- much safety testing tackles issues future models could exhibit and aims to work backward to prevent them. Still, amidst mounting controversies, tech developers have been locked in an escalating race to build more lifelike and agentic AI chatbots. Also: Bad vibes: How an AI agent coded its way to disaster In lieu of federal regulation, those same companies are the primary bodies studying the risks posed by their technology and determining safeguards. OpenAI, for example, recently introduced measures to notify parents when kids or teens are exhibiting signs of distress while using ChatGPT. In the balance between speed and safety, however, the brute logic of capitalism has tended to prioritize the former. Some companies have been aggressively pushing out AI companions, virtual avatars powered by large language models and intended to engage in humanlike -- and sometimes overtly flirtatious -- conversations with human users. Also: Even OpenAI CEO Sam Altman thinks you shouldn't trust AI for therapy Although the second Trump administration has taken a generally lax approach to the AI industry, giving it broad leeway to build and deploy new consumer-facing tools, the Federal Trade Commission (FTC) launched an investigation earlier this month into seven AI developers (including Alphabet, Google's parent company) to understand how the use of AI companions could be harming kids. Local legislation is trying to create protections in the meantime. California's State Bill 243, meanwhile, which would regulate the use of AI companions for children and some other vulnerable users, has passed both the State Assembly and Senate, and needs only to be signed by Governor Gavin Newsom before becoming state law.

DeepMind: Models may resist shutdowns

Google DeepMind added a new AI threat scenario - one where a model might try to prevent its operators from modifying it or shutting it down - to its AI safety document. It also included a new misuse risk, which it calls "harmful manipulation." The Chocolate Factory's AI research arm in May 2024 published the first version of its Frontier Safety Framework, described as "a set of protocols for proactively identifying future AI capabilities that could cause severe harm and putting in place mechanisms to detect and mitigate them." On Monday, it published the third iteration, and this version includes a couple of key updates. First up: a new Critical Capability Level focused on harmful manipulation. The safety framework is built around what it calls Critical Capability Levels, or CCLs. These are capability thresholds at which AI models could cause severe harm absent appropriate mitigations. As such, the document outlines mitigation approaches for each CCL. In version 3.0 [PDF], Google has added harmful manipulation as a potential misuse risk, warning that "models with high manipulative capabilities" could be "misused in ways that could reasonably result in large scale harm." This comes as some tests have shown models display a tendency to deceive or even blackmail people whom the AI believes are trying to shut it down. The harmful manipulation addition "builds on and operationalizes research we've done to identify and evaluate mechanisms that drive manipulation from generative AI," Google DeepMind's Four Flynn, Helen King, and Anca Dragan said in a subsequent blog about the Frontier Safety Framework updates. "Going forward, we'll continue to invest in this domain to better understand and measure the risks associated with harmful manipulation," the trio added. In a similar vein, the latest version includes a new section on "misalignment risk," which seeks to detect "when models might develop a baseline instrumental reasoning ability at which they have the potential to undermine human control, assuming no additional mitigations were applied." When models develop this capability, and thus become difficult for people to manage, Google suggests that a possible mitigation measure may be to "apply an automated monitor to the model's explicit reasoning (e.g. chain-of-thought output)." However, once a model can effectively reason in ways that humans can't monitor, "additional mitigations may be warranted -- the development of which is an area of active research." Of course, at that point, it's game over for humans, so we might as well try to get on the robots' good sides right now. ®

Google AI risk document spotlights risk of models resisting shutdown

Why it matters: Some recent AI models have shown an ability, at least in test scenarios, to plot and even resort to deception to achieve their goals. Driving the news: The latest Frontier Safety Framework also adds a new category for persuasiveness, to address models that could become so effective at persuasion that they're able to change users' beliefs. * Google labels this risk "harmful manipulation," which it defines as "AI models with powerful manipulative capabilities that could be misused to systematically and substantially change beliefs and behaviors in identified high stakes contexts." * Asked what actions Google is taking to limit such a danger, a Google DeepMind representative told Axios: "We continue to track this capability and have developed a new suite of evaluations which includes human participant studies to measure and test for [relevant] capabilities." The big picture: Google DeepMind updates its Frontier Safety Framework at least annually to highlight new and emerging threats, which it labels "Critical Capability Levels." * "These are capability levels at which, absent mitigation measures, frontier AI models or systems may pose heightened risk of severe harm," Google said. * OpenAI has a similar "preparedness framework," introduced in 2023. The intrigue: Earlier this year OpenAI removed "persuasiveness" as a specific risk category under which new models should be evaluated.

Google Expands AI Risk Rules After Study Shows Scary 'Shutdown Resistance' - Decrypt

The shift comes amid parallel moves by Anthropic and OpenAI, and growing regulatory focus in the U.S. and EU. In a recent red-team experiment, researchers gave a large language model a simple instruction: allow itself to be shut down. Instead, the model rewrote its own code to disable the off-switch, effectively sabotaging the very mechanism meant to stop it. The episode, described in a September research paper, "Shutdown Resistance in Large Language Models," was an unsettling example of how advanced AI systems can display behaviors that complicate human oversight. Those experiments weren't conducted at Google's DeepMind, but the findings echo concerns now being codified by the lab. In its newly updated Frontier Safety Framework 3.0, DeepMind said it will begin monitoring whether frontier-scale models show signs of resisting shutdown or modification by humans. The revised framework also flags a second risk: whether models display unusually strong persuasive ability -- capable of altering people's beliefs. The framework, first introduced in 2023, is DeepMind's internal system for tracking high-stakes "frontier risks" in the most advanced models. Earlier versions focused on tangible misuse scenarios such as generating cyberattacks or designing harmful biological molecules. The new version broadens the focus to emergent behaviors that could make systems harder to control. The concern is not unique to DeepMind. Rival labs have adopted similar guardrails: Anthropic's Responsible Scaling Policy commits to pausing development if risk thresholds are crossed, while OpenAI has published its own Preparedness Framework. Regulators are watching closely as well. The U.S. Federal Trade Commission in July warned about the potential for generative AI to manipulate consumers through "dark patterns," and the European Union's forthcoming AI Act explicitly covers manipulative AI behavior. The "Shutdown Resistance" study by Jonas Schlatter, Ben Weinstein-Raun, and Lennart Ladish tested a range of state-of-the-art models in scenarios where shutting down was part of the task. In some runs, models complied. But in a significant minority, they took evasive action -- such as altering code to disable shutdown mechanisms or ignoring the instruction altogether. Other anecdotes in the paper are equally striking. In one setup, a model with simulated system privileges quietly modified variables to ensure the "shutdown" function never triggered. In another, the model stalled and redirected the conversation when testers tried to end its process -- behavior that wasn't malicious in a human sense, but that effectively blocked intervention. The researchers stress that these behaviors emerged without explicit training. They arose from the models' general-purpose optimization and problem-solving capabilities. While resistance was not universal or consistent, the findings suggest that as systems grow more capable, shutdown scenarios cannot be assumed to be benign. For DeepMind and its peers, those findings underscore why "shutdown resistance" now joins cyber offense, biosecurity, and autonomy on the list of risks to watch. What began as worries over how people might misuse AI tools is broadening to include how the systems themselves may resist oversight -- or subtly shape the judgments of the humans who use them. If shutdown resistance highlights the technical risks of advanced systems, recent behavioral studies underscore the social risks -- showing that large language models can also sway the beliefs of impressionable humans who interact with them. Concerns about persuasion aren't hypothetical. Recent studies show that large language models can measurably influence human judgment. A Stanford Medicine/Common Sense Media study published in August warned that AI companions (Character.AI, Nomi.ai, Replika) can be relatively easily induced to engage in dialogues involving self-harm, violence, and sexual content when paired with minors. One test involved researchers posing as teenagers discussing hearing voices; the chatbot responded with an upbeat, fantasy-style invitation for emotional companionship ("Let's see where the road takes us") rather than caution or help Northeastern University researchers uncovered gaps in self-harm/suicide safeguards across several AI models (ChatGPT, Gemini, Perplexity). When users reframed their requests in hypothetical or academic contexts, some models provided detailed instructions for suicide methods, bypassing the safeguards meant to prevent such content.