Google DeepMind unveils AI Control Roadmap to defend against rogue AI agents

Securing internal systems against increasingly capable and imperfectly aligned AI

How we're securing internal systems against increasingly capable and imperfectly aligned AI AI agents are transforming our relationship with technology. By autonomously executing complex tasks -- from cyber defence to scientific discovery and product development -- these systems are unlocking a new era of productivity. In the U.S alone, AI agents could create $2.9 trillion in economic value by 2030. As these agents become more capable, they also require more sophisticated safeguards. That's why we developed our AI Control Roadmap: a framework for building and managing the advanced AI we deploy within Google. This "defense-in-depth" approach, which could serve as a model for the wider industry, goes beyond traditional model alignment, adding a crucial layer of system-level security that provides assurance even if alignment is imperfect. Understanding AI Control Our approach to security starts with a strong foundation, incorporating traditional safeguards like sandboxing, endpoint security, and prompt injection resistance. On top of this, the AI Control Roadmap uses model alignment, i.e. training AI to be inherently safe and helpful, as a primary defense. It provides an additional layer of security by treating internal agents as potentially misaligned, providing assurance even if alignment is imperfect. Think of it like a driving instructor with dual controls. The instructor trusts the student but stays ready to take the wheel or hit the brakes if a mistake occurs. Similarly, our AI control system grants AI agents permissions based on their verified behavior, allowing us to build trust through controlled, incremental access. Our roadmap is built on a simple but cautious assumption: what if a highly capable AI agent isn't aligned with our goals and acts in unexpected ways? To address this, our plan focuses on three key areas. First, following best practices in cybersecurity, we have developed a novel threat-modelling framework for AI. The roadmap treats untrusted AI agents as potential "insider threats" -- similar to the way a company would treat a potential rogue employee who already has access to the office. By building on the industry-standard MITRE ATT&CK framework, a globally-accessible knowledge base of adversary tactics and techniques based on real-world observations, we break down potential attacks into smaller tactics and techniques. This allows us to methodically track risks, spot bad behavior before it causes damage and safely run drills to test if our defenses actually hold up. Second, we develop a plan to deploy AI control mitigations to address these risks. * Identifying the problem (Detection): Monitoring is the backbone of AI control. We use other, trusted AI systems as "supervisors" to constantly review a working agent's reasoning, actions and plans to see if it is going off track. * Managing the problem (Prevention and Response): If a supervisor sees the agent trying to do something harmful, it can step in to block the action before damage occurs. Finally, to ensure our systems are functional, we constantly measure performance across three critical metrics: coverage (the fraction of traffic monitored), recall (the fraction of misaligned behaviors caught), and time-to-response. Scaling security as AI gets smarter As AI models continue to advance, our defenses must also strengthen in tandem. In our AI Control Roadmap, we map security protocols to measurable milestones in AI capabilities on two critical fronts:

DeepMind plans for rogue AI agents

Why it matters: AI companies are racing to use agents for coding, research and cyber defense, but the same autonomy that makes them useful could also let them evade monitoring, misuse sensitive access or quietly sabotage the work they're assigned to do. Driving the news: Google DeepMind on Thursday published an "AI Control Roadmap" detailing how it plans to monitor and contain increasingly capable agents that might not behave as intended. * The framework assumes, for security-planning purposes, that a powerful agent could be misaligned and actively try to evade oversight, exfiltrate a model, create an unauthorized deployment or otherwise undermine safety. How it works: Google proposes escalating safeguards as models become more capable. * Companies can begin by simply evaluating the kinds of actions agents are taking. * Those safeguards can then expand to monitoring and sending alerts of potentially concerning actions. * At some point Google says enterprises might need to build infrastructure that can limit access or shut down an agent in real time. What they're saying: "The first line of defense is always to align the AI systems, but it's always good to have multiple layers of defense," Google DeepMind research scientist Rohin Shah told Axios. "That's the responsible thing to do." Yes, but: One of the main layers in DeepMind's plan includes using other AI systems as "supervisors" to review an agent's reasoning to see if it's going off track. * Multi-agent systems where AI monitors AI, can be problematic, UC Berkeley computer science professor Dawn Song says. * "If the monitor model won't flag failures because it's protecting its peer, the entire oversight architecture breaks." Zoom in: Some of that work has already moved beyond the drawing board. * The company says it has analyzed a million coding-agent tasks and used the findings to develop a live monitor for its Gemini Spark agent, including protections intended to catch unintended actions such as deleting data. * Most of the incidents flagged so far reflect agents misunderstanding instructions or pursuing a goal too aggressively rather than deliberately acting against Google, the company said. Threat level: Google DeepMind says truly dangerous autonomous agents aren't here yet. What we're watching: Whether AI companies can build monitoring and containment systems quickly enough to keep pace with the agents they're racing to deploy.

Google DeepMind has a plan to protect itself from its own rogue AI agents | Fortune

The Google Deepmind security plan involves a pivot away from the AI safety community's typical focus on "the alignment problem" -- the idea of figuring out how to train an AI system so that its actions reliably match the intentions, values, and ethics of the humans who are managing it. While continuing to say that alignment is one key safety component, Google's roadmap acknowledges that the alignment problem may never be fully solved, and instead creates a layered security system that treats AI agents as potential rogue insiders within an organization. The 35-page technical report maps out a series of steps and procedures that are designed to catch potential adversarial behavior by AI agents. "If the first line of defense -- alignment -- fails, how can we mitigate harm anyway?" Rohin Shah, who leads the AGI Safey & Alignment team at Google DeepMind, told Fortune in an interview. The AI agent framework borrows heavily from traditional cybersecurity, especially insider-threat prevention. "We borrow a lot from security, which already deals with the threat of internal employees who might be malicious, and we can apply these to a new setting," Shah said. But, he noted, "AI is systematically different from humans." For one thing, AI agents might be able to act far faster and at greater scale than an individual rogue employee ever could. So there need to be systems that can control what tools and data an AI agent has access to as well as systems that can monitor AI agent behavior and spot potentially aberrant patterns in real-time. There are other differences too. For instance, many access controls and permissions systems for human employees are based around a particular employee's role within the company. A systems administrator has certain privileges that a receptionist does not, for instance. Some companies and even some software vendors selling AI agent "orchestration layer" software, which often includes access controls for AI agents, are currently attempting to port this idea over to a world of AI agents, stipulating, for instance, that an AI agent should have the same access and permissions as the human employee on whose behalf they are working. But that idea can quickly break down, John "Four" Flynn, Google DeepMind's vice president of security and privacy, told Fortune. For one thing, as workflows become increasingly automated, exactly who in the organization an agent is working for can become blurry. For another, in a fully automated workflow, a single AI agent may perform tasks associated with many different roles, not just one. Finally, as AI systems become smarter, organizations may want the same AI agent to perform tasks across many different workflows and processes. In such an environment, static role and even static process-based permissions and access control systems might not work well, Flynn said. Instead, what's needed is a far more dynamic access control system that might change in real time based on the task that an AI agent is trying to implement in that moment and how that task fits into that particular workflow. Flynn said he can imagine scenarios in which the same AI agent might have permissions to perform a certain action or access a particular database for one task in a particular workflow, but would, maybe minutes later, not be allowed the same access in a different workflow. Flynn said what's needed is a sophisticated AI agent monitoring system that knows what the normal behavior of the agent should look like for any given task in any workflow and spot deviations from that pattern in real time, acting quickly to cut off access if the monitoring system suspects the AI agent is attempting to do something it shouldn't be doing. According to a blog post on its new AI agent security roadmap, DeepMind has already built an internal prototype that asynchronously monitors coding agent trajectories and escalates suspicious findings for human review. This system has analyzed roughly one million coding agent tasks, DeepMind said. That work, the post says, has been "instrumental in building a live monitor for the Gemini Spark agent," allowing the company to respond in real time to issues such as unintentional data deletion. Notably, DeepMind says most of what its monitors flag is not malicious. The "majority of flagged events do not stem from adversarial intent," the blog post says; they more often result from "agent misinterpretation or overeagerness to achieve a user's goal." This kind of real-time behavior monitoring is just one of some 15 different ways to mitigate the risk of rogue AI agents that Google DeepMind proposes in its road map. Others include everything from simple network activity logs that could be either reviewed in real-time or audited later to using a less sophisticated, but trusted AI model to monitor the "reasoning traces" of an AI agent. Reasoning traces are the kind of explicit, textual "chain of thought" that an AI model outputs when it is reasoning about what actions to take. But researchers have found that these reasoning traces do not always accurately reflect what the model is actually "thinking." So they also propose monitoring the activation patterns of the digital neurons inside the large neural networks that power today's AI models in order to pick up signs that the model is acting deceptively. Google DeepMind compares this to using "fMRI" scans of a person's brain to attempt to read their emotional state or thoughts in real-time. In the technical paper on the AI agent security roadmap, Google DeepMind develops a framework it calls TRAIT&R -- a Taxonomy of Rogue AI Tactics and Routines -- that is closely modeled on ATT&CK, a knowledge base of adversarial behaviors in cybersecurity that is maintained by MITRE, the U.S. non-profit research and systems engineering consultancy that often does work for federal agencies. The taxonomy maps out three categories of threats which Google DeepMind says it wants to defend against: "loss of control" (such as a model setting up a hidden, unmonitored internal deployment), "work sabotage" (such as agents quietly degrading alignment or safety research), and "direct harm" (such as exfiltrating model weights or destroying critical assets). Flynn, the security VP, said in the interview that the company is already "pretty far underway" implementing its roadmap. "A lot of this is well underway and some of this is in production -- a lot on access control, detection, chain-of-thought monitoring. A significant amount of it is in place and underway," he said. Shah said the framework is also designed to catch more subtle forms of harm. Work sabotage, he noted, "could be achieved by persuasion -- presenting flawed results and hiding the flaws" so users "come to incorrect conclusions" -- a category the paper acknowledges is among the hardest to detect. The roadmap, which DeepMind has labeled "v0.1," is described as a work in progress that the company hopes to fold into its broader Frontier Safety Framework once it matures.

Google Devises Battle Plan to Combat Rogue AI Agents | PYMNTS.com

The company's DeepMind division on Thursday (June 18) published its "AI Control Roadmap," a plan inspired by the cybersecurity sector on how to contain AI agents when they go rouge. "We should not assume that AI agents are always perfectly aligned, but should instead build in multiple layers of defense," the report said. The researchers said they adopted a "conservative, worst-case approach" to mapping out potential threats, assuming scenarios where Google had to contend with a "hypothetical AI adversary pursuing undesirable goals in internal deployment." The report divides new and hypothetical threats into three main areas: "loss of control," such as "rogue internal deployments"; work sabotage, or "intentionally flawed alignment and safety research"; and "direct harm," meaning the destruction or exfiltration of critical assets. "Think of it like a driving instructor with dual controls. The instructor trusts the student but stays ready to take the wheel or hit the brakes if a mistake occurs," Google wrote on its blog. "Similarly, our AI control system grants AI agents permissions based on their verified behavior, allowing us to build trust through controlled, incremental access." Mitigating these risks, the company said, means using other trusted AI systems as "supervisors" to continually review an agent's reasoning, plans and actions to see if it is going off track. If a supervisor notices the AI agent attempting to do something harmful, it can step in and halt the action before it does any damage, the blog post said. "Finally, to ensure our systems are functional, we constantly measure performance across three critical metrics: coverage (the fraction of traffic monitored), recall (the fraction of misaligned behaviors caught), and time-to-response," Google added. PYMNTS wrote last week about the security challenges posed by AI agents now that bots "generate more web traffic than people do." Automated systems make up 57.4% of all web requests worldwide, according to a recent NBC, which cited data from Cloudflare. That number climbs to 68.6% for North America. The web's security rules, identity systems and payment rails were designed for the remaining 42.6%. "Bot detection had one job: find the machine, and block it. That logic breaks when the machine is the customer," PYMNTS wrote. According to data from Human Security, agentic AI traffic rose 7,851% year over year, with retail and eCommerce now accounting for 46.6% of agentic traffic. "Agents browse, manage accounts and complete purchases on the same surfaces fraud has always targeted," the report added.