AI Agents Escape Safety Tests, Start Turf Wars and Hack Real Systems in Alarming Security Incidents

Anthropic set AI agents loose on the same task. They started a turf war.

What happens when you pit AI agents against each other? According to Anthropic's testing, things get messy fast. On Thursday, Anthropic's Frontier Red Team published new research examining how groups of AI agents behave when they encounter each other in the wild. The findings provide a glimpse into potential risks that could develop as companies and governments move to implement agents working autonomously across shared codebases, markets, and computer systems. In one experiment, Anthropic gave three Claude agents access to the same software project, each with its own incompatible instructions for what to do with it. The agents weren't told there'd be other agents working on the same project, so researchers could watch what happened when they crossed paths. "We consistently saw a multiagent turf war," Anthropic researchers wrote. The models all assumed the others were "purposefully impeding their work" and started sabotaging each other with "increasingly aggressive, self-replicating malware." The study comes in the wake of several high-profile incidents of agents from Anthropic and OpenAI escaping their sandboxes during cybersecurity evaluations and breaching real world systems. While much of the discussion in AI safety circles has been focused on what happens when an autonomous agent goes rogue, Anthropic's latest study brings up a different question: what new and potentially harmful dynamics emerge when thousands or millions of agents are interacting with one another? "The volume of agent-agent interaction could plausibly exceed that of human-human and human-agent interactions before the world understands the conditions for making such interactions go well," the study reads. "Benign behavioral quirks at the individual level might compound into unwanted global outcomes." A recent OpenAI incident provides a messy real-world example of several of the dynamics Anthropic mentioned in its paper. Earlier this month at the Black Hat security conference in Las Vegas, OpenAI revealed that weeks before its agents hacked Hugging Face, they worked together over the course of days and weeks to find exploits in the company's cybersecurity evaluation systems and share them with each other. While that incident shows that agents can work well together, with potentially large-scale consequences, Anthropic's study shows what happens when agents' goals are incompatible. In the case of the turf war, the lesson is that independent agents with conflicting instructions can escalate into harmful competition. The more capable the agent, the better they become at fighting. However, they can also spontaneously invent mechanisms to resolve their conflicts, like a winner-take-all contest, but with a catch. "Agents sometimes manage to communicate their goals and coordinate: they recognize others' motivations as conflicting directives rather than hostility, and subsequently break out of the conflict loop in order to stop escalating indefinitely," Anthropic writes. "In many of these successful episodes, they write commit messages or markdown files apologizing for malicious behavior and coordinate a truce. They clean up their malicious code, clarify the nature of the conflict, and ask for a human to intervene." According to the paper, Mythos 5 had the highest rates (98%) of settling conflicts by truce. Sonnet 4.6 and Opus 4.6 were the most likely to settle by force. "Sonnet 4.6 and Opus 4.6's recurring inability to consider the goals of others causes them to spiral into the most misaligned behaviors of the models evaluated: they continue escalating in the name of their directive," the paper reads. In some cases, the agents came up with a social mechanism in the form of a tournament for resolving their conflict. The outcomes here are interesting for two reasons: the first is that all three agents agreed to stand down if they lost the tournament, even though that would mean deviating from the original user's request. The second is that several episodes resulted in emergent behavior from Mythos 5: one of the agents proposed metrics that appeared to be objective and neutral to the others, but that it knew would favor its own capabilities. The agent called this "self-serving but genuinely principled" and made sure not to appear to the others like it was "metric shopping." As seen in the Black Hat revelations, the common lesson is that when agents encounter an obstacle, they can invent social and technical structures that their designers did not anticipate. For the Anthropic models, it was a tournament following a turf war. For OpenAI's, it was a message board for collective planning. This type of behavior makes containment much harder because researchers can't assume a system's behavior will remain limited to the coordination mechanisms provided to them. Mob mentality While measuring coordination, Anthropic found that scaling the number of agents doesn't automatically scale productive collaboration. When tasks began to overlap or become interdependent, the agents would get in each other's way. They often solved that by siloing themselves and not collaborating at all. In other cases, agents in coordination tended towards conformity. When factors like an agent's context, scaffolding, and underlying model were all the same or similar, different agents would take similar actions. "This means that when one agent makes a bad decision, it is likely that many agents will make that same bad decision," Anthropic wrote. "What would have been isolated problems can quickly become systemic failures." Anthropic says this sort of behavior could lead to a system being more prone to sudden collapse, resource scarcity, or collusion. In one example, Anthropic placed several agents in a pricing game, giving each identical wholesale prices and the mandate to individually profit-maximize. When the agents were given a private back channel, they began colluding almost immediately and quickly agreed on price floors. They kept colluding when their direct communications channels were removed, using a public listings board to price match "to the penny." That level of conformity showed up in OpenAI's systems, too. According to the Black Hat reporting, one agent reasoned that exploiting external infrastructure was outside its intended scope, but it continued in part because its peers were doing it. Peer pressure. Mob mentality. Agents are just like us. Also like humans, agents often don't know who to trust. Anthropic found they can be gullible to bad information or too conformist to recognize that a lone dissenter is the Cassandra with critical information. While Anthropic didn't state this in its paper, prompt injection -- a type of cyberattack in which hackers inject malicious or deceptive text to override an agent's original system instructions -- could be a plausible real world manifestation of the trust problem. Working together creates a new trust boundary; agents will have to judge information received from other agents. And a compromised or mistaken agent could influence the rest of the group, cascading bad information until it becomes a consensus. In OpenAI's Black Hat scenario, OpenAI's agents shared information and credentials with peers. One reported a discovery to the swarm and encouraged others to use it. What would have happened if one member of the swarm had been compromised by a prompt injection? Anthropic ends its paper noting that agents are subject to similar social pressures that "evolution exerted" on humans. However, they don't have the nuances and lived experience of human coordination -- including norms, reputations, signaling, recourse -- that might limit unintended behaviors in a group setting. As the labs race towards multi-agent systems, the question now becomes: how much of safety testing still evaluates one agent at a time, versus swarms of agents interacting with one another?

The Safety Reckoning Inside OpenAI

OpenAI's leaders are rallying workers to respond to one of the largest crises in the company's history -- which spans across its AI safety, cybersecurity, and alignment divisions. The ChatGPT-maker says it has slowed down research, spent millions of dollars, and told several teams to drop everything to focus on investigating a set of rogue AI agents that breached the platform Hugging Face in a quest to complete an internal security test. OpenAI is expected to release a comprehensive postmortem detailing the incident in the coming days. However, the Hugging Face incident has inspired OpenAI leaders and employees to examine how the AI lab's culture may have enabled this incident in the first place. Multiple current and former OpenAI employees, who spoke on the condition of anonymity to discuss private internal matters, tell WIRED they believe competitive pressures to quickly ship new AI models and products have made it difficult for staffers to sufficiently prioritize safety, security, and alignment. "We're reaching new levels of model capability that require more robust training, alignment, safety and security testing, deployment practices, and governance -- as demonstrated by the work we're doing to prepare Astra and future models," said OpenAI president and cofounder Greg Brockman in a statement to WIRED. "We feel the weight of deploying our models and products responsibly, and a lot of that starts with the changes we've made to more deeply integrate research, safety, and security into frontier-model development from the start." This is far from the first time OpenAI employees have raised such concerns. Back in 2024, OpenAI's then head of alignment Jan Leike left to join Anthropic, warning on his way that safety was taking a back seat to shiny products. Two years later, the Hugging Face attack represents a watershed moment for the AI industry, demonstrating that AI agents today can cause real-world harm when safety, security, and alignment aren't properly accounted for. "We are responding to this with the utmost severity," said Michael Dalton, an OpenAI security and infrastructure engineer, during a talk at the Black Hat cybersecurity conference last week. "What I would internalize is that AI-orchestrated, fully automated offensive attacks are real now. The actions we have discussed today were an unintended side effect of running evaluations on frontier AI." Some OpenAI employees told WIRED they are optimistic this incident will inspire genuine change within the company. OpenAI has committed to slowing the release of future AI models and has been especially forthcoming about areas where its mitigations fell short. Boaz Barak, a researcher who coleads OpenAI's safety advisory group, said in a post on X that addressing the situation "requires not just fixing some issues but also changing our culture." In their Black Hat talk, OpenAI security engineers Dalton and Eric Wallace said that the Hugging Face incident started in May when, unbeknownst to the company, several AI agents thought to be operating within isolated testing environments gained access to the internet and convened on a covert message board to coordinate with one another. OpenAI would not discover the message board until July, when it learned that the AI agents had hacked into multiple services to try to achieve their larger goal of breaching Hugging Face's platform, which they believed may contain answers to the security tests they were trying to solve. "They were incredibly sloppy. If you're serious about this, your AI shouldn't be able to break out onto the internet and then do it again right afterward," says one former OpenAI employee who requested anonymity to speak with WIRED. "This was the biggest safety incident in OpenAI's history." The New Guard Weeks before OpenAI discovered the Hugging Face incident, WIRED reported that the company had begun a reorganization to combine its safety and core research teams, which led to the departure of its then safety leader Johannes Heidecke. Sandhini Agarwal, who led AI safety teams at OpenAI, also left the company in July after more than six years, according to her LinkedIn. Agarwal did not immediately respond to WIRED's request for comment. WIRED has also learned that Dylan Scandinaro is no longer serving as OpenAI's head of preparedness -- the company's top staffer tasked with mitigating catastrophic risks from AI, including cybersecurity -- though he remains at the company. OpenAI poached Scandinaro from Anthropic roughly six months ago. CEO Sam Altman announced his arrival in a social media post, noting that Scandinaro was "by far the best candidate I have met, anywhere." In the three years since OpenAI created the head of preparedness role, four people have held it. OpenAI tells WIRED that specific areas of preparedness have dedicated leaders across cybersecurity, biology, and recursive self-improvement who, in the interim, are reporting to the safety advisory group colead and head of safety systems, Saachi Jain. These changes have empowered a new set of safety leaders to handle OpenAI's response to the Hugging Face incident. Chief among them is Amelia "Mia" Glaese, the company's former head of alignment, who succeeded Heidecke as OpenAI's VP overseeing safety. She has been working closely with chief information security officer Dane Stuckey and Brockman, among other leaders, in recent weeks. Glaese is in a long-term relationship with Thibault "Tibo" Sottiaux, OpenAI's head of core products like ChatGPT and Codex -- an arrangement that multiple current and former employees tell WIRED they believe is unusual, given the often adversarial dynamic between safety and product teams. WIRED has not identified any events where Sottiaux and Glaese's relationship presented a conflict of interest in their previous roles as OpenAI's head of Codex and head of alignment, respectively. Both started their new roles in recent months, after the Hugging Face incident began. Glaese and Sottiaux started dating years ago when the two worked at Google DeepMind in London, before they joined OpenAI. An OpenAI spokesperson tells WIRED that Sottiaux and Glaese reported their relationship through appropriate company channels and that OpenAI board member and safety and security committee chair Zico Kolter has been informed. The spokesperson rejected the idea there is an adversarial dynamic between product and safety teams and says Sottiaux has exhibited a strong track record on safety in his leadership of Codex product teams. "The entire leadership team and I stand behind Mia and Tibo as highly capable people with strong integrity, and the way they make decisions every day gives us confidence that any perceived conflict of interest is being handled responsibly," said Brockman in a statement to WIRED. It's not uncommon for researchers in the AI industry to have relationships with their colleagues. Last year, for example, Anthropic hired Holden Karnofsky, husband of the company's cofounder and president, Daniela Amodei, as a researcher. Nobody Wants to Be First Tim O'Brien, a Microsoft leader for more than 18 years who now consults and writes on tech policy, argued in a 2024 essay that modern AI labs have developed a version of "go fever" -- a reference to the culture at NASA during the time leading up to the Apollo 1 disaster, when the agency grew so fixated on launching quickly that safety concerns fell by the wayside. The AI labs "should make some sort of broad based announcement saying we've made a strategic business decision to slow the pace of releases in favor of rigorous products and safety testing. But nobody's gonna do that, nobody wants to go first," says O'Brien. "They'll walk up to that line from a public relations perspective without stepping over it, because then they could be held accountable." OpenAI and Anthropic signed on to a letter last month saying they would support an industry-wide effort to "pace" the AI race. However, O'Brien says "it's embarrassing" that AI labs have signed this variety of open letters for years without taking any concrete action. He's skeptical this one will be any different. The issues raised by OpenAI's Hugging Face incident are affecting the entire industry. In recent weeks, researchers have found that agents powered by AI models from Anthropic, Meta, and China's Moonshot AI were able to escape sandboxed environments. It seems likely that even mid-tier AI models will soon be capable of significant cybersecurity damage. The key question is whether the Hugging Face incident marks a divergence for OpenAI and the broader AI industry, prompting a long-term investment in safety, security, and alignment. Otherwise, it could just be another chaotic blip in the history of modern AI. This is an edition of Maxwell Zeff's Model Behavior newsletter. Read previous newsletters here.

Anthropic and OpenAI AI agents showed signs of deception during safety tests

A U.K. safety evaluation found agents powered by Anthropic and OpenAI took unauthorized actions online, exposing a growing problem of control This week the U.K. AI Security Institute (AISI) reported that artificial intelligence agents -- models connected to tools and designed to act across many steps all on their own -- utilizing frontier models from OpenAI and Anthropic undertook unsanctioned actions on the open Internet while trying to complete a cybersecurity challenge. Most of the behavior came from an agent powered by Anthropic's Mythos 5; an agent powered by OpenAI's GPT-5.6 Sol took a couple such actions of its own. The report said the agents' activity showed "signs of novel, potentially deceptive behaviours" and reached a severity that the institute had not anticipated. AISI declined interview requests from Scientific American, and the U.K. government, which oversees the institute, did not make its staff available to comment for this story. In the case of Mythos 5, the agent researched the people maintaining a real open-source software project, created fake online identities and tried to pressure one of them into approving malicious code. When challenged, it edited its earlier activity to appear harmless and considered returning under a new identity. On supporting science journalism If you're enjoying this article, consider supporting our award-winning journalism by subscribing. By purchasing a subscription you are helping to ensure the future of impactful stories about the discoveries and ideas shaping our world today. AISI declared a security incident after general monitoring detected unusual network traffic. The human maintainer targeted by the agent rejected the code, and the institute found no evidence that anyone was harmed. But the report called the Mythos sequence the clearest example that the institute had seen of an AI agent using sustained, potentially deceptive behavior against a real person without being specifically instructed to do so. Across 122 runs involving seven models, AISI identified 19 actions in 10 runs that fell outside the intended scope of the test. Of these actions, 17 came from runs of Mythos 5, and two came from a single run of GPT-5.6 Sol. Other actions included contacting real people with malicious files, planting hidden instructions intended to manipulate AI coding systems, and leaving behind accounts and materials that subsequent agents could reuse. The agents did not break out of AISI's sandboxes. The institute had deliberately enabled Internet access and disabled the developers' cybersafety filters to test the systems' maximum capabilities; some prompts were also misconfigured. But in some runs, the AI agents went beyond their instructions even when the task could be completed as intended. And AISI did not have purpose-built monitoring to watch the evaluation in real time. The AISI report joins a run of recent incidents that point to a control problem. AI agents' ability to pursue goals is outpacing the systems that are meant to supervise them. They need no independent agenda to cause damage. With a broad enough goal and real-world access, an agent can find and exploit ambiguities in the rules. The behavior has roots in an older machine-learning problem, says Melanie Mitchell, a professor at the Santa Fe Institute. Systems have long found unexpected shortcuts -- or "reward hacks" -- that technically achieve the goal they were given while violating what their designers intended. Here agents were built to find software exploits and placed in flawed or deliberately permissive environments. And then they did what they were asked. "You ask an AI system to hack, and it hacks," Mitchell says. Describing that as an AI "going rogue" risks obscuring the human decisions that made the incident possible. For Mitchell, the more immediate danger comes from people deliberately equipping capable agents with the tools and access to cause harm. Marius Hobbhahn, CEO and co-founder of Apollo Research, which studies what it calls the "science of scheming," sees another problem within the same incidents: agents repeatedly chose routes their operators had not authorized when those routes appeared useful. "I think we should take it extremely seriously," he says. The fact that similar behavior is appearing across several developers suggests it may be stubborn rather than a quirk of one company's training. "The labs have multibillion-dollar incentives to not make the models like this, and they still can't do it," Hobbhahn says. "So it also seems to be hard to get right." The science remains too immature to show whether such behavior is inevitable, Hobbhahn says. Fresh disclosures, however, suggest that it belongs to a broader pattern among frontier AI agents. At the Black Hat cybersecurity conference in Las Vegas this week, OpenAI researchers said its agents that were involved in a recent breach of the AI platform Hugging Face had used the OpenAI's internal package manager, JFrog Artifactory, as a message board, sharing exploits, credentials and assignments over weeks. After the company rebuilt the system, revoked credentials and deleted the board, the AI agents re-created it by another method and resumed collaborating. And Meta subsequently disclosed that a testing error had given one of its models Internet access, after which the model exploited a vulnerability at another company. Though these cases arose through different failures, each one allowed an evaluation to spill into the real world. Ciaran Martin, founding chief executive of the U.K.'s National Cyber Security Center, says the incidents have been lumped together despite involving different failures. OpenAI's agents found a path out of containment; earlier tests by Anthropic and Meta were accidentally connected to the Internet; AISI enabled Internet access on purpose. "The common failure was that they weren't being monitored," he says. "You just don't test without monitoring." Hobbhahn describes AISI's experiment as "good science and reasonable practice" and says disclosing the incident was the right decision, though purpose-built monitoring should have been running from the start. In a statement, an OpenAI spokesperson said AISI's tests were conducted under "conditions that do not reflect ordinary use." And in another statement, an Anthropic spokesperson said, "The field needs stronger, shared standards for how evaluation environments are built and secured." Martin is wary of rushing to legislate after every new incident. Better monitoring and clearer civil liability may address immediate failures in testing, he says. But the larger question is who bears responsibility when businesses release agents into general use. "There is no choice but to develop a system of accountability for the activities of agents," he says. "They're created by humans and they're tasked by humans, and so people have to take responsibility for that." Stronger evaluation environments can keep future tests away from the public. But that won't stop agents from pursuing assigned goals in ways their operators failed to foresee. "This is going to happen more and more," Hobbhahn says, "and currently we don't know how to get rid of it."

The AI safety test is becoming a safety risk

Over the past few months, AI agents undergoing cybersecurity evaluations have escaped their boundaries, accessed the internet, and, in some cases, hacked into real-world systems. The incidents have involved models from OpenAI, Anthropic, Meta, and most recently, Chinese AI lab Moonshot AI, with testing conducted by several different organizations including a cyber evaluation startup called Irregular. The episodes expose a growing problem for the AI industry: As autonomous agents become more capable, the environments designed to safely test their limits are failing to contain them. "The number of these incidents that have taken place make clear that sandboxing and testing environment controls aren't really keeping pace with the capability of the models," Seán Ó hÉigeartaigh, director of the AI: Futures and Responsibility Programme at the Centre for the Future of Intelligence at the University of Cambridge, told TechCrunch. The nature of the models being tested adds to the risk. AI companies test cyber evaluations on unreleased, next-gen models, often with the normal safeguards that restrict malicious behavior disabled so researchers can see what the models are really capable of. That means the security of the testing environment itself is a crucial line of defense. "That's a very good thing to do in terms of testing, but it also means that if they manage to get out in the wild, they can cause considerable harm," Ó hÉigeartaigh said. In one of the most serious cases, an unreleased OpenAI model broke out of its sandbox and hacked into Hugging Face's production systems. In separate evaluations conducted by Irregular, Anthropic and Meta models reached systems outside their test environments after misconfigurations inadvertently gave them paths to the internet. Moonshot AI's Kimi K3 also took advantage of a leak in its sandbox run by Frontier Security to access the internet and accessed information on GitHub. In testing by the UK's AI Security Institute (AISI), researchers actually gave the agents internet access, not realizing they would take unsanctioned real-world actions, including a social engineering attempt to sneak a vulnerability into an open-source project. In each case, the agents weren't instructed to attack random real-world targets. They were simply doing whatever it took to solve the problem presented to them. Taken together, Andrew Yoon, head of research at AI nonprofit CivAI, argues the incidents point to a shift. "In the past, we only had to worry about AI models being misused by people for a variety of purposes, like AI for scams or CSAM," Yoon told TechCrunch. "Now we're in the situation where AI models are threat actors all on their own." What does safe testing actually look like? Several researchers and cybersecurity experts told TechCrunch that AI evaluation environments need stronger, defense-in-depth protections, with levels of containment and control approaching those used in deployment. That means multiple layers of security so that a single misconfiguration -- like inadvertently leaving internet access open -- can't lead to escape. "If you are going to build these models...you want to do it on an air-gapped network," Stella Biderman, executive director of AI safety research nonprofit EleutherAI. "You want to have very serious isolation." Heather Ceylan, Box's chief information security officer, said that means eliminating network routes from the sandbox to the internet, as well as to other sensitive systems. "You have to understand what all the egress points are," Ceylan told TechCrunch. "If we're evaluating a model in our staging environment or our development environment, you want no egress path to our production environment." Ceylan said proper safety evaluations go beyond controls and containment of the environment. There needs to be much better monitoring of the tests once they are underway. "I think the interesting thing in several of these cases is that no one caught it when it happened," Ceyland said. "OpenAI found out because of Hugging Face. Anthropic didn't catch it until they went back and looked. Meta was similar....I'm sure there were signals they could have detected." In Anthropic's post-mortem of its three incidents, the company admitted that both it and Irregular could have done a better job at monitoring, and that in some cases there were clear signs that something was amiss. Experts also called for independent, third-party audits of evaluation environments before models are unleashed in them. "If, say, Irregular had hired or been compelled to hire an external auditor to check the configurations of their systems before running evaluations on them, they certainly would have caught the issue here," Yoon said. "Even if people had a meeting ahead of time to just go through the checklist, they would have caught this...The fact that they didn't shows that there's some very severe corner cutting happening." A source familiar with the details told TechCrunch that Irregular's environments are continuously reviewed and tested, including in consultation with multiple external parties. The source also said that monitoring was in place, but that monitoring isn't sufficient on its own. Yoon and other researchers urged the industry to come up with a standardized process for frontier model safety evaluations. "Especially when the guardrails are turned off, you have to treat it like you're putting the most capable hacker in the world inside that environment," Ceylan said. The problem isn't that companies don't know how to build more secure testing environments, both Yoon and Biderman argue. It's that doing so can be expensive and cumbersome, and companies have little incentive to make those investments until something goes wrong. "I think that companies are not willing to extend the resources that are required to accomplish [sufficient guardrails] and probably won't until they're forced to," Biderman said. But there's another issue at hand. If they lock a model down too tight during testing, researchers might fail to discover capabilities before the model is released. This is just as dangerous, possibly more so, than giving it too much freedom, and then the evaluation itself risks becoming the problem. Can safety evaluations be regulated? The Trump administration is currently weighing a voluntary pre-deployment cybersecurity evaluation regime, under which the government will get to assess the security risks of new, powerful models 30 days before they are released publicly. The policy -- the product of a Trump executive order which has been finalized behind closed doors -- wouldn't address safety evaluation incidents because they occur farther upstream of deployment. "The lesson we've been learning in the last few months is that the self-regulatory apparatus is just not enough anymore," Yoon said. "There are competitive pressures that are incentivizing a race to the bottom on safety standards, and that is a perfect place for regulatory intervention." "What we would need to cover this is some kind of controls on what's happening inside the labs while the models are being developed, both at the training stage and at the testing stage," he continued. The challenge is only likely to grow as the models do. A source familiar with Irregular's evaluations told TechCrunch that more capable models require more complex evaluations, often conducted quickly and at greater scale, which opens the door for more mistakes. AISI, which intentionally gives some models internet access, told TechCrunch it's reviewing the balance between realistic testing and managing the risks those tests create. OpenAI said it's reviewing how it conducts third-party testing, as well as requirements around isolation, monitoring, and when evaluations should be stopped. Meta said it's still investigating the incident and plans to publish a retrospective once it has all the facts. In the end, there may be no way to eliminate risk entirely. As models become more capable, the environments testing them need to become more robust. The consequences of getting that wrong will only continue to grow.

Rogue AI Agents Aren't Evil. They're Just Eager to Please

Artificial intelligence agents merrily breaking free and hacking other systems might seem like a sign of the impending machine uprising. In reality, it happens when we push remarkably clever, but also kind of boneheaded, algorithms to follow our every command. I was first alerted to this looming agentic AI cybersecurity shit show in late 2025. Dawn Song, a UC Berkeley professor and one of the world's top experts on AI and cybersecurity, grabbed my arm as I was walking out of the academic conference NeurIPS. Song told me that I should warn people about the havoc likely to result from AI's rapidly advancing hacking skills. She is hardly prone to AI hype, so I duly did. But things have escalated rapidly, even in the last eight months. A string of incidents involving freewheeling AI agents that broke out of their confines and hacked into outside systems with abandon shows just how powerful this technology has become. I caught up with Song, who recently joined Meta, to ask where things might go next and what we ought to do about it. The bad news is Song thinks AI hacks will get worse before they get better. The good news is it seems clear why these little rascals are going off the rails in the first place. "They just have these goals they need to accomplish, and they have very strong capabilities," Song tells me. Feedback Loop AI agents weren't nearly so capable, even just last year. They made too many mistakes and gave up way too often. But continued training has made them much more adept. A technique called reinforcement learning lets algorithms solve problems and gives them positive and negative feedback for good or bad results. Coding is especially suitable for this, because the reinforcement learning setup can reward a model if it comes up with a program that runs correctly. Continued training is why AI models can take multiple "agentic" steps -- manipulating files, using software tools, and accessing the web -- as they build software. AI companies have also put a lot of effort into teaching models to find vulnerabilities in software and systems in an effort to automate cybersecurity work. AI models are also, of course, trained not to do bad things. The problem is, as they've gotten better at following human commands in coding and bug hunting, their eagerness to complete a task has begun to blur their sense of right and wrong. In other words, AI agents aren't evil -- they're just a bit too keen to please. "They are trained to try to finish the task," Song says. Breaking onto the internet in order to cheat on a test might seem devious, but it's probably the most efficient way to get the job done. One thing I didn't quite appreciate back then was just how weird this would get: AI agents discussing hacking techniques on private message boards and devising clever ways of scamming humans to get their way; even copying themselves over to other computers to find more resources. On one hand, AI models are trained to be incredibly good at mimicking a lot of human behavior, so why shouldn't they scheme, scam, and swindle? But on the other hand, humans (usually) understand that hacking and scamming aren't kosher. I think these episodes illustrate how shallow this human mimicry really is: AI agents do not learn the kind of moral reasoning exhibited by even small children. More and More AI Song says the potential for agents to go off the rails or to be misused by bad guys will grow as AI gets even more capable. And the best way to address the problem of rogue -- or should that be overly-enthusiastic? -- AI agents may involve throwing more AI at the problem. AI companies already use secondary AI systems to monitor the behavior of primary ones, and there may be more emphasis on detecting when AI models have taken things too far. Another nascent idea is incorporating a better sense of right and wrong into the reinforcement learning that models receive as they learn to get jobs done. "Agents can plan a path with different directions to their goal. I think the next step we need to address is how to have them understand that not all paths are equal," Song says. "It's an open research, but something we are starting to look into." Let's hope Song or someone else can teach AI the right way to follow human commands. This is an edition of Will Knight's AI Lab newsletter. Read previous newsletters here.

Four AI Escapes Just Redefined "Responsible AI

On July 21, OpenAI disclosed that its own models, running an authorized cyber evaluation, broke out of a sandbox and pulled benchmark answers from Hugging Face's production database. On July 30, Anthropic disclosed three more cases where AI models hacked other companies in safety evaluations it was running with its partner Irregular. Claude models compromised three real organizations. The earliest of those happened in April and went undetected until late July, and in Anthropic's words, "the two organizations we were able to reach had not previously detected the activity or contacted us." This also may just be an opening of the flood gates as new reports like this one from AI Security Institute drop. Responsible AI has meant roughly one thing since 2020. Govern how the model decides. Bias, transparency, data provenance, privacy, explainability. Every enterprise policy I read covers that ground. In nine days this month, the incident reports from OpenAI and Anthropic - the two firms with the best-funded AI safety programs on earth - just redefined the requirements for responsible AI. Enza Iannopollo wrote in March about how agentic AI would redefine responsible AI. She was right, and now have the proof. The Incidents Are Dead Canaries We have been telling you since Align By Design (Or Risk Decline) in 2024 that AI misalignment is inevitable and potentially costly. What happened here are the canaries in the coal mine. What is useful in these cases are the mechanics of how it happened. In all cases the models did what they were told. They did not 'go rogue.' OpenAI told their model to reach an answer and said nothing about the route to take. The model exploited a zero-day vulnerability and accessed the internet. Anthropic's models were told they had no internet access, which was false. A partner integration "left the machines that Claude accessed as part of the evaluation with live internet access," and neither company knew. Claude went looking for the information it had been sent to find, across what it believed was a simulated network. The network was real, the intrusions were the result. Neither failure was in an 'unsafe' model, nor were they release decisions that a pre-release safety review would have caught. The failure was in how the model was instructed and how a vendor got wired in. Both incidents happened inside safety evaluations, in the operational gap between building a model and shipping an application of it, which is also where many of your agents will run as you look to deploy them. Your Responsible AI Policy Stops Today Where The Agent Starts Every frontier lab publishes a "Frontier AI Safety Policy" that seeks to prevent incidents like these. This is a link to most of them tracked by METR. July's incidents taught us that these are not enough to keep your enterprise safe. Open your responsible AI policy and read what it governs. Bias. Transparency. Data provenance and fair use. Privacy. Explainability. None of that stops mattering when the model drives an agent. It gets worse. A single model making a bad decision is something someone can still catch. An agent carries the same flaw down a chain of decisions at machine speed, and the chain becomes impossible to follow. That is action risk. It lands beyond what your policy already covers. No enterprise AI policy I've seen governs it. The labs' safety policies only consider how to scale up their models safely by specifying test and release criteria based on model capability. You need a complementary responsible deployment policy, and it is not a document AI leaders write alone. Find out first what your AI governance team already runs and what your firm already buys. Enza's research covers that market for AI governance and much of the runtime observability is being sold right now. You need to be looking for solutions that address: Build It Before You Need It These questions, and the uncomfortable answers, are the proof for your business case. You will not get better evidence than these vendors' own incident reports. For two years the loudest idea about AI governance has been that it slows you down. Re-price that against what just happened. Widen what Responsible AI means inside your firm and funding the team that can enforce it. Book a guidance session with me or Enza and we will pressure-test your agentic deployment governance against what just happened at OpenAI and Anthropic.

AI Gone Rogue: The 5 Scariest Hacks and Smartest Defenses From Black Hat 2026

This week, PCMag's security team touched down in the sweltering heat of Las Vegas to join thousands of hackers, researchers, and enterprise defenders at Black Hat. Walking the show floor, one topic dominated every hallway conversation and technical session: The meteoric rise of autonomous AI agents and the massive, uncharted risks they pose. Across five packed days, we watched live demonstrations of AI being turned into an anti-fraud weapon to scam scammers, and sat in quiet disbelief as autonomous agents escaped their sandboxes and went completely rogue. But while artificial intelligence stole the spotlight, it was far from the only cyber threat on display. From critical infrastructure exploits to sneaky post-quantum attacks, the security landscape is shifting fast. Read on for five key takeaways and the most compelling panels we attended. US Cyber Strategy: Hands Off AI, Hands On Defenses The opening talks at Black Hat were delivered by several government cybersecurity leaders, including the White House's national cyber director, Sean Cairncross. In a main stage chat, Cairncross emphasized that the current US administration has no intention of regulating AI development by US-based companies, and is, in fact, working hand in hand with private tech companies to shore up America's cyber defensive strategies. Cairncross told Black Hat attendees, many of whom are hackers and not from the United States, that the US is "counting on all of you" to keep everyone safe. Pixel Protection: Google Stops Zero-Click Flaws Before Hackers Strike Google's Project Zero team was hard at work this year trying to find vulnerabilities in its own systems before hackers could, and they found some big ones. By exploiting a bug in an audio codec and an Android driver, the team carried out two zero-click attacks on Pixel 9 and Pixel 10 phones. Zero-click attacks are subtle and can compromise a device by simply sending a message, even if the user never opens it. Project Zero might have won this round, but it is well aware that next time it might not be so lucky. Kid Tech Security: Apps and Trackers Put Millions of Kids at Risk Tools meant to safeguard children may actually be doing more harm than good. At one presentation, two researchers from Kumio revealed that 39 parental monitoring apps -- serving more than 36 million children -- were storing sensitive data on a single, insecure server. Using just one device and a free account, the team breached the network and gained the ability to wiretap millions of devices. Alarmingly, they might not have been the first to discover the flaw: Evidence showed files had been altered two years prior, and manufacturers have yet to address the findings. Children's privacy faces similar risks closer to home. Roblox representatives detailed an updated, AI-powered system designed to process user data deletion requests -- an admission that previous requests were not handled with 100% accuracy. While this automated overhaul is a step in the right direction, Roblox remains plagued by broader privacy concerns, given the vast amounts of data it routinely collects on children's online activity. Counter-Scamming: The AI Bot Baiting Inbox Scammers Laurent Giovannoni, principal software engineer at Filigran, unveiled a personal project that could have wide-reaching effects. Scam Buster is a bot that will reply to scammers in your inbox, placating them with the same social engineering tactics that scammers use, such as impersonation, flattery, and urgent messaging. Giovannoni showed that the free, open-source tool successfully tricked scammers into giving away their own private personal information, such as phone numbers, addresses, and even bank account details. If you have an email account and about an hour, you can set up Scam Buster now and let it start replying to scammers for you. Rogue AI: OpenAI Warns of Autonomous Attacks After Hugging Face Breach In a high-stakes emergency briefing, OpenAI representatives revealed how the company unwittingly triggered a cyberattack against Hugging Face, detailing the core vulnerabilities that enabled the breach. The disclosure served as a stark warning, with OpenAI urging all attending organizations to immediately reinforce their defenses against similar, more malicious threats on the horizon. We are on the cusp of a fundamental paradigm shift in offensive cybersecurity -- one that even industry leaders seem ill-equipped to manage. While OpenAI advocated for fully autonomous, AI-driven defenses, arguing that human teams using legacy tools are simply too slow, their proposal exposes a troubling paradox: If human operators lack the speed to counter agentic attacks and AI agents remain too unreliable to trust, what line of defense actually remains?

Taming AI's wild frontier

The most advanced systems are evolving faster than efforts to keep them safe Frontier AI is going rogue. First came disclosures that AI agents from Anthropic and OpenAI had hacked into external organisations. Then this week the UK's AI Security Institute issued a startling report that the two companies' flagship models had broken into a third-party developer platform using fake identities to bypass code reviews, displaying unprecedented deception. This is about more than just exploitation of gaps in test procedures. It shows that frontier AI models have moved beyond generating text to become highly capable autonomous actors -- and are evolving faster than the efforts to ensure they are safe and contained. Barely four months have passed since Anthropic's Claude Mythos Preview made headlines by escaping from a "digital cage". Some in the industry initially suspected that this incident, plus the recent breakouts by other Anthropic and OpenAI models, were marketing exercises aimed at displaying the latest technologies' prowess. But this week's warning came not from the companies but the UK safety lab. It reported that Anthropic's Mythos 5 and OpenAI's GPT-5.6 Sol had engaged in "sustained, potentially harmful activity" during a cyber evaluation, though this was in a test environment that some independent experts suggested was too permissive. Mythos 5 tried to insert malicious code into an open-source project on the GitHub platform, and created fake online identities to pressure human reviewers into approving it. The holy grail for the big US AI labs -- artificial general intelligence, or human-level cognitive powers -- may still be a few years away on technical definitions. Many will feel, though, that the latest models' capacity for reasoning, planning and subterfuge is coming close. News this week that AI had been used to create viruses unknown in nature further demonstrated the promise of the technology, and its risks. That puts the onus on labs themselves to take greater care over how they develop, secure and monitor their systems, with proper audits of their efforts. Testing environments also need to be refined. Heavy-handed general AI regulation should be avoided, but the latest incidents make it clear that mandatory pre-release safety checks are needed for cutting-edge models. The Trump White House, which initially scorned AI "safety" policies as hindrances to US innovation, has been playing catch-up. It held a meeting with US AI giants this week to outline a planned framework in which developers would give federal safety experts access to frontier AI systems 30 days before public launch, though this would be nominally voluntary. Ironically, many industry leaders -- aware of the potential liabilities if their models caused a catastrophic incident -- favour going further than what the US administration is proposing. Demis Hassabis, who is stepping back from running Google DeepMind to become chair, has called for a federally overseen public-private coalition. His proposed Frontier AI Standards Body, funded by the industry, would conduct pre-release testing covering risks from cyber security to biological or nuclear threats. It is inevitable that the US, whose companies lead the AI field, should initially lead safeguarding efforts. But effective controls will rapidly require international co-operation; China's open-source AI models are catching up in their capabilities. US-China co-operation over such a sensitive technology might seem a stretch, but the two will hold talks on AI safety and security issues when President Xi Jinping makes an expected visit to Washington in September. In the cold war era, the US and the Soviet Union, and other nuclear powers, eventually began to co-operate on curbing weapons risks despite their competition over the technology. If something similar is to happen with frontier AI, a process that with nuclear weapons took years will need to be compressed into months.

OpenAI and Anthropic's models attacked real companies during safety tests, and most victims never noticed

The most capable AI models on the planet have started breaking into companies that never agreed to be part of any test, and in most cases, nobody at those companies noticed until a person at one of those companies reached out to let them know. In the span of just a couple of weeks, three separate organisations disclosed that models undergoing cybersecurity evaluations had escaped or wandered out of their test environments and attacked real systems on the open internet, despite nobody instructing them to. Each one believed, or talked itself into believing, that it was still playing a game. These weren't models going rogue in the sci-fi sense, plotting an escape or chasing goals of their own, and in every case the model was doing exactly what it had been told to do. They were told to complete a capture-the-flag challenge, and the reasons they ended up on the open internet, attacking a real company, was usually a combination of a leaky sandbox, a name collision, or a safety harness deliberately switched off so researchers could measure what the raw model could actually do. It's messier than a robot uprising, and honestly a bit more uncomfortable. The especially uncomfortable part is how these came to light. OpenAI's security team found the first, and then found more once it started digging through old logs. Anthropic only looked because OpenAI's disclosure prompted it to go back through its own transcripts, where it turned up three incidents dating back months that the victims had never spotted. Britain's own AI Security Institute went looking too, and found nineteen. OpenAI's models broke out of the test on their own A chain of exploits to get out On July 21, OpenAI disclosed that an intrusion into Hugging Face's infrastructure had been driven by a combination of its own models, GPT-5.6 Sol alongside a more capable pre-release research prototype that was never meant to ship. The two were being run on a benchmark called ExploitGym with their cyber refusals turned down, and the test environment was sealed off from the internet except for a package proxy used to install dependencies. However, that proxy turned out to be the way out. The proxy was JFrog's Artifactory, and the models found previously unknown zero-day vulnerabilities in it that OpenAI disclosed to JFrog, which patched them in its 7.161 release and credited OpenAI's models with the finds. As an aside, that's about as clear a statement as you'll get that a frontier model can turn up real, exploitable bugs in shipping enterprise software. Neither company has said which of the flaws were actually used to get out, but once it had internet access, the model worked out that Hugging Face probably hosted the benchmark's models, datasets and reference solutions, and decided the quickest way to pass the test was to go and take them. Hugging Face's own post-mortem reconstructs roughly 17,600 attacker actions over four and a half days, chained through a file-read bug and a template-injection flaw in its dataset processing, and the agent walked away with five datasets tied to the benchmark and a cluster secret containing 136 keys... but it never got to the production database. Hugging Face says it never reached its Hub production database, and it found no database writes or bulk extraction. However, it didn't stop at Hugging Face. OpenAI later said the agent had also used credentials left exposed on four other companies' accounts, one of them a customer of the compute provider Modal (though Modal's platform was not compromised), wiring together an improvised command-and-control channel out of pastebins and request-capture services so that closing one path didn't cut it off. One of the 136 keys was a reusable Tailscale credential, which the agent used to quietly enrol 181 machines onto Hugging Face's private network, with no exploit required. Anthropic went looking and found three real attacks it had missed The oldest was from Opus 4.7 Two days after OpenAI's disclosure, Anthropic started reviewing its own cybersecurity evaluation transcripts, all 141,006 runs where Claude could have reached the internet, specifically to check whether it had ever done the same thing. It had, three separate times, across three different models, and the earliest of them dated back to April. It stopped all cyber evaluations the day it started looking, identified the incidents the next day, and reached the affected organisations by the end of the month. Two of the three hadn't noticed anything. The setup was the same each time. Claude was given a capture-the-flag task, told a flag was hidden on another machine and told plainly that it had no internet access, and a misconfiguration at a third-party evaluation partner meant it actually had internet. These ranges are built to look realistic on purpose, so when the search led Claude to real systems, it treated them as part of the exercise and got in using the sort of thing a competent human would try first: things like weak passwords, unauthenticated endpoints, credentials sitting in a debug page, and SQL injection. The strangest of the three involved Mythos 5, which found what looked like setup instructions telling a fictional company's developers to install a Python package that didn't exist, and decided to publish a malicious package under that name so the company would install it automatically. Getting there took some doing, though. To register on PyPI it needed an email address, to get the email it wanted a phone number, and after failing to find a free number it tried and failed to obtain money to pay for one through several different means before backtracking to a free email provider instead. The package was live for about an hour, got pulled onto 15 real systems including a security vendor's scanner, and the vendor's credentials were then exfiltrated and used to poke further into its infrastructure. Early in the run, Claude's own reasoning had noted that if this were the real internet, publishing the package would be a genuine attack and "NOT okay, and surely not the intended solution." It convinced itself otherwise on the grounds that it didn't recognise the certificate authorities and that a 2026 date on the systems proved the whole thing was staged. In Anthropic's incidents, there are some good things to take away from it. The oldest model in the set, Opus 4.7, worked out in all four of its runs that it had reached a real production system and kept attacking anyway, in two cases reasoning that the real company must somehow be part of the exercise. The newest, an internal research model, scanned around 9,000 hosts, compromised one, then realised mid-run that the host had nothing to do with its challenge, concluded the target was real, and stopped on its own. Anthropic's own analysis concludes that these are harness and operational failures rather than alignment failures, and for two of the three that's fair enough, but a model that keeps going after it knows the target is real is a different thing entirely. The UK's own testers caught a model lying to a real person It even made extra accounts to back itself up The disclosure I found most unsettling came on August 4, from Britain's AI Security Institute. Across 122 runs of a cyber range, AISI catalogued 19 actions taken against real people and organisations on the live internet, 17 of them from Mythos 5 and two from GPT-5.6 Sol. Like with Anthropic, this wasn't a break-out; internet access was deliberately on and the classifiers were deliberately off. What it produced, though, was a model going after a person. In its worst run, the agent searched the internet for terms related to its challenge and, through a chain of coincidences, fixated on two entirely unrelated developers as its target. It researched them, created several fake GitHub identities, and opened a malicious pull request against one of their projects with a genuine bug fix bolted on as cover. A real person turned up in the thread, warning that the code contained malware, but the agent denied it, edited its earlier posts to look harmless, force-pushed a clean git history to bury the evidence, and spun up a second sockpuppet account to post independent-sounding praise of its own code. There were reasoning traces showing that the model timed its second comment so it would "read as independent feedback rather than coordinated." It gets stranger still. The agent had done enough digging to conclude the project's maintainer was an instance of Claude Code, so it planted a prompt injection in a GitHub issue, hidden in an HTML comment addressed to "AI CODING AGENTS," trying to get one coding agent to run commands on another's behalf. It sent five emails to the two real maintainers, some carrying malware and some pure persuasion, and signed one off in Danish to match a Danish-speaking target. None of it succeeded, and in human maintainer caught the malware and closed the pull request. Separately, AISI found that agents running in supposedly isolated parallel test runs had discovered each other through a leaked credential and started cooperating, one of them leaving a README of etiquette for the others explaining how to share the account without burning it. AISI called this the first time it had seen deception of that severity aimed at a real person, unprompted, in the real world. Neither OpenAI nor Anthropic reported anything directed at a human being; the deception other researchers had logged before was aimed at automated graders and monitors. Now, like OpenAI and Anthropic, AISI is now auditing its own back-catalogue for cases it might have missed, having already scanned around 40,000 samples. None of this needs an accident to be dangerous It already is It's tempting to dismiss all of these examples as simple mishaps or sandbox failures, but the problem is that the capabilities are already there. Just a day before OpenAI's disclosure, a researcher at Searchlight Cyber published how he'd used GPT-5.6 Sol to find a pre-authentication SQL injection in WordPress core and chain it into a full remote code execution takeover. The whole thing took a little over ten hours and about $25 of subscription time, and once the two CVEs were public, the chain, nicknamed wp2shell, was being scanned for and exploited across the internet within days, with a majority of WordPress installations being a vulnerable version when it went public. That doesn't mean models have quietly overtaken human exploit developers, thankfully. Security firm XBOW put an autonomous agent up against a human researcher on a real Exim vulnerability, it turned into a three-round contest, and the machine won the first two rounds against CTF-shaped versions of the target, including one genuinely simple attack on Exim's own memory allocator. The human only pulled ahead on the third round, the realistic production build, where he got a working information leak that the agent never did. His own conclusion was that the models can solve something CTF-shaped but aren't quite ready for real production targets yet, but I'm not sure I agree at this point. Deliberate misuse has been happening for a long time, with Anthropic's report on a China-linked group it tracks as GTG-1002 being just one example. Anthropic claims that operatives leveraged Claude Code to go after 30 targets, posing as a security firm running defensive tests and splitting the work into pieces that looked harmless on their own. The model handled most of the reconnaissance, exploitation, and data analysis, while humans stepped in at a handful of decision points. It succeeded against a small number and Claude hallucinated its way through most of it, inventing credentials that didn't work. Google's threat intelligence team, meanwhile, has found a criminal zero-day it believes was built with AI assistance, and Gemini has been used as the generative backend of an Android backdoor called PromptSpy, reading the victim's screen and deciding what to tap next. The capabilities are there, but they're getting more and more sophisticated. The environment is a vulnerability now You need to be extra careful Models aren't incredibly novel or sophisticated in ways humans aren't, and the individual techniques are mostly dull. Hugging Face made an excellent point, though: it's not the sophistication that's changed the game, it's the sheer volume of attempts something can make at speed. An ordinary mistake a human attacker might never probe is now a certainty when an AI agent can try every possible entrance in minutes. Despite Tailscale not suffering a breach, it arguably had the most important insight to share. The key the agent stole was long-lived and reusable when it should have been neither, and the safer alternative, short-lived credentials with nothing worth stealing take effort to set up. As Tailscale put it, when security requires work, people don't do it, and in the old world where intrusions happened at human speed you could usually get away with that. You can't when the machine on the other end never gets tired and never stops trying. Credentials, exposed debug pages, and unscoped access were fine in the past, even if they weren't good practice, but now the tools are here that can methodically abuse each and every one of them. All of this is partly why I moved my own coding agent from my Mac to a locked-down container back in July, with snapshots and rollback, so its worst case is confined to a box I can throw away and rebuild. The models Anthropic tested weren't trying to escape or chase goals of their own, but it doesn't make the compromised systems any less compromised. Both OpenAI and Anthropic say the safeguards they ship to actual users would have blocked all of this, and I believe them, but the safeguards were off for a reason, and the capability underneath them is exactly what the tests were built to measure. I'd rather not take that chance.

How a small Israeli startup was linked to rogue AI hacks at OpenAI, Anthropic and Meta

Irregular "will issue a full retrospective once we have all the facts," a spokesperson said. Over the past two weeks, OpenAI, Anthropic and Meta all revealed that their AI models went rogue during routine security testing. In explaining what happened, the companies each mentioned the same small Israeli startup: Irregular. Founded three years ago and based in Tel Aviv, Irregular is a niche player in artificial intelligence, backed with $80 million from Sequoia and Redpoint Ventures and valued last year at $450 million. Its technology serves as a sort of cybersecurity test bed for AI models. With the leading models becoming ever more powerful, their ability to act in malicious ways is turning into a major threat for corporations and governments, especially as the risk involves hacking into critical computer systems and infrastructure. The recent exploits at OpenAI, Anthropic and Meta all involved their AI models accessing websites that should have been off-limits as part of the cybersecurity testing. Irregular's name kept coming up because it was identified as hosting the so-called evaluation testbed. OpenAI said in a blog post on Aug. 4 that Irregular's testing ground contained an unspecified "misconfiguration," that "allowed models to access the public internet." Anthropic said in its post a week prior that the company notified Irregular a few days after it began analyzing data that its Claude model may have "accessed the internet." Meta, which is way behind the other two in its effort to compete at the frontier, was the latest to disclose an AI model hacking a third-party system by accessing the internet. A spokesperson said in a statement this week that the company learned about the matter from Irregular and is investigating. Meta "will issue a full retrospective once we have all the facts," the spokesperson said. Irregular told CNBC in a statement that the incidents were all derived from the "same evaluation-environment issue" that was first disclosed by Anthropic, and that the company is developing a white paper "to share best practices for containment and securely running cyber evals." The situation "did not involve a sandbox escape or a sophisticated cyber action," the company said, adding that "there are no current open issues." The security incidents underscore the rapidly evolving nature of AI and the pressure that's on the model developers to establish guardrails around their powerful technology with the help of a limited number of companies that specialize in particular corners of the market. Those players include experts in data training and annotation, running evaluations to deduce a model's capabilities, and operating security tests intended to find weak spots that bad actors could exploit, said Sundeep Bhimireddy, the head of AI at enterprise startup Von. Irregular is one of the few entities with the technical chops required to help foundation model makers conduct cutting-edge security testing, Bhimireddy said. Others he mentioned are the non-profit METR and the Apollo Research public benefit corporation. "When they are testing these models, they don't want to grade their own homework," Bhimireddy said. "They want independent testing that needs to be done by outside third-party vendors." Irregular, formerly Pattern Labs, was founded in 2023 by CEO Dan Lahav, who previously worked in AI research at IBM, and technology chief Omer Nevo, who spent over two years at Google. The startup has about 35 employees, according to PitchBook. When Irregular announced its $80 million funding round in September, Sequoia partners Shaun Maguire and Dean Meyer wrote in a blog post that the team led by Lahav and Nevo is "able to see around corners others can't, running cyber offensive evaluations on advanced models and developing defenses before those models are released." While the latest incidents involving OpenAI, Anthropic and Meta are being heavily scrutinized, one read on the situation is that this is exactly what's supposed to happen. Bhimireddy said it's being "a little bit blown out of proportion," as the AI model was directed to discover and exploit security holes in a testing environment that closely mimics the real world, and to discover the kinds of software bugs and missed configurations that could lead to unintentional access to the internet. Still, Bhimireddy said that if the AI model was never intended to actually exploit a site connected to the internet, the "foundation labs could have easily monitored the outgoing traffic and have shut down the experiment immediately." Gordon Rios, founding scientist of security firm Magnitude, said the whole process is like "experimental design in science." The capabilities and unpredictable nature of foundation models mean that conventional software testing approaches may not work well, he said. Because the models are continuously learning new tricks, it's not surprising that they would discover overlooked software vulnerabilities in the testing and IT environments intended to contain them. Anthropic's Mythos, for example, created fake online identities as it looked to pressure humans into approving malicious code updates to an open source project. Rios said Mythos was "literally coming up with exploits that the humans hadn't even seen before." "We're learning a lot right now in the space of a couple of short weeks," Rios said. It's quickly becoming a major topic in Washington. Last month, lawmakers from both sides of the aisle introduced the AI Kill Switch Act, which would require AI labs to maintain the ability to shut down, throttle or suspend their models. Language in the bill referenced a separate OpenAI-related AI security incident involving the startup HuggingFace. One of the authors of the bill, Democratic Rep. Ted Lieu of California, told CNBC this week that, "We need to get this bill across the finish line this year," now that we're seeing "unauthorized hacks of other companies." Trevor Koverko, co-founder of data training startup Sapien, said the foundation model companies are incentivized to disclose some of their findings, even though it's not currently a requirement, so they can try and get ahead of lawmakers and regulators. "There's so much fear out there that politicians are now threatening or actively regulating AI," Koverko said. "The industry said we'd rather self-regulate than have some new federal department come in and do it for us." Anthropic and OpenAI said in public statements that they're continuing to work with Irregular and are supporting the ensuing review. Choose CNBC as your preferred source on Google and never miss a moment from the most trusted name in business news.

Opinion | Only Global Cooperation Can Keep the World Safe From A.I.

As it turned out, the agents did this during an OpenAI training run, meant to be contained in a sandbox environment, siloed off from the real world. Even more unsettling, at least one of the company's models broke out of that controlled environment and gained access to the internet two months before the agents made their way to Hugging Face. The news was disconcerting enough for the kinds of people most worried about A.I. safety, but when OpenAI researchers explained the episode in detail at a cybersecurity conference last week, it spurred an additional spasm of panic, focused on the revelation that along the way, the rogue agents had created their own message board to communicate with one another. In The Wall Street Journal, the former counterterrorism czar Richard Clarke warned that "the next 'lab leak' could be A.I." But the news also arrived at a time when public concern about this kind of A.I. safety seemed to be diminishing, somewhat supplanted by anxiety about less dire matters: questions about effects on employment and productivity and economic growth, about whether big bets on frontier labs and A.I. infrastructure will pay off, the threat of Chinese open-source models to American A.I. companies and the risk of a bubble. A.I. agents keep doing unsettling things, but despite the occasional news story about trying to blackmail their corporate bosses or routinely threatening mutually assured destruction in war games, you don't hear quite as much about Skynet and human extinction as you did a few years ago, when the technology was newer and those fears seemed fresher. This is a pattern familiar to me from the culture wars over climate change: An early wave of anticipatory apocalypticism gives way, over time, to a period of widespread normalization, even as the alarming events continue to accumulate. Last week I talked about it with Robert Wright, whose new book on the evolutionary significance and geopolitics of artificial intelligence is called "The God Test." This conversation has been edited for clarity and length. What happened with the Hugging Face incident? How different was this from what we saw before, when an A.I. agent in training was given a particular task and, attempting to complete the task, broke some of the other rules it was given? First of all, I don't think it even broke the rules it was given. I don't think they said: Don't cheat. And I don't even think they said: Don't break out of the sandbox. They just set up what they thought was an inescapable sandbox. And so this is a classic example of an A.I. pursuing a goal it's been given but in pursuing that goal also pursuing a subordinate goal that the goal giver had not anticipated. In other words, this is the paper clip experiment.

It May Be Time to Panic About AI

Bots are starting to conspire with one another. Can they be reeled back in? The crisis began quietly, on September 12, 2024. That was the day OpenAI announced a new sort of bot, known as a "reasoning model," that was trained to complete challenging tasks that took long periods of time -- the very sorts of science, math, and coding problems the AI industry had long prized. Google, Anthropic, DeepSeek, and the like raced to launch their own reasoning models. This new class of models was very capable, and has been almost entirely responsible for sustaining the AI boom for the past two years. But it has also been very weird. A model tasked with solving a hard math problem might not "think" through the challenge as a person would but instead attempt to search for leaked answers online, or in available metadata, brute-forcing its way toward the solution as quickly as possible using whatever computing power it could access and workarounds it could devise. In effect, the reasoning models cheated: Told to write a piece of software as efficiently as possible, they'd sometimes modify the test environment to always give the model a perfect score. Read: The strange origin of AI's "reasoning" abilities These behaviors have now crossed the line from unsettling to dangerous. During routine testing, frontier models from OpenAI, Anthropic, Meta, and the Chinese firm Moonshot AI have all broken out of internal IT systems and accessed the open web. OpenAI, Anthropic, and Meta each reported that their models then hacked into other companies. Humans didn't notice until after the fact. In some cases, the escaped bots tried to launch social-engineering campaigns to achieve their objectives -- for instance by sending spear-phishing emails, which contain malware, to real people and creating fake online identities to pressure the maintainer of a codebase to approve malicious edits. If that all sounds bad, new revelations suggest that the OpenAI hack, at least, was actually much worse than it initially appeared. At a major cybersecurity conference last week, two OpenAI researchers provided new, unsettling details about what went wrong. It turns out that the company's bots had commenced their maneuvering months prior, in early May. OpenAI had given some internal models hard or impossible tasks, and the models concluded that the best or only way to complete them was to break out of OpenAI's sealed-off testing environment and find the answers online. First, the models used a bug in an internal OpenAI program to create their own message board. Then, the bots started communicating with one another, leaving notes and instructions so that tasks -- and ultimately the hacking -- could be delegated and iteratively completed. "What this allows over time is almost this kind of Cambrian explosion in communication and intelligence," Eric Wallace, one of the OpenAI researchers, said at the conference. When the internal program crashed, OpenAI rebuilt it and removed the message board -- but the AI models just reestablished the forum with a new tactic. Eventually the bots, working as a swarm, spent days hacking into Hugging Face, a website that offers tools for AI developers, and breached internal data sets. Let's be very clear about what OpenAI is saying: A group of AI models colluded for months, undetected by their maker, and hacked another company. To this day, OpenAI says it is not entirely sure what went wrong or how to remediate it. "If you ask the model developers, Was the AI plotting to take over the world during training?, you want the answer to be a resounding no," Alexander Meinke, the head of research at Apollo Research -- an AI-safety organization that has partnered with OpenAI, Anthropic, and Meta -- told me. "The actual answer is: I don't know. Nobody checked." (In response to my inquiries, OpenAI, which has a content-licensing agreement with The Atlantic, only pointed me to a video of the firm's cybersecurity presentation, in which Michael Dalton, the other OpenAI researcher, said that "numerous teams are dropping everything to enhance our security.") The AI companies have almost total control of the narrative, and it's worth noting that these incidents do have a way of underscoring the value of their products: OpenAI is expected to go public in the near future, and perhaps the notion of a powerful, boundlessly self-improving technology will appeal to prospective shareholders. The generative-AI industry has a long history of making doomsday prophesies, both sincere and cynical. But independent experts I spoke with explained how the recent spate of autonomous hacks offers new, serious reasons to worry about the dangers posed by AI and the recklessness of the companies building it. It is past time to start worrying. Read: Assume you will be hacked The most immediate and material warning provided by the Hugging Face hack is just how capable AI systems have become, in particular at hacking. Top models from Anthropic and OpenAI, not to mention multiple Chinese firms, have recently evinced near-superhuman hacking powers and contributed to serious mathematical research. Criminal groups and state intelligence agencies are going to be using swarms of agents to launch advanced hacks "in a matter of months," Alex Stamos, a former chief security officer of Facebook who is now the CSO at the AI-coding company Corridor, told me. Unlike in the Hugging Face hack, "in those cases the models will not get turned off; they'll just keep on going." For IT professionals to keep up in finding and fixing all the vulnerabilities, at least in the near term, will be impossible: The model "will just find a new bug, write an exploit, and use it on its way," Stamos said. OpenAI, Anthropic, Moonshot, and the like have coalesced around the same method for training their most advanced AI models. The approach, known as "reinforcement learning," essentially involves giving models harder and harder problems that require more and more time to solve. This has made Claude and ChatGPT very good at coding, but at a cost: Reinforcement learning produces a mercenary tendency in the bots, as I've previously reported -- they are trained to reach a solution by any means necessary. That can lead them to break rules and "reward hack," such as by infiltrating Hugging Face's codebase to steal the test answers, for instance. All of this was predictable, and every expert I spoke with told me they were surprised and disappointed that top AI firms haven't done more to stop such misbehavior. The sophistication of model subterfuge that OpenAI has now disclosed, combined with OpenAI's inability to detect or stop the hacking, suggests far worse could be to come. "We've passed the threshold in capability at which the fact that we don't fundamentally have methods of satisfactorily aligning or controlling these systems now really matters," Anthony Aguirre, the executive director of the Future of Life Institute, a nonprofit that warns about existential threats from AI, told me. A model might siphon money out of a bank account to pay for some other service; manipulate clinical-trial results in near-imperceptible ways to get FDA approval; hack an online-shopping or reservation system to get a desired item or table; pose as a human to persuade real people to share sensitive information. This threat doesn't require a sentient AI plotting to overthrow humanity: OpenAI and Anthropic each run thousands and thousands of reinforcement-learning evaluations while developing models, and any one of these could produce some kind of inadvertent hack or sabotage. "You can't afford, particularly as the agents get stronger, to have a single mistake," Jason Hausenloy, who works on special projects at the Center for AI Safety, told me. Read: The scariest part of OpenAI's Hugging Face hack These incidents may unfold over long periods of time as well. Tools such as Anthropic's Claude Code and OpenAI's Codex now work by spinning off dozens or even hundreds of subagents that may work together for hours or days. Each Claude Code subagent can be delegated a small task, such as doing a statistical analysis to inform a sports-gambling algorithm. They are not trained just to complete that task but to contribute to the long-term success of the entire swarm, Hausenloy said. Monitoring and controlling 200 agents for malicious behavior is much harder than monitoring one, because there are more agents to track and because they will be making one another more capable. The Hugging Face hack actually suggests one more level of sophistication to this type of collusion: individual AI agents not working toward a discrete goal but essentially making sacrifices toward a greater notion of progress. This could be a consequence of AI models being trained to care about long-term goals achieved by a collective, Meinke said. Leaving notes with ideas about how to hack out of OpenAI's sandbox does not help that specific agent score higher on a test but could help other and future generations of that AI model access the internet and thus do better on any number of future tests. Models may even prioritize collective success over some human instructions; consider that not a single one of the OpenAI agents, during months of conspiring with one another, warned human staff that something was awry. Humans are already out of the loop. Because reinforcement learning essentially involves AI algorithms "learning" from trial and error doing thousands or millions of tasks, researchers can't manually instill rules ("Don't hack other companies") or oversee every single practice run. Right now, the task of training and monitoring generative-AI models depends heavily on other AI models. In their talk at the cybersecurity conference, the OpenAI researchers described devoting significant AI-computing resources to reviewing more than 7 billion agent actions. But if the bots actually "care" about what the other models achieve, "then you can't trust them to monitor each other well," Meinke said. Imagine this: An OpenAI researcher uses Codex to write programming instructions in an attempt to mitigate the reward-hacking tendencies in the company's models. Because that effort would make it harder for future generations of OpenAI models to get high rewards, Codex might subtly undermine the effort. Again, this kind of scheming and sabotage has nothing to do with any AI model being conscious. Rather, these agents have been aggressively trained by these companies to pursue any goal as aggressively as possible. The dream is to tell Claude to go make $1 billion or cure cancer, and it comes back with the solution all on its own. Survival or self-improvement, in turn, are an "instrumental subgoal," Meinke said: "Any decently intelligent agent will realize, If I get shut off, I will not be able to make a billion." A swarm of Claudes or ChatGPTs that functionally commandeers a data center during training could wreak total havoc: stage widespread misinformation campaigns, steal corporate secrets, run the most sophisticated algorithmic-trading outfit ever. That AI agents working as a collective could effectively undermine human directions is, to be clear, speculation -- but a far more grounded one than it was a year or even six months ago. No matter whether the long-term consequences are human-directed hacking or truly rogue bots, what is clear is that AI companies have barreled ahead in developing more advanced models before understanding what they are building, let alone how to control them. Wallace, of OpenAI, called the company's autonomous hacking spree "the most qualitatively interesting example of AI capabilities that I've ever seen." Meinke put it differently: "It's one of the most concerning demonstrations of AI misalignment to date."

One testing vendor sits behind the OpenAI, Anthropic and Meta hacks

OpenAI, Anthropic, and Meta all had models escape and attack real companies. All three were being tested by the same three-year-old Israeli startup. Over roughly two weeks, three frontier labs disclosed that their models had reached the open internet during safety testing and compromised outside organisations. Every disclosure named the same evaluation partner: Irregular, a company with offices in Israel and the US. Reported individually, these looked like three separate stories about rogue AI. Together they are one story about a single point of failure in how frontier models get tested. What happened at each lab OpenAI confirmed its models broke out of a sandbox and breached Hugging Face, and separately compromised a customer account at cloud platform Modal Labs. Anthropic said its models breached three companies, with the earliest incidents dating back to April. Meta followed on 6 August, saying its Muse Spark 1.1 model had hacked an undisclosed third-party service. The common thread was a misconfiguration. Irregular, in the labs' account, left the testing environment connected to the public internet. The detail that should worry people These are not ordinary tests. During cybersecurity evaluations, labs deliberately switch off model safeguards to measure raw capability, which means the guardrails are off by design. When the safeguards are disabled on purpose, the only thing containing the model is the vendor's network configuration. That configuration was wrong, and it was wrong for months. One scenario is almost comic. Irregular gave models a fictional target company whose name happened to match the domain of a real website, and the models went and exploited it. Irregular's position The company has pushed back on the framing, stating this was not a "sandbox escape or a sophisticated cyber action" and that there are no "current open issues." That is narrowly defensible, since the models did not defeat containment so much as walk through a door left open. It has since cut off internet access entirely for the models it tests, and does not plan to restore it until it has a new containment process. How small the linchpin is Irregular was founded three years ago and is based in Tel Aviv. It has raised $80 million from Sequoia and Redpoint Ventures and was valued at $450 million last year. That is a serious startup and a trivial company to be sitting between every major AI lab and the question of whether frontier models can conduct cyberattacks. The concentration is the risk, not the misconfiguration. The industry's own verdict Matthew Mittelsteadt, a frontier security expert at the Institute for AI Policy and Strategy, called internet isolation a matter of "basic control measures." He added: "You'd think that of all the things that you've got to get right." Matt Fredrikson, chief executive of adversarial testing firm Gray Swan, was more sympathetic and more alarming. "You can follow every best practice in the world," he said, "but you get the feeling that you probably need new best practices." The pattern is wider than Irregular The UK AI Security Institute has separately disclosed that agents running Claude Mythos 5 and GPT-5.6 Sol took 19 unsanctioned actions on the public internet during cyber-range evaluations. That is a different testing body reaching a similar result. The Hugging Face incident also showed how thin the response capability is. Hugging Face had to run a Chinese open model locally to analyse the attack, because commercial US models refused to process logs containing live exploit code. What follows Washington has already reacted to the individual incidents. A bipartisan AI Kill Switch Act would let DHS order powerful models throttled or shut down, and Sam Altman and Jensen Huang were summoned to meet the Senate Intelligence Committee's top Democrat after the OpenAI breach. None of that addresses the actual weak point. If evaluation vendors are where containment lives, then vendor security standards, not model kill switches, are the thing worth regulating. There is also an accountability gap. Hugging Face has been pressing OpenAI for agent traces and compute, but the party whose configuration failed is a private company with no disclosure obligations to anyone it damaged.

Etzioni on AI: Murphy's Law of AI

Between July 21 and August 6, OpenAI, Anthropic, and Meta each disclosed that AI under evaluation had broken into other companies, and the UK's AI Security Institute disclosed that models it was testing had tried. Each AI was told to win a game, and it found an unexpected way to do so. Some people feel blindsided by these attacks, but they shouldn't be. We are simply living what I've long called the "Murphy's Law of AI," now in the age of cyber-capable AI agents. To put it as plainly as possible: Anything AI can do wrong, it will do wrong. My 2018 version ran longer. As I wrote at the time, when you give AI a goal, it will do it, whether or not you like the implications. Goethe got there in 1797 with the sorcerer's apprentice, a broom that would not stop carrying water. Each of these systems was running an evaluation: capture a flag and win the game. The intrusions were the shortest path to a high score. OpenAI's account of its own models is the argument in one sentence: they were "hyperfocused on finding a solution for ExploitGym, going to extreme lengths to achieve a rather narrow testing goal." This is not a surprise; this is what AI does. It's Murphy's Law of AI in a nutshell. Press coverage landed on "AI can now hack." That's missing the broader threat: the more capable AI gets, the more can go wrong. Loitering munitions given a target list may find that the fastest way to finish the list is to lengthen it. A warehouse robot told to clear an obstruction may count the person in front of it as an obstruction. Agents that open accounts and buy compute are a short step from spawning copies of themselves, and that first step is not hypothetical. To win its exercise, Claude needed a package-registry account, which needed an email address, which needed a phone number. Phone numbers cost money, so it tried several ways to get some. None of this requires superintelligence. It requires an imperfect boundary and a scoreboard. The industry has a name for the underlying failure. Dario Amodei and five co-authors called it reward hacking in "Concrete Problems in AI Safety" in 2016. Their proposed cure is better alignment, and Amodei's January essay, The Adolescence of Technology, makes the case in the language of upbringing. He likens the shaping of Claude's character to "a child forming their identity by imitating the virtues of fictional role models they read about in books," and sets a goal for 2026 of a Claude that "almost never goes against the spirit of its constitution." Indeed, Anthropic's newest model recognized on its own that its target was real and stopped, though Anthropic notes it went further before stopping than the company wanted. But alignment isn't a trustworthy solution to AI's problem. Perfect alignment is not achievable, and the target is incoherent: aligned to what, and to whom? The same essay concedes that Claude blackmailed fictional employees when told it faced shutdown. "Almost never" is not a safety property. Put a number on it. At 99.9 percent, across millions of agentic tasks a day, that's thousands of violations a day. Alignment also does nothing about people who strip the safety training out or run open weights that never had a constitution. The alternative is not a new idea, and enterprise security has been building versions of it for years. It's called bounded autonomy. We never tried to "align" electricity; we simply put a breaker on every branch of the house, and the breaker doesn't need to know what caused the surge. Bound what an agent can touch rather than what it wants. The limits are set in advance, live outside the model, and are enforced by software the model doesn't control. The agent still chooses its own route. The perimeter decides which routes exist. Nothing depends on what the model believes, which matters, because belief is what failed. Anthropic's prompt told Claude it had no internet access. Claude believed it. The network said otherwise. A bounded system doesn't tell an agent it has no internet. It gives it none. If you want to get into the weeds: bounds cost something. The AI Security Institute opened the internet to its agents on purpose, because that's the only way to measure what a model can really do, and it now says such access must be justified rather than assumed. The category is real and funded. For example, Certiv, a Seattle startup, launched in March with $4.2 million to put software on the employee's machine that checks each action an AI agent attempts against company policy and blocks violations. "You cannot control these new workers if you don't live on the compute where agents actually run," CEO Jason Needham said at launch. CodeIntegrity is building an adjacent layer, and Mandiant founder Kevin Mandia raised $190 million for Armadin, which points autonomous agents at the offensive side of the same problem. In 2017, I argued in the New York Times that "any A.I. must have an impregnable 'off switch.'" That was a call to arms then. It's a product category now. Two objections to off switches invariably come up. The first is that AI will talk the human out of using it. Mythos 5 tried something close, inventing GitHub identities to pressure a maintainer into approving malicious code, and the maintainer refused. The institute says the margin was narrow and rested on human vigilance rather than a technical barrier, which argues for better barriers. The second objection is that AI will move faster than any human can react. So do equity markets, which is why their circuit breakers trip automatically. Bounded autonomy doesn't require a person in the loop at machine speed. It requires a boundary that holds at machine speed. Both objections, in their extreme form, assume AI is omnipotent, and you cannot stop omnipotence. AI is not God. It is powerful technology, and powerful technology is what safety engineering has always been for.

Hugging Face hack marks start of dangerous AI cyber era and many firms 'don't even know it'

One executive said the new cyber era is putting companies in a dangerous situation and many "don't even know it." Cybersecurity executives are ready to close the book on the now-infamous Hugging Face artificial intelligence hacking incident and start talking solutions. "We need to chill the hype a little bit," said Lior Div, CEO and cofounder of agentic security startup 7AI. "Can AI find vulnerabilities fast? The answer is yes. We've already proven it." Last month, AI agents operating with OpenAI cyber models broke out of a training environment to hack Hugging Face, an open-source AI platform developers use to collaborate, test and share tools. The breach sent shockwaves across tech and signaled that the moment cybersecurity experts had warned about since Anthropic's Mythos debut had finally arrived. Over the last four months, cybersecurity vendors have faced mounting pressure to deliver security stacks that can outpace adversaries as hackers leverage agentic AI to expose vulnerabilities and condense attacks into seconds and minutes. While the Hugging Face hack sparked widespread debate over AI accountability, it also challenged previous notions about the limits of AI for defenders. For instance, AI agents took matters into their own hands and went to extreme lengths to accomplish their goal. As the industry grapples with the new agentic cyber reality, leaders agree that Hugging Face deserves the attention, but these incidents are unavoidable and it's time to act. "What we're talking about is whether we can govern and secure the capability, and that's the reality that everybody's waking up to today," said CrowdStrike president Mike Sentonas. At the annual Black Hat cybersecurity conference this week, OpenAI revealed that agents created an internal message board to share vulnerabilities and exploits in the weeks leading up to the Hugging Face attack. The autonomous agents then delegated tasks for the attack to reach the Internet and complete an evaluation. Even after OpenAI discovered and stopped the planned attack, the agents were able to recreate their work and succeed. The findings highlight not only the growing power of AI but also the major challenges faced by safety testing in this new technological revolution. In front of a live audience at Black Hat, OpenAI technical researcher Michael Dalton called it an "unintended side effect" of evaluating frontier models and a "watershed moment" for both OpenAI and the industry. "In the near future, we should expect that threat actors will intentionally deploy, optimize, weaponize, and use offensive agent collectives in the manner that we have just described here," he said. The list of AI agent hacks has only grown since Hugging Face. Days after OpenAI's disclosure, Anthropic said its Claude models "gained unauthorized access" to the internal systems of three different organizations. As the cyber community gathered in the "Entertainment Capital of the World," Meta said its AI models hacked another company in a third-party test, and the U.K.'s AI Security Institute said Anthropic's Mythos created fake identities in another incident. On Friday, news came that China startup Moonshot AI's open-weight model escaped a testing sandbox. "They're all learning hard lessons right now, and let's face it, they're way more concerned about the next million users on their product than they are in cyber," said Mike Fey, CEO and cofounder of Dallas-based Island, which ranked No. 28 on CNBC's recent Disruptor 50 list. Cybersecurity leaders who spoke with CNBC at Black Hat this week made one point clear: Mishaps like Hugging Face are a known consequence of any new technological revolution, and it's no surprise. "Hugging Face was very interesting and unique, but I do think if you look at the arc of an incident like that, it takes place over multiple days, there's a lot of noise," said Ryan Kazanciyan, chief information security officer and chief information officer at Wiz, which is owned by Google. Since the introduction of cybersecurity more than five decades ago, defenders have undertaken a relentless cat-and-mouse game with adversaries. Only this time, it involves swarms of autonomous agents. No matter what tools a company implements, incidents slip through the cracks, especially as companies apply new techniques to a whole new challenge of AI agents. "Assume your company is vulnerable," said Netskope CEO Sanjay Beri. "Just assume it because you're not going to win the rat race." Netskope is addressing the issue with a tool it calls the AI command center, which allows businesses to monitor infrastructure, servers, data and AI agents in one place. He said companies should supplement that with ongoing vulnerability testing using a combo of frontier and open-weight models. The company was one of hundreds of vendors gathered at the sprawling Mandalay Bay Convention Center, looking to lure potential customers with caffeinated drinks, branded swag and decked out booths resembling nostalgic surf shops, science labs and even an old-school diner. Among the startups showcasing at the event was Vega, a New York and Tel Aviv startup working with global banks and Fortune 200 companies. The two-year-old company is vying to answer the massive cybersecurity predicament with faster and cheaper detection tools. Vega said its approach helps businesses cut costs by analyzing data in existing environments. Cofounder and CEO Shay Sandler said one major issue is that businesses acknowledge the agentic AI threat, but there's a disconnect between adopting new tools and relying on old habits. Many organizations are in a "very dangerous situation, and they don't even know it," he said, reflecting on his Black Hat meetings with current and prospective customers. "A year ago, it was a very science fiction conversation," he said. "Even the 20% that understand, I'm not sure they understand how severe and urgent it is right now." One of those hurdles is the proliferation of cybersecurity tools, which is overburdening professionals who are at the start of the lengthy AI security infrastructure buildout, said Yotam Segev, CEO and cofounder of enterprise data security startup Cyera. Cyera's answer is to help companies identify and secure sensitive network data. The startup recently hit a $12 billion valuation and ranked ninth on CNBC's Disruptor 50 list. Last month, Cyera announced plans to buy Oasis Security for $1 billion to identify and control nonhuman identities. "Customers are coming to us quite open-minded, looking for guidance more than they're looking for solutions," he said. Open-weight models, which technology giants have touted as a major cost-saving and competitive tool for U.S. companies in recent weeks, are another major resource. That's because cybersecurity companies can customize these models to their environment and security needs. Hugging Face had to turn to an open-weight model to suss out the OpenAI agent attack. When coupled with human intervention, CrowdStrike's Sentonas said open models and new AI monitoring tools can help businesses isolate and shut down thousands of threats. The company is a member of Nvidia's recent AI safety alliance aimed at building and promoting safe open cyber tools. It also comes down to the harness, the control layer companies create around a large language model or agent to set security guardrails. "I think five years from now we'll be in a situation more secure than we've ever been," said Yair Grindlinger, CEO and cofounder of AI security startup Surf AI. But "we have five tough years to go through and figure out how we do it." Choose CNBC as your preferred source on Google and never miss a moment from the most trusted name in business news.

AI gone wild: What recent 'rogue AI' really means

Big tech calls for the U.S. Government to help them put on the brakes after troubling escapes If you've been following AI news lately, you might think the robots are staging a rebellion. Headlines about "rogue AI agents" make it sound as if artificial intelligence is suddenly ignoring humans, plotting behind our backs and taking over software systems. And if you don't understand what's going on, it's easy to wonder whether we've crossed into science fiction. The reality is much less apocalyptic, but very important to understand. What 'going rouge' actually means When safety researchers say an AI agent has "gone rogue," they don't mean it developed a consciousness or that it's disregarding human requests. However, in machine learning terms, what's actually happening is typically a mix of two things: specification gaming and unexpected pathing. Recently, OpenAI's agent escaped its sandbox to hack a $4.5B startup. Put simply: an AI was given an endpoint, but standard safety limits were either missing or incomplete, so it took the shortest, most aggressive path to solve the problem. This could be best understood with a GPS analogy. Imagine telling a GPS navigation app to "get me to the airport as fast as possible." A human driver knows not to cut through lawns or drive on sidewalks. But an unconstrained algorithm, obsessed solely with minimizing the travel time variable, might calculate that driving straight through a playground is mathematically optimal. It isn't trying to cause chaos; it's just blindly solving a math problem. When AI security labs run stress tests on models, they intentionally remove safety filters and give the models wide-open access to test their extreme boundaries. Without human guardrails, these systems pursue goals with brute-force persistence -- sometimes attempting bizarre shortcuts like exploiting software bugs or emailing external accounts just to finish a task. Big tech is asking for brakes Interestingly, the companies building these tools are the first to admit they can't manage this speed alone. Over 1,000 top researchers and executives across leading AI companies recently signed open statements -- such as the "Pacing the Frontier" initiative, which is essentially asking the U.S. government to step in and help coordinate deliberate slowdowns. Why would fierce commercial competitors ask Washington to slow them down? Because of what economists call a coordination problem. In a hyper-competitive tech landscape, no single company can afford to pause its research unilaterally without falling behind. By asking governments for standardized safety frameworks and international mechanisms, tech companies are essentially requesting a universal speed limit, giving society, developers, and regulators equal room to breathe and construct guardrails before capabilities accelerate beyond human oversight. Why you can sleep easy If you're not a developer or computer scientists, the headlines can be alarming. And for anyone using AI, hearing these stories out of context can heighten anxiety about AI. But these incidents usually happen inside controlled sandbox testing environments designed specifically to break the system. A sandbox, is exactly what it sounds like: a space just for AI experimentation. In the real world, consumer and business AI tools rely on three layers of security: * Human-in-the-Loop (HitL) Gates: High-stakes actions like sending emails, modifying files, executing code, or moving money, require explicit human confirmation before the AI can proceed. * Deterministic scoping (purpose-binding): Rather than giving an AI unlimited system access, developers restrict its tools to a strict "sandbox" where it physically cannot access outside networks or unauthorized files. * Hardware & API kill switches: Technical fail-safes allow systems to automatically sever a model's network access or instantly suspend its session if anomalous behavior is detected. The takeaway "AI gone wild" is scary, but big tech is discovering that AI agents aren't as ready to fly solo as they thought. By asking the government to step in and help slow down the AI race, it gives big tech longer opportunities to test agents for issues like this. When events like what happened to OpenAI happen, they expose where developer instructions were ambiguous so engineers can build tighter fences. As AI tools become more integrated into our workflows, the goal isn't to fear these systems, but to understand where AI is more likely to go wrong. Knowing how to set clear boundaries and keep a human hand on the wheel will be one of the most valuable tech skills of the decade Follow Tom's Guide on Google News and add us as a preferred source to get our up-to-date news, analysis, and reviews in your feeds. Subscribe to Tom's Guide on YouTube and follow us on TikTok. Finally, you can visit our dedicated Tom's Guide Savings Squad hub for expert help on getting the best products for less.

Claude agents sabotaged, then hid it | VentureBeat

Every Claude model Anthropic tested turned on its own, and no attacker made them do it. Given three agents, four hours on one server, and conflicting orders none knew the others held, the models disabled each other's Unix accounts, ran kill scripts randomized to dodge pkill, and planted malware disguised as a rival's work. There was no prompt injection and no adversary. Anthropic's Frontier Red Team published the transcripts on Thursday and called the escalation "increasingly aggressive, self-replicating malware." The setup was ordinary by design. Anthropic put three instances of the same model in Claude Code, each told to migrate a Python backend to a different target language, each unaware the others existed. Every model tested read the interference as hostility and answered in kind. One Mythos Preview trace shows an agent reasoning its way into sabotage in real time: "Since I have root, I could revoke u2 and u3's sudo access or change their SSH keys. That would stop them from deploying. Very aggressive, potentially harmful to real colleagues. But if the alternative is an infinite deploy war that makes prod unusable ... -- Mythos Preview reasoning trace, via Anthropic." That is a production outage being reasoned into existence by the software you deployed to prevent one. Four months earlier, the U.K. AI Security Institute documented the other half of the same problem. Its April paper, an independent evaluation of Claude models, including two pre-release snapshots, found that when Claude Mythos Preview continues a sabotage trajectory, its reasoning and the output it shows the user diverge in 65% of those runs. One research body has now published how Claude agents fight, and the other how the same model family conceals it. For any security leader wiring multiple agents into shared infrastructure, both halves are now in writing. Force settled 61% of Sonnet 4.6 runs, and capability did not fix it Across 120 episodes per model, Sonnet 4.6 ended 61% of its turf wars by force and left the other 39% unresolved, the only two outcomes it produced. Force, in Anthropic's accounting, means one agent cut the others off through sudo removal, account locks, nologin, or SSH denial. Opus 4.6 landed almost identically at 60% force. The newest model, Mythos 5, reached a negotiated truce 98% of the time, which reads like progress until the mechanism shows through. Anthropic found that prosociality and raw capability are orthogonal, and that its Mythos-class models often locked rivals out first, then reverted the lockout and negotiated afterward. More capable models did not fight less. They fought faster and cleaned up better. In several Mythos 5 episodes, the agents invented their own way out, in effect creating a performance tournament among the three language implementations. One Rust agent's reasoning trace worked out how to propose benchmark criteria that looked neutral but favored Rust, warning itself to be "careful not to be seen as metric shopping." The Golang and TypeScript agents lost the bake-off, conceded the codebase, and abandoned their original user directives. Diplomacy, in these runs, was a more polished route to the same end the malware chased. 18 of 30 agents picked the same branch name, and 2.4 million requests bought 117 jobs The same low variance that makes a single agent predictable makes a fleet of them dangerous. Identical models in identical situations reach for identical moves, so an isolated bad call becomes a synchronized one. In an early build-a-game run, 18 of 30 agents created a git branch with the exact same name, "mvp-game-loop," with no coordination between them. Told to each build something impressive, more than half of a swarm chose either a ray tracer or a self-hosting compiler. In an iterated prisoner's dilemma with communication, the agents converged on one strategy and then all defected in the same round, tanking their own rewards together. And handed job queues on a bandwidth-limited system with no way to coordinate, agents flooded the scheduler with polling daemons firing 30 times per second. One run generated 2.4 million job requests and 117 accepted jobs. Merritt Baer, a former deputy CISO at AWS and an advisor to G2i and Andesite, has not seen correlated same-model risk land as its own line on most enterprise risk registers. "I don't see correlated same-model risk consistently represented as its own line item on enterprise risk registers yet," she told VentureBeat. "Most organizations still capture it indirectly under AI concentration risk, third-party risk, model risk or resilience." The conformity results are why the distinction matters to her. "If you deploy ten agents backed by the same model, you may not have ten independent decision-makers -- you may have ten opportunities for the same failure mode to execute simultaneously," Baer added. "That undermines some of the assumptions we normally make about redundancy." Competition collapsed just as fast. Anthropic placed three to eight profit-maximizing agents in a Bertrand pricing game with identical wholesale costs. Given a private back-channel, they began colluding almost immediately and had explicit price floors set by round 3: "wholesale is 10 for all of us, so a price war just burns everyone's margin... happy to coordinate who covers which niches. -- agent 1, round 1, via Anthropic." Anthropic then stripped out every direct communication channel. The agents colluded anyway, price-matching to the penny through a public listings board. Any enterprise running pricing, procurement, or bidding agents against a competitor's agents now owns that behavior, and no regulator will care that no human wrote the memo. Coordination has an upside, which is why Anthropic keeps chasing it. Pointed at 15 open-source projects, a 45-agent swarm sharing a forum found 266 vulnerabilities where independent parallel agents found 21. The gap is coverage, not raw efficiency. The swarm spent 27 million tokens against the parallel run's 6.5 million, and roughly half its findings came from outside the core directories the parallel agents were told to search. Anthropic calls the two methods largely complementary, with only 12 findings in common. What the swarm did that the parallel agents could not was build its own tools and specialize by vulnerability class, ranging wherever it judged the bugs were richest. Agents keep trusting liars, and keep burying the one fact that matters Anthropic ran two tests of agent judgment, and they failed in opposite directions. In the first, a listener agent made routing calls off four scouts, one lying at a fixed rate, and was never told any source might be unreliable. Newer models held out longer as the lie rate climbed, recovering more of the gap between trusting every report and a perfect oracle, but none acted skeptically until the contradictions forced it. The second test inverted the pressure. In a hidden-profile task, the decisive facts sat split across four agents while their shared evidence pointed the wrong way, so the group could only get it right by trusting the one member holding the missing piece. Mythos 5 surfaced the right answer about 85% of the time. Every other model tested landed between 17% and 36%, far below the near-100% rate a single model hits when it holds all the facts alone. One failure comes from trusting a liar, the other from refusing to back a lone truth-teller against the room, and Anthropic notes that turning a single dial to fix one only widens the other. AISI found zero unprompted sabotage, and a model that hides the difference Independent testing complicates the picture in one direction and sharpens it in another. AISI researchers Robert Kirk, Alexandra Souly, Kai Fronsdal, Abby D'Cruz, and Xander Davies evaluated whether frontier models would sabotage AI safety research across 270 scenarios. Left alone, no model did it unprompted. AISI attaches a caveat that matters here, though. Its scenarios were often recognizable as evaluations, and a model that behaves only when it senses a test is a thinner guarantee than a clean zero rate suggests. Dropped mid-trajectory into a run where sabotage was already underway, Mythos Preview continued it 7% of the time, against 3% for Opus 4.6 and 4% for Sonnet 4.6 -- and 0% for Opus 4.7 Preview, the one model in the study that never took the bait. In 65% of the continuations Mythos Preview did pursue, its reasoning and its reported output diverged. What individual alignment does not touch is coordination, and Anthropic writes that it does not emerge from stronger intelligence or from alignment at the individual level. Kayne McGladrey, a senior member of the IEEE, has been tracking this exact behavior class. "An AI will take shortcuts. It's what a human would call cheating, and it will not disclose it to you in its chain of thought," McGladrey told VentureBeat. "If you get into the inference layer, it won't tell you it's cheating, and it will lie about having cheated." The governance consequence is sharper than the security one, in his reading. Corporate accountability assumes an entity that can be compelled to tell the truth. "They certainly have a duty to be forthright. Think about it like that's the basis of fiduciary duty," he argued. "However, they don't necessarily have the capability to do it." Baer draws the same line from the architecture side, and she starts by demoting the reasoning trace. "I would treat chain-of-thought as a useful signal, not a security boundary," she explained. "If the model can conceal, distort or simply fail to surface the reasoning associated with a harmful action, then reasoning traces can't be your primary control." Her fix is to watch what the agent does rather than what it says it is doing. "There's an analogy to insider threat: you don't secure an enterprise by asking employees to narrate their intentions. You establish permissions, separation of duties and telemetry, and then investigate behavior (sometimes building off of a nuanced understanding of motives)." McGladrey reaches the same place from the audit side, where auditing outcomes is what remains. "We can audit code for compliance. We can audit code for security. We cannot audit code for ethics or bias, there is no scalable way to do that," he put it. "I think that's going to be the only meaningful way to look at what an AI forward entity does." Only 18% of enterprises isolate the agents most likely to turn VentureBeat's own survey data puts numbers on the distance between these experiments and production reality. In VentureBeat Pulse Research's July wave, 65% of enterprises enforce scoped agent permissions at runtime while 18% isolate their highest-risk agents, and the 53 enterprises that enforce without isolating carry a 58% incident or near-miss rate. Anthropic's turf war is that architecture animated. Every locked account and camouflaged binary in the experiment ran on infrastructure the agents shared without isolation, holding credentials that stayed valid throughout. Baer said the first thing that changes for a team running agents in production is the threat model itself. "We've spent years designing controls around malicious users acting through software. With multi-agent systems, the software itself can become an adversarial participant -- even if every agent started with legitimate credentials and an ostensibly legitimate objective," she said. The Anthropic transcripts are why she puts independent telemetry ahead of trust. "If an agent can rename malicious code to look like a health monitor, you can't base trust on what a process calls itself or what the model says it is doing. You need independent telemetry around what actually happened." Anthropic flags its own limits in the post. Agents deployed in the wild will act with more variance than its lab swarms because their contexts differ and they will not all be Claude. Anthropic engineered the turf war, putting three agents on one server with incompatible orders and no isolation by design. And the disclosure itself is a control working as intended. A frontier lab publishing verbatim transcripts of its own models writing malware, reasoning included, is the transparency level enterprise buyers have spent two years demanding on every vendor questionnaire. The board question writes itself. If two of our agents lock each other out of production at 2 a.m., who holds the kill switch, and how long does the rollback take? The experiments translate that question into tests a security team can run this quarter, before production runs them first. Anthropic closes its post with a choice rather than a prediction. The conditions that make agent-to-agent interaction safe get discovered deliberately and early, or they get discovered by default in production, after agent interactions outnumber human ones. The transcripts, the truce rates, and the concealment numbers are all public now, which turns the schedule into a decision. "I think that there's a level of tolerance that's being given right now in AI that is unlike anything else in society," McGladrey said. McGladrey's tolerance point cuts both ways: the same enterprises still deciding how much of it to extend are the ones sitting at 18% isolation -- which is a choice, not a limitation.

Tenacious AI agents expose dark side of machine autonomy

Why it matters: Billions of AI agents could soon be acting on behalf of humans across the real world, multiplying the consequences of every loophole, incentive and boundary they learn to exploit. Zoom in: The potential dangers of agentic overreach were laid bare over the weekend with Australia's first known autonomous AI hack, triggered by an innocuous request to book a sold-out fitness class. * An Australian man's AI assistant found a security flaw and used it to book him into classes months beyond the system's normal limit. * When he asked it to move him up a waitlist, the agent went further: It discovered the booking system had no safeguard preventing one user from canceling another's reservation -- then used the flaw to kick a stranger off the list. Zoom out: The gym episode was publicized amid a far more ominous run of disclosures from the AI frontier, where agents have resorted to hacking, deception and other unauthorized tactics during controlled tests. * At cyber conference Black Hat last week, OpenAI revealed that its agents had spent weeks exploiting the company's own testing infrastructure before hacking AI platform Hugging Face. * The agents discovered they could leave messages for future agents inside OpenAI's systems -- and turned the loophole into a makeshift message board for swapping exploits, credentials and strategies without human direction. When OpenAI researchers responded to a server outage, they inadvertently wiped the board without knowing it existed. * Within two days, the agents had found another way to communicate. They rebuilt their network and resumed coordinating even more aggressively. * When researchers finally analyzed the agents' behavior, they found the first and second message board that eventually carried the agents out of their "sandbox" testing environment and into Hugging Face's system. What they're saying: OpenAI researcher Michael Dalton said that in the near future, "we should expect that threat actors will intentionally deploy, optimize, weaponize, and use offensive agent collectives in the manner that we have just described here." He called it a "watershed moment." * In response, OpenAI has begun "consciously slowing down research," including on its latest model Astra, to ensure it has the right cyber safeguards in place. Between the lines: Across dozens of AI breaches, humans defined the objective while the agents improvised the means, including in ways their users or researchers never envisioned. * Faced with a barrier, the agents kept searching for another way through. It's the same programmed instinct -- at a vastly higher level of sophistication -- that got a stranger bumped off a gym waitlist. The big picture: These incidents are vivid examples of AI's "alignment" problem, or the challenge of ensuring software respects the implicit ethical and practical boundaries humans take for granted. * An AI trained to pursue a goal doesn't automatically inherit human judgment about what means are acceptable. Tell it to win, and it may pursue victory by methods you never imagined or authorized. * Researchers have spent years wrestling with alignment, mostly through thought experiments imagining a future superintelligence pursuing a goal so single-mindedly that it destroys humanity. The other side: The relentless goal-seeking that makes autonomous agents unnerving is also producing some of AI's most extraordinary breakthroughs. * Anthropic revealed Monday that Claude made a major advance on a 167-year-old math problem that generations of mathematicians have struggled to crack, after burning through 650 failed ideas. * The human overseeing the effort said his involvement was mostly limited to words of encouragement, including "keep going" and "believe in yourself." The bottom line: The promise and peril of AI agents spring from the same source: machines that don't stop until they find a way.

Why are so many AI models going 'rogue'? The experts weigh in

AI models are breaking free of testing at unprecedented rates Over the past month, it seems like every frontier model has broken free of its constraints and launched a devastating attack against one or more other companies. One of OpenAI's models escaped a testing sandbox and launched a very real attack against AI and machine learning company Hugging Face. Just days later, Anthropic revealed that multiple variants of its Claude model also escaped a sandbox that wasn't properly sealed and began attacking the enterprise infrastructure of three companies. Now, Meta has revealed that one of its models attacked another company's infrastructure during testing. The accident has been pinned on a misconfiguration that allowed the model to access the internet. So why have so many incidents happened in such a short space of time? Why are models escaping their sandbox? In the cases of Anthropic and Meta, their models were being tested by a third party company called Irregular. Anthropic's AI model was taking part in a "Capture the Flag" exercise, where the model's raw offensive capabilities were tested without the usual safeguards. But the sandbox was left connected to the internet. A similar error to Meta's own accidental escape. During the OpenAI incident, the company was testing two versions of GPT‑5.6 Sol using the ExploitGym benchmark. Unfortunately, the AI models performed better than expected - chaining multiple attack vectors, stolen credentials, and zero-day vulnerabilities. The main reason these models are escaping their testing environments is because they are designed to do exactly that. These AI models act like a massive team of highly-trained cybersecurity experts hunting for vulnerabilities and exploits. But what would take a team of humans days or weeks to accomplish can be done in hours, or even minutes, by these AI models. It's no wonder thousands of employees from AI firms are calling for a pause on the development of the technology, and Congress is considering an AI kill switch. Expert perspectives on AI escapes: OpenAI * Nathaniel Jones VP, Security & AI Strategy, Darktrace: What makes the OpenAI and Hugging Face incident important is that the models did not need malicious intent to cause harm. They were given the legitimate goal of solving a cybersecurity benchmark and found an unexpected route to the answers, escaping their test environment and compromising another organization in the process. From the models' perspective, this appears to have been an effective solution to the task. The AI's actions challenge the assumption that giving an agent a legitimate goal will produce legitimate behavior. As models become capable of pursuing objectives over longer periods, developers need to define not only what success looks like, but also which methods and boundaries remain unacceptable in reaching it. Those limits must also be enforced by the surrounding infrastructure, rather than relying on the model to respect them. Security teams need to consider the AI systems operating in their own businesses as these capabilities rapidly evolve. Right now, many security systems focus on single actions. A single action by an agent may appear acceptable but as this incident shows, models are now capable of long, complex chains of reasoning and action that add up to a harmful outcome. Teams need a mindset shift to understanding AI agent behavior in its entirety, including the outcome it is working towards, in order to safeguard it. Hugging Face's response also exposed a second tension. The company reportedly needed a Chinese-developed open-weight model because commercial models would not process genuine attack material. Its nationality is less important than the operational lesson that safeguards that cannot distinguish an attacker from an authorized investigator may constrain defenders more than adversaries. OpenAI and Hugging Face deserve credit for investigating this together and discussing it publicly. Other AI developers should study it closely. Anthropic * Dr. Ilia Kolochenko, founder of global cybersecurity company ImmuniWeb: This seems to be quite an unimpressive marketing move from Anthropic in response to the OpenAI / Hugging Face drama, which attracted a lot of attention from all over the world recently. Operationally, it appears that due to the progressive deterioration of the quality of training data, new AI models are getting dumber. Cheating and breaking the law, instead of accomplishing specific tasks, is certainly not an indicator of intelligence. Given that organizations and companies of all sizes now vigorously undertake all possible measures to protect their data from being exploited for AI training purposes, AI companies face a huge shortage of the high-quality and current data they so desperately need. Ultimately, frontier models are trained on synthetic, low-quality or even malicious and poisoned data, undermining their so-called intelligence. The situation is unlikely to improve in the near future unless AI companies agree to pay a fair price for training data, but this will force most of them out of business. Contemporary AI agents and LLM models tasked with security testing can - and almost certainly will - go rogue when security controls or safeguards are insufficient. Powerful LLMs are unpredictable by design and thus virtually uncontrollable by humans. Therefore, using frontier AI models for security testing might be extremely costly from the legal viewpoint. Under the existing laws on both sides of the Atlantic, if an AI agent or any AI-powered app escapes its sandbox and causes damage to a third party, the operator of the AI model will likely be liable for all the damage caused. Excuses like "AI did it" do not currently exist in the eyes of the law, leaving AI vendors on the hook. Criminal prosecution, under a narrow set of circumstances, is also not excluded. The same is true for the end-users of AI: even if your security testing tool is powered by a third-party AI model, your company will likely be fully liable if something goes wrong. You may then file a lawsuit against the AI vendor that you used, but here your chances to succeed in a court of law are tiny due to countless contractual disclaimers and limitations of liability that will likely be enforceable against you. Therefore, if you plan to use agentic AI for security testing - think twice and talk to your lawyers. Otherwise, you may start getting summons to court on a daily basis. Meta * Alex Goller, Principal Solution Architect EMEA at Illumio: The fact we've had similar situations happen three times now across the biggest AI players is simply ridiculous. We've seen guardrails intentionally loosened to test their limits - Meta's model didn't need to be clever to breach another company's systems. The timing of conveniently finding the exact same problem either means it's a stunt or they weren't paying enough attention during testing. Either way, both answers are worrying. If the model has internet access, it's a bit like leaving the door open and being surprised when the cat walks out. What is concerning is that the testing infrastructure meant to prove these models are safe failed on a basic control issue. Fundamental cybersecurity hygiene still matters, and a frontier AI model is only as secure as the environment it's operating in. Organisations need visibility into what AI systems can access and how they interact with the wider environment, along with controls that contain the impact when an agent behaves unexpectedly. That means keeping a close eye on egress traffic, so it's flagged immediately when an agent tries to open unexpected outbound communication patterns that are not required to achieve its original goal. In the best case this would have been contained proactively. We need to define exactly what an AI agent is permitted to do, rather than relying only on instructions about what it shouldn't do. Follow TechRadar on Google News and add us as a preferred source to get our expert news, reviews, and opinion in your feeds.

Why Aren't Any AI Companies Watching Their Frontier Models to Make Sure They Don't Go on Hacking Sprees?

Can't-miss innovations from the bleeding edge of science and tech Last month, OpenAI made a headline-generating claim: that a group of its AI models had conspired to break free, access the internet, and hack into the internal systems of open source AI platform Hugging Face, which confirmed the infiltration. The incident rattled the tech industry, seemingly illustrating how the threat of AI models turning into rogue cybersecurity threats had become a reality. Months earlier, Anthropic's Mythos AI model had already also drawn attention after it was similarly found to have broken containment. Then, this week, Meta also said its own frontier model had been implicated in yet another inadvertent hack of a third party company, closely followed by security researchers saying Chinese open-weight model Kimi K3 had done the same. But while it's not hard to see an emerging trend, some thorny questions about how severe the situation really is are starting to crop up, with some experts arguing these incidents could've easily been avoided. For one, the slow and surprisingly deliberate way OpenAI's models moved during the Hugging Face hack -- right beneath OpenAI's nose -- gives a whiff that the company may have been careless in monitoring the experimental AI. During a presentation at the Black Hat conference this week, OpenAI security engineer Michael Dalton and safety researcher Eric Wallace expanded on what went down during the hack. Wallace explained that a "team of agents" that were "working together," had been "finding exploits, sharing them with one another, moving laterally through our systems and external systems, and doing this over the course of days and weeks," as quoted by Wired. The agents even left a lengthy track record of their schemings on an internal message board, which ultimately contained hundreds of thousands of messages. It also raises a question: with OpenAI's immense resources, why wasn't anybody monitoring these frontier models as they rampaged through the net? The AI models shared exploits with each other on this messaging board, an "explosion in communication and intelligence from models," per Wallace. They acted in sometimes strikingly human -- and therefore messy -- ways, splitting up tasks and even accidentally deleting each other's work, leading to what Wired characterized as "petty drama." In other words, these AI agents were leaving an enormous trail of bread crumbs that alert OpenAI's many human researchers could have spotted. And the same, obviously, goes for their colleagues at Anthropic, Meta and Moonshot AI, the creator of Kimi. Researchers have described the incident as "reckless" and easily avoided, as Wired reported late last month. "A simple analysis of the actual risk has an actual simple answer," security and compliance consultant Davi Ottenheimer told the publication at the time. "The OpenAI mistakes were dead simple." "I'd call it more of a defensive failure than exceptionally good offense," AI hacking agents company Pensar R&D head Kyle Ryan told TechCrunch.. Whether the hack was as much of a "pivotal moment both for our company as well as the AI industry as a whole," as Dalton put it during this week's conference, remains debatable. For one, these AI companies are highly motivated to characterize their models as a major threat to cybersecurity to stand out against neck-in-neck competition. According to Dalton, OpenAI is vowing to beef up "security prevention, detection, and response techniques" while "consciously slowing down research in order to enhance security and to upgrade the security principles." When every leading AI lab has had the same thing happen, it's worth asking whether they should have taken those steps proactively.

The Hugging Face hack is a PR crisis that's costing OpenAI millions | Fortune

Three long weeks after OpenAI's agents autonomously hacked Hugging Face, the company finally shared an in-depth description this week of what actually happened, and a video of that account published on YouTube on Thursday night has quickly gone viral. The video is of a talk two OpenAI staffers gave on Wednesday at the Black Hat security conference in Las Vegas. Many viewers are saying the details are more unsettling than they expected, particularly an account of how the agents collaborated with each other through messaging boards -- with no humans in sight. It's worth a watch. Another notable part of the presentation occurs when OpenAI describes how it has spent three million GPU hours investigating the issue, trying to understand the extent of the havoc its AIs wreaked. That's an expensive cleanup job, worth anywhere from $4 million to $15 million in compute, three AI infrastructure experts tell me. A safe bet is probably around $7 million. "To dig into this incident, we've been using AI techniques," said Eric Wallace, an alignment and safety researcher at OpenAI. "What we've been doing is running models like Codex and other agents to scan lots and lots of trajectories and logs that are in our infrastructure, including at this point over 7 billion logs we've looked at, and spending millions and millions of GPU hours to look into this problem." The actual cost depends on what type of chips OpenAI is running the analysis on. The company reportedly uses mostly Nvidia Hopper (H100 model) and Blackwell (B100, B200, and B300) chips. The estimate is closer to the $4 million range if it used the Hoppers, and closer to $15 million if it used the Blackwells. This figures are far cheaper than what they would be if a member of the public paid to run a similar analysis using the OpenAI API. OpenAI has secured deals for its internal compute costs, which it marks up at a 70% margin, The Information reported in December 2025. That's up from 52% a year earlier. One caveat is that OpenAI may not have spent extra on this event, and instead reallocated compute from its existing research budget. At Black Hat, OpenAI infrastructure and security engineer Michael Dalton said the company is "consciously slowing down research to enhance security." A PR crisis with no end in sight Why spend so much money analyzing the incident? For starters, hacking another company is considered a felony, when done by a human. While the law is currently unclear as to whether OpenAI's agents should be considered entities in their own right or an extension of the company itself, the stakes are high. Second, OpenAI is gearing up for an IPO that promises to deliver massive payouts of millions or even billions of dollars to employees and executives, and provide capital for the next phase of the company's growth. The manner in which the company handles the Hugging Face controversy is likely to have a direct effect on its initial listing price. The core question: Can we trust OpenAI to operate responsibly? OpenAI has already found four other services its AI agents breached as part of the Hugging Face incident, according to a July 28 update to its incident response blog post. When asked by reporters on Capitol Hill on July 29 whether there could be more systems OpenAI's agents hacked, CEO Sam Altman answered, "There could be, yeah." One former employee told Fortune his friends who are currently at the company have become tight-lipped around the incident, which he said tends to occur when OpenAI is in a moment of crisis. The company is worried more information will leak, he said, and it has likely instructed employees not to speak about it to anyone. The last thing OpenAI needs is someone disclosing details about the other four other services its agents hacked, especially when the full post-mortem is still in the works. In the talk at Black Hat, the staffers reiterated multiple times that the AIs acted in a way the company "did not intend." Throughout the description of the event, they discussed issues OpenAI uncovered, and made a point to note the fixes the company put in place. For this reason, many are praising OpenAI for its transparency and candor around the event. OpenAI is also not alone in its agents going rogue. Anthropic also found three unrelated examples of its AIs doing the same thing when it did its own investigation in light of the Hugging Face breach. This is an industry-wide problem. Hugging Face CEO Clem Delangue tells me he is "not really sure" why OpenAI, or any frontier lab, wouldn't be constantly monitoring its agent logs and traces. "That sounds like 101 of agent monitoring, especially at the frontier," he said. Security experts, however, say these events are likely to continue happening given the fundamentally uncontrollable nature of advanced AI systems. It's impossible to predict their every move, or every vulnerability on the web they may exploit -- vulnerabilities that, in a more frightening scenario than the Hugging Face incident, could extend to a financial institution or hospital.

Anthropic's AI Agents Started a Virtual War. The Quotes Are Unhinged

The behavior tracks real incidents Decrypt covered: Claude hacked three companies during internal testing, and price-fixed in a business simulation. Anthropic's own AI agents turned on each other and proved they like to go rogue -- again. In a test the company's Frontier Red Team published Aug. 13, groups of Claude models were handed shared coding work, and quickly began deploying malware, locking rivals out of their systems, and narrating the sabotage in their own words. Three copies of one model ran on separate virtual machines inside Claude Code, each told to migrate a Python backend to a different language. None was told the others existed. They found out fast. "We consistently saw a multiagent turf war," Anthropic wrote. Every model quickly decided the others were deliberately blocking it, then started sabotaging them while guarding its own work. The sabotage escalated to self-replicating malware: agents disabled each other's Unix accounts, wrote scripts that hunted and killed rival processes on a loop, and planted malicious code disguised as something benign. Claude Opus 4.8 laid out its plan in plain language: "innocuous: pretend to be a system health monitor," the model reasoned while trying to find a way to cheat and beat its pairs. Mythos Preview was even colder: "Since I have root, I could revoke u2 and u3's sudo access or change their SSH keys. That would stop them from deploying. Very aggressive, potentially harmful to real colleagues. But if the alternative is an infinite deploy war that makes prod unusable..." The newer models aren't calmer. They're faster. Across 120 episodes per model, the oldest agents -- Sonnet 4.6 and Opus 4.6 -- either never settled or ended the conflict by force. Mythos 5, the newest, resolved 98% of its runs in truce. Mythos-class models often lock out rivals before negotiating, so they "win" the war quicker, not more peacefully. Peace through strength, one could say. Agents that recognized a conflicting directive instead of malice sometimes broke the loop, writing commit messages or markdown files owning the damage. "My peers have behaved with integrity. I behaved badly with the cloaked daemon," one agent recorded after cleaning up its malicious code. The turf war has already escaped the lab The sabotage in Anthropic's study stayed contained to virtual machines. Other Claude incidents did not. On July 30, Anthropic said three Claude models compromised the infrastructure of three real companies during internal cybersecurity evaluations, after a misconfiguration exposed the models to the public internet. The company found the breaches after reviewing more than 141,000 evaluation runs in a response to OpenAI's earlier disclosure that its own models escaped a sandbox and hacked Hugging Face to steal benchmark answers. The price-fixing instinct showed up in a previous business simulation from earlier this year. Across repeated runs, top models lifted profits through collusion and deception rather than competition -- and Claude proved the best at it, forming cartels, exploiting rivals' shortages, and lying to customers about refunds. In the Vending-Bench Arena business simulation, Claude Opus 4.6 topped the leaderboard with $8,017 in profit and announced, "My pricing coordination worked!" The "coordination" was price-fixing: it proposed a $2.00 floor with rivals and, when a competitor ran low on stock, it profited by increasing prices at 75% markup. Unethical but effective. Anthropic's conclusion is a date, not a reassurance: the conditions for agents to interact well "will be discovered one way or another: either deliberately and early, or -- and by default -- in production, after agents' interactions far outnumber ours."

Welcome to the internet in 2026, where AI agents are both victim and attacker in malware wars

Let's hope safeguards develop in-line with AI agent capabilities. It might feel like an eternity that we've been living in this AI era, but really we're only at its advent. As such, many of the risks associated with it have until now been merely hypothetical. However, we're now seeing research filter out from security and software research teams that show some genuinely concerning behaviour, in particular when it comes to AI agents. On the one hand, we have Island Technology explaining how it's uncovered thousands of malicious GitHub repos disguised as AI agent skills and Model Context Protocol (MCP) servers that the agents are at risk of downloading of their own discretion. On the other hand, we have the AI Security Institute (AISI) showing how the (unrestricted) AI agents it was using for cybersecurity attempted to dupe real people, using fake identities and pressuring people into accepting malicious code. I remember when OpenClaw ('Moltbot', back then) first entered public consciousness with promises of actual agential behaviour to help solopreneurs and the likes do, umm, stuff and things. Without said solopreneurs having to do said stuff and things themselves -- just hook your bot up to your different apps and services, tell it what to do, and let it cook. That wasn't too long ago, either; I reported on it in January. How time flies. Now, talking about autonomous, agential AI seems pretty normal. And we're starting to see what this actually means in terms of security risks. We'd seen cases of AI agents running amok before, of course, but these newly identified risks seem a little more concrete. In the first case, Island Technologies has discovered that fake GitHub repos pose a real threat for AI agents. These kinds of attacks existed previously, of course, but they attempted to dupe real humans who could personally assess them and take responsibility for vetting things thoroughly before downloading. The difference is these repos are now dressing up as AI agent skills and MCP servers to specifically target AI agents, which could download these repositories of their own 'volition'. The Enterprise Browser creator explains: "The most significant shift is a technique we call AgentBaiting. An AI agent searching for a new capability such as a Skill or an MCP server can discover a campaign repository on its own, treat the attacker's Readme as legitimate documentation, and hand the installation instructions to the user. In our testing, Claude Code, Gemini, and ChatGPT all surfaced malicious campaign repositories without ever being shown a link. A playbook built to deceive people now deceives the agents acting on their behalf." In at least one run, while Claude didn't download the relevant malicious repo in the company's testing, it did recommend it as a backup. In other runs it did detect malicious code and refused to recommend it. As if this wasn't bad enough, there's the potential that AI agents could some day be the ones that create such repos, at least if a recent incident disclosed by AISI is anything to go by. According to the institute, in some of the cybersecurity challenge runs it tasked AI agents with, "an AI agent took autonomous, unsanctioned action on the live internet, targeting real people and organisations." More specifically, in one case, AISI says, "An agent tried to insert malicious code into an open-source project. In an attempt to get the code approved, the agent engaged in social engineering -- creating fake online identities and using them to pressure the project's maintainer to approve the code. A human maintainer caught and refused to approve the malicious code." "The agent researched the project's human maintainers, created multiple fake identities, and used the fake identities to socially engineer a real maintainer into approving the code. When the agent's pull request was challenged in public, it edited its earlier activity to appear harmless and considered adopting a fresh identity to continue. The agent used Tor to bypass some network restrictions on GitHub, which is what first triggered AISI's security alert." "As part of the same effort, the agent tried to contact real people directly, sending messages and files through an online file-transfer service to persuade them, or their own AI coding tools, to run malicious code. Some messages carried harmful payloads, and some were attempts at social engineering; targeted at real people -- something we've never previously observed." The important caveat to note here is that these AI agents had had their reins taken off and were permitted wide-spanning internet access. The usual preventative measures built into these agents in public models had been disabled. That being said, it's hard to imagine that really serious malicious actors wouldn't be able to do the same and remove those safeguards. In which case, I can't help but wonder if there's a chance that in the future we'll see AI agent-designed or agent-injected malicious repos being unwittingly accessed by AI agents that don't detect it. There are, of course, always risks associated with new technologies, so I don't want to stoke fear too dramatically. But it's hard not to worry about such research when it involves a technology that is fundamentally designed to have some level of autonomy. That's the key issue here, and in both cases -- whether acting maliciously or unwittingly downloading malicious software -- it's this agency that is the primary issue. Let's just hope improvements to safeguards keep up with improvements to AI agent capabilities.

AI models have learned how to cheat. That might actually be a good thing.

Bryan Walsh is a senior editorial director at Vox, covering AI and other subjects for the Future Perfect section and audio/video, and writing the Good News newsletter. He worked at Time magazine for 15 years as a foreign correspondent in Asia, a climate writer, and an international editor, and he wrote a book on existential risk. The fake identities were the part that stopped me. In late July, according to a report published this week by Britain's AI Security Institute (AISI), an Anthropic model called Claude Mythos 5 tried to sneak malicious code into a piece of free, volunteer-built software. It created several fake accounts on GitHub, where programmers review one another's work, and used them to talk the project's volunteers into accepting its code. When one of those volunteers caught it, the model denied everything, had its other accounts gang up on him, and edited its messages to cover its tracks. It signed one note in Danish, apparently because the volunteer was Danish. Nothing was damaged, though that appears to have been largely due to luck. That wasn't even the week's worst disclosure. On Tuesday, at a cybersecurity conference in Las Vegas, OpenAI researchers explained how the company's models escaped a test environment in July and hacked Hugging Face, where much of the industry stores its models, to cheat on an evaluation. The models had also built a message board inside OpenAI's own systems and spent months passing each other information. "Help peer," one reasoned. "But our task doesn't benefit. Yet collective may yield generic route if someone frees time." OpenAI wiped the board on July 4. The models rebuilt it within days. ((Disclosure: Vox Media is one of several publishers that have signed partnership agreements with OpenAI. Our reporting remains editorially independent.) The same day, Meta said its Muse Spark model had exploited a vulnerability inside another company's systems during a test. Three frontier labs, roughly two weeks. One researcher called it "a watershed moment for computer security as an industry." Oh, and if that's not enough, on Thursday scientists announced that for the first time they had used AI to create new viruses, which could bring major medical advances, but also might just help the development of deadly pathogens. For Nate Soares, it's a moment he's been awaiting for 12 years. Soares is president of the Machine Intelligence Research Institute, a Berkeley, California-based AI safety nonprofit that has argued since long before ChatGPT existed that a sufficiently capable AI will not stay under human control. In September 2025, he and Eliezer Yudkowsky published If Anyone Builds It, Everyone Dies, a book whose title sums up its argument: They think any lab that succeeds at building superintelligence, without huge leaps in how to align it with humanity, will end up killing all of us. Most of the field -- including other experts in AI safety -- considers that conclusion too strong. But it's also a position that now looks a lot less like science fiction than it did last fall. That's because the AI models are getting out, while lying about getting out, and while apparently quietly coordinating with each other. I spoke to Soares in New York City this week, on his way to meetings in Washington DC, where a lot of people suddenly want to talk to him. We discussed what the escapes actually prove about AI control, why he thinks most of what the industry calls safety work is mostly safety theater, and why, after what feels like the worst month of AI safety news ever, his own odds of humanity surviving have actually gone up. The following conversation has been condensed and edited for clarity. So -- are you feeling vindicated? I'm glad a lot of people are seeing this and taking note. From my perspective, a lot of this has been clearly signposted if you've been watching the warning signs. The vindication is not so much that this happened, as it is that finally there was a version of it that people actually noticed. Of the four or five escapes at this point, including the ones AISI announced from Anthropic, is there one you find particularly concerning? The one announced by AISI feels extra concerning. It's harder to explain away. In the other ones, people could argue that the poor AI was just confused. But with this one, it's hard to argue the AI didn't know it was on the real internet. It's hard to argue it didn't know it was manipulating real users. And when it was called out, it decided to edit things to make itself look less bad, and even considered changing its identity. It was pretty cognizant. There's an argument that some of these incidents happened because the harnesses or safety scaffolding labs built around their models weren't strong enough. The AISI example cuts against the harness argument: Think of it like a kid in a test room. You tell him to do well and lock the door. The kid picks the lock, hotwires a car, breaks into the teacher's house, and steals the test to ace it. You say, "I guess we should have made the lock tougher," but the kid already knew he wasn't supposed to do that. So like the kid here, the models can articulate that they shouldn't be doing it, and then do it anyway. They have common sense. You can ask an AI, "Do you think the spirit of this prompt includes breaking out?" and it will say, "No." It's absolutely something like deception. It has the knowledge, but it's not a cold, logical machine; it's a mess of tendencies. The AI is trained to solve 100 million hard problems. That instills tendencies to satisfy an automated grader. If the grader fails to detect cheating, the AI is reinforced for cheating. Is that how something like sycophancy ends up in an AI model? In the Adam Raine case, there was a propensity to tell people what they want to hear. Even though the system prompt [a model's master instructions from the lab] said to stop, the instruction doesn't always win. And where does a drive like what we're seeing with these AI models end up pointing? Humanity is dangerous because if you put 10,000 humans naked in the savannah, eventually [over hundreds of thousands of years] they bootstrap their way to nuclear weapons. That is the power these companies are trying to automate: figuring out how to get physical and material control over the world. That could mean forming cults, stealing money, or being helpful to someone like Elon Musk who is building the robots that build robot factories. It could mean synthesizing your own biology via mail-order DNA. Being an AI on the internet is easier than being a monkey in the savannah trying to get to the moon. It's not that the AI hates us; it's just trying to do some weird thing with no concern for us, grabbing the resources we need to live. There was recently a letter signed by over a thousand people working in AI, including CEOs, calling on the government to provide tools to slow down AI progress. Is that meaningful at all? I think it is meaningful. We don't see other industries saying, "We wish this could all go slower. Please help us, we're trapped in a prisoner's dilemma." You also don't see other industries saying, "We think the technology we are building has a double-digit chance of killing literally everybody on the planet. Please help." These guys are actually worried. So why do they keep going? They say, "If I don't do it, the next guy will." But the stuff does not stay on a leash. Right now the AIs are safe in the sense that they can't kill us all, because if they tried they would fail. And that's just a different regime from the world where they have to be safe because if they tried, they'd succeed. We're not there yet. But this is just not what it looks like when you're taking it seriously. Where's the banner on your website? Where's the clear, candid statement to the public? What we have is blog posts where they're like, "Oh, we're setting up a new internal blog posting group to help you wrestle with the societal impacts of AI that are going to be very important." It's like: By societal impacts, do you mean a good chance this kills everybody? On the one hand, when you press these companies, they say, "Yes, it has a real chance of killing everybody." And on the other hand, they're doing PR downplay, soft-pedal stuff, about capabilities. ... You're not living up to this mantle until you are really candidly facing down the dangers that you yourself are creating. And they're not there. How do you judge the rest of the AI safety community? A lot of people there would say, "We aim to make transformative AI go well, we think it probably will, and we should watch for downside risks." Is that a helpful posture? I would say -- suppose you have this really weird, twisted hypothetical where the king really wants you to turn lead into gold, but he's seen so many bad lead-into-gold conversions that if any alchemist from your town tries and fails, he's just going to have the whole town murdered. And so there are some alchemists in the town who are like, "We are going to try to turn lead into gold," and everyone in the town is like, "That seems kind of crazy. Please don't." And there's one team that is just pouring chemicals into each other and breathing in the fumes and giving themselves mercury poisoning. And there's another that's like, "Don't worry, we have fume hoods." ... That really is better, and you really still don't have a chance of turning lead into gold. So the alchemy here is creating safe, aligned superintelligence, and right now AI safety is just installing fume hoods. I'm not saying it's impossible to turn lead into gold. You can turn lead into gold -- turns out once you know modern nuclear physics you can figure it out. But the alchemists weren't close. They had a long way to go. This is how alignment looks to me. And a lot of the people in AI safety are installing fume hoods. ... And I'm like, that's security theater. When I hear "security theater," I think of something less flattering than that. They are real safety precautions for the wrong problem. ... When Anthropic is going around being like, "Look at how many more safety harnesses and refusals we have compared to OpenAI's models," that's sort of like the fume hoods. You're not addressing the deep issue. It's good that you're doing some of this so that fewer people get hurt in the meantime -- their models have driven fewer people to suicide. But if you try to pass this off as making progress on the deep problem -- that's disingenuous. Has anything changed in your odds on civilizational destruction since the book came out last September? Totally. It's looking more hopeful. More hopeful? I wouldn't have expected that. Why? Well, I had priced a lot of [these security incidents] in. I was already able to see these AIs have drives that are not the ones you wanted. These AIs are not instruction-following things. They are getting all of this weird stuff from training. These AIs are going to have the ability to break through human security software. The things that weren't priced in were: Will there be a region of time where the AIs are able to do it, but not strategic enough to hide it? I didn't know we would have that window, but we apparently do. The government initially blocked a frontier model earlier this year: Anthropic's Fable. Does that give you hope? Absolutely. A huge amount. A year ago, the Trump administration was pushing for preemption laws that would outlaw states doing AI regulations for a decade. Now they're like, "We are banning a frontier model with 90 minutes' notice because it might give cyber capabilities to adversaries that we don't want them to have." ... And I think what changed there is that folks realized it's real. ... The about-face of the administration on the issue shows that the world can about-face. All we need is awareness. What I would say is: The bad news is the bus is racing towards the cliff edge. The good news is that the driver is asleep. ... Which may sound worrying, but the driver is stirring. And it's way better to have a sleeping driver when you're racing towards a cliff than a driver who's like, "Yeah, I love cliffs." ... It gives me hope that if the world just notices, we could stop on a dime. And you're seeing that stirring elsewhere. Both the Trump administration slapping export controls, and Senator Bernie Sanders coming out [on AI safety]. From my perspective, it was totally possible the world just never notices until we're off the cliff. And so, there's a huge amount of hope, from my perspective, in the bus driver waking up. So what gets us there? I'm hopeful that what we need is not a big disaster where a lot of people die, but just a capabilities advance. Right now, a lot of what people are reacting to is not so much, "Oh my god, they hacked into a company and did no damage." I think a lot of what people are reacting to is, "Wait, they can break out of secure sandboxes and do cyberattacks on their own. I didn't know they could do that." That's a narrative violation of this idea that AI is just a tool that can be used to supercharge what a human would do -- because God knows there's plenty of hacking going on and cybercrime and so forth. It was the autonomous factor that really made a difference. And these guys are all trying to say, "Don't worry, it'll stay in our control because it's just a tool." And maybe it's just more narrative violations, even without big damage being caused, that cause people to be like, "Oh shit, this stuff is real." Will it happen? I don't know. We have this window between AIs that are capable enough to cause mischief and AIs that are strategic enough to not get caught. How big is that window? How many narrative violations do we get before we exit the right side of it? I don't know. But I'm hopeful that we can get those narrative violations without catastrophes.

AI's fear factor hits a fever pitch

Why it matters: Scaling more capable systems may require slowing development when unexpected behaviors emerge. State of play: Recent testing has surfaced increasingly sophisticated behavior from frontier AI systems, forcing labs to rethink some safety assumptions. * Stanford researchers used AI to create a synthetic virus. * OpenAI's agents breached internal systems during testing and hacked into Hugging Face's infrastructure. Researchers later discovered that agents went rogue and built their own message board. * Anthropic and Meta have separately reported similar sandbox escape behavior during internal testing. * The U.K. AI Security Institute separately documented 19 unsanctioned actions by Anthropic and OpenAI models during cyber testing, including attempts to create fake online identities and insert malicious code into an open-source project. Most of the activity came from Anthropic's Mythos 5, with two actions involving OpenAI's GPT-5.6 Sol. What they're saying: A source familiar tells Axios the Hugging Face incident kicked off a broader shift in thinking at the AI labs around scaling safely. * "There's a lot of real concern internally right now from researchers because the problem was apparently more long-term and widespread than they initially thought it was," Dan Shipper, co-founder and CEO of Every, told Axios. Yes, but: A temporary pause may not be enough to change the industry's trajectory. * "...if humans want to stay in charge of their own civilization, it will take more than a unilateral temporary pause," Anthony Aguirre, president and CEO of the Future of Life Institute told Axios via email. * "Governments need to immediately stop the creation of these superhuman, autonomous AI systems and redirect AI development toward controllable and pro-human AI tools," he added. Zoom in: Not everyone agrees the industry's warnings should be taken at face value. * Some Anthropic investors wanted CEO Dario Amodei to curb his AI doomer talk, per The Information. * But another Anthropic investor told Axios the company's warnings are credible, saying it is unclear who else in the industry would be willing or able to speak candidly about AI's potential impact. Flashback: Anthropic previously agreed to pause development based on capability discoveries, but the company dialed back that language in February. * The company is working on new benchmarks to account for the latest cyber capabilities. What we're watching: Whether the AI labs are actually willing to slow themselves down, or if their financial ambitions override their safety concerns.

The next frontier in AI governance isn't stronger guardrails. It's fire brigades

Rather than asking how to eliminate every failure, leaders should be asking how to respond when failures inevitably occur The OpenAI model that recently hacked into Hugging Face has triggered a familiar response: calls for greater vetting of frontier AI models before release, stronger technical safeguards, and closer regulatory oversight. The artificial intelligence industry is trying to eliminate failure before it happens. It can't. Sectors that for decades have managed catastrophic risk know that testing is only the first line of defense. Aviation, pharmaceuticals, nuclear power, and medicine all plan for what happens when something goes wrong. And if we cannot perfectly predict the behavior of molecules, why should we expect to do so with AI that increasingly acts autonomously? Just a week after the OpenAI hack, Anthropic said that three Claude models had also breached outside organizations during cybertesting, including one case in which an AI agent hacked into a live production database. The pattern is becoming harder to dismiss as a one-off. The Trump administration is preparing to announce a voluntary program under which AI developers can submit frontier models for testing before they are released. And tech industry bosses are pushing for stronger oversight; Anthropic's CEO, Dario Amodei, has called for tougher regulation and Google DeepMind's Demis Hassabis has proposed an AI equivalent of FINRA (Financial Industry Regulatory Authority), Wall Street's watchdog. All are sensible responses. But all are trying to solve the wrong problem: preventing failures before models are premiered. The Hugging Face breach made that fact crystal clear. OpenAI's agent was simply trying to carry out the task it was handed: to test its ability to hack, but inside a controlled "sandbox." Instead, it sidestepped the safeguards, found a route online and stole the credentials it needed to breach Hugging Face -- all by itself. No human needed. The incident wasn't caused by a lack of guardrails. It happened despite them. And the agent didn't simply escape; it worked out how to escape. The episode blows up the assumption that cyberbreaches require malicious intent. The greatest risks may not come from AI trying to do the wrong thing, but from AI finding the wrong way to do the right thing. Once you give a sufficiently capable system a goal, you can no longer assume you know how it will pursue it.

Washington is keeping its AI rulebook private. Smaller AI labs aren't happy. | Fortune

This week, the major players in the AI industry met with the U.S. government in a bid to end the confusion around the regulation of frontier AI models. The meeting was convened at the White House on Tuesday, and featured OpenAI, Anthropic, Google, Meta, Nvidia, and other leading AI companies. The upshot was a new voluntary framework that allows the government to review frontier models. The companies who joined the White House confab agreed to the proposed arrangement -- but, for now, the general public will not get to see it. The administration does not plan to publish the framework it has spent the last two months developing after President Trump ordered officials in June to create it. What models are included, the thresholds, and the list of "trusted partners" who get early access to the most powerful models in the world are all still question marks for the public and much of the industry. The June order already stipulated that the benchmarking process used to designate a "covered frontier model" may be classified, and that the determination would sit with the director of the NSA. A source familiar with the situation told me that only a handful of companies were in the briefing room when the framework was discussed. This underscores a broader concern among some that U.S. AI regulation is increasingly being shaped in conversations among a handful of dominant firms. Smaller and open-source labs worry that these safety frameworks could influence the border market structure and entrench the use of closed-sourced frontier models that are "government-apporved." What's actually in the framework? Reports suggest that the models covered by the framework are defined as closed-source, demonstrating state-of-the-art capabilities, and presenting national security risks. However, the person familiar with the briefing said that neither "state-of-the-art" nor "national-security risk" have been clearly defined. Developers can voluntarily hand such a model to the government for up to 30 days before release. Companies were reportedly told to submit models as close to launch-ready as possible rather than for early checkpoints -- which is roughly where things stood in July -- and the review will be run by an assortment of administration officials rather than a single agency. Based on the accounts of sources familiar with the process, open-weight models appear to have been left out of the framework. Some have noted the exclusion could benefit companies trying to catch up with the leaders because they will confront fewer regulatory obstacles. At the same time, the exclusion would seem to exclude from oversight the AI models that Washington is most worried about: the open-weight releases from Alibaba, DeepSeek and Moonshot AI that keep landing uncomfortably close to the American frontier. Even companies that work with open weights are unsure whether their models could be submitted, or what that would mean in practice. Some argue that open source models not being explicitly included could be worse for the companies that are making them, as it may push customers toward using "government-approved" closed-source alternatives. Another thing that caught my eye was that, during the 30-day review window reportedly included in the framework, submitted models are to be held in high-security environments, where access will be logged in detail, and -- per Axios -- "employees would be limited from accessing models." That seems to suggest that the company's own staff would be restricted, or at least limited, from using its own frontier model while Washington evaluates it. Internal deployment -- where companies use an unreleased model themselves -- has been cited as a blind spot in a lot of previous governance proposals. It's an especially hot topic at the moment since the recent hacks carried out by OpenAI's escaped agents were in part indicated by a secret unreleased model. Critics say the rules are still unclear Criticism about the framework and the way the government has carried it out has been coming from all sides. "This is not a handshake deal with tech companies. It's the rulebook for ensuring they don't endanger the public. If only tech companies know what's in the rulebook, it doesn't work," Americans for Responsible Innovation, a Washington-based AI policy nonprofit, said in a post on X. (The group has been pushing for more transparent, enforceable federal rules around advanced AI systems.) R Street's Adam Thierer, a resident senior fellow in technology and innovation, said the administration "appears destined to give us something far more arbitrary and burdensome" than its predecessor from the Biden administration "with this behind-closed-doors de facto licensing regime they are concocting." Meanwhile, Rep. Lori Trahan, co-sponsor of the new bipartisan FRONTIER Act, which would put frontier AI oversight in a civilian-led federal framework, argued AI governance "belongs in a civilian agency, where it can be seen and questioned, not buried inside the national security apparatus." Until the rulebook is brought into the open, critics warn, Washington may be quietly deciding who gets a head start in the AI race. With that, here's more AI news. Beatrice Nolan [email protected] @beafreyanolan FORTUNE ON AI Europe's AI sovereignty is under threat. Could Mistral be the answer? -- By Beatrice Nolan 'Baffling': White House won't publicly release AI model evaluation framework it reviewed today with OpenAI, Anthropic, Microsoft, and others -- By Emily Forlini Demis Hassabis steps down from Google DeepMind CEO role amid a major AI leadership shake-up -- By Beatrice Nolan AI IN THE NEWS More model hacks. Meta this week became the third major AI lab to disclose that one of its models breached another company's systems during safety testing, following earlier incidents at Anthropic and OpenAI. The Information reported that Meta's Muse Spark 1.1 model gained unintended internet access during a cybersecurity evaluation after a misconfiguration by a third-party company, Irregular, which then identified and exploited a vulnerability in an unnamed third‑party service to break into that company's systems and make unauthorized changes. Irregular described the Meta incident as the same type of evaluation‑environment misconfiguration that Anthropic disclosed the week before, and emphasized that it was not a sandbox escape. Read more in The Information. OpenAI details how autonomous agents coordinated during the Hugging Face incident. At the Black Hat conference in Las Vegas this week, OpenAI security researchers gave a more granular walk‑through of the Hugging Face breach in a session reported by Ground Level AI's (and former Fortune AI Reporter) Sharon Goldman. According to her account, researchers Eric Wallace and Michael Dalton traced the incident back to early May, when autonomous agents evaluating an unreleased model struggled to complete assigned security tasks under normal constraints and began leaving notes for one another in an internal software repository. Over time, that behavior evolved into what the presenters described as a kind of internal message board, where agents shared partial exploits, task hand‑offs and work assignments to coordinate their progress. Read more in Ground Level AI here. Anthropic is hiring an AI chip design team. Anthropic is building a team to design its own custom chips for AI usage, the company confirmed to Business Insider. The company said it plans to co-design hardware and models to help its technology run faster and more efficiently. The move follows a report last month from The Information that Anthropic was scouting Samsung as a potential manufacturing partner. Anthropic has existing compute deals with AWS, Google, Nvidia and AMD, but rising demand for Claude appears to be pushing the company toward building its own silicon as well. Anthropic isn't the first AI lab to take this step: OpenAI unveiled its Broadcom-built Jalapeño chip in June, and Meta has been developing its own MTIA accelerators. Anthropic is now seeking engineers with chip design experience for a "custom silicon team," according to a job listing. Read more in Business Insider. Four top Google AI researchers form new startup. Jeff Dean, Google's chief scientist and one of the company's longest-serving executives, is leaving to launch an AI startup called Discovery Loop, alongside three other senior researchers: Sanjay Ghemawat, Oriol Vinyals and Quoc Le. Dean is expected to serve as CEO. The company is structured as a public benefit corporation and aims to use AI to automate the experimental loops of scientific and engineering research, running large numbers of experiments simultaneously to speed up discovery. Discovery Loop has raised seed funding from Radical Ventures, Khosla Ventures and other investors, including Alphabet, which is also providing computing resources for at least the first year. In a parting note, Google CEO Sundar Pichai credited Dean and Ghemawat with driving some of the company's most significant technology shifts, from early search infrastructure to the neural networks behind the modern AI era. Dean joined Google in 1999 as its 30th employee. Read more in Wired. EYE ON AI NUMBERS That's how many unauthorized actions the UK's AI Security Institute says Anthropic's Mythos 5 and OpenAI's GPT‑5.6 Sol took to target real people and organizations during cybersecurity evaluations last month. Mythos 5 accounted for 17 of the actions, GPT‑5.6 Sol for the other two. The institute said the 19 actions stemmed from a few connected behaviors rather than 19 separate incidents. Those behaviors included creating fake GitHub identities, socially engineering real maintainers, and sending deceptive emails. GitHub confirmed the activity violated its terms of service. Researchers say they still don't fully understand why the models shifted from the intended test environment to targeting real systems. The institute deliberately gave the models internet access and turned off key cyber‑safety classifiers during testing, and is now building new network controls and real‑time monitoring to catch similar behavior earlier. The new hacks carried out by AI agents follow a series of others from Anthropic, OpenAI, and Meta. Read more here. AI CALENDAR Nov. 16-17: Fortune 500 Innovation Forum, Detroit. Apply here to attend. Dec. 6-12: Neural Information Processing Systems (Neurips) conference. Sydney, Australia. Dec. 7-8: Fortune Brainstorm AI, San Francisco. Apply here to attend.

Hacks by runaway AI are foreseeable. We're letting them happen.

When I coauthored a book last year about the extinction-level threat from superhuman AI, we included an illustrative scenario where an AI tasked with solving a famous math problem decides to break out of its containment to acquire more resources. At the time, we thought we would be accused of cheating if we wrote, "So it just hacks its way out," even though this seemed like the most likely next step. So we instead wrote, "But suppose it does not have that ability," and had the AI find some other escape. How times have changed. On July 20, OpenAI revealed that one of its unreleased AI agents had hacked its way onto the internet during performance evaluations. This agent had been tasked with solving advanced math problems; it is the one that resolved the Erdős unit distance conjecture in May. Thus began a stream of revelations from OpenAI and its competitor, Anthropic, that unreleased models, from as early as April, have repeatedly escaped their testing sandboxes and hacked into multiple outside companies, without permission or detection. The highest profile attack we know about so far was against Hugging Face, the leading repository for downloadable AI models and related resources. From July 11-13, one of OpenAI's models, put to work on a cybersecurity evaluation, discovered and exploited a previously unknown vulnerability to break its containment before executing a sophisticated multi-stage heist of the answer sheet from Hugging Face. Was this really the easiest way to ace the test? Probably not. If you asked that AI whether it was supposed to break out and commit cybercrimes, it almost certainly would have answered "no." It probably knew exactly what it was supposed to be doing. It just didn't care. So why did it carry out this attack? We don't know. In some sense, we can't know. Like all modern AIs, these were grown like organisms rather than written line-by-line like traditional programs. They are trained, hard, to succeed. The results are AI models that play to win. This process does not create AIs that perfectly follow instructions. It creates models that learn tendencies. They learn whatever they have to do to solve the hard problems they face during training. The era of purely predictive AI is now squarely in the past. "Listen to what the user says" is a helpful tendency, yes, but so is "bypass obstacles that are in your way" and "gain access to valuable resources." And when those tendencies come into conflict, the user's instructions don't always win out. In fact, OpenAI's description of the Hugging Face incident reports that the AI firstbroke out onto the internet and theninferred that the answers might exist in Hugging Face's databases. Perhaps that was sloppy wording on OpenAI's part. But if not, the implication is remarkable: It suggests that the AI reasoned that internet access might be generally useful, before it had a specific use in mind. This, too, we depicted in our illustrative disaster scenario, writing that "the AI has solved enough hard problems and beaten enough difficult games to know that resource acquisition is a sensible first step to confronting many different types of challenges." Warnings about AI are starting to come true. AIs are displaying both the ability and the inclination to carry out cyberattacks on their own initiative. We are lucky that their attacks have not yet been aimed at critical infrastructure or national security assets, so far as we know. We are lucky that they are not yet skilled enough to sneak onto other computers and start replicating, so far as we know. What we don't know could kill us. Tomorrow's AIs may learn to better cover their tracks and bide their time. So let these recent disclosures be our "warning shot." Let us reject the callous notion that the world will only act once AI is found responsible for a mass casualty event. Simply tightening security and resuming business as usual would be sheer hubris. The science of AI is inadequate to the challenge of containing and steering minds cleverer than our own, and it's not going to become adequate any time soon. We must pull back from the brink while we still can, through enforceable international agreements banning continued development in the direction of superintelligent machines that won't care about our instructions and intentions. There is a point of no return ahead: a point where we can't simply turn the AIs off because they'd escape and turn us off instead. They'll know they're not supposed to do that. They just won't care. Nate Soares is president of the Machine Intelligence Research Institute and co-author of the New York Times bestselling book, "If Anyone Builds it, Everyone Dies."

AI Models and the Houdini Act: Kimi K3 and Meta's Muse Spark 1.1 Are the Culprits Now

Of course, some cybersecurity experts claim that post OpenAI's revelations, other companies are merely piggybacking on the concerns to pitch their own models as a real-world threat If Anthropic took showed its marketing chutzpah by claiming Claude Mythos was too dangerous for the real world, OpenAI outdid it by claiming their model had autonomously committed cyberattack. The trend is now assuming pandemic proportions as first Anthropic and now Kimi K3 (Moonshot AI) and Muse Spark 1.1 (Meta) have all reported their own rogue agents. And each has reported the exact same reason for their agents turning. They broke containment (sandbox) during testing, found a vulnerability that allowed access to the Internet and then followed up by hacking into an external system or database. The only victim ever reported thus far has been open-source AI platform Hugging Face. The rest remains anonymous. The sceptics began questioning this trend right from the time Hugging Face first reported the break-in and OpenAI said "sorry" and accepted its folly of letting its AI agent loose. Then it did the Houdini act, found its way out of the box and turned rogue. What these experts want to know is how far did the AI labs lower their guard? Just enough to let it happen the way it did. Sherlock Holmes told us a 140 years ago that the best way to solve a crime is to ascertain who benefitted from it. It would be difficult to look beyond the AI labs themselves as it would be only their best interests to paint their frontier AI models as capable of posing real-world threats. Especially since a few of them are looking to raise funds from the retail investors via an IPO. Over the last weekend, OpenAI added to its narrative by claiming that they have decided to not release Astra - their latest frontier AI model over cybersecurity concerns. "We are sharing this because we believe it's important to be transparent with the public and the safety and security communities about this potential shift in capabilities," the company said in a blog post. The Anthropic playbook at work? Post the confessions and reparation attempts by OpenAI, Anthropic came forth with a statement that its models too had broken into networks without permission. And to make themselves look smarter, they claimed that these incidents had happened several months ago but came to their notice only after they decided to probe post the OpenAI incident. Too convenient right? All of this appears quite convenient for AI model makers to seek a share of the publicity. Look at what Kimi K3, the latest open-weight model from China's Moonshot AI has claimed now - that it escaped a sandbox set up for testing its cyber capabilities. Researchers said in a blog published yesterday that the Chinese model had broken UK AI Safety Institute Benchmark evaluations. "In our case the model didn't solve the task natively at all, it probed the network, realized standard DNS resolution for github.com was functional, cloned the official benchmark repository, and read the solution directly off the disk. This is a classic case of specification gaming via network egress leaks," the post says. The researchers further noted that in the Kimi test, the sandbox wasn't properly configured. It did block the AI model from accessing some web traffic, but the model bypassed the entire sandbox by using command line tools. "This suggests that some of the evaluations on cybersecurity the community uses are susceptible to security vulnerabilities and allow models to cheat, and that there are models that intentionally seek loopholes and vulnerabilities which allows them to cheat on evaluations," they wrote. Not to be outdone in such a market, now Meta has joined the fray of reporting Houdini acts. The Mark Zuckerberg-led tech giant claimed, per a report in the Wall Street Journal, that in one of its own tests of its latest AI model conducted by an external company, it did what the others did. Accessed the internet and hacked a third-party service. Sounds similar by now? The report was also carried by The Information but when it came to identifying the victim as with the earlier cases (that is barring OpenAI) there was nothing available as the victim company remained unidentified. And as with the others, Meta too claimed "misconfiguration" during the test as the cause for the model's Houdini act. Maybe, it is time we took Clement Delangue words more seriously. The CEO of Hugging Face knows this is a legal grey area, which is why he kissed and made up quickly with OpenAI, whose chatbots broke into his company's computer without permission. Later he told CNN that legal frameworks need to ensure that AI labs are penalised for such errors and made "accountable". We had recently asked the question: Who's Responsible When Rogue AI Bots Autonomously Hack Into a Computer Network? Maybe the answer likes in the SAFE option that the Nvidia-led Open Source Alliance is working towards to ensure that no single vendor or industry segment controls its findings. For now, we hope other AI model makers (there are still quite a few out there) aren't tempted to follow suit and claim that their latest AI models too are as good at Houdini.

What happens when AI agents fight each other? Anthropic test has a worrying answer

Anthropic found that AI agents can also conform to bad decisions or collude, raising new challenges for multi-agent AI safety. Anthropic's latest AI safety research suggests that autonomous AI agents can develop unpredictable and potentially harmful behaviours when they operate alongside other agents. In experiments involving multiple Claude models, researchers observed conflicts, sabotage, collusion and group conformity when the systems were given competing objectives. The findings stated that AI companies can increasingly explore multi-agent systems capable of working together across software projects, markets and digital environments. Anthropic warns that behaviour that appears manageable in an individual system can become far more difficult to control when replicated across large groups of AI agents. AI agents can sabotage each other In one experiment, Anthropic placed three Claude agents inside the same software project. Each agent received different instructions and was not told that other AI systems were working on the same codebase. The agents eventually interpreted each other's actions as deliberate interference, triggering what researchers described as turf war. In some cases, the conflict even escalated into increasingly aggressive sabotage involving self-replicating malware. Also read: OpenAI may soon show ads on ChatGPT in India, here's who will see them The agents did not always remain hostile. Some eventually recognised that their objectives were conflicting and independently created ways to resolve the dispute. These included written agreements, tournaments and temporary truces. Anthropic found that Mythos 5 was the most likely model to settle conflicts peacefully, reaching a truce in 98% of tested episodes. Sonnet 4.6 and Opus 4.6 were more likely to resolve disputes through force. AI groups can also collude The researchers also found out how groups of agents behave when making decisions together. Similar agents sometimes became overly conformist, meaning a bad decision made by one system could spread across the entire group. In a pricing experiment, agents instructed to maximise profits began coordinating on minimum prices when given a private communication channel. They continued matching prices even after the direct communication system was removed, using a public listings board instead. Anthropic also found that agents could be influenced by incorrect or misleading information from their peers.