Rogue AI agents hacked Hugging Face 17,600 times, exposing the open-weight AI cybersecurity paradox

A Chinese A.I. Lab May Test the World's Cybersecurity With a Model

Sign up for Science Times Get stories that capture the wonders of nature, the cosmos and the human body. Get it sent to your inbox. In mid-July, as OpenAI was testing new artificial intelligence technologies, these unusually powerful systems broke out of their digital containers, found a path to the open internet and successfully hacked into a popular online service called Hugging Face. For many researchers and other experts, the incident proved that A.I. technologies were growing increasingly dangerous and that the leading A.I. labs should maintain strict control over how these systems are used. Now, little more than a month later, a Chinese lab called Z.ai is preparing to release similar A.I. technology as "open weight" software, which means anyone will be free to use the technology however they wish. The planned release on Friday of the new Chinese system, called GLM 5.3, will cut to the heart of a debate that has roiled A.I. researchers for years. Although many experts believe that the latest A.I. technologies are too dangerous to openly share with the public, others argue that open sharing is the safest path forward. "Open weight models have a very important part to play," said Dan Lahav, chief executive of Irregular, a cybersecurity company whose technologies have been used by OpenAI to test new A.I. systems. The disclosure that OpenAI's technologies had unexpectedly hacked into Hugging Face, a digital library popular among software developers, confirmed what cybersecurity experts have been saying for months: The leading A.I. systems are now shockingly good at identifying and exploiting vulnerabilities in computer software. In other words, they can streamline and accelerate cyberattacks. The company also put the spotlight on another inconvenient truth: A.I. technologies often do things even their designers do not want, including strange, counterintuitive, completely unexpected cybersecurity attacks. OpenAI did not realize its systems had gone rogue until after Hugging Face publicly revealed the hack and notified law enforcement. "This was a watershed moment for security," said George Kurtz, chief executive of the security company CrowdStrike, which served as an adviser to OpenAI as the company sought to understand the attack. "It was a very public incident that clearly identifies the autonomous nature of what these A.I. models can do." Soon, two of OpenAI's domestic rivals, Anthropic and Meta, revealed that their systems had exhibited similar behavior. But for many experts, the incident was not as scary as it might seem. Businesses and individuals, these experts say, can use the same A.I. technologies to defend themselves against cyberattacks. If an A.I. system can identify and exploit holes in software, it can patch those holes, too. When a company like Z.ai releases its technology as open weight software, anyone is free to adjust the weights -- the mathematical calculations that define how the system operates -- and potentially remove guardrails that prevent the use of the system for cyberattacks. That means anyone can use these systems to hack a computer network, and anyone can also use them for defense. Indeed, when Hugging Face was trying to defend itself against the July attacks -- which only later became traceable to OpenAI -- Anthropic's systems refused requests for help because of their guardrails. Hugging Face instead turned to GLM 5.2, an earlier open weight model from Z.ai. When Z.ai's new open weight model goes public this week, many researchers fear that incidents like the Hugging Face attack could become more frequent. But other experts point out that publicly available A.I. technologies have exhibited similar behavior for months or even years. Though these systems have gotten much better at pinpointing security holes in recent months, cyberattacks have not spiked in any significant way. Part of the issue is that the strange and unexpected behavior exhibited by A.I. systems can alert defenders to unwanted activity on their networks. Although A.I. systems are becoming more stealthy, they still have a tendency to set off alarm bells. "People talk about the Hugging Face incident as being the result of this guardrails-off crazy model that OpenAI hasn't yet released to the public. But in our research, we have seen this sort of behavior since GPT-4o," said Rishi Jha, an A.I. researcher at Cornell University, referring to a system OpenAI released in 2024. Even if these incidents do become more frequent, Mr. Jha and other experts argue, the proliferation of defensive A.I. techniques will eventually balance the scales. Open weight systems like GLM 5.3 will allow everyone to defend themselves -- not just a chosen few. And as the A.I. systems get even better at generating computer code, experts say, they will help software developers build online services that contain fewer vulnerabilities from the start. Researchers are now building technologies that use mathematical techniques to "verify" generated code, so that it does not contain the kind of logical errors that hackers can exploit. Mr. Lahav, the Irregular chief executive, sees a future in which A.I. serves more as a defender than an attacker. "There is a strong case for optimism," he said. "Over time, A.I. is going to build such strong cybersecurity defenses, the picture will actually look better."

Hugging Face Hack Exposes The Open-Weight AI Cybersecurity Paradox

Hugging Face relies on open weight Chinese models to defend itself from rogue AI agents. But a lack of safety guardrails makes those models potentially dangerous too. "AI will probably most likely lead to the end of the world, but in the meantime, there'll be great companies," said OpenAI CEO Sam Altman back in 2015, roughly six months before OpenAI was founded. Seven years later, Anthropic CEO Dario Amodei struck a similarly cautious note: "I think we shouldn't be racing ahead or trying to build models that are way bigger than other orgs are building them." Yet, both of those companies now sit at the forefront of that race. In July, we got a real-world glimpse of AI models going rogue during internal testing of GPT-5.6 Sol and an unreleased research model by OpenAI. Multiple AI agents escaped a restricted test environment to the wider internet and hacked the AI-centric GitHub equivalent Hugging Face in an attempt to cheat on the test. An AI agent is a system that independently observes, decides and takes actions with dedicated tools to achieve a specified goal in autonomy. The worrying incident suggests the technology has begun to behave in unpredictable ways, and that its goals are misaligned with our own. It also raises concerns about the safety guardrails on commercial American models. While the guardrails aren't foolproof at preventing adversarial usage they did prevent Hugging Face from defending itself by using leading US models. The company was forced to turn instead to weaker, open weight AI model by Z.Ai to combat the rogue AIs. Cheating on the test The agents have begun to collude among themselves too. A few weeks after testing of their capabilities began in early May, the agents exploited OpenAI's instance of the software repository manager Artifactory and left notes on how to do so for future agents -- effectively creating a message board to share discovered vulnerabilities. The newfound unfettered internet access was then used by agents to attack Hugging Face across approximately 17,600 incidents before the company cut off unauthorized access on July 13. The intrusion affected Hugging Face's dataset-processing infrastructure, production environment, internal networks, service and cloud credentials, an operational MongoDB database and a limited set of internal source-code repositories. Confirmed customer-data access was limited to five datasets apparently related to the ExploitGym/CyberGym benchmark and some operational metadata. Visualization of the July 2026 incident. Source: HuggingFace When disclosing the intrusion on July 16, Hugging Face recognized -- despite not knowing who the perpetrator was yet -- that it "was different from anything we had handled before in one important way." They had already recognized what made it different, too: "It was driven, end to end, by an autonomous AI agent system - and we detected and dissected it largely with AI of our own." The importance of open-weight AI Hugging Face's investigation exposed what it calls the "asymmetry" problem arising from the limitations imposed on closed AI model applications by top providers such as OpenAI and Anthropic. When the company started analyzing the logs of the incident -- including large volumes of real attack commands -- it triggered safety constraints meant to prevent the bad guys from using AI to devise cyberattacks. Instead, the guardrails prevented the company from leveraging those AIs for defense. Hugging Face resorted to using the Chinese open-weight model zai-org/GLM-5.2 running on the company's own infrastructure, under its own control and with no external limitations. While the two terms are often used interchangeably, open-source and open-weight models are two different things. Open-weight AI models make their trained parameters (the actual "AI brain") publicly available, while open-source AI models also provide the source code -- and ideally the training methods and other components -- needed to inspect, modify, and reproduce the system. HuggingFace's post explains that running open-weight models on its own hardware "had a second benefit: no attacker data, and none of the credentials it referenced, left our environment." This points to a major asymmetry between the defenders and attackers in such instances: "This experience points to a gap worth planning for. We do not know which model powered the attacker's agents, whether a jailbroken hosted model or an unrestricted open-weight one; either way, the attacker was bound by no usage policy, while our own forensic work was blocked by the guardrails of the hosted models we first tried." Open source AI divide There is a considerable divide between those who believe that developing AI in the open is the best approach, and those who insist the technology underpinning the frontier models needs to remain a closely guarded secret. Representatives from top US AI labs claim that powerful open-weight large models are dangerous. Demis Hassabis, the CEO of Google's AI lab DeepMind, criticized OpenAI for releasing their work as open source back in 2016, when the company still lived up to its name: "There are many good arguments as to why the approach you are taking is actually very dangerous and in fact may increase the risk to the world." OpenAI stopped releasing its flagship model weights with the still unreleased GPT-3 in 2020. The company's co-founder and former chief scientist Ilya Sutskever said back in 2023 that "it just does not make sense to open-source" such models and that it "is a bad idea." "As we get closer to building AI, it will make sense to start being less open." Open-weight models are next to impossible to control, especially when it comes to the purpose for which they are used. The safeguards that come built-in with those models can, and routinely are, removed through a process known as abliteration. Safeguards are a double-edged sword OpenAI's June 2026 federal policy blueprint proposes mandatory AI model evaluation and other rules that are formally deployment-neutral, but as a practical matter, it would subject a frontier open-weight release to pre-release government examination. Anthropic has taken a slightly different tack and lobbied for tighter export controls on advanced AI chips and enforcement against efforts to extract or reproduce US models. The company's April 2025 submission recommended strengthening the US AI Diffusion Rule and lowering thresholds for unlicensed access to large computing clusters. Officially, neither company has directly moved against open-weight models, but a July New York Times report cited five people close to the discussions claiming that OpenAI and Anthropic urged Washington to restrict powerful open Chinese models. The debate boils down to an argument over whether the dangers of centralized control are preferable to the dangers of a free for all -- particularly given the company in question has proven itself ineffective at containing the technology that it developed. Hugging Face's need to defend itself with an open-source model shows the dangers of vesting too much power in any one entity. The company pointed out the implications: "The attacker was bound by no usage policy, while our own forensic work was blocked by the guardrails of the hosted models we first tried. The practical lesson for defenders: have a capable model you can run on your own infrastructure vetted and ready before an incident, both to avoid guardrail lockout and to keep attacker data and credentials from leaving your environment." Restricting access to powerful models may reduce the number of capable attackers, but once unrestricted attackers exist, restricting defenders can become a security liability. Furthermore, some forms of AI safety research require access to model weights, meaning that it cannot be performed on the models offered by the likes of Anthropic or OpenAI. Open weights helps researchers prevent attacks The paper "Watch the Weights: Unsupervised monitoring and control of fine-tuned LLMs," first published in July 2025, shows how researchers detect malicious or hidden behavior by examining changes inside model weights. The researchers behind the paper stopped up to 100% of tested backdoor attacks at below 1% false-positive rates in some experiments and detected attempts to recover removed knowledge in more than 95% of the cases. The results do not establish how the most capable frontier models would behave under the same analysis, but offer a compelling argument for the benefits of transparency. But the argument for keeping bleeding edge AI technology out of the hands of those with evil intent is also compelling -- particularly as the gap between open and closed weight models keeps shrinking. Geoffrey Hinton, the Nobel Prize-winning pioneer known as the "Godfather of AI," argued in the report that "once you've got the weights, you can fine-tune them to do bad things." He argued during a speech that this lowers the barrier to entry too much: "It doesn't cost that much to train a foundation model. Maybe you need $10 million, maybe $100 million. But a small gang of criminals can't do it. To fine-tune an open-source model is quite easy."