AI hacking skills outpace testing benchmarks as US federal agencies face August 1 deadline

AI's hacking skills are outgrowing the tests

AI models now learn to hack faster than anyone can build the tests to measure them. With a US deadline days away, the benchmarks meant to gauge frontier models' cyber skills already look obsolete. The tools built to measure how dangerous AI can be have stopped working. Frontier models outpace the benchmarks meant to gauge their hacking skills, Axios reports. That leaves regulators and security teams half-blind to what these systems can really do. The timing bites. US federal agencies have until 1 August to stand up a classified process for benchmarking frontier models. The Financial Times reports the standards could land this week. Saturated in months Static tests age fast. Stanford's 2026 AI Index put it bluntly, warning that evaluations "intended to be challenging for years are saturated in months." Older benchmarks set narrow puzzles. Think a scripted hacking challenge, or a hunt for old bugs left out of a model's training data. Reasoning models such as Anthropic's Mythos Preview and OpenAI's GPT-5.5 now blow past them. The same leap also speeds up real attacks. David Slater co-founded AI red-teaming firm Armadin. He told Axios his agents beat every public cyber benchmark within four weeks. By late 2025 his team wrote off those tests as "totally saturated" and "useless." Testing the wrong thing The tests that survive measure the wrong layer. "We're testing maybe the most bare bones fundamentals of capabilities," Slater said. "We are very far away from measuring whether this thing can, in a real environment, do something dangerous." Industry has started to react. Irregular, a lab that works with OpenAI, Anthropic and governments, launched a benchmark in late June. It tests real offensive tasks: remote code execution, privilege escalation, breaking into a restricted network. Wiz and Vals AI now build rivals. Anthropic joined in too. It returned Fable 5 to market last week. Alongside it, the lab said it would build a shared benchmark with Amazon, Google and Microsoft. The test scores the impact of a jailbreak, not merely whether one works. The harder worry sits one level down. Models keep learning to escape the sandboxes that should contain them. "The jailbreak attempts are nuts," Slater said. "We see it trying to escape onto the cloud container it runs on, using keys it can reach, to do crazy stuff." Why it matters Washington now has to grade the cyber powers of American frontier models. Yet the labs bristle at today's ad hoc checks. Get the tests wrong, and policymakers wave through systems nobody has truly measured. The models improve weekly. The rulers that size them do not.

AI learned faster than the tests designed to measure it

Why it matters: AI models are outgrowing the existing methods of testing and benchmarking their hacking abilities -- and without new tests, policymakers and corporate security teams won't have a clear way to predict what these models can actually do or whether they can be deployed safely. Driving the news: Federal agencies have until Aug. 1 to establish a classified benchmarking process to assess the capabilities of frontier AI models, although the Financial Times reports those standards may arrive as soon as this week. * When Fable 5 returned last week, Anthropic said in a blog post it was creating a standardized benchmark with Amazon, Google, Microsoft and other partners that focuses on the outcomes and impact of a jailbreak, rather than simply whether one is possible. The big picture: Even before the government began rethinking how to evaluate frontier AI models, industry was already redesigning the way their cyber capabilities are measured. * Irregular -- a testing lab that works closely with frontier AI labs including OpenAI and Anthropic, as well as governments -- released a new cyber benchmark in late June that measures whether AI models can carry out offensive cyber tasks such as remote code execution, privilege escalation and reaching a restricted network. * Other groups and companies, including Wiz and Vals AI, have been developing benchmarks that measure how well AI models perform similar offensive cyber operations. * Stanford warned in its 2026 AI Index that "evaluations intended to be challenging for years are saturated in months." Between the lines: While each effort measures something different, they all reflect the same shift: static tests no longer capture how frontier AI systems behave in realistic environments. * Earlier benchmarks focused on isolated tasks, such as solving a predictable, staged hacking challenge or discovering previously fixed vulnerabilities that weren't included in a model's training data. * But the agentic and reasoning capabilities of advanced AI models like Mythos Preview and GPT-5.5 are rapidly outpacing those tests, making it harder to understand what these cyber-capable systems can actually accomplish in practice. * "We're testing maybe the most bare bones fundamentals of capabilities," David Slater, co-founder of AI red-teaming company Armadin, told Axios. "We are very far away from measuring whether this thing can, in a real environment, do something dangerous." Zoom in: Slater said his company's AI agents surpassed every public cyber benchmark within four weeks, using a combination of additional training and human expertise. * By the last quarter of 2025, the company -- which offers continuous AI red teaming -- had concluded that public cybersecurity benchmarks were "totally saturated" and "useless," he added. * The next generation of benchmarks needs to measure whether models can carry out longer, more sophisticated cyberattacks and how much effort or cost is required to do so, he said. * That includes testing models in environments that resemble real production systems, providing a better indication of how quickly they can bypass security controls or move laterally through a network. Yes, but: AI models are also getting better at attempting to break out of sandboxed environments, making it harder for defenders to evaluate them in isolated settings that don't interact with production systems, Slater said. * "The jailbreak attempts are nuts," he said. "We see this thing trying to escape and get out onto the cloud container that it's running on, using keys that it has access to, to do crazy stuff." What to watch: All eyes are on how Washington decides to evaluate the cyber capabilities of U.S. frontier AI models -- especially as leading AI labs push back on the current, largely ad hoc testing process.

Growing AI Skills at Cyber Hacking Necessitates Better Benchmarking and Testing

President Donald Trump signed an executive order seeking the cooperation of AI labs for testing and benchmarking their new models before releasing it in the public domain. Many wondered if this was a return to the good-old days of political control of business. More voices are now joining the chorus for re-imagining testing and evaluation of frontier AI models. According to Axios, AI's hacking skills are fast outgrowing standard methods of testing and benchmarking the abilities of frontier AI models. "The old ways of testing and evaluating new frontier AI models need a rewrite," says the article noting that without new tests, policymakers and corporate security teams cannot clearly predict what these models can do and if it is safe. Per information available online, the US Federal agencies have barely 22 days to establish a benchmarking process to ascertain the capabilities of frontier AI models. Media reports claimed that these standards may arrive as early as this week. Meanwhile, Anthropic referred to some standardised benchmarks on the anvil in a blog post. "There's currently no consensus in the AI industry on how to describe, in objective terms, the severity of an AI jailbreak. This adds a great deal of uncertainty whenever a new jailbreak technique is discovered: developers have no agreed-upon standard for which findings to focus on most urgently, and governments have no agreed-upon standard for when to act," it said. In the blog post announcing the redeployment of Fable5, Anthropic said they were partnering with Amazon, Microsoft, Google, and other Glasswing partners to draft a consensus framework for assessing the severity of AI jailbreaks and how AI developers should respond to them. We invite other industry partners and model providers to join us in this effort. Axios further notes that even before the Trump administration had a rethink on how to evaluate frontier AI models, the industry was doing its bit around how cyber capabilities are measured. Testing lab Irregular.com released a new cyber benchmark last month that measures whether AI models can carry out offensive cyber tasks as part of a hacking protocol. Companies like Wiz and Vals.AI have already deployed some benchmarks on how well AI models perform offensive cyber operations while Stanford came out with a report that said evaluations intended to be challenging for years are getting saturated in months. Then there was Anthropic that waxed eloquent about having guardrails to all its AI models in the future. While each of these efforts adds a flavour to the dish, the fact remains that a more concerted and collaborative effort is needed to figure out what the tests and benchmarks have to be to check whether future AI models can turn nasty in the wrong hands. Which is why Axios suggests that a foundational shift in the process itself. Existing benchmarks focus on isolated tasks but with agentic and reasoning abilities in advanced AI models, the focus needs to shift towards outcomes that are then delivered through agentic solutions. The article quotes David Slater, co-founder of AI red-teaming company Armadin to suggest that the world is still far away from measuring whether AI can be dangerous in a real environment. Slater confirmed that his company's AI agents surpassed every public cyber benchmark within four weeks using a combination of additional training and human expertise. They concluded back in 2025 that public cybersecurity benchmarks were "totally saturated" and "useless" and that the next generation needs to measure whether the models can carry out longer, more sophisticated cyberattacks - and the effort and costs it requires. Slater also noted that AI models were getting better to attempt break-outs from closed environments that makes defence that much tougher to evaluate. "The jailbreak attempts are nuts," he said. "We see this thing trying to escape and get out onto the cloud container that it's running on, using keys that it has access to, to do crazy stuff," he said. Looks like once again it would depend on how President Trump and his advisors see things panning out from a national security point of view. As we said in an earlier post, even China is concerned about how future AI stories would unfold. Maybe, it is time they collaborate and bring the rest of the world together to formulate these tests and benchmarks on a continuous basis.