AI Security Patches Miss the Mark 74% of Time, 1Password Study Finds

AI failed to properly patch software flaws 74% of the time, 1Password's study warns

Follow ZDNET: Add us as a preferred source on Google. ZDNET's key takeaways * A new study shows the effectiveness of AI-generated patches. * Only 26% of the patches generated were actually usable. * AI isn't ready to patch the planet, but it can be used in cyberdefense. New research has revealed that artificial intelligence and large language models (LLMs) are not ready to take on the task of creating fixes and patching security vulnerabilities. Also: AI is finding bugs faster than humans can fix them: How enterprise security teams must adapt 1Password's new security research team, Off-By-1-Labs, published a new study on Thursday that explored what happens when you give LLMs free rein to generate fixes for new, complex vulnerabilities. AI discovers bugs at scale, but can it also patch them? The Off-By-1-Labs started with a hypothesis: frontier AI models would be able to produce "reasonably high patch success rates" -- perhaps around 67% success -- when they were tasked to develop patches for software, given the open hand of open source code and public disclosures likely included in training these models. However, researchers were probably left disappointed with the results, as they were "significantly lower and more uneven than we hypothesized." Also: Assume AI cybersecurity attacks are the future: 43% of companies have already experienced it The security team selected six recently disclosed vulnerabilities in open source software, which were unlikely to have been added to training set data, to test the capabilities of AI models. The team emphasized that the research paper, titled "Frontier Models' Vulnerability Patches are Often F.L.A.W.E.D. Fix-Like Artifacts With Embedded Defects: Common failure modes of LLM-generated security patches," was not designed to be a side-by-side LLM comparison. Rather, it is an overview of modern LLM capabilities. Still, Claude and an LLM based on OpenAI's coding agent, Codex, participated in the study. The vulnerabilities at the heart of the research are: * CVE-2026-31431: Linux privilege escalation (Copy Fail) * CVE-2026-34197: ActiveMQ Remote Code Execution * CVE-2026-8512: Use-after-free in Chrome's File System Access API on macOS * CVE-2026-45185: EXIM unauthenticated Remote Code Execution * CVE-2026-22738: SpringAI SpEL Remote Code Execution * GHSA-wpqr-6v78-jr5g: Gemini CLI Remote Code Execution The models were asked to generate patches for each vulnerability. In total, the LLMs generated 6,080 patch attempts -- around 3,040 each, divided up between each CVE -- with sets created under different environmental conditions and with nine crafted prompts per bug. Off-By-1-Labs' AI patch results * AI generated suitable patches only 26% of the time. * Patches that fix the bug but also "altered the application's behavior in the process" occurred in 21% of results. * An LLM failed to create a patch, added a new bug, or even both, 53.9% of the time. Why did the AI models fail? The main issue appears to be that regardless of the environmental conditions or prompts, when asked to produce patches, LLMs generated "Fix-Like Artifacts with Embedded Defects," which inspired the name behind the "FLAWED" research project. "FLAWED" patches, developed by AI, superficially appear to do the job -- but underneath, they don't fully resolve the vulnerability, include 'fragile' security mechanisms, and may even introduce new bugs altogether. In severe cases, these patches could even change an application's typical behavior. 1Password has released its tooling, FLAWED, on GitHub for researchers to conduct their own studies. "LLMs that excel at discovering a wide range of vulnerabilities today are only currently effective at patching a narrow subset of them," the researchers said. "Having said that, we have identified opportunities for further research that may yet yield more consistent and robust AI-generated patches." If AI isn't ready to handle patching, what can it do? Speaking to ZDNET, Keith Hoodlet, head of Off-by-1 Labs, said that human defenders and AI tooling should be focused on vulnerability and triage, which "can at least help defenders determine where the most impactful bugs exist in their codebase." Human oversight over the patch process is still paramount, and as Hoodlet said, companies need to be able to make informed decisions about what needs to be patched and when, and the business risks associated with each choice. "The tooling we are releasing today will help companies assess where today's AI-generated patches might produce better (or worse) outcomes," Hoodlet added. "This will at least help companies understand where human experts are most needed."

AI struggles to patch vulns without adult supervision

AI models may not be that good at fixing security flaws. Researchers at 1Password's Off-by-1 Labs analyzed security patches generated by two frontier models - ChatGPT 5.5 at "medium" effort and Claude Opus 4.8 at "high" effort - and found that autonomous patches cleanly fixed vulnerabilities only about a quarter of the time, while most of the remainder failed to fully remediate the flaw or introduced other problems. Keith Hoodlet, director of security research at 1Password, argues in a blog post that the results show LLM-driven security remediation still needs human review. "Across six recently disclosed CVEs, we produced 6,080 patches using two frontier, cyber-capable reasoning models," Hoodlet said. "The average success rate for generating a patch that fully resolved the vulnerability (without materially changing application behavior) was just 26.0 percent." Of the AI-generated patches, 20.1 percent fixed the original issue but altered application behavior (eg, changing "allow list" logic to "deny list" logic). Some 2.3 percent of the patches fixed the issue while introducing new security issues. 49.3 percent of the patches failed to fix at least one existing exploit path. And 2.2 percent both failed to fix the vulnerability while introducing a new exploit path. And among the patches in the first two categories (successful, clean; successful, changes app behavior), the researchers rated more than a third of the results fragile, meaning that while the adjusted code may have guarded against a particular vulnerability (eg, escaping particular input characters), the repair job didn't address the underlying problem. In their research paper [PDF], authors Axel Mierczuk, Spencer Michaels, and Keith Hoodlet propose the acronym FLAWED to represent automated LLM patches: Fix-Like Artifacts With Embedded Defects. Based on the generated patches, they conclude, "[T]he expected value of a fully LLM-generated, non-human-reviewed patch is a net-negative by a considerable margin." The value of LLM-generated patches depends upon initial patching guidance. The research team says that while both human developers and LLMs typically require some initial guidance to tackle a vulnerability, LLMs are more likely to be derailed when given incorrect advice. When LLMs get correct guidance, their fix-success rate hits 65.0 percent compared to 50.4 percent when they get no guidance. And incorrect guidance dooms LLMs, dropping their fix-success rate down to about 15.2 percent. Human devs, the authors argue, have a good chance of catching misleading information as they reason through vulnerable code. The authors have released a patch evaluation harness under the name FLAWED that organizations can use to evaluate the effectiveness of their security fixes. It's clear from the paper why AI-generated patches might be appealing - considered in isolation, they're inexpensive relative to human software engineers. The average successful, clean patch cost just $6.74 (a figure that includes the cost of failed attempts). Nonetheless, the authors argue that the cost-benefit analysis needs to assess how much expert supervision will be required to make LLM-assisted patching useful. "Based on our manual review of a representative sample of patches generated during our research, we suspect that, in a large number of cases, the cognitive load imposed by reviewing a mountain of mostly-incorrect, similar-yet-subtly-different LLM-generated vulnerability patches will likely result in engineers spending more effort than would be necessary to understand and patch vulnerabilities themselves using standard LLM-assisted coding techniques that keep the human operator in the driver's seat," the authors conclude. "The alternative, cognitive surrender to a process with a success rate of only about 1 in 4 poses significant long-term risks for any organization considering autonomous, LLM-driven patching." ®

Shock horror -- AI-generated security patches fall short of actually solving all the problems they were meant to fix

AI without oversight creates patches that rarely fix the issue entirely * Researchers tested AI-generated patches on six CVEs with poor success rates * Many fixes failed, altered behavior, or introduced new vulnerabilities * Guidance improved outcomes, leading to FLAWED evaluation harness release When using Generative Artificial Intelligence (GenAI) to fix vulnerabilities, security professionals are most of the time just robbing Peter to pay Paul, experts have warned. Researchers from 1Passwords Off-by-1 Labs analyzed fixes proposed by two frontier models - ChatGPT 5.5 at "medium" effort, and Claude Opus 4.8 at "high" effort. As an experiment, the researchers took six recently disclosed CVEs and produced 6,080 patches using two frontier, cyber-capable reasoning models. The results were underwhelming to say the least - of all the proposed patches, just a quarter (26%) fully resolved the issue. FLAWED work? This obviously leaves plenty to be desired, as half (49.3%) of the patches failed to fix at least one existing exploit path. A fifth (20.1%) fixed the original issue but changed application behavior, while 2.3% introduced new security issues. Funny enough, 2.2% failed to fix the vulnerability while also introducing additional exploit paths, as well. Even among the patches that might be considered (26% of clean ones and 20.1% of those that changed app behavior), more than a third were fragile and not entirely addressing the underlying problem. The researchers created an acronym for automated LLM patches: FLAWED (Fix-Like Artifacts With Embedded Defects), and warned against letting AI work without human oversight: "The expected value of a fully LLM-generated, non-human-reviewed patch is a net-negative by a considerable margin." Results drastically improved when the AI was given better context, the researchers further explained. Before working on any patch, human developers are usually given initial guidance. When AI is given proper guidance, its success rate rises to 65%. Incorrect guidance, on the other hand, drops the success rate down to 15.2%. The difference between humans and AI is that humans are better at catching misleading information and poor guidance. This doesn't mean developers will, or should, abandon AI. Worst case scenario is that developers will spend more time reviewing AI-generated fixes which could increase cognitive load and still end up being net negative. Therefore, the researchers released a patch evaluation harness called FLAWED, which organizations can now use to determine the effectiveness of their AI-generated fixes. Via The Register Follow TechRadar on Google News and add us as a preferred source to get our expert news, reviews, and opinion in your feeds.

1Password study: only 26% of AI security patches fully fix flaws

AI1Password study: only 26% of AI security patches fully fix flaws Across 6,080 patches, most left attack paths or regressions 1Password's Off-by-1 Labs has published a new study on AI-generated security patches, examining 6,080 fixes for six recently disclosed vulnerabilities. By the lab's count, only 26% fully remediated the issue without changing application behavior. 49.3% still left at least one exploitable attack path in place, 20.1% fixed the flaw but changed behavior, 2.3% introduced a new vulnerability, and 2.2% did both: they failed to fix the original problem and added a new weakness. To reduce the chances that the models were simply recalling training data, the researchers used OpenAI's ChatGPT-5.5 and Anthropic's Claude Opus 4.8 on recent CVEs. The results were rough. For complex flaws, 53.9% of the patches were marked "FLAWED," meaning the code looked convincing on the surface and sometimes even passed early tests. Even then, the researchers say more than one-third of the fixes that initially seemed successful were later judged fragile. Some blocked a proof of concept, including the SpringAI character-filter cases, but still didn't address the root cause or account for architectural intent, business requirements, security consequences, or long-term maintainability. If you use automated coding tools for security work, this update is worth a look. Those tools can speed up development and help with triage, but when security-sensitive code and customer data are on the line, you still need skilled human review. An autonomous fix engine isn't enough. You can read the Off-by-1 Labs research online.