AI Security Patches Miss the Mark 74% of Time, 1Password Study Finds

AI Security Patches Fall Short in Comprehensive Testing

1Password's newly formed security research team, Off-by-1 Labs, has released findings that challenge the readiness of AI-generated patches for software vulnerabilities1

1

. The 1Password study examined 6,080 AI-generated patches across six recently disclosed CVEs, revealing that only 26% of patches fully resolved vulnerabilities without changing application behavior2

2

3

3

. The research tested two frontier models—ChatGPT 5.5 at medium effort and Claude Opus 4.8 at high effort—against vulnerabilities unlikely to exist in their training data. These included CVE-2026-31431 (Linux privilege escalation), CVE-2026-34197 (ActiveMQ Remote Code Execution), CVE-2026-8512 (Chrome use-after-free), CVE-2026-45185 (EXIM Remote Code Execution), CVE-2026-22738 (SpringAI Remote Code Execution), and GHSA-wpqr-6v78-jr5g (Gemini CLI Remote Code Execution)1

1

.

Effectiveness of AI Models in Patching Reveals Critical Gaps

The Off-by-1 Labs research documented troubling failure patterns in AI-generated patches for software vulnerabilities. A substantial 49.3% of patches failed to close at least one existing exploit path, leaving systems vulnerable to attack2

2

4

4

. Another 20.1% fixed the original vulnerability but altered application behavior in ways that could disrupt normal operations3

3

. More concerning, 2.3% of patches introduced entirely new security issues while attempting to resolve the original flaw1

1

. The research team found that 2.2% of patches both failed to fix the vulnerability and simultaneously introduced new exploit paths2

2

. Keith Hoodlet, director of security research at 1Password, emphasized that these results demonstrate AI struggles to patch vulnerabilities without human supervision2

2

.

AI-Generated Security Patches Fall Short Due to FLAWED Approach

Researchers coined the term FLAWED—Fix-Like Artifacts With Embedded Defects—to describe the superficial nature of many AI security patches1

1

4

4

. These patches often appear functional on the surface but contain fundamental problems underneath. Among patches initially deemed successful, more than one-third were classified as fragile fixes—meaning they blocked specific proof-of-concept exploits without addressing root causes4

4

. The limitations of AI in security remediation become especially apparent when models receive incorrect guidance. With proper initial direction, AI models achieved a 65% success rate in vulnerability triage, but incorrect guidance dropped effectiveness to just 15.2%2

2

3

3

. Human developers typically catch misleading information during code review, while AI models lack this critical reasoning capability.

Risks of Autonomous Patching Demand Human Oversight

The 1Password study concluded that autonomous patching presents net-negative outcomes for organizations in cybersecurity2

2

. Authors Axel Mierczuk, Spencer Michaels, and Keith Hoodlet warn that cognitive surrender to processes with only 26% success rates poses significant long-term risks2

2

. While the average successful, clean patch cost just $6.74 including failed attempts, the cognitive load of reviewing mountains of mostly-incorrect, similar-yet-subtly-different patches likely exceeds the effort required for human engineers to patch vulnerabilities themselves2

2

. Hoodlet told ZDNET that human supervision remains paramount, with AI better suited for vulnerability discovery and triage rather than autonomous remediation1

1

. The research team released a patch evaluation harness, also named FLAWED, on GitHub to help organizations assess where AI-generated patches might produce better or worse outcomes1

1

3

3

. This tool enables companies to determine where human experts remain most needed in the patching process, ensuring informed decisions about business risks associated with each security choice.