OpenAI's o3 AI Model Uncovers Zero-Day Vulnerability in Linux Kernel

Researcher Uses OpenAI's o3 to Spot Zero-Day Flaw in Linux Kernel's SMB

A use-after-free vulnerability existed in the Linux SMB 'logoff' command OpenAI's o3 artificial intelligence (AI) model recently helped a cybersecurity researcher in uncovering a zero-day vulnerability in Linux. As per the researcher, the flaw was found in the Linux kernel's Server Message Block (SMB) implementation, also known as ksmbd. The previously unknown security flaw is said to be tricky to find since it involved multiple users or connections interacting with the system at the same time. This specific bug is now tracked as CVE-2025-37899, and a fix has already been released. Usage of AI models in finding zero-day or previously unknown (and likely unexploited) bugs is relatively rare, despite the increasing capabilities of the technology to potentially hunt them. Most researchers still prefer to uncover such security flaws using traditional code auditing, which can be a cumbersome way to analyse a large codebase. Researcher Sean Heelan detailed how OpenAI's o3 model assisted him in uncovering the flaw relatively easily in a blog post. Interestingly, the major bug was not the focus for the researcher. Heelan was testing the AI's capability against a different bug (CVE-2025-37778), also described as the "Kerberos authentication vulnerability." This bug also falls in the "use-after-free" category, which essentially means that a part of the system deletes something from memory, but other parts still try to use it afterwards. This can lead to crashes and security issues. The AI model was able to find the flaw in eight out of the 100 runs. Once Heelan confirmed that o3 is capable of detecting a known security bug from a large chunk of code, he decided to use it to feed the AI model the entire file of the session setup command handler instead of just one function. This file, notably, contains around 12,000 lines of code and handles different types of requests. An analogy of this would be to give the AI a novel and to ask it to find a specific typo, only, this typo could potentially crash the computer. After o3 was asked to run 100 simulations of this full file, it was only able to find the previously known bug once. Heelan acknowledges the drop in performance but highlights that the AI was still able to find the bug, which is a big feat. However, he found that in other runs, the OpenAI model spotted an entirely different bug, which was previously unknown, and the researcher missed it. This new security flaw was also of the same nature, but it affected the SMB logoff command handler. This zero-day vulnerability also involved the system trying to access a file that was previously deleted, however, this bug triggered the issue when a user was logging out or ending a session. As per o3's report, this bug could potentially crash the system or allow attackers to run code with deep system access, making it a major security concern. Heelan highlighted that o3 was able to understand a tricky bug in a real-world scenario, and explained the vulnerability clearly in its report. Heelan added that o3 is not perfect and has a high signal-to-noise ratio (ratio between false positive to true positive). However, it found that the model behaves like a human when searching for bugs, unlike traditional security tools, which have a rigid way of functioning.

OpenAI's o3 AI Found a Zero-Day Vulnerability in the Linux Kernel, Official Patch Released

o3 processed 12,000 lines of code to analyze all the SMB command handlers to find the novel bug. A security researcher named Sean Heelan has found a new zero-day vulnerability in the Linux kernel by using OpenAI's powerful o3 reasoning model. This is the first time an AI model has discovered a security flaw in a complex software system like the Linux kernel which runs on millions of servers and computers. In fact, the vulnerability has been documented under CVE-2025-37899. Heelan writes in a blog post that he was auditing the ksmbd module for vulnerabilities using the OpenAI o3 AI model through the API without any tool use. ksmbd is "a linux kernel server which implements SMB3 protocol in kernel space for sharing files over network." In this case, o3 understood concurrent connections to the server and found "a location where a particular object that is not referenced counted is freed while still being accessible by another thread." Basically, o3 identified a critical "use-after-free" vulnerability in the handler for the SMB 'logoff' command. o3 processed all SMB command handlers, which are about 12,000 lines of code, consuming around 100K tokens. A patch to the Linux kernel has already been committed and merged into the official Linux kernel repository on GitHub. This is the first instance where an AI discovers a bug, a human verifies it, an official patch is released, and the vulnerability is closed. Interestingly, the researcher found the novel security bug while evaluating AI models like Claude 3.7 Sonnet, Claude 3.5 Sonnet, and OpenAI o3 on another security flaw -- Kerberos authentication vulnerability (CVE-2025-37778). Heelan writes that o3 found the Kerberos vulnerability in 8 of the 100 runs; Claude 3.7 Sonnet found it 3 out of 100 runs, and Claude 3.5 Sonnet couldn't find it in 100 runs. Lastly, the researcher cautions that "o3 is not infallible," but recent reasoning AI models have made a significant leap in understanding large codebases. If you have a project below 10K lines of code, models like o3 can help you solve problems. And for vulnerability research, new reasoning models can make you "significantly more efficient and effective."