Google's Big Sleep AI Makes History by Discovering SQLite Security Flaw

Google's Big Sleep AI model sets world first with discovery of SQLite security flaw - SiliconANGLE

Google's Big Sleep AI model sets world first with discovery of SQLite security flaw Google LLC has revealed that it uncovered a previously unknown vulnerability using artificial intelligence, a claimed world first that could mark the beginning of AI being used at the forefront of security vulnerability detection. The vulnerability, a buffer overflow issue in SQLite, was uncovered using a large language model called "Big Sleep," a collaboration between Google Project Zero and DeepMind. The Big Sleep model uses advanced variant-analysis techniques - techniques involving using insights from previously discovered vulnerabilities to identify similar, potentially exploitable flaws in related code sections. By leveraging this approach, Big Sleep was able to detect a flaw that had eluded traditional fuzzing methods, methods that involve automatically generating and testing large volumes of random or semi-random inputs to a program to uncover bugs or vulnerabilities by observing unexpected crashes or behaviors. The system works by first reviewing specific changes in the codebase, such as commit messages and diffs, to identify areas of potential concern. The model then analyzes these sections using its pre-trained knowledge of code patterns and past vulnerabilities, allowing it to pinpoint subtle flaws that conventional testing tools might miss. During its analysis, Big Sleep discovered an issue in SQLite's "seriesBestIndex" function, where it failed to properly handle edge cases involving negative indices that could lead to a write operation outside the intended memory bounds, creating a potential exploit. The AI identified the vulnerability by simulating real-world usage scenarios and scrutinizing how different inputs interacted with the vulnerable code. In addition, Big Sleep was also able to perform root-cause analysis, not just identifying vulnerabilities but also understanding the underlying issues that lead to them. The capability is said by Google to enable developers to address the core problem and hence reduce the likelihood of similar vulnerabilities in the future. Interestingly, the discovery of the vulnerability occurred before it could be exploited in an official release, arguably demonstrating the effectiveness of AI in proactive defense. "We hope that in the future this effort will lead to a significant advantage to defenders - with the potential not only to find crashing test cases but also to provide high-quality root-cause analysis, triaging and fixing issues could be much cheaper and more effective in the future," the Big Sleep team wrote in a blog post. "We aim to continue sharing our research in this space, keeping the gap between the public state-of-the-art and private state-of-the-art as small as possible."

Google claims AI first after SQLite security bug discovered

Google claims one of its AI models is the first of its kind to spot a memory safety vulnerability in the wild - specifically an exploitable stack buffer underflow in SQLite - which was then fixed before the buggy code's official release. The Chocolate Factory's LLM-based bug-hunting tool, dubbed Big Sleep, is a collaboration between Google's Project Zero and DeepMind. This software is said to be an evolution of earlier Project Naptime, announced in June. SQLite is an open source database engine, and the stack buffer underflow vulnerability could have allowed an attacker to cause a crash or perhaps even achieve arbitrary code execution. More specifically, the crash or code execution would happen in the SQLite executable (not the library) due to a magic value of -1 accidentally being used at one point as an array index. There is an assert() in the code to catch the use of -1 as an index, but in release builds, this debug-level check would be removed. Thus, a miscreant could cause a crash or achieve code execution on a victim's machine by, perhaps, triggering that bad index bug with a maliciously crafted database shared with that user or through some SQL injection. Even the Googlers admit the flaw is non-trivial to exploit, so be aware that the severity of the hole is not really the news here - it's that the web giant believes its AI has scored a first. We're told that fuzzing - feeding random and/or carefully crafted data into software to uncover exploitable bugs - didn't find the issue. The LLM, however, did. According to Google, this is the first time an AI agent has found a previously unknown exploitable memory-safety flaw in widely used real-world software. After Big Sleep clocked the bug in early October, having been told to go through a bunch of commits to the project's source code, SQLite's developers fixed it on the same day. Thus the flaw was removed before an official release. "We think that this work has tremendous defensive potential," the Big Sleep team crowed in a November 1 write-up. "Fuzzing has helped significantly, but we need an approach that can help defenders to find the bugs that are difficult (or impossible) to find by fuzzing, and we're hopeful that AI can narrow this gap." We should note that in October, Seattle-based Protect AI announced a free, open source tool that it claimed can find zero-day vulnerabilities in Python codebases with an assist from Anthropic's Claude AI model. This tool is called Vulnhuntr and, according to its developers, it has found more than a dozen zero-day bugs in large, open source Python projects. The two tools have different purposes, according to Google. "Our assertion in the blog post is that Big Sleep discovered the first unknown exploitable memory-safety issue in widely used real-world software," a Google spokesperson told The Register, with our emphasis added. "The Python LLM finds different types of bugs that aren't related to memory safety." Big Sleep, which is still in the research stage, has thus far used small programs with known vulnerabilities to evaluate its bug-finding prowess. This was its first real-world experiment. For the test, the team collected several recent commits to the SQLite repository. After manually removing trivial and document-only changes, "we then adjusted the prompt to provide the agent with both the commit message and a diff for the change, and asked the agent to review the current repository (at HEAD) for related issues that might not have been fixed," the team wrote. The LLM, based on Gemini 1.5 Pro, ultimately found the bug, which was loosely related to changes in the seed commit [1976c3f7]. "This is not uncommon in manual variant analysis, understanding one bug in a codebase often leads a researcher to other problems," the Googlers explained. In the write-up, the Big Sleep team also detailed the "highlights" of the steps that the agent took to evaluate the code, find the vulnerability, crash the system, and then produce a root-cause analysis. "However, we want to reiterate that these are highly experimental results," they wrote. "The position of the Big Sleep team is that at present, it's likely that a target-specific fuzzer would be at least as effective (at finding vulnerabilities)." ®

Google's Big Sleep AI: The first to detect 0-day vulnerability

Google's Big Sleep AI has detected a zero-day vulnerability in the SQLite database, marking a new chapter in memory-safety flaw detection. Learn how this breakthrough could redefine bug-hunting. Big Sleep, an evolution of Google's Project Naptime, was developed through a collaboration between Google's Project Zero and DeepMind. Its capability to analyze code commits and pinpoint flaws previously undetected by traditional fuzzing methods brings a new approach to identifying complex vulnerabilities. Big Sleep is Google's experimental bug-hunting AI tool that leverages the capabilities of LLMs to identify vulnerabilities in software. Google created this tool to go beyond traditional techniques, such as fuzzing, by simulating human behavior and understanding code at a deeper level. Unlike fuzzing, which works by randomly injecting data to trigger software errors, Big Sleep reviews code commits to detect potential security threats. In October 2024, Big Sleep successfully identified a stack buffer underflow vulnerability in SQLite. This flaw, if left unchecked, could have allowed attackers to crash the SQLite database or potentially execute arbitrary code. The discovery is notable because it was made in a pre-release version of SQLite, ensuring that the vulnerability was patched before reaching users. Google tasked Big Sleep with analyzing recent commits to the SQLite source code. The AI combed through changes, aided by a tailored prompt that provided context for each code alteration. By running Python scripts and sandboxed debugging sessions, Big Sleep identified a subtle flaw: a negative index, "-1," used in the code, which could cause a crash or potentially allow code execution. The Big Sleep team documented this discovery process in a recent blog post, explaining how the AI agent evaluated each commit, tested for code vulnerabilities, and then traced the cause of the bug. This stack buffer underflow vulnerability, categorized as CWE-787, arises when software references memory locations outside of allocated buffers, resulting in unstable behavior or arbitrary code execution. Google asserts that Big Sleep's focus is on detecting memory-safety issues in widely used software, an area often challenging for conventional AI tools. For instance, Protect AI's Vulnhuntr, an AI tool supported by Anthropic's Claude, is designed to detect zero-day vulnerabilities in Python codebases, but it focuses on non-memory-related flaws. According to a Google spokesperson, "Big Sleep discovered the first unknown exploitable memory-safety issue in widely used real-world software." By targeting specific bug types, Big Sleep and Vulnhuntr complement each other, suggesting a future where AI-powered agents can specialize in different aspects of cybersecurity. Google sees Big Sleep's success as a significant step toward integrating AI into cybersecurity defenses. Google's Big Sleep team stated, "We believe this work has tremendous defensive potential. Fuzzing has helped significantly, but we need an approach that can help defenders find the bugs that are difficult (or impossible) to find by fuzzing." The team highlighted the importance of AI in preemptive security measures, where vulnerabilities are identified and patched before attackers can discover them. While the success of Big Sleep in spotting the SQLite vulnerability is promising, Google has noted that the technology remains experimental. The AI model is still undergoing refinement, and the team acknowledged that a target-specific fuzzer could match or exceed its current capabilities in certain cases. Despite these caveats, the team remains optimistic, viewing this as the beginning of AI's larger role in vulnerability detection. By continually testing Big Sleep's abilities on both known and unknown vulnerabilities, Google aims to enhance its bug-hunting capabilities, potentially making it a vital tool for developers and security teams worldwide. Big Sleep's successful SQLite vulnerability detection may signal a paradigm shift in cybersecurity, where AI agents autonomously identify and address security issues. This transition to automated security measures could offer unprecedented protection, closing the gap between bug discovery and exploitation.

One of Google's "big AI" projects uncovered some serious security threats seeminlgy all on its own

AI could revolutionize software development by discovering critical flaws A collaborative "big AI" project between Google Project Zero and Google DeepMind has discovered a critical vulnerability in a piece of software before public release. The Big Sleep AI agent was set to work analyzing the SQLite open source database engine, where it discovered a stack buffer underflow flaw which was subsequently patched the same day. This discovery potentially marks the first ever time an AI has uncovered a memory-safety flaw in a widely used application. Big Sleep found the stack buffer underflow vulnerability in SQLite which had been 'fuzzed' multiple times. Fuzzing is an automated software testing method that can discover potential flaws or vulnerabilities such as memory safety issues that are typically exploited by attackers. However, it is not a foolproof method of vulnerability hunting, and a fuzzed vulnerability that is found and patched could also exist as a variant elsewhere in the software and go undiscovered. The methodology used by Google in this instance was to provide a previously patched vulnerability as a starting point for the Big Sleep agent, and then set it loose hunting for similar vulnerabilities elsewhere in the software. While hunting for a similar vulnerability, Big Sleep encountered a vulnerability and traced the steps it took to recreate the vulnerability in a test case, gradually narrowing down the potential causes to a single issue and generating an accurate summary of the vulnerability. Google Project Zero points out that the bug wasn't previously spotted using traditional fuzzing techniques as the fuzzing harness was not configured to access the same extensions. However, when fuzzing was re-run with the same configurations, the vulnerability remained undiscovered despite 150 CPU-hours of fuzzing. "We hope that in the future this effort will lead to a significant advantage to defenders - with the potential not only to find crashing testcases, but also to provide high-quality root-cause analysis, triaging and fixing issues could be much cheaper and more effective in the future," the Big Sleep team said. "We aim to continue sharing our research in this space, keeping the gap between the public state-of-the-art and private state-of-the-art as small as possible." The full testing methodology and vulnerability discovery details can be found here.