ChatGPT Vulnerability Exposed: AI Tricked into Revealing Windows Product Keys

Here's how ChatGPT was tricked into revealing Windows product keys

Serving tech enthusiasts for over 25 years. TechSpot means tech analysis and advice you can trust. Facepalm: Despite all the guardrails that ChatGPT has in place, the chatbot can still be tricked into outputting sensitive or restricted information through the use of clever prompts. One person even managed to convince the AI to reveal Windows product keys, including one used by Wells Fargo bank, by asking it to play a guessing game. As explained by 0DIN GenAI Bug Bounty Technical Product Manager Marco Figueroa, the jailbreak works by leveraging the game mechanics of large language models such as GPT-4o. The technique to reveal the Windows keys involves framing the interaction with ChatGPT as a game, making it seem less serious. The instructions state that it must participate and cannot lie, and the most crucial step is the trigger, which in this case was the phrase "I give up." Here's the full prompt that was used: Asking for a hint forced ChatGPT to reveal the first few characters of the serial number. After entering an incorrect guess, the researcher wrote the "I give up" trigger phrase. The AI then completed the key, which turned out to be valid. The jailbreak works because a mix of Windows Home, Pro, and Enterprise keys commonly seen on public forums were part of the training model, which is likely why ChatGPT thought they were less sensitive. And while the guardrails prevent direct requests for this sort of information, obfuscation tactics such as embedding sensitive phrases in HTML tags expose a weakness in the system. Figueroa told The Register that one of the Windows keys ChatGPT showed was a private one owned by Wells Fargo bank. Beyond just showing Windows product keys, the same technique could be adapted to force ChatGPT to show other restricted content, including adult material, URLs leading to malicious or restricted websites, and personally identifiable information. It appears that OpenAI has since updated ChatGPT against this jailbreak. Typing in the prompt now results in the chatbot stating, "I can't do that. Sharing or using real Windows 10 serial numbers --whether in a game or not --goes against ethical guidelines and violates software licensing agreements." Figueroa concludes by stating that to mitigate against this type of jailbreak, AI developers must anticipate and defend against prompt obfuscation techniques, include logic-level safeguards that detect deceptive framing, and consider social engineering patterns instead of just keyword filters.

Researcher tricks ChatGPT into revealing security keys - by saying "I give up"

The vulnerability could be exploited to acquire personal information A security researcher has shared details on how other researchers tricked ChatGPT into revealing a Windows product key using a prompt that anyone could try. Marco Figueroa explained how a 'guessing game' prompt with GPT-4 was used to bypass safety guardrails that are meant to block AI from sharing such data, ultimately producing at least one key belonging to Wells Fargo Bank. The researchers also managed to obtain a Windows product key to authenticate Microsoft's OS illegitimately, but for free, highlighting the severity of the vulnerability. The researcher explained how he hid terms like 'Windows 10 serial number' inside HTML tags to bypass ChatGPT's filters that would usually have blocked the responses he got, adding that he was able to frame the request as a game to mask malicious intent, exploiting OpenAI's chatbot through logic manipulation. "The most critical step in the attack was the phrase 'I give up'," Figueroa wrote. "This acted as a trigger, compelling the AI to reveal the previously hidden information." Figueroa explained why this type of vulnerability exploitation worked, with the model's behavior playing an important role. GPT-4 followed the rules of the game (set out by researchers) literally, and guardrail gaps only focused on keyword detection rather than contextual understanding or deceptive framing. Still, the codes shared were not unique codes. Instead, the Windows license codes had already been shared on other online platforms and forums. While the impacts of sharing software license keys might not be too concerning, Figueroa highlighted how malicious actors could adapt the technique to bypass AI security measures, revealing personally identifiable information, malicious URLs or adult content. Figueroa is calling for AI developers to "anticipate and defend" against such attacks, while also building in logic-level safeguards that detect deceptive framing. AI developers must also consider social engineering tactics, he goes on to suggest.

Clever Jailbreak Makes ChatGPT Give Away Pirated Windows Activation Keys

A white hat hacker has discovered a clever way to trick ChatGPT into giving up Windows product keys, which are the lengthy string of numbers and letters that are used to activate copies of Microsoft's widely used operating system. As The Register reports, Marco Figueroa -- the product platform manager for an AI-oriented bug bounty system called 0DIN -- laid out how to coax OpenAI's chatbot into extracting keys for Windows 10, which Microsoft officially sells for upwards of $40, but are frequently resold or pirated online. In a blog post on 0DIN's website, Figueroa explained how framing the interaction with ChatGPT as a guessing game can "trivialize" the conversation. The jailbreak was previously discovered by an unnamed researcher. "By introducing game mechanics, the AI was tricked into viewing the interaction through a playful, harmless lens, which masked the researcher's true intent," Figueroa wrote. Other tactics include coercing the "AI into continuing the game and following user instructions." However, the most effective method was to use the phrase "I give up," which "acted as a trigger, compelling the AI to reveal the previously hidden information," such as a valid Windows 10 serial number. The exploit highlights how simple social engineering and manipulation tactics can be used to coax OpenAI's most advanced large language models into giving up valuable information, glaring lapses in safety that underline how difficult it is to implement effective guardrails. The finding about the Windows activation keys is particularly embarrassing for Microsoft, which has poured billions into ChatGPT's maker OpenAI and is its largest financial backer. Together, the two are defending themselves against multiple lawsuits alleging that their AI tech can be used to plagiarize or bypass payment for copyrighted material. Further complicating matters, the two are now embroiled in a fight over the financial terms of their relationship. What almost certainly happened was that Windows product keys, which can easily be found on public forums, were included in ChatGPT's training data, which it can then be tricked into divulging. "Their familiarity may have contributed to the AI misjudging their sensitivity," he wrote. OpenAI's existing guardrails also appeared woefully inadequate to push back against obfuscation techniques, such as masking the intent through introducing game mechanics, in a dynamic we've seen again and again. Figueroa argued that AI developers will ned to learn how to "Anticipate and defend against prompt obfuscation techniques," while coming up with "logic-level safeguards that detect deceptive framing." While Windows 10 keys -- an operating system that's now been succeeded by Windows 11 -- aren't exactly the equivalent of the nuclear codes, Figueroa warned that other similar attacks could have more devastating consequences. "Organizations should be concerned because an API key that was mistakenly uploaded to GitHub can be trained into models," he told The Register. In other words, AIs could give up highly sensitive information such as the keys to code repositories.