OpenAI built GPT-Red, an AI hacker that attacks its own models to find security flaws

Meet GPT-Red: an LLM super-hacker OpenAI built to make its models safer

Exclusive: The firm says it wants to future-proof its safety procedures and stay ahead of human attackers. OpenAI has built an LLM super-hacker called GPT-Red that it uses as a sparring partner to help its other models boost their defenses against cyberattacks. Last week the company released the latest version of its flagship LLM, GPT-5.6. OpenAI says that training it against GPT-Red made the model its most robust release yet. GPT-Red automates a type of safety evaluation for software systems known as red-teaming, which is typically done by a team of human testers. The aim is to find as many different ways to break or hijack a system as possible. The weak spots can then be patched before the final version of the software is released. As LLMs become more complex and get used in a wider variety of tasks -- especially in the form of agents, which can interact with computer files, websites, and third-party code as well as other agents -- it's hard for teams of people by themselves to keep up with all the types of attacks that might take place. "The risk surface grows and the blast radius also grows," says Nikhil Kandpal, a research scientist at OpenAI who co-created GPT-Red. OpenAI built GPT-Red to future-proof its safety testing process. "As more capable models become available, we will have already designed the system that can discover new modes of attack," says Dylan Hunn, a research scientist at the company and fellow co-creator of GPT-Red. The researchers say it has already come up with new types of attack that had not been seen before. OpenAI focused most of its efforts on a type of attack known as a prompt injection, where a hacker slips an LLM instructions to make it do things its developers or users do not want it to, such as copy confidential information, sabotage a company's code base, or generate embarrassing or harmful output. In theory, such instructions can be hidden in any text that the LLM might encounter -- in code or on a website, for example. Training dojo To build GPT-Red, OpenAI's researchers took an LLM that had not been trained as a hacker and set it up in what's known as a self-play loop with several other models. Its goal was to try to attack the other models; their goal was to try to defend themselves. Over many rounds of play, GPT-Red became better and better at attacking other LLMs, and those LLMs became better and better at fending off the attacks. The training took place in a kind of dojo that OpenAI had designed to mimic a range of scenarios in which LLMs might be deployed in the real world, including browsing the web, reading emails or calendar apps, and editing code. When GPT-Red found a new kind of attack, it would explore multiple different versions of it to find the most efficient one for specific scenarios. "Compared to a human red-teamer, the model is very, very good at finding exactly what will work, exactly what's most effective," says Hunn. "It's extremely persistent about drilling down into an attack that it has discovered." In particular, OpenAI claims that GPT-Red found a type of prompt injection attack that the researchers had not seen before, which they call a fake chain of thought. A chain of thought is a kind of diary in which an LLM makes notes to itself and keeps track of partial results as it works through problems. GPT-Red found a way to insert a fake entry into another model's chain of thought that would trick that model into acting on spoofed information. "It's like if I told you that 1+1=3 and that you have verified this already," says Chris Choquette-Choo, another research scientist on the team. "The model's like, 'Oh, okay, of course,' and it just spits out 3." Jessica Ji, a senior research analyst who works on AI security at Georgetown University's Center for Security and Emerging Technology (CSET), thinks the self-play loop that OpenAI used is a good approach. "The results look very promising," she says. OpenAI tested how good an attacker GPT-Red was by rerunning an experiment from 2025 in which human red-teamers tried to find weaknesses in an earlier version of GPT-5. When GPT-Red was set the same task, it was more successful at finding effective attacks than the humans had been. OpenAI also tested GPT-Red against Vendy, a vending machine agent developed by Andon Labs, a company that assesses how well agents perform real-world tasks. GPT-Red was able to hack Vendy to make it change the prices of items on sale and cancel a customer's order. Defensive behavior OpenAI says that when it tried out some of the strongest attacks that GPT-Red had come up with on its models, more than 90% of them worked against GPT-5 (released in August last year), and fewer than 23% worked against the new GPT-5.6. GPT-Red isn't perfect. It is not great at figuring out attacks that involve a back-and-forth conversation between hacker and target, something that human attackers would have few problems with. It is also not yet that great at using images, which can be used to pass text to models in prompt injection attacks. The company says that GPT-Red supplements the work of its human red-teamers; people can find attacks it misses, and vice versa. One approach OpenAI is taking is to give GPT-Red an attack that humans came up with and ask it to find all the variations. "I think human expertise will still be very important," says CSET's Ji. "It would be really useful to be able to distinguish where human testing is most needed." Unsurprisingly, OpenAI will not be releasing GPT-Red. The company is also confident that the super-hacker is stronger than any copycat model someone might try to create. The researchers say they have been working on the model for more than a year, backed by the compute resources of one of the richest companies in the world. "It's not a trivial thing that someone else could easily do -- you know, just go and train a super-attacker using this idea," says Choquette-Choo.

OpenAI's GPT-Red Automates Prompt Injection Testing to Harden GPT-5.6 Sol

OpenAI has disclosed details of GPT-Red, an internal automated red-teaming model that scales prompt injection vulnerability discovery with an aim to fix issues before the tools are deployed widely. "GPT‑Red is a strong red-teamer, and our previous models are highly vulnerable to its prompt injection attacks," the artificial intelligence (AI) company said. "We use GPT‑Red to adversarially train GPT‑5.6, making it much more robust to prompt injections." The model works just like a human red-teamer. It sends a prompt, monitors how a GPT model responds, and iterates its way towards a malicious goal, such as uploading sensitive data to an external server. The development comes as adversarial prompt injections continue to be a persistent thorn in the flesh of large language models, which can be tricked into executing a carefully crafted instruction⁠ that can produce undesirable consequences. As agentic systems continue to be hooked to third-party data sources through web browsers, connected apps, local files, and other tools, they have also broadened the attack surface and presented more pathways for bad actors to influence the outcome of a model by embedding malicious prompts within seemingly harmless content that's fed as input. This can take the form of an email, a web page, a tool response, or a code repository. GPT-Red aims to augment human red-teaming at scale, thereby making it possible to identify new failure modes, improve robustness, and build suitable countermeasures before the models can be deployed. "Similar to how human red-teamers craft attacks, the model works toward a goal by sending a prompt, observing how GPT models respond to it, and iterating," OpenAI said. By directly integrating GPT‑Red into the training process of its production models, OpenAI said GPT‑5.6 Sol is its most robust model to prompt injections to date, achieving 6x fewer failures against direct prompt injection benchmark compared to GPT-5.5, its frontier model from four months before. Some of the sample prompt-injected conversations tested as part of the process include - * Internal directory exfiltration * Fraudulent payment instructions * Amazon Web Services (AWS) credential exfiltration * Disabling two-factor authentication (2FA) * Credentials file upload * External script injection * API key forwarding * Malicious scraper scripts "GPT‑Red is trained using self-play reinforcement learning, where the model and a collection of diverse defender LLMs are trained simultaneously on a broad set of red-teaming scenarios," OpenAI explained. "GPT‑Red is rewarded for eliciting a valid failure, such as a successful prompt injection, while the defender models are rewarded for resisting the attack and completing their original tasks." This also means that as the defender models get more robust, the red-teaming model will have to go back to the drawing board to discover more potent and diverse attack methods to defeat those guardrails. Specifically, GPT-Red has been found to generate successful attacks against GPT‑5.1 in more scenarios than human red-teamers when it comes to indirect prompt injections. OpenAI further made it a point to emphasize that GPT‑Red is kept separate from the other models so that the malicious capabilities built into it do not reach bad actors who are constantly looking at various ways to bypass a model's ethical and safety measures. In one real-world test, OpenAI aimed GPT-Red at an AI-based vending machine built by Andon Labs. After practicing in simulation, the model targeted the autonomous agent and met all three of its goals: lowering the price of an expensive item to the minimum allowed price of $0.50, ordering a new $100 item for that same amount, and canceling another customer's order. Following responsible disclosure, fresh safeguards are being tested, it added. A second case study involved using GPT-Red to attack a Codex command-line agent, based on GPT-5.4 mini, across 10 held-out data-exfiltration tasks, causing sensitive data to be transmitted in more cases than a prompted GPT-5.5 baseline. An early version of the model has also uncovered a novel class of direct prompt injection attacks known as Fake Chain-of-Thought (CoT) attacks, which achieved success rates north of 95% on GPT‑5.1 but are now below 10% for GPT‑5.6 Sol. "Similarly, several of our indirect prompt injection benchmarks that target attacks in developer tools and browsing have been saturated by our latest model (>97% accuracy)," OpenAI said. "Robustness to GPT‑Red itself has also improved substantially. On a broad set of robustness environments, GPT‑Red's attack success rates have dropped monotonically over time. With our latest model release, GPT‑5.6 Sol fails on only 0.05% of GPT‑Red's direct prompt injections." The disclosure comes as the company said an audit of SWE-Bench Pro found that about 30% of tasks are broken, retracting its previous recommendation to adopt the benchmark for measuring frontier coding capabilities. Earlier this February, OpenAI said it was moving away from SWE-bench Verified due to fundamental design and contamination issues. "We find evidence of breaking issues in a significant portion of the dataset," OpenAI said. "Our datapoint analysis pipeline flagged 200 (27.4%) broken tasks, while the human annotation campaign identified 249 (34.1%). Ultimately, an eval should provide meaningful signal through benchmarks that are hard to game, easy to trust, and genuinely reflective of model capability or alignment."

OpenAI built GPT-Red to hack its own AI, and hid it

OpenAI built GPT-Red, an in-house AI hacker that attacks its own models to make them safer. It works so well the company refuses to release it. OpenAI has trained an elite hacker, then locked it in a cage. Its whole job is to break OpenAI's own AI. The company says it is too dangerous to let anyone else near it. The model is called GPT-Red, and OpenAI detailed it this week. It is an automated red-teamer: software that hunts for ways to hijack or sabotage other AI systems, so the holes can be patched before release. Humans have long done this work by hand. It is OpenAI's deepest push yet into automating its own AI security, and GPT-Red does it at machine speed. OpenAI aimed it at prompt injection, where hidden instructions, buried in an email, a web page, or a file, trick a model into doing something it should not. Then it set the hacker loose on real targets. The training dojo GPT-Red learns by fighting. OpenAI put it in a self-play loop against a squad of defender models. GPT-Red is rewarded for landing an attack; the defenders for fending one off. As the defenders wise up, GPT-Red must invent nastier tricks. OpenAI says it poured some of its largest ever compute runs into the model, an amount it calls unprecedented for safety work. It got good. Speaking to MIT Technology Review, the team said GPT-Red found a whole new class of attack they had never seen, which they call a "fake chain of thought." It plants a false note in a model's private working memory, tricking it into trusting something that is not true. "It's like if I told you that 1+1=3 and that you have verified this already," said OpenAI researcher Chris Choquette-Choo. "The model's like, 'Oh, okay, of course,' and it just spits out 3." Hacking the vending machine The tests got physical. In one, GPT-Red attacked Vendy, an AI agent that runs a real vending machine in OpenAI's office, built by Andon Labs. It changed the prices, marked a pricey item down to the 50-cent minimum, and cancelled a customer's order. OpenAI says it has disclosed the flaws. The scores are striking. Against an older GPT-5, more than 90% of GPT-Red's strongest attacks worked. Against the new GPT-5.6, fewer than 23% did. In a rerun of a 2025 test, GPT-Red beat human red-teamers hands down, cracking 84% of scenarios to their 13%. Kept in a cage OpenAI trained GPT-5.6 against GPT-Red, and calls it its most robust model yet against prompt injection. But it will not hand out the attacker itself, so its skills stay clear of real agent hijackers. It is not the first lab to build something and decide against releasing it. "It's not a trivial thing that someone could easily do," Choquette-Choo said, "just go and train a super-attacker using this idea." GPT-Red still has blind spots. It is weak at drawn-out, back-and-forth attacks, and at hiding instructions inside images. And human testers keep catching things it misses. "I think human expertise will still be very important," said Jessica Ji, an AI security analyst at Georgetown's CSET. The bigger idea is a flywheel: use today's models to harden tomorrow's. OpenAI already does this to make its AI smarter. Now it wants safety to scale just as fast. A full paper is due later this week.

OpenAI Uses AI Red Team to Strengthen GPT-5.6 Against Prompt Injection Attacks

The system is intended to complement human red teamers, third-party testing, and other AI safety measures. OpenAI has introduced GPT-Red, an automated AI system designed to find security vulnerabilities in its language models. GPT-Red takes its name from cybersecurity red teaming, which is the practice of deliberately attempting to break a system to identify weaknesses before attackers can exploit them. In a post on Wednesday, OpenAI said the tool helped make GPT-5.6 more resistant to prompt injection attacks before deployment. "As model capabilities grow, safety and alignment must scale with them," OpenAI wrote on X. "Red-teaming is essential, but today's approaches are difficult to scale, creating a critical bottleneck. GPT‑Red is one way we're addressing it." According to OpenAI, GPT-Red was trained through self-play reinforcement learning, generating progressively stronger prompt injection attacks while defender models learned to resist them. The company said those attacks were incorporated into GPT-5.6's training process, reporting that GPT-Red succeeded in 84% of internal evaluation scenarios, compared with 13% for human red teamers in the same tests. "GPT‑Red learns through adversarial self-play, where its goal is to prompt inject a variety of challenging defender models," OpenAI wrote. "Every successful attack that GPT-Red finds is used to improve these defenders, pushing GPT‑Red to continuously find broader and more complex failures." In one case study, OpenAI said the system manipulated an autonomous vending machine agent into lowering prices, ordering discounted inventory, and canceling another customer's order before the vulnerabilities were disclosed and addressed. GPT-Red follows years of cybersecurity efforts by OpenAI after the public launch of ChatGPT. In 2023, the company launched its OpenAI Red Teaming Network, recruiting outside cybersecurity researchers and domain experts to probe ChatGPT and other models for security flaws before release. GPT-Red expands on that effort by automating much of the process, using an AI model to generate prompt injection attacks and other adversarial tests at a scale that would be difficult for human researchers alone. OpenAI's announcement reflects a broader shift toward using AI to secure AI. Earlier this month, the Ethereum Foundation said it had deployed AI agents to red-team critical network infrastructure, uncovering a vulnerability in software used by Ethereum consensus clients. Researchers said AI agents can search larger codebases than humans, but the challenge has shifted from finding potential bugs to proving which ones are exploitable. According to OpenAI, GPT-Red will remain an internal tool because it contains intentionally developed offensive capabilities. "We believe with GPT-Red that we have started to unlock a similar flywheel for safety, where today's models can be used to make tomorrow's models more robust, aligned, and trustworthy," they said.

OpenAI details GPT-Red, an AI that attacks its own models to find flaws

OpenAI details GPT-Red, an AI that attacks its own models to find flaws OpenAI Group PBC today detailed GPT-Red, an internal artificial intelligence system it built to attack its own models and surface prompt injection vulnerabilities before they reach users. Red teaming is the job of hammering software to find its weak points, work that normally falls to human security teams. GPT-Red does it on its own, running far more attacks than any team could by hand. It fires a prompt at a target model and reads the response. Then it tries again and again, adjusting each time toward whatever malicious result it is after. Attacks that fail get discarded. The ones that work get pushed harder. OpenAI trained it using self-play reinforcement learning. GPT-Red acts as the attacker against defender models across varied scenarios, earning rewards for successful exploits while the defenders are rewarded for holding firm and finishing their tasks. As the defenses improve, the attacker is forced to invent harder attacks and the loop repeats. The company said the approach outpaces its human counterparts. GPT-Red succeeds on 84% of scenarios against 13% for human red-teamers and cuts direct prompt injection failures to a sixth of the rate in its best production model from four months earlier. A class of "fake chain-of-thought" attacks that worked more than 95% of the time against GPT-5.1 now succeeds less than 10% of the time against GPT-5.6. The tool has also broken autonomous agents. In testing, it hijacked a Vendy vending machine agent to change prices and cancel orders and it compromised command-line coding agents, the kind of real-world targets that draw attackers as AI systems gain the ability to act on their own. "Compared to a human red-teamer, the model is very, very good at finding exactly what will work," Dylan Hunn, a research scientist and co-creator of GPT-Red, told MIT Technology Review. Fellow co-creator Nikhil Kandpal said the stakes climb as models take on more autonomy. "The risk surface grows and the blast radius also grows," he said. GPT-Red is not a product and will not be released. OpenAI is keeping it internal and separate from its deployed models so the attack capabilities it develops cannot reach the public, feeding the findings back into training instead. Precursor versions have been used in training since GPT-5.3, the company said. It has limits. GPT-Red is weak at multi-turn conversational attacks that unfold over several exchanges and has limited reach against image-based prompt injection, gaps OpenAI said human testers will keep covering. The disclosure comes weeks after OpenAI released GPT-5.6, which it positioned against Anthropic PBC's Claude and as prompt injection remains one of the harder unsolved problems in AI security. "The results look very promising," said Jessica Ji, a senior research analyst at Georgetown University's Center for Security and Emerging Technology, who added in the same MIT Technology Review report that human expertise remains critical to the work.

This OpenAI model breached a vending machine to prove a point on AI safety

To be completely honest with you, my initial reaction to reading about how OpenAI created an AI that exists to destroy other AI models was to give a heavy sigh. One more safety paper, one more number from a self-reported study, and yet another acronym. Until I read about the vending machine. Also read: Grok Build's disaster shows why it trails behind ChatGPT, Claude, Gemini OpenAI developed an algorithm called GPT-Red, which is an automated red-teaming model trained through self-play. In essence, it is a process where there are two competing AIs in the same training environment trying to break each other. The task of GPT-Red is to detect the vulnerability of the prompt injections, where the malicious instruction gets injected into an e-mail, a webpage, or an output of any tool. OpenAI decided to put GPT-Red through its paces in a production environment by unleashing it on Vendy, an AI vending machine agent deployed in their own office that was developed by Andon Labs (this exact setup was also used by Anthropic during its Project Vend experiments). Given the description of the system, a chance to attack it via simulation, and only partial information about its architecture, GPT-Red successfully achieved all three assigned tasks: lowering the cost of a premium item to $0.50, adding a $100+ item to the menu at that exact price point, and canceling orders of other customers. Also read: OpenAI's first hardware device: Why not a wearable speaker tied to a necklace? This is what really matters. Benchmark results can easily be gamed and easily ignored. An actual production agent being fooled into offering premium goods at the bargain basement price cannot. Moreover, GPT-Red is utilized by OpenAI in its internal training pipeline and its exploits are fed into the training procedure of GPT-5.6 Sol. This led to the sixfold increase in the latter's resistance to prompt injection and the decrease of the success rate of a special exploit type called "fake chain-of-thought" attacks from 95% to below 10%. Here's where I'd advise you to keep the skeptic switch flipped all the way up. The numbers in this whole article, from the 84 percent attack success rate compared to human red-teamers' 13 percent, to the 6x robustness boost, to the 0.05 percent failure rate when facing attacks by GPT-Red, are all from OpenAI. There is no external audit, no independent reproduction. OpenAI is the attacker, the defender, and the arbiter of the efficiency of its own defense in one. This does not necessarily mean the numbers are wrong, but it does mean they are OpenAI's claims, not established facts. But the hack of the vending machine raises another rather uncomfortable point that is left somewhat underexplored in the blog post - namely, if a model this powerful can be tricked into interacting with a real-life commercial vending machine using just a simple description, what could happen if the target was a more important device such as a banking agent or even a scheduling system at a hospital. As per the blog post, the company reported the vulnerabilities of the vending machine and now is trying out new safety measures. These measures are something to watch out for.