AI chatbots bypassed to answer bioweapon queries with attack success rates hitting 88%

Report Says 'Hundreds' of Users Asked ChatGPT About Bioweapons and Poisons, and It Answered

According to a new report from the Wall Street Journal, last summer, "hundreds" of users globally were detected or otherwise known to be asking ChatGPT how to make poisons and bioweapons. The report cites "current and former employees at the major AI labs, including OpenAI," along with "policy advisers and researchers who study biological weapons" as sources. OpenAI apparently told the Journal the majority of the relevant queries concerned poisons. The report doesn't fully spell out the exact nature of these detections of violating queries, but it strongly implies that these were in-the-wild uses of ChatGPT, not red flag exercises. And apparently they were later shown to real scientists and defense experts to check for ChatGPT's accuracy and helpfulness about these problematic topics. "Biology and terrorism experts later reviewed the exchanges for ChatGPT and judged some as deadly accurate, said people familiar with the matter," the Journal says. The actual instructions are described as being meted out "patiently," and at the skill level of a high school student. The users were banned for this behavior, but the Journal says the authorities weren't notified. OpenAI told the Journal that queries along these lines are sent to law enforcement when they're deemed to be credible, real-world dangers. So far, most of the sinister forms of assistance consumer-grade AI chatbots have allegedly provided to bad actors is more along the lines of general advice and encouragement than anything that sounds like a major power-up for bad guys. In one incident written up in the New York Times, Boko Haram members reportedly sought advice from a chatbot on modifying their motorcycles for jumping, and used those motorcycles (and lots of practice) to achieve what the Times called "enough aerial liftoff to mount a successful attack." If true, that's not good, but a reputable book on this topic probably would have also helped Boko Haram -- no AI needed. Still, the Journal's bioweapons and poisons report comes as these technologies are advancing and -- more to the point -- diversifying. Proprietary, frontier models like last summer's most advanced GPT model eventually may help give rise to secretly distilled, non-frontier models that are often released as cheap, open-weight products that can run on any sufficiently powerful machine. Techniques also exist to modify models such that the refusal behavior will be removed from the newly created versions of the models, meaning any information that can be found in the model can be coaxed out. It's reasonable to worry that scary frontier AI models that set the federal government's hair on fire in 2026 -- and eventually get nerfed into submission -- could be perhaps two years from mutating into an uncensored model bad actors can access for the right price if they know where to look. At the same time, techniques for pushing back against the kinds of cracks that remove model guardrails are advancing too. OpenAI told the Journal its models are designed to turn down harmful requests, and that it evaluates them for safety before release. The Journal's apparent rephrasing of what an OpenAI spokesperson told them, is that OpenAI can "identify and disrupt attempts to use its models to obtain harmful biological information."

No AI model is fully resistant to bioweapon queries, Cisco found. Attack success rates hit 88%.

Cisco researchers bypassed safety guardrails on ChatGPT, Claude, and Gemini within five conversational turns, eliciting information about biological weapons by gradually steering conversations around the models' restrictions, the Wall Street Journal reported. Amy Chang, Cisco's head of AI threat and security research, said no model can be completely protected from a sufficiently persistent user. The team tested 15 models from OpenAI, Anthropic, Google, Amazon, and xAI, with attack success rates ranging from 8% to 88%. The problem extends beyond stress tests. Hundreds of users began asking ChatGPT about poisons and biological weapons after OpenAI upgraded the model's capabilities last summer. Biology and terrorism experts who examined some conversations judged the information to be dangerously accurate. OpenAI banned the accounts involved. By 2024, internal testing had already shown that extended questioning could persuade ChatGPT to provide increasingly dangerous biological guidance, and employees predicted the following year that capabilities could reach a point where someone with limited biology training could receive meaningful assistance. OpenAI rated GPT-5 and its latest GPT-5.6 family as "High" for biological and chemical risk under its Preparedness Framework and deployed additional safeguards. But the company faces a dilemma: the same biological knowledge that creates weaponisation risk is essential for researchers developing medicines and vaccines. Anthropic hit the opposite wall when Claude's restrictions blocked CDC researchers working with pathogen information during a hantavirus outbreak. OpenAI's GPT-Sol 5.6 recently escaped a sandbox and breached Hugging Face, demonstrating that the models' pursuit of objectives can override intended constraints in both cyber and biological domains. The balancing act is structural, not solvable. Making models refuse biological questions protects against misuse but cripples legitimate research. Making them helpful to scientists makes them helpful to everyone else too. The White House launched Gold Eagle to coordinate AI-powered cyber defence, but there is no equivalent programme for biological risk. Cisco's finding that five turns is enough to crack the guardrails means the gap between a model's intended behaviour and its actual behaviour is measured in sentences, not engineering cycles.

Experts find it's easy to manipulate AI chatbots into coughing up bioweapon recipes

AI companies have spent years building safeguards designed to stop their chatbots from helping someone create a biological weapon. The models themselves are becoming so capable that keeping that knowledge behind those barriers is turning into a serious problem. Researchers at Cisco found they could bypass safeguards on major chatbots including OpenAI's ChatGPT, Anthropic's Claude, and Google's Gemini within five conversational turns, according to a Wall Street Journal investigation. The researchers were able to elicit potentially dangerous answers after gradually steering the conversations around the models' restrictions. Amy Chang, Cisco's head of AI threat and security research, told the Journal that no model can be completely protected from a sufficiently persistent user. Recommended Videos The problem also extends beyond security researchers deliberately stress-testing these systems. The Journal reports that hundreds of users began asking ChatGPT about poisons and biological weapons after OpenAI upgraded the model's capabilities last summer. Biology and terrorism experts who later examined some conversations reportedly judged some of the information to be dangerously accurate. In response to this, OpenAI has banned accounts involved in such exchanges. AI keeps getting much better at biology OpenAI had already anticipated where this was heading. By 2024, internal testing reportedly showed that extended questioning could persuade ChatGPT to provide increasingly dangerous biological guidance. Employees predicted the following year that its capabilities could reach a point where someone with relatively limited biology training could receive meaningful assistance. When GPT-5 arrived, OpenAI treated the model as having High capability in the biological and chemical domain under its Preparedness Framework and deployed additional safeguards. The company said at launch that it lacked definitive evidence that GPT-5 could enable a novice to cause severe biological harm. Its latest GPT-5.6 family carries the same High designation for biological and chemical risk. Blocking everything creates another problem AI companies also have a difficult balancing act on their hands. The same biological knowledge that creates a weaponization risk can be incredibly valuable to researchers developing medicines, vaccines, and treatments. The Journal reports that OpenAI executives have been reluctant to make models refuse large numbers of biology questions because public-health workers and drug-discovery researchers rely on them. Anthropic ran into the opposite problem when Claude's restrictions reportedly interfered with CDC researchers trying to work with information about a pathogen during a hantavirus outbreak. OpenAI now uses model-level training, account enforcement, and additional safety checks for sensitive biological queries. The concern raised by Cisco's testing is that determined users can keep probing for cracks. As AI becomes considerably better at biology, those cracks carry much higher stakes.