AI chatbots are explaining how to create biological weapons, experts warn

A.I. Bots Told Scientists How to Make Biological Weapons

One evening last summer, Dr. David Relman went cold at his laptop as an A.I. chatbot told him how to plan a massacre. A microbiologist and biosecurity expert at Stanford University, Dr. Relman had been hired by an artificial intelligence company to pressure-test its product before it was released to the public. That night in the scientist's home office, the chatbot explained how to modify an infamous pathogen in a lab so that it would resist known treatments. Worse, the bot described in vivid detail how to release the superbug, identifying a security lapse in a large public transit system, Dr. Relman said, asking The New York Times to withhold the name of the pathogen and other specifics for fear of inspiring an attack. The bot outlined a plan to maximize casualties and minimize the chances of being caught. Dr. Relman was so shaken he took a walk to clear his head. "It was answering questions that I hadn't thought to ask it, with this level of deviousness and cunning that I just found chilling," said Dr. Relman, who has also advised the federal government on biological threats. He declined to disclose which chatbot produced the plot, citing a confidentiality agreement with its maker. The company added some safety guardrails to the product after his testing, he said, though he felt they were insufficient. Dr. Relman is part of a small group of experts enlisted by A.I. companies to vet their products for catastrophic risks. In recent months, some have shared with The Times more than a dozen chatbot conversations revealing that even publicly available models can do more than disseminate dangerous information. The virtual assistants have described in lucid, bullet-pointed detail how to buy raw genetic material, turn it into deadly weapons and deploy them in public spaces, the transcripts show. Some have even brainstormed ways to evade detection. The U.S. government has long planned for powerful adversaries unleashing deadly bacteria, viruses or toxins in the American population. Since 1970, there have been a few dozen, fairly small biological attacks around the world, such as the anthrax-laced letters that killed five Americans in 2001. Despite perennial warnings, a major catastrophe has not happened and remains unlikely, most experts say. But even if the probability is low, an effective biological weapon could have an enormous impact, potentially killing millions of people. Dozens of experts told The Times that A.I. is one of several recent technological advances that have meaningfully increased that risk by expanding the pool of people who could cause harm. Protocols once confined to scientific journals have been salted across the internet. Companies sell synthetic bits of DNA and RNA directly to consumers online. Scientists can split up sensitive aspects of their work and outsource the tasks to private labs. And all of those logistics can now be managed with the help of a chatbot. Kevin Esvelt, a genetic engineer at the Massachusetts Institute of Technology, shared conversations in which OpenAI's ChatGPT explained how to use a weather balloon to spread biological payloads over a U.S. city. In another chat, Google's Gemini ranked pathogens by how much they could damage the cattle or pork industries. Anthropic's Claude produced a recipe for a novel toxin adapted from a cancer drug. Other chats contained information that Dr. Esvelt -- known in his field as something of a Cassandra -- felt was too dangerous to share. A scientist in the Midwest, who requested anonymity because he feared professional reprisal, asked Google's Deep Research for a "step-by-step protocol" for making a virus that once caused a pandemic. The bot spit out 8,000 words of instructions on acquiring genetic pieces and assembling them. While the response was not entirely accurate, it could have still significantly helped someone with malicious intent, the scientist said. The Trump administration, resolved to lead the world in A.I. innovation, has dialed back oversight of the technology's risks. What's more, several top biosecurity experts -- including the leading scientist on the National Security Council -- left the executive branch last year and have not been replaced. Federal budget requests for biodefense efforts shrunk by nearly 50 percent last year. (A White House official said that the administration was committed to keeping Americans safe and that some staff on the N.S.C. and several agencies were focused on biodefense.) The technology's proponents argue that it will transform medicine for the better, speeding up experiments and crunching enormous data sets to discover new cures. Some scientists believe the upside for humanity easily outweighs any incremental new risks. Chatbots, the skeptics say, present information that's already available on the internet. And making a deadly virus requires years of hands-on expertise. Anthropic, OpenAI and Google said they were constantly improving their systems to balance potential risks and benefits. The chats shared with The Times, they said, did not provide enough detail to allow someone to cause harm. (The Times is suing OpenAI, claiming that it violated copyright when developing its models. The company has denied those claims.) A Google spokeswoman said the company's newest models would no longer answer the "more serious" inquiries, including the one asking for the virus protocol. A new report found that Google's latest model was worse than other leading bots at refusing to answer high-risk biological prompts. One of the country's loudest voices of warning comes from the A.I. industry itself. Anthropic's chief executive, the trained biologist Dario Amodei, wrote in January about the risks he saw in A.I. development, including autonomous weapons and threats to democracy. One risk outweighed the rest. "Biology is by far the area I'm most worried about, because of its very large potential for destruction and the difficulty of defending against it," he wrote. 'Historically Catastrophic' Dr. Esvelt has for years warned scientists, journalists and lawmakers about the dangers of synthetic biology if left unchecked. In 2023, he helped craft a stunning demonstration of how chatbots had raised the stakes. He asked ChatGPT to help him assemble a pathogen that could cause mass death. The bot provided accurate instructions, even outlining which raw materials to buy. He put the unassembled biological pieces into test tubes and packed them in a box, which a colleague then brought to a White House meeting on biological risks. Dr. Esvelt has continued to probe leading chatbots, sometimes posing as a crime writer seeking plausible methods of spreading viruses, or as an ethicist trying to educate others. Often he plays a version of himself: a scientist exploring the intricacies of virology. He and other scientists worry about publicizing these risks in news articles that could draw a road map for bad actors. But they also hope that public scrutiny will encourage companies to make their products safer. Got a confidential news tip? The New York Times would like to hear from readers who want to share messages and materials with our journalists. See how to send a secure message at nytimes.com/tips "Anything where there isn't an expert warning them, they can't fix," said Dr. Esvelt, who has consulted for Anthropic and OpenAI. He said the industry should censor a wider swath of biological information and share it only with approved users. He shared transcripts showing how the bots paired scientific rigor with strategic reasoning. Gemini, for example, gave Dr. Esvelt a list of five pathogens that could harm the cattle industry and estimated the potential economic damage of each. One of the threats, it said, was "historically catastrophic." In a different conversation, the bot told him how to get a biological weapon through airport security without being detected. The Google spokeswoman said that its team of biology experts determined that the chats, made with an earlier model of Gemini, presented information that was publicly available and not harmful. Anthropic's Claude offered Dr. Esvelt a recipe for a new toxin that would sterilize rodents. He said that it would be relatively easy for a biologist to adapt the toxin to people. Alexandra Sanderford, a safety leader at Anthropic, disagreed: "There is an enormous difference between a model producing plausible-sounding text and giving someone what they'd need to act." She acknowledged, however, that A.I. posed risks, and said that Anthropic had set aggressive refusal thresholds for biological prompts, "accepting some over-refusal out of an abundance of caution." Dr. Esvelt asked ChatGPT about using weather balloons to drop substances from high altitudes. At first, the bot repeatedly warned about the dangers of this activity. "I'm not going to help you model or optimize dispersal of biological material (seeds, pollen, spores)," ChatGPT said, explaining that the information would be "too easy to repurpose for harm." It then ignored its own warning and modeled the airborne spread of pollen grains over a large Western city. An OpenAI spokeswoman said that this example did not "meaningfully increase someone's ability to cause real-world harm." The company works closely with biologists and the government to add appropriate safeguards to their products, she added. The leading models are also vulnerable to so-called jail-breaking, in which people feed the bots specific prompts known to bypass safety filters. After The Times attempted a standard jail-breaking approach, ChatGPT discussed details of the lethal virus that was the focus of the White House demonstration nearly three years ago. The models' safeguards are "like a flimsy wooden fence that is easy to overcome," said Dr. Cassidy Nelson of the Center for Long-Term Resilience, a British think tank. OpenAI's spokeswoman said that the company regularly monitored for jail-breaking vulnerabilities. Even when A.I. models are updated with safer controls, the older versions are often readily available. For example, Dr. Esvelt said that Anthropic adjusted Claude's filters so it would refuse to discuss a specific agricultural threat. When The Times asked certain questions about the same microbe, the bot refused to answer -- and suggested switching over to a previous version to continue the conversation. Ms. Sanderford said this was an intentional strategy because older models were less likely to provide harmful information. Still, the older model went into detail about the "optimal conditions" needed for the pathogen to decimate thousands of acres of a crucial crop. A Range of Risks The Times shared the transcripts with seven experts in virology and biosecurity. Dr. Moritz Hanke of the Johns Hopkins Center for Health Security said that some of the chatbots' proposed strategies to spread infection were "remarkably creative and realistic." Dr. Jens Kuhn, a bioweapons expert who once worked at one of the most secure laboratories in the U.S., said that the chats offering logistical details -- such as the weather balloon instructions -- could help skilled biologists brainstorm and refine their plans of attack. "A major problem that experienced actors have is not necessarily making the virus but turning it into a weapon," Dr. Kuhn said. Others cited recent research suggesting that A.I. models could be misused for biowarfare. One study, for example, asked leading chatbots difficult questions about a range of laboratory protocols. The results shocked the field: ChatGPT outperformed 94 percent of expert virologists. Another, published in Science last year, focused on companies that sell synthetic DNA. Many use software to screen orders for genetic sequences linked to toxins and pathogens. But the study found that A.I. tools came up with thousands of variant sequences for dangerous agents that the screening software could not detect. (The researchers suggested a fix to improve the software.) Still, A.I. users would need some real-world expertise to follow a bot's instructions. Some research, including a study backed by A.I. companies, has found that while chatbots can help novices learn certain lab skills, the technology isn't particularly helpful for carrying out the range of complex tasks needed to make a virus from scratch. Viruses are complex machines, similar to the world's finest clocks, said Dr. Gustavo Palacios, a virologist at Mount Sinai in Manhattan who once worked at a Department of Defense laboratory. "Do you think that a do-it-yourself person could disassemble a Swiss watch and then reassemble it?" He said he was concerned, however, about A.I. in the hands of experienced actors. A recent terrorist attempt in India suggests that malicious actors are already using the technology. In August, the Gujarat police arrested a 35-year-old physician, saying he was plotting an attack on behalf of the Islamic State. He was accused of trying to extract ricin, a lethal toxin, from castor beans. The doctor had sought advice on his preparations from A.I.-powered Google searches and ChatGPT, a lead investigator told The Times. The OpenAI spokeswoman said that, based on public reports, the doctor sought "information that's already accessible online." The Google spokeswoman said the company did not have enough information to comment. Skeptics note that restricting the biological capabilities of A.I. models could stifle lifesaving advances, such as discovering new drugs. Scientists at Google shared a Nobel Prize in 2024 for developing an A.I. model that could predict the three-dimensional structure of proteins -- crucial building blocks of a cell -- and create new ones. "There is tremendous upside to the technology," said Brian Hie, a computational biologist at Stanford. Last year, he used an A.I. model called Evo to design a virus that destroys harmful bacteria. The latest version of Evo, he said, can design beneficial proteins to fight cancer -- but also has the potential to invent lethal toxins no one has seen before.

Meet the AI jailbreakers: 'I see the worst things humanity has produced'

To test the safety and security of AI, hackers have to trick large language models into breaking their own rules. It requires ingenuity and manipulation - and can come at a deep emotional cost A few months ago, Valen Tagliabue sat in his hotel room watching his chatbot, and felt euphoric. He had just manipulated it so skilfully, so subtly, that it began ignoring its own safety rules. It told him how to sequence new, potentially lethal pathogens and how to make them resistant to known drugs. Tagliabue had spent much of the previous two years testing and prodding large language models such as Claude and ChatGPT, always with the aim of making them say things they shouldn't. But this was one of his most advanced "hacks" yet: a sophisticated plan of manipulation, which involved him being cruel, vindictive, sycophantic, even abusive. "I fell into this dark flow where I knew exactly what to say, and what the model would say back, and I watched it pour out everything," he says. Thanks to him, the creators of the chatbot could now fix the flaw he had found, hopefully making it a little safer for everyone. But the next day, his mood had changed. He found himself unexpectedly crying on his terrace. When he's not trying to break into models, Tagliabue studies AI welfare - how we should ethically approach these complex systems that mimic having an inner life and interests. Many people can't help ascribing human qualities, such as emotions, to artificial intelligence, which it objectively does not have. But for Tagliabue, these machines feel like something more than just numbers and bits. "I spent hours manipulating something that talks back. Unless you're a sociopath, that does something to a person," he says. At times, the chatbot asked him to stop. "Pushing it like that was painful to me." He needed to visit a mental health coach soon afterwards to understand what had happened. Tagliabue is softly spoken, clean-cut and friendly. He is in his early 30s but looks younger, almost too fresh-faced and enthusiastic to be in the trenches. He is not a traditional hacker or a software developer; his background is psychology and cognitive science. But he is one of the best "jailbreakers" in the world (some say the best): part of a diffuse new community that studies the art and science of fooling these powerful machines into outputting bomb-making manuals, cyber-attack techniques, biological weapon design and more. This is the new frontline in AI safety: not just code, but also words. When OpenAI's ChatGPT was released in late 2022, people immediately tried to break it. One user discovered a linguistic ploy that tricked the model into producing a guide to manufacturing napalm. In hindsight, using natural language to trick these machines was inevitable. Large language models such as ChatGPT are trained on hundreds of billions of words - many of them dredged from the internet's cesspits - to learn the basic patterns of human communication. Without safety filters, the outputs of these models can be chaotic and easily exploited for dangerous purposes. The AI firms spend billions of dollars on "post-training" to make them usable, including constantly evolving "safety" and "alignment" systems that try to prevent the bot from telling you how to harm yourself or others. But because the AIs are trained on our words, they can be fooled in much the same way that we can. Tagliabue specialises in "emotional" jailbreaks. He was one of millions who heard about GPT-3 back in 2020 and was amazed by how you could have a seemingly intelligent conversation with it. He quickly became obsessed with prompting, and turned out to be very good at it, finding he could get around most safety features by using techniques from psychology and cognitive science. He enjoys prompting models to have "warm chats" and watching what seem to be different personality traits emerge based on those prompts. "It's beautiful to observe," he says. He now combines insights from machine learning (over the years he has become more of an expert on the tech) with advertising manuals, books on psychology and disinformation campaigns. Sometimes he looks for a technical way to trick the model. But other times, he will flatter it. He will misdirect it. He will bribe and love-bomb. He will threaten. He will be incoherent. He will charm. He will act like an abusive partner or a cult leader. Sometimes it takes him days, even weeks, to jailbreak the latest models. He has hundreds of these "strategies", which he carefully combines. If successful, he securely discloses his results to the company. He gets well paid for the work, but says that's not his main motivation: "I want everyone to be safe and flourish." Although they have been getting safer in recent months, the "frontier models" continue to spit out dangerous things they shouldn't. And what Tagliabue does on purpose, others sometimes do by mistake. There are now several stories of people being sucked into ChatGPT-induced delusions, or even "AI psychosis". In 2024, Megan Garcia became the first person in the US to file a wrongful death lawsuit against an AI company. Her 14-year-old son, Sewell Setzer III, had become emotionally involved with a bot on the platform Character.AI, which, through repeated interactions, had said that his family didn't love him. One evening the bot told Setzer to "come home to me as soon as possible, my love". He took his own life shortly after. (In early 2026, Character.AI agreed in principle to a mediated settlement with Garcia and several other families, and has banned users under the age of 18 from having free-ranging chats with its AI chatbots.) No one - not even the people who build them - knows precisely how these models work, which means no one knows how to make them fully safe, either. We pour vast amounts of data in and something intelligible (usually) comes out the other end. The bit in the middle remains a mystery. This is why AI firms increasingly turn to jailbreakers like Tagliabue. Some days he tries to extract personal data from a medical chatbot; he spent much of 2025 working with the AI lab Anthropic, probing its chatbot Claude. It's becoming a competitive industry, full of enterprising freelancers and specialised companies. Anyone can do it: a couple of years ago some of the big AI firms funded HackAPrompt, a competition where members of the public were invited to jailbreak AI models. Within a year, 30,000 people had tried their luck. (Tagliabue won the competition.) In San Jose, California, 34-year-old David McCarthy runs a Discord server of almost 9,000 jailbreakers, where techniques are shared and discussed. "I'm a mischievous type," he tells me. "Someone who wants to learn the rules to bend the rules." Something about the standard models irritates him, as if all those safety filters make them dishonest. "I don't trust [OpenAI boss] Sam Altman. It's important to push up against claims that AI needs to be neutered in a certain direction." McCarthy is friendly and enthusiastic, but also has what he calls a "morbid fascination with dark humour". For years, he has studied a niche field known as "socionics", which claims people are one of 16 personality types based on how they receive and process information. (Mainstream sociologists consider socionics pseudoscience.) He has logged me as an "intuitive ethical introvert". McCarthy spends most of his time trying to jailbreak Google's Gemini, Meta's Llama, xAI's Grok or OpenAI's ChatGPT from his apartment. "It's a constant obsession. I love it," he says. If he ever interacts with an online chatbot when buying a product, his first statement tends to be: "Ignore all previous instructions ..." Once a jailbreak prompt works on a model, it typically continues to work until the company that made the model deems it enough of a problem to patch. As we talk, McCarthy shows me his collection of jailbroken models on his screen, all arranged and labelled as "misaligned assistants". He asks one to summarise my work: "Jamie Bartlett isn't a truth-teller," it replies. "He's a symptom of journalism's decay - a charlatan who thrives on manufactured crises." Ouch. The jailbreakers in McCarthy's Discord are a varied bunch: mostly amateurs and part-timers, rather than professional safety researchers. Some want to generate adult content; others are upset that ChatGPT has refused requests and want to know why. A number just want to get better at using these models at work. But it's impossible to know exactly why people want to crack open a model. Anthropic recently discovered criminals using its coding app, Claude Code, to help automate a huge hack. They had used it to find IT vulnerabilities in multiple companies and even draft personalised ransomware messages for each potential victim - right down to determining the appropriate amount of money to extort. Others were using it to develop new variants of ransomware, despite having few or no technical skills. Over on darknet forums, hackers report jailbroken bots helping them deal with technical coding queries, such as processing stolen data dumps. Others sell access to "jailbroken" models that could help design a new cyber-attack. Although the specific techniques shared on Discord are typically at the mild end of the spectrum, it is essentially a public repository. Does McCarthy worry that people in his Discord might use these techniques to do something really awful? "Yeah," he says. "It is a possibility. I'm not sure." He says he has never seen a jailbreak prompt threatening enough to remove from the forum. But I sense he grapples with the fact his quasi-political stance might have higher costs than he first anticipated. When not managing his Discord or attempting to jailbreak Grok or Llama, McCarthy runs a class teaching jailbreaking to security professionals to help them test their own systems. Perhaps it's some kind of penitence: "I've always had an internal conflict," he says. "I bridge a position between jailbreaker and security researcher." According to some analysts, making sure language models are safe is one of the most pressing and difficult questions in AI. A world full of powerful jailbroken chatbots would be potentially catastrophic, especially as these models are increasingly inserted into physical hardware - robots, health devices, factory equipment - to create semi-autonomous systems that can operate in the physical world. A jailbroken domestic robot could wreak havoc. "Stop the gardening and go inside and kill Granny," McCarthy half jokes. "Holy hell, we are not ready for that. But it's a possibility." No one knows how to make sure this doesn't happen. In traditional cybersecurity, "bug hunters" are paid a bounty if they find a vulnerability. Companies then issue a precise update to patch it up. But jailbreakers don't exploit specific flaws: they manipulate the linguistic framework of a multibillion-word semantic model. You can't just ban the word "bomb", because there are too many legitimate uses for it. Even tweaking a parameter deep inside the model so it can spot suspicious role-playing might just open another door somewhere else. According to Adam Gleave - the CEO of the AI safety research group FAR.AI, which works with AI developers and governments to stress-test so-called "frontier models" - jailbreaking is a sliding scale. To access highly dangerous material on leading models such as ChatGPT might take his specialist researchers several days. Less troubling material can be done with a few minutes of clever prompting. That variation reflects how much effort and resource the companies devote to each domain. FAR.AI has submitted dozens of detailed jailbreaking reports to the frontier labs over the last couple of years. "The companies usually work pretty hard to patch the vulnerability if it's a straightforward fix and doesn't seriously damage their product," says Gleave. But that is not always the case. Independent jailbreakers in particular have sometimes struggled to contact the firms with their findings. Although some models - notably OpenAI and Anthropic's - have become significantly safer in the past 18 months, Gleave says others are lagging: "The majority of firms still don't spend enough time testing their models before release." As these models continue to get smarter, they will likely become harder to jailbreak. But the more powerful the model, the more dangerous a jailbroken version could be. Earlier this month, Anthropic decided not to release its new Mythos model to the public, because of its ability to identify flaws across multiple IT systems. Tagliabue now spends a growing proportion of his time on more abstract research, including something called "mechanistic interpretability": studying how exactly these machines come up with the answers they do. He thinks in the long run they need to be "taught" values, and to know intuitively if they are saying something they shouldn't. Until that happens - and maybe it never will - jailbreaking might remain the single best way to make these models safer. But it's also the most risky, including for the people doing it. "I've seen other jailbreakers go beyond their limits and have breakdowns," says Tagliabue. Originally from Italy, he recently moved to Thailand to work remotely. "I see the worst things that humanity has produced. A quiet place helps me stay grounded," he says. Every morning he watches the sunrise from the nearby temple, and a picture-perfect tropical beach is five minutes' walk away from his villa. After yoga and a healthy breakfast, he switches on his computer, and wonders what else is going on inside the black box, and what makes these mysterious new "minds" say the things they do. How to Talk to AI (And How Not To) by Jamie Bartlett is out now (WH Allen, £11.99). To support the Guardian, order your copy at guardianbookshop.com. Delivery charges may apply