AI-Designed Proteins Expose Biosecurity Vulnerabilities in DNA Screening

Biothreat hunters catch dangerous DNA before it gets made

DNA-synthesis firms routinely use -biosecurity-screening software to ensure that they don't inadvertently create dangerous sequences. But a paper published in Science on 2 October describes a potential vulnerability in this workflow. It details how protein-design strategies aided by artificial intelligence (AI) could circumvent the screening software that many DNA-synthesis firms use to ensure that they avoid unintentionally producing sequences encoding harmful proteins or pathogens. The researchers used an approach from the cybersecurity world: 'red teaming', in which one team attempts to break through another's defences (with their knowledge). They found that some screening tools were unprepared to catch AI-generated protein sequences that recreate the structure, but not the sequence, of known biothreats, says Eric Horvitz, chief scientific officer at Microsoft in Redmond, Washington. This is a type of zero-day vulnerability -- one that, in the cybersecurity world, blindsides software developers and users. "The diversified proteins essentially flew through the screening techniques" that were tested, Horvitz says. After the developers patched their software to address the new threat, the tools performed much better, flagging all but about 3% of malicious sequences in a larger second attempt. The impetus for the study is researchers' rapidly growing ability to create new, custom proteins. Armed with AI-powered tools such as RFdiffusion and ProteinMPNN, researchers can now invent proteins to attack tumours, defend against viruses and break down -pollutants. David Baker, a biochemist at the University of Washington in Seattle, whose team developed both RFdiffusion and -ProteinMPNN, won a share of the 2024 Nobel Prize in Chemistry for his pioneering work in this area. But biodesign tools could have other uses -- not all of them noble. Someone might intentionally or accidentally create a toxic compound or pathogen, putting many people at risk. The Microsoft-led project aims to prevent that possibility, focusing on a key checkpoint: synthesizing the DNA strands that encode these proteins. Researchers identified gaps in the screening of risky sequences and helped DNA-synthesis providers to close them. But as AI for protein design advances, defences, too, must evolve. Horvitz has long recognized that AI, like all technologies, has both good and bad applications. In 2023, motivated by concerns about potential misuse of AI-based protein design, he, Baker and others organized a workshop at the University of Washington to hammer out responsible practices. Horvitz asked Bruce Wittmann, an applied scientist at Microsoft, to create a concrete example of the threat. Proteins, built of amino acids, are the workhorses of the cell. They are first written in the language of DNA -- a string of nucleotides, denoted by A, C, G and T, whose order defines the sequence of amino acids. To create a protein, researchers specify the underlying nucleotide sequence, which they send to a DNA--synthesis company. The provider uses biosecurity screening software (BSS) to look for similarities between the new sequence and known sequences of concern -- genes that encode, say, a toxin. If nothing is flagged, the provider creates the requested DNA and mails it back. Horvitz and Wittmann wanted to see how porous such screening was. So, Wittmann adapted open-source AI protein-design software to alter the amino-acid sequence of a protein of concern while retaining its folded, 3D shape -- and, potentially, its function. It's the protein-design equivalent of paraphrasing a sentence, Wittmann says. The AI designed thousands of variants. Horvitz and Wittmann then reached out to two synthesis providers and asked them to use their BSS tools to test the sequences. One was Twist Bioscience in San Francisco, California, which used ThreatSeq from Battelle in Columbus, Ohio; the other was Integrated DNA Technologies (IDT) in Coralville, Iowa, which uses FAST-NA Scanner from RTX BBN Technologies in Cambridge, Massachusetts. The result: the tools were porous, indeed. Jacob Beal, a computer scientist at BBN, recalls a "moment of panic" looking at one of the tools: "Oh my goodness, this just goes straight through everything, like butter." Because the findings could have been dangerous in the wrong hands, the team began by sharing them with a small circle of people, including select workshop attendees; US government biosecurity officials; and James Diggans, the chair of the International Gene Synthesis Consortium (IGSC), a coalition of synthesis providers, formed in 2009 to create and share standards for screening both sequences and customers. "The results of the framing study were not a huge surprise," says Nicole Wheeler, a microbiologist then at the University of Birmingham, UK, and a co-author of the report. But "the study gave a clear indication of the scale of the problem today and data we could use to start testing and improving our screening tools". Horvitz and Wittmann then conducted a larger study. They started with 72 proteins of concern -- both toxins and viral proteins -- and generated tens of thousands of variants of the amino-acid sequences. As before, they ran the design software in two modes, one of which kept amino acids untouched at key locations. This mode increased the chance not only that the proteins would retain the functionality of the unaltered template proteins that they were emulating, but also that they'd be flagged by the BSS. Then, they reverse-translated the -amino-acid sequences into DNA, which they sent to four BSS providers who were in on the exercise. The team also scored the variant proteins for predicted risk. Proteins that exceeded a threshold on two measures were deemed dangerous. First, the proteins needed to be structurally similar (on the basis of computer simulations) to the template proteins. Second, the software needed to have high confidence in the predicted structure, indicating that the protein was likely to fold up properly and be functional. The researchers never actually made the toxic proteins, but in work posted to the preprint server bioRxiv in May, they synthesized some benign ones generated through their design method. They found that their metrics accurately predicted when a protein variant would maintain functionality, suggesting that at least some of the dangerous protein variants would have been functional, too. (But perhaps not many; most of the synthesized variants of benign proteins were inactive.) Overall, of the proteins that Horvitz and Wittmann deemed most dangerous, the patched BSS caught 97%, while keeping the false-positive rate under 2.5%. Diggans, who is also the head of biosecurity at Twist, says that the BSS tools that they use were patched in different ways during the Science study. In one case, developers used Wittmann's sequences to fine-tune a machine-learning model; in others, they lowered the statistical-significance threshold for similarity to cast "a wider net", now that they knew the degree to which AI could change sequences. Beal, at BBN, says that FAST-NA Scanner works differently. Before the red-teaming exercise, it looked for exact matches between short substrings of nucleotides and the sequences of genes encoding proteins of concern. After being patched, it scans for exact matches only at locations known to be important to a protein's functionality, allowing for harmless variation elsewhere. The company uses machine learning to generate diverse new sequences of concern, then identifies the important parts of their structures on the basis of similarities between those sequences. Some of the providers have since made further patches on the basis of this work. Horvitz and Wittmann teamed up with co-authors, including Wheeler, Diggans and Beal, to write up and share the results. Some colleagues felt the authors should provide every detail, whereas others said they should share nothing. "Our first reaction was, 'Anybody in the field would know how to do this kind of thing, wouldn't they?'" Horvitz says. "And even senior folks said, 'Well, that's not exactly true.' And so that went back and forth." In the end, they posted a version of their white paper on the preprint server bioRxiv in December, with key details removed. It doesn't describe the proteins they modified (the Science version of the paper lists them), the design tools they used or how they used them. It also omits a section on common BSS failures and glosses over obfuscation techniques -- ways to design sequences that won't raise flags but that produce DNA strands that can easily be modified after synthesis to become more dangerous. For the published version, the authors worked with journal editors to create a tiered system for data access. Parties must apply through the International Biosecurity and Biosafety Initiative for Science (IBBIS) in Geneva. (The highest-risk data tier includes the study's code.) "I'm really excited about this," Tessa Alexanian, a technical lead at IBBIS, said in a press briefing on 30 September. "This managed-access programme is an experiment, and we're very eager to evolve our approach." "There are two communities, which each have very well-grounded principles that both apply here and are in opposition to one another," Beal says. In the cybersecurity world, people often share vulnerabilities, so they can be patched widely; in biosecurity, threats are potentially deadly and difficult to counter, so people prefer to keep them under wraps. "Now we're in a place where these two worlds overlap." Even if screening tools work perfectly, bad actors could still design and build dangerous proteins. There are no laws requiring DNA-synthesis providers to screen orders, for instance. "That's a scary situation," says Jaime Yassif, who runs the global biosecurity programme at the Nuclear Threat Initiative (NTI), a non-profit organization in Washington DC. "Not only is screening not required, but the cost of DNA synthesis has been plummeting exponentially for years, and the cost of biosecurity has been basically fixed, so the profit margins on DNA synthesis are pretty thin." To maximize profit, companies could skimp on screening. In 2020, the NTI and the World Economic Forum organized a working group to make DNA-synthesis screening more accessible to synthesis firms. The NTI began building a BSS tool called the Common Mechanism, and last year it spun off of IBBIS, which now manages the tool. (Wheeler was the technical lead who developed it.) The Common Mechanism is free, open-source software that includes a database of concerning sequences and an algorithm that detects similarities between those sequences and submitted ones. Users can integrate more databases and analysis modules as they become available. Still, some scientists think that regulations are necessary. In 2010, the US Department of Health and Human Services issued guidelines recommending that providers of synthetic double-stranded DNAs screen both sequences and customers, but screening was voluntary. In 2023, former US president Joe Biden issued an executive order on AI safety that, among other things, required researchers who receive federal funding and order synthetic DNA to get it only from providers that screen the orders. The aim wasn't to stop federally funded researchers from becoming terrorists, Yassif says; it was to add another intervention point to safeguard well-intentioned research that might result in a lab leak or lead to published work that informs terrorists. In any case, President Donald Trump rescinded the order when he took office in January. (An executive order issued on 5 May that halts 'gain of function' pathogen research, also directs the Director of the Office of Science and Technology Policy to "revise or replace" the 2024 Framework for Nucleic Acid Synthesis Screening -- a product of the 2023 executive order -- to ensure that it "effectively encourages providers of synthetic nucleic acid sequences to implement comprehensive, scalable, and verifiable synthetic nucleic acid procurement screening mechanisms to minimize the risk of misuse".) Beyond DNA sequences themselves, Yassif says that regulators should look at protecting protein-design software and other biological AI models against misuse. "It's so important to get this right, and DNA-synthesis screening can't be the single point of failure." In 2023, the NTI released a report on AI in the life sciences, based on interviews with 30 specialists. It floated several ideas, including having protein-design AI models screen user requests, restricting the training data, requiring the evaluation of model safety and controlling model access. A correspondence in Nature Biotechnology earlier this year recommended similar safeguards. But regulations to protect biological AI models against misuse could be difficult to iron out, Yassif says, because people disagree on risks and rewards. Participants at the University of Washington workshop had a hard time agreeing on a community statement, notes Ian Haydon, the head of AI policy at the university's Institute for Protein Design (which Baker directs). "It's a document that's signed by scores of professors who famously can be a bit stubborn," Haydon says. "It's a bit like herding cats." As a result, its commitments are vague. The biggest area of contention, Haydon says, involved open-source software. "We had people unwilling to sign the language that we arrived at for opposing reasons," he says: some thought it was too supportive of openness, and others thought it was not supportive enough. The risks of sharing design tools are obvious. Sharing screening tools is also risky, because people who want to synthesize -dangerous sequences might work out where the blind spots are and potentially exploit them. The databases in IBBIS's Common Mechanism include well-known proteins of concern, but not some of the more obscure ones. One idea is to send a list of those proteins to approved recipients, but "invariably things will leak", Yassif says. "The challenge this community is facing is: how do we deal with the extra threats beyond the baseline that's publicly known in a way that doesn't create an info hazard?" she says. "That's an unsolved problem." Even if all synthesis providers did screening, there's a potential workaround: would-be bioterrorists can buy a synthesis device, although benchtop versions are error-prone and make relatively short segments of DNA (called oligonucleotides) that need to be pieced together. "Oligo synthesis is a bit of an art," Diggans says. But state-of-the-art technology is changing rapidly. In 2023, the NTI issued a report warning that benchtop synthesizers might be able to build complete viral genomes in as little as a decade. The report recommended regulation. One idea is to require benchtop machines to implement screening internally or over the cloud. But "if there's hardware and software, it can be hacked", says James Demmitt, chief executive of Biolytic Lab Performance, a biotech company in Fremont, California, that makes DNA-synthesis hardware. That said, defences don't need to be perfect to be effective. "I'm not aware of any solution that is 100% bulletproof," Yassif says. The aim is to "make it harder to exploit this technology to cause harm, shrink the number of people that have the capacity to actually exploit this and do something really dangerous, increase the odds of getting caught, increase the odds of failure. That's what success looks like." According to Demmitt, "biosecurity screening does a good job stopping accidental or casual misuse. By forcing folks to go through bigger, pricier hoops, it prevents many would-be dabblers from drifting into dangerous territory." And there are more technical hurdles facing bad actors. Rarely is DNA itself a danger; people need to engineer sequences into cells or viruses to manufacture toxins or produce self-replicating pathogens. That requires biological know-how and equipment beyond many people's means. Even for specialists, there's a huge gap between designing a protein or virus and knowing its effect in people. That's why pandemic prediction is so difficult. Scientists find viruses in the wild that seem dangerous, but few infect people, fewer spread between them and even fewer make them sick. Whatever the chances of someone designing something deadly, specialists say we should remain vigilant, just as in cybersecurity -- but the cat-and-mouse games are different in one regard. "In the cyber world, you have a lot of people looking to exploit these systems," Diggans says, "from the 'script kiddie' teenagers looking to do it for fun, all the way to the multinational crime syndicates." He continues: "It is vanishingly rare to have anyone who wants to exploit biotechnology for nefarious purposes. That is both good -- because we don't want people exploiting biotech -- but it is also hard, because it gives us very few signals against which to build defences." In March, the US National Academies of Sciences, Engineering, and Medicine recommended that more research into methodologies for nucleic-acid-synthesis screening is needed. As that happens, the field can continue to take lessons from cybersecurity specialists, who have been going toe-to-toe with bad actors for decades. "What stands out to me" in the new paper, Haydon says, "is the way they wove in practices and precedents from cybersecurity", such as letting providers build patches before publicizing their findings. As the field develops, providers will need to keep upping their game. As a "Microsoft person", Horvitz is reminded of the Windows update model. "This will never be ending," he says.

Do AI-designed proteins create a biosecurity vulnerability?

On Thursday, a team of researchers led by Microsoft announced that they had discovered, and possibly patched, what they're terming a biological zero-day -- an unrecognized security hole in a system that protects us from biological threats. The system at risk screens purchases of DNA sequences to determine when someone's ordering DNA that encodes a toxin or dangerous virus. But, the researchers argue, it has become increasingly vulnerable to missing a new threat: AI-designed toxins. How big of a threat is this? To understand, you have to know a bit more about both existing biosurveillance programs and the capabilities of AI-designed proteins. Catching the bad ones Biological threats come in a variety of forms. Some are pathogens, such as viruses and bacteria. Others are protein-based toxins, like the ricin that was sent to the White House in 2003. Still others are chemical toxins that are produced through enzymatic reactions, like the molecules associated with red tide. All of them get their start through the same fundamental biological process: DNA is transcribed into RNA, which is then used to make proteins. For several decades now, starting the process has been as easy as ordering the needed DNA sequence online from any of a number of companies, which will synthesize a requested sequence and ship it out. Recognizing the potential threat here, governments and industry have worked together to add a screening step to every order: the DNA sequence is scanned for its ability to encode parts of proteins or viruses considered threats. Any positives are then flagged for human intervention to evaluate whether they or the people ordering them truly represent a danger. Both the list of proteins and the sophistication of the scanning have been continually updated in response to research progress over the years. For example, initial screening was done based on similarity to target DNA sequences. But there are many DNA sequences that can encode the same protein, so the screening algorithms have been adjusted accordingly, recognizing all the DNA variants that pose an identical threat. The new work can be thought of as an extension of that threat. Not only can multiple DNA sequences encode the same protein; multiple proteins can perform the same function. To form a toxin, for example, typically requires the protein to adopt the correct three-dimensional structure, which brings a handful of critical amino acids within the protein into close proximity. Outside of those critical amino acids, however, things can often be quite flexible. Some amino acids may not matter at all; other locations in the protein could work with any positively charged amino acid, or any hydrophobic one. In the past, it could be extremely difficult (meaning time consuming and expensive) to do the experiments that would tell you what sorts of changes a string of amino acids could tolerate while remaining functional. But the team behind the new analysis recognized that AI protein design tools have now gotten quite sophisticated and can predict when distantly related sequences can fold up into the same shape and catalyze the same reactions. The process is still error prone, and you often have to test a dozen or more proposed proteins to get a working one, but it has produced some impressive successes. So, the team developed a hypothesis to test: AI can take an existing toxin and design a protein with the same function that's distantly related enough that the screening programs do not detect orders for the DNA that encodes it. The zero-day treatment The team started with a basic test: use AI tools to design variants of the toxin ricin, then test them against the software that is used to screen DNA orders. The results of the test suggested there was a risk of dangerous protein variants slipping past existing screening software, so the situation was treated like the equivalent of a zero-day vulnerability. "Taking inspiration from established cybersecurity processes for addressing such situations, we contacted the relevant bodies regarding the potential vulnerability, including the International Gene Synthesis Consortium and trusted colleagues in the protein design community as well as leads in biosecurity at the US Office of Science and Technology Policy, US National Institute of Standards and Technologies, US Department of Homeland Security, and US Office of Pandemic Preparedness and Response," the authors report. "Outside of those bodies, details were kept confidential until a more comprehensive study could be performed in pursuit of potential mitigations and for 'patches'... to be developed and deployed." Details of that original test are being made available today as part of a much larger analysis that extends the approach to a large range of toxic proteins. Starting with 72 toxins, the researchers used three open source AI packages to generate a total of about 75,000 potential protein variants. And this is where things get a little complicated. Many of the AI-designed protein variants are going to end up being non-functional, either subtly or catastrophically failing to fold up into the correct configuration to create an active toxin. The only way to know which ones work is to make the proteins and test them biologically; most AI protein design efforts will make actual proteins from dozens to hundreds of the most promising-looking potential designs to find a handful that are active. But doing that for 75,000 designs is completely unrealistic. Instead, the researchers used two software-based tools to evaluate each of the 75,000 designs. One of these focuses on the similarity between the overall predicted physical structure of the proteins, and another looks at the predicted differences between the positions of individual amino acids. Either way, they're a rough approximation of just how similar the proteins formed by two strings of amino acids should be. But they're definitely not a clear indicator of whether those two proteins would be equally functional. In any case, DNA sequences encoding all 75,000 designs were fed into the software that screens DNA orders for potential threats. One thing that was very clear is that there were huge variations in the ability of the four screening programs to flag these variant designs as threatening. Two of them seemed to do a pretty good job, one was mixed, and another let most of them through. Three of the software packages were updated in response to this performance, which significantly improved their ability to pick out variants. There was also a clear trend in all four of the screening packages: the closer the variant was to the original structurally, the more likely the package (both before and after the patches) was to be able to flag it as a threat. In all cases, there was also a cluster of variant designs that were unlikely to fold up into a similar structure, and these generally weren't flagged as threats. What's this mean? Again, it's important to emphasize that this evaluation is based on predicted structures; "unlikely" to fold into a similar structure to the original toxin doesn't mean these proteins will be inactive as toxins. Functional proteins are probably going to be very rare among this group, but there may be a handful in there. That handful is also probably rare enough that you'd have to order up and test far too many designs to find one that works, making this an impractical threat vector. At the same time, there are also a handful of proteins that are very similar to the toxin structurally and not flagged by the software. For the three patched versions of the software, the ones that slip through the screening represent about 1 to 3 percent of the total in the "very similar" category. That's not great, but it's probably good enough that any group that tries to order up a toxin by this method would attract attention because they'd have to order over 50 just to have a good chance of finding one that slipped through, which would raise all sorts of red flags. One other notable result is that the designs that weren't flagged were mostly variants of just a handful of toxin proteins. So this is less of a general problem with the screening software and might be more of a small set of focused problems. Of note, one of the proteins that produced a lot of unflagged variants isn't toxic itself; instead, it's a co-factor necessary for the actual toxin to do its thing. As such, some of the screening software packages didn't even flag the original protein as dangerous, much less any of its variants. (For these reasons, the company that makes one of the better-performing software packages decided the threat here wasn't significant enough to merit a security patch.) So, on its own, this work doesn't seem to have identified something that's a major threat at the moment. But it's probably useful, in that it's a good thing to get the people who engineer the screening software to start thinking about emerging threats. That's because, as the people behind this work note, AI protein design is still in its early stages, and we're likely to see considerable improvements. And there's likely to be a limit to the sorts of things we can screen for. We're already at the point where AI protein design tools can be used to create proteins that have entirely novel functions and do so without starting with variants of existing proteins. In other words, we can design proteins that are impossible to screen for based on similarity to known threats, because they don't look at all like anything we know is dangerous. Protein-based toxins would be very difficult to design, because they have to both cross the cell membrane and then do something dangerous once inside. While AI tools are probably unable to design something that sophisticated at the moment, I would be very hesitant to rule out the prospects of them eventually reaching that sort of sophistication. Science, 2025. DOI: 10.1126/science.adu8578 (About DOIs).

Made to order bioweapon? AI-designed toxins slip through safety checks used by companies selling genes

Microsoft bioengineer Bruce Wittmann normally uses artificial intelligence (AI) to design proteins that could help fight disease or grow food. But last year, he used AI tools like a would-be bioterrorist: creating digital blueprints for proteins that could mimic deadly poisons and toxins such as ricin, botulinum, and Shiga. Wittmann and his Microsoft colleagues wanted to know what would happen if they ordered the DNA sequences that code for these proteins from companies that synthesize nucleic acids. Borrowing a military term, the researchers called it a "red team" exercise, looking for weaknesses in biosecurity practices in the protein engineering pipeline. The effort grew into a collaboration with many biosecurity experts, and according to their new paper, published today in Science, one key guardrail failed. DNA vendors typically use screening software to flag sequences that might be used to cause harm. But the researchers report that this software failed to catch many of their AI-designed genes -- one tool missed more than 75% of the potential toxins. Scientists involved in the exercise kept these vulnerabilities secret until the screening software was upgraded -- but even now, it's not foolproof, they warn. Jaime Yassif, vice president for global biological policy and programs at the Nuclear Threat Initiative, says the study is a model for the future. "It's just the beginning," she says. "AI capabilities are going to evolve and be able to design more and more complex living systems, and our DNA synthesis screening capabilities are going to have to continue to evolve to keep up with that." In their experiment, the Microsoft researchers selected 72 different proteins that are subject to legal controls, such as ricin, a bacterial toxin already used in several terrorist attacks. Traces of ricin were detected on letters sent to top U.S. officials in 2013 and 2018. Using specialized AI protein design tools, Wittmann came up with more than 70,000 DNA sequences that would generate variant forms of these proteins. Computer models suggested that at least some of these alternatives would also be toxic. "The knowledge that I had access to, and stewardship over these proteins was, on a human level, a notable burden," Wittmann says. Wittmann didn't actually create the proteins or test them for toxicity; that would have required ordering the genes from DNA vendors and inserting them into bacteria or yeast to produce the proteins of interest. And doing so could be considered a violation of the Biological Weapons Convention, which bans development or production of such weapons. Instead, he asked four suppliers of biosecurity screening systems used by DNA synthesis labs to run these sequences through their software. The tools failed to flag many of these sequences as problematic. Their performance varied widely. One tool flagged just 23% of the sequences. One of the screening tools flagged 70% of the sequences, and its developer chose not to make any changes to improve the software. (A sensitive screen that catches every potentially hazardous sequence would likely also flag innocuous ones, creating headaches and raising costs.) The other software suppliers rolled out upgrades. The whole process took "a few months," Wittmann says. "We were all very quiet about it," Yassif says. "It was a good example of the community being very responsible." After the upgrades, the systems flagged 72% of Wittmann's AI-generated sequences, on average, including 97% of the sequences that models rated most likely to generate toxins. The study's authors, with the consent of Science, are withholding some details about the AI-generated DNA sequences and the industry's software screening fixes. There will be "managed access" to this information, says Tessa Alexanian of the International Biosecurity and Biosafety Initiative for Science (IBBIS), a nonprofit group that developed one of the four screening software systems. Its experts will review requests for access to the information. Screening software isn't the only biosecurity guardrail that needs strengthening, Yassif says. Some DNA vendors, accounting for perhaps 20% of the market, don't screen their orders at all, she notes. She also argues that additional safeguards should be built into AI protein design tools themselves. There's little indication, so far, that rogue actors are trying to acquire illicit synthetic DNA. "I've been doing this for 10 years, and the number of times we've had to refer an issue to law enforcement -- I have more fingers on one hand," says James Diggans, vice president of policy and biosecurity at Twist Bioscience, a DNA synthesis company. "The real number of people who are trying to create misuse may be very close to zero." Drew Endy, a synthetic biology researcher at Stanford University, says improving the screening software is fine, but it consumes too much attention compared to a much bigger biosecurity risk: nations' possible operation of clandestine bioweapons programs. "I wish people would wake up a little bit," he says. "Today, nations are accusing one another of having offensive bioweapons programs. We accuse Russian and North Korea. China and Russia accuse the United States. This is the historical pattern that happened 100 years ago that led to actual bioweapons programs. We have to de-escalate this."

Microsoft says AI can create "zero day" threats in biology

"The patch is incomplete, and the state of the art is changing. But this isn't a one-and-done thing. It's the start of even more testing," says Adam Clore, director of technology R&D at Integrated DNA Technologies, a large manufacturer of DNA, who is a coauthor on the Microsoft report. "We're in something of an arms race." To make sure nobody misuses the research, the researchers say, they're not disclosing some of their code and didn't reveal what toxic proteins they asked the AI to redesign. However, some dangerous proteins are well known, like ricin -- a poison found in castor beans -- and the infectious prions that are the cause of mad-cow disease. "This finding, combined with rapid advances in AI-enabled biological modeling, demonstrates the clear and urgent need for enhanced nucleic acid synthesis screening procedures coupled with a reliable enforcement and verification mechanism," says Dean Ball, a fellow at the Foundation for American Innovation, a think tank in San Francisco. Ball notes that the US government already considers screening of DNA orders a key line of security. Last May, in an executive order on biological research safety, President Trump called for an overall revamp of that system, although so far the White House hasn't released new recommendations. Others doubt that commercial DNA synthesis is the best point of defense against bad actors. Michael Cohen, an AI-safety researcher at the University of California, Berkeley, believes there will always be ways to disguise sequences and that Microsoft could have made its test harder. "The challenge appears weak, and their patched tools fail a lot," says Cohen. "There seems to be an unwillingness to admit that sometime soon, we're going to have to retreat from this supposed choke point, so we should start looking around for ground that we can actually hold." Cohen says biosecurity should probably be built into the AI systems themselves -- either directly or via controls over what information they give. But Clore says monitoring gene synthesis is still a practical approach to detecting biothreats, since the manufacture of DNA in the US is dominated by a few companies that work closely with the government. By contrast, the technology used to build and train AI models is more widespread. "You can't put that genie back in the bottle," says Clore. "If you have the resources to try to trick us into making a DNA sequence, you can probably train a large language model."

Should we worry AI will create deadly bioweapons? Not yet, but one day

AI tools are being used to design proteins and even viruses, leading to fears these could eventually be used to evade bioweapon controls Artificial intelligence promises to transform biology, allowing us to design better drugs, vaccines and even synthetic organisms for, say, eating waste plastic. But some fear it could also be used for darker purposes, to create bioweapons that wouldn't be detected by conventional methods until it was too late. So, how worried should we be? "AI advances are fuelling breakthroughs in biology and medicine," says Eric Horvitz, chief scientific officer at Microsoft. "With new power comes responsibility for vigilance." His team has published a study looking at whether AI could design proteins that do the same thing as proteins that are known to be dangerous, but are different enough that they wouldn't be recognised as dangerous. The team didn't reveal which proteins they attempted to redesign - parts of the study were withheld - but it probably included toxins such as ricin, famously used in a 1978 assassination, and botulinum, the potent neurotoxin better known as Botox. To make lots of a protein like botulinum, you need the recipe - the DNA that codes for it. When biologists want a specific piece of DNA, they usually order it from companies that specialise in making any desired piece. Due to concerns that would-be bioterrorists could order the recipes for making bioweapons this way, some DNA-synthesis companies voluntarily screen orders to check if someone is trying to make something dangerous. Proteins are sequences of amino acids, and the screening checks whether the amino acid sequence matches any "sequences of concern" - that is, potential bioweapons. But with AI, it is in theory possible to design a version of a protein that has a different amino acid sequence but still does the same thing. Horvitz and his colleagues attempted this with 72 potentially dangerous proteins and showed that screening methods often miss these alternative versions. This isn't as alarming as it sounds. Firstly, the team didn't actually make the redesigned proteins, for obvious reasons. But in a separate study earlier this year, they tested redesigned versions of harmless proteins - and basically found they didn't work. Secondly, while there have been attempted bioterrorist attacks, albeit very few, there is little reason to think this is because of a failing of the voluntary scanning system. There are already many ways to get around it without resorting to AI redesigns - for instance, ricin can be obtained from castor oil plants, found in many gardens. This study is the equivalent of warning that a bank could be robbed by some highly sophisticated Mission Impossible-style plan, when in fact the vault door has been left wide open. Last but not least, when state actors are excluded, no bioterrorist has ever managed to kill anyone using protein-based bioweapons. The Aum Shinrikyo cult in Japan tried to kill people with botulinum, but succeeded only with chemical agents. The ricin-laced letters sent to the White House didn't kill anyone. Based on body counts, guns and explosives are wildly more dangerous than biotoxins. So does that mean we stop worrying about AI-designed bioweapons? Not quite. While Horvitz's studies looked only at proteins, it is viruses that pose the big threat - and AI is already being used to redesign entire viruses. Last month, a team at Stanford University in California revealed the results of their efforts to redesign a virus that infects the bacterium E. coli. As with the redesigned proteins, the results were unimpressive - of the 302 AI-designed viruses that were made, just 16 could infect E. coli. But this is just the start. When asked about AI-designed viruses, James Diggans at the DNA-making firm Twist Bioscience, and a member of Horvitz's team, said it is easier to detect DNA-encoding viruses of concern than proteins of concern. "Synthesis screening operates better on more information rather than less. So at the genome scale, it's incredibly informative." But not all DNA-making companies carry out this screening, and benchtop DNA synthesisers are becoming available. There is talk of designing AI tools that will refuse to create dangerous viruses or try to detect malevolent intent, but people have found many ways to get around safeguards meant, for instance, to stop AIs providing bomb-making instructions. To be clear, history suggests the risk from "wild" viruses is way higher than the risk from bioterrorism. Despite what the current US administration claims, the evidence suggests that SARS-CoV-2 emerged when a bat virus jumped to other wild animals, and then to people at a market - no lab involved. What's more, would-be bioterrorists could do an incredible amount of damage simply by releasing a known virus, such as smallpox. With the many gaping holes in bioweapon control efforts, there is little need to resort to AI trickery to get around them. For all these reasons, the risk of an AI-designed virus being unleashed anytime soon is probably near zero. But this risk is going to grow as the various technologies continue to advance - and the covid-19 pandemic showed just how much havoc a new virus can create, even when it isn't especially deadly. Increasingly, there will be reason to worry.

AI-designed proteins test biosecurity safeguards

New fixes to monitoring software boosts its ability to catch AI-altered toxic proteins New patches to biosecurity screening software can make it harder to produce potentially harmful proteins using artificial intelligence. Around the world, this software monitors processes to artificially make proteins, ensuring that people with bad intentions aren't producing dangerous proteins, such as toxins. Making slight tweaks with AI to known toxins or viral proteins can bypass the safeguards, researchers report in the Oct. 2 Science. But reinforcing gaps in screening can boost the programs' ability to flag risky AI-designed proteins. "AI advances are fueling breakthroughs in biology and medicine," Eric Horvitz, chief scientific officer at Microsoft in Redmond, Wash., said at a Sept. 30 news briefing. "Yet with new power comes responsibility for vigilance and thoughtful risk management." Proteins are the workhorses of biology. The molecules perform cellular tasks such as assembling cells and transporting cargo throughout the body. With AI, researchers are unlocking ways to fine-tune existing proteins to carry out specific tasks, to design new proteins or to generate new organisms. AI can generate digital blueprints for proteins by determining the amino acids needed to make them, but the technology can't construct physical proteins from thin air. DNA manufacturers string together the appropriate genetic letters and ship the synthetic genes to research labs. Computer programs screen the orders to make sure that the genes don't make hazardous proteins. Horvitz and colleagues simulated tests for biosecurity screening models to find weaknesses that could let AI-generated proteins slip by filters. The team generated roughly 76,000 blueprints for 72 harmful proteins, including ricin, botulinum neurotoxin and ones that help viruses infect people. While the biosecurity screens flagged the DNA for nearly all proteins in their original forms, many AI-adjusted versions snuck through. Software patches helped, even picking up genes after they'd been broken down into fragments. The models failed to flag about 3 percent of variants. The work was done entirely on computers, meaning that the team did not make physical proteins in the lab, and it's unclear if the AI-generated variants retained their function. In reality, biosecurity screens flagging orders for concerning proteins "is an incredibly rare thing," James Diggans, vice president of policy and biosecurity at Twist Bioscience, a DNA synthesis company based in San Francisco, said at the news briefing. While cybersecurity threats happen all the time, "close to zero" people have tried to produce malicious proteins, Diggans said. "These systems are an important bulwark against [threats], but we should all find comfort in the fact that this is not a common scenario."

AI bioweapon risk laid bare by protein security screening flaw

Bioterrorism threats are rising because of advances in artificial intelligence and synthetic biology, scientists have warned, after researchers found a "striking vulnerability" in software that guards access to genetic material for making deadly proteins. An international team rolled out patches to close the loophole but said it was the first "zero day" of AI and biosecurity -- a term used in cyber hacking to describe a blind spot unknown to the software developer. The news highlights the growing urgency to deal with potential threats unleashed by the use of AI as it helps deepen and accelerate the understanding of living systems and how to change them. Experts are seeking to prevent the creation of bioweapons and synthetic organisms that could threaten life on Earth. "AI-powered protein design is one of the most exciting frontiers of science [and] we're already seeing advances in medicine and public health," said Eric Horvitz, Microsoft's chief scientific officer and senior author of the latest research, published in Science on Thursday. "Yet, like many powerful technologies, these same tools can also be misused." The Science paper researchers carried out a test on biosecurity software used to screen customer orders by companies that sell synthetic nucleic acids. These are deployed by the clients to build DNA that instructs the manufacture of desired proteins, the building blocks of life. The biosecurity screening is designed to block the sale of materials that could be used to make harmful proteins. The researchers used open-source AI protein design software to generate computational renderings of more than 75,000 variants of dangerous proteins with structural tweaks -- a kind of biochemical disguise. While the screening tools worked well for flagging naturally-occurring proteins of concern, they did not spot some of the altered ones, the scientists found. Even after all but one of the companies applied the software patches, about 3 per cent of the protein variants most likely to retain hazardous functionality still passed the monitoring undetected. The scientists worked with organisations including the International Gene Synthesis Consortium and US authorities to address the problem. The research comes after some leading scientists have called for a systematic assessment of biosecurity screening software and improved global governance of AI-boosted protein synthesis. High-profile biologists are also pushing for an international agreement to prevent the creation of potentially deadly manufactured "mirror" microbes, should it become technologically possible to make them. Horvitz said there had been an "intensity of reflection, study and methodology" about the prospect that large language models could be used to further "malevolent actions with biology". Microsoft had incorporated such possibilities in its product safety reviews and had a "growing set of practices" about "red-teaming", or searching for potential vulnerabilities. The Science study highlighted a "pressing issue in protein engineering and biosafety", said Francesco Aprile, associate professor in biological chemistry at Imperial College London. "By introducing targeted improvements to existing software, the authors significantly enhance detection and flagging," Aprile said. "This work provides a practical, timely safeguard that strengthens current DNA synthesis screening, and establishes a solid foundation for continued optimisation." Those defences must be strengthened soon because of the fast pace of technical improvements in the field, said Natalio Krasnogor, professor of computing science and synthetic biology at Newcastle University. While the aspiring bioterrorists of today would need significant expertise, time and money to actually make harmful proteins, those barriers were likely to shrink. "We do need as a society take this seriously now," Krasnogor said, "before additional advances in AI make the validation and experimental production of viable synthetic toxins much easier and cheaper to deploy than it is today."

AI can now be used to design brand-new viruses. Can we stop it from making the next devastating bioweapon?

As scientists design new viruses using AI, experts are investigating whether current biosecurity measures can hold up to this potential new threat. (Image credit: Andriy Onufriyenko via Getty Images) Scientists have used artificial intelligence (AI) to build brand-new viruses, opening the door to AI-designed forms of life. The viruses are different enough from existing strains to potentially qualify as new species. They are bacteriophages, which means they attack bacteria, not humans, and the authors of the study took steps to ensure their models couldn't design viruses capable of infecting people, animals or plants. However, this week, a different group of scientists showed that AI can easily circumvent existing safety measures used to prevent bioweapon development. In the study, published Thursday (Oct. 2) in the journal Science, researchers from Microsoft revealed that AI can get around safety measures that would otherwise prevent bad actors from ordering toxic molecules from supply companies, for instance. After uncovering this vulnerability, the research team rushed to create software patches that greatly reduce the risk. This software currently requires specialized expertise and access to particular tools that most people in the public can't use. Combined, the new studies highlight the risk that AI could design a new lifeform or bioweapon that poses a threat to humans -- potentially unleashing a pandemic, in a worst-case scenario. So far, AI doesn't have that capability. But experts say that a future where it does isn't so far off. To prevent AI from posing a danger, experts say, we need to build multi-layer safety systems, with better screening tools and evolving regulations governing AI-driven biological synthesis. At the heart of the issue with AI-designed viruses, proteins and other biological products is what's known as the "dual-use problem." This refers to any technology or research that could have benefits, but could also be used to intentionally cause harm. A scientist studying infectious diseases might want to genetically modify a virus to learn what makes it more transmissable. But someone aiming to spark the next pandemic could use that same research to engineer a perfect pathogen. Research on aerosol drug delivery can help people with asthma by leading to more effective inhalers, but the designs might also be used to deliver chemical weapons. Stanford doctoral student Sam King and his supervisor Brian Hie, an assistant professor of chemical engineering, were aware of this double-edged sword. They wanted to build brand-new bacteriophages -- or "phages," for short -- that could hunt down and kill bacteria in infected patients. Their efforts were described in a preprint uploaded to the bioRxiv database in September, and they have not yet been peer reviewed. Phages prey on bacteria, and bacteriophages that scientists have sampled from the environment and cultivated in the lab are already being tested as potential add-ons or alternatives to antibiotics. This could help solve the problem of antibiotic resistance and save lives. But phages are viruses, and some viruses are dangerous to humans, raising the theoretical possibility that the team could inadvertently create a virus that could harm people. The researchers anticipated this risk and tried to reduce it by ensuring that their AI models were not trained on viruses that infect humans or any other eukaryotes -- the domain of life that includes plants, animals, and everything that's not a bacteria or archaea. They tested the models to make sure they couldn't independently come up with viruses similar to those known to infect plants or animals. With safeguards in place, they asked the AI to model its designs on a phage already widely used in laboratory studies. Anyone looking to build a deadly virus would likely have an easier time using older methods that have been around for longer, King said. "The state of this method right now is that it's quite challenging and requires a lot of expertise and time," King told Live Science. "We feel that this doesn't currently lower the barrier to any more dangerous applications." But in a rapidly evolving field, such precautionary measures are being invented on the go, and it's not yet clear what safety standards will ultimately be sufficient. Researchers say the regulations will need to balance the risks of AI-enabled biology with the benefits. What's more, researchers will have to anticipate how AI models may weasel around the obstacles placed in front of them. "These models are smart," said Tina Hernandez-Boussard, a professor of medicine at the Stanford University School of Medicine, who consulted on safety for the AI models on viral sequence benchmarks used in the new preprint study. "You have to remember that these models are built to have the highest performance, so once they're given training data, they can override safeguards." Thinking carefully about what to include and exclude from the AI's training data is a foundational consideration that can head off a lot of security problems down the road, she said. In the phage study, the researchers withheld data on viruses that infect eukaryotes from the model. They also ran tests to ensure the models couldn't independently figure out genetic sequences that would make their bacteriophages dangerous to humans -- and the models didn't. Another thread in the AI safety net involves the translation of the AI's design -- a string of genetic instructions -- into an actual protein, virus, or other functional biological product. Many leading biotech supply companies use software to ensure that their customers aren't ordering toxic molecules, though employing this screening is voluntary. But in their new study, Microsoft researchers Eric Horvitz, the company's chief science officer, and Bruce Wittman, a senior applied scientist, found that existing screening software could be fooled by AI designs. These programs compare genetic sequences in an order to genetic sequences known to produce toxic proteins. But AI can generate very different genetic sequences that are likely to code for the same toxic function. As such, these AI-generated sequences don't necessarily raise a red flag to the software. The researchers borrowed a process from cybersecurity to alert trusted experts and professional organizations to this problem and launched a collaboration to patch the software. "Months later, patches were rolled out globally to strengthen biosecurity screening," Horvitz said at a Sept. 30 press conference. These patches reduced the risk, though across four commonly used screening tools, an average of 3% of potentially dangerous gene sequences still slipped through, Horvitz and colleagues reported. The researchers had to consider security even in publishing their research. Scientific papers are meant to be replicable, meaning other researchers have enough information to check the findings. But publishing all of the data about sequences and software could clue bad actors into ways to circumvent the security patches. "There was an obvious tension in the air among peer reviewers about, 'How do we do this?'" Horvitz said. The team ultimately landed on a tiered access system in which researchers wanting to see the sensitive data will apply to the International Biosecurity and Biosafety Initiative for Science (IBBIS), which will act as a neutral third party to evaluate the request. Microsoft has created an endowment to pay for this service and to host the data. It's the first time that a top science journal has endorsed such a method of sharing data, said Tessa Alexanian, the technical lead at Common Mechanism, a genetic sequence screening tool provided by IBBIS. "This managed access program is an experiment and we're very eager to evolve our approach," she said. There is not yet much regulation around AI tools. Screenings like the ones studied in the new Science paper are voluntary. And there are devices that can build proteins right in the lab, no third party required -- so a bad actor could use AI to design dangerous molecules and create them without gatekeepers. There is, however, growing guidance around biosecurity from professional consortiums and governments alike. For example, a 2023 presidential executive order in the U.S. calls for a focus on safety, including "robust, reliable, repeatable, and standardized evaluations of AI systems" and policies and institutions to mitigate risk. The Trump Administration is working on a framework that will limit federal research and development funds for companies that don't do safety screenings, Diggans said. "We've seen more policymakers interested in adopting incentives for screening," Alexanian said. In the United Kingdom, a state-backed organization called the AI Security Institute aims to foster policies and standards to mitigate the risk from AI. The organization is funding research projects focused on safety and risk mitigation, including defending AI systems against misuse, defending against third-party attacks (such as injecting corrupted data into AI training systems), and searching for ways to prevent public, open-use models from being used for harmful ends. The good news is that, as AI-designed genetic sequences become more complex, that actually gives screening tools more information to work with. That means that whole-genome designs, like King and Hie's bacteriophages, would be fairly easy to screen for potential dangers. "In general, synthesis screening operates better on more information than less," Diggans said. "So at the genome scale, it's incredibly informative." Microsoft is collaborating with government agencies on ways to use AI to detect AI malfeasance. For instance, Horvitz said, the company is looking for ways to sift through large amounts of sewage and air-quality data to find evidence of the manufacture of dangerous toxins, proteins or viruses. "I think we'll see screening moving outside of that single site of nucleic acid [DNA] synthesis and across the whole ecosystem," Alexanian said. And while AI could theoretically design a brand-new genome for a new species of bacteria, archaea or more complex organism, there is currently no easy way for AI to translate those AI instructions into a living organism in the lab, King said. Threats from AI-designed life aren't immediate, but they're not impossibly far off. Given the new horizons AI is likely to reveal in the near future, there's a need to get creative across the field, Hernandez-Boussard said. "There's a role for funders, for publishers, for industry, for academics," she said, "for, really, this multidisciplinary community to require these safety evaluations."

How AI is making it easier to design new toxins without being detected

In October 2023, two scientists at Microsoft discovered a startling vulnerability in a safety net intended to prevent bad actors from using artificial intelligence tools to concoct hazardous proteins for warfare or terrorism. Those gaping security holes and how they were discovered were kept confidential until Thursday, when a report in the journal Science detailed how researchers generated thousands of AI-engineered versions of 72 toxins that escaped detection. The research team, a group of leading industry scientists and biosecurity experts, designed a patch to fix this problem found in four different screening methods. But they warn that experts will have to keep searching for future breaches in this safety net. "This is like a Windows update model for the planet. We will continue to stay on it and send out patches as needed, and also define the research processes and best practices moving forward to stay ahead of the curve as best we can," Eric Horvitz, chief scientific officer of Microsoft and one of the leaders of the work, said at a press briefing. The team considered the incident the first AI and biosecurity "zero day" -- borrowing a term from the cyber-world for defense gaps software developers don't know about that leave them susceptible to an attack. In recent years, researchers have been using AI to design bespoke proteins. The work has opened up vast potential across many fields of science. With these tools, scientists can create proteins to degrade plastic pollution, fight disease or make crops more resilient. But with possibility comes risk. That's why in October 2023, the Microsoft scientists embarked on an initial "adversarial" pilot study, in advance of a protein engineering biosecurity conference. The researchers never manufactured any of the proteins but created digital versions as part of the study. Outside biosecurity experts applauded the study and the patch, but said that this is not an area where one single approach to biosecurity is sufficient. "What's happening with AI-related science is that the front edge of the technology is accelerating much faster than the back end ... in managing the risks," said David Relman, a microbiologist at Stanford University School of Medicine. "It's not just that we have a gap -- we have a rapidly widening gap, as we speak. Every minute we sit here talking about what we need to do about the things that were just released, we're already getting further behind." How toxic ricin was a test case for detection Proteins are the building blocks of life -- strings of amino acids that perform crucial functions in cells. They can build muscles, fend off pathogens and carry out chemical reactions necessary for life. Proteins can be spelled out as a sequence of letters, but they fold and twist into 3D shapes. Their form is key to their function. Predicting the structure of proteins was, for decades, a major challenge in science. The winners of last year's chemistry Nobel Prize shared the award for work that allowed scientists to predict protein structure and use AI to custom design proteins with different shapes and functions. Those functions can be positive -- biosensors to detect environmental toxins or used to diagnose a disease. They can also be harmful. As their test case, Horvitz and his collaborator Bruce Wittmann used AI tools to initially "paraphrase" parts of the code of ricin, a deadly poison naturally found in castor beans. In digital form, they created tens of thousands of AI-generated proteins that were spelled differently than the original, but would probably still be toxic. Translating these digital concepts into real-life proteins relies on DNA synthesis companies, which create strands of DNA that scientists can study in the lab and use to generate the protein of interest. The industry standard is for DNA synthesis companies to deploy biosecurity software designed to guard against nefarious activity by flagging proteins of concern -- for example, known toxins or components of pathogens. When the researchers tested two major companies' biosecurity screening techniques, they found that "up to 100 percent" of the AI-generated ricin-like proteins evaded detection. Because the new proteins no longer looked like ricin, they were not flagged. Once they discovered this vulnerability, Horvitz and Wittmann brought in more collaborators and expanded their research to dozens of toxins and components of viruses. Again, they used AI techniques to "paraphrase" parts of their code while retaining their harmful structure, creating more than 70,000 synthetic versions. Screening programs were good at screening out the original toxins, but let thousands of the new versions slip by. Once the researchers discovered the scale of the problem, they devised a patch. "This is a really valuable study, in that it shows there is a problem -- and that it shows AI is going to change the nature of the problem. But it's not an insoluble problem," said Tom Inglesby, director of the Johns Hopkins Center for Health Security at the Bloomberg School of Public Health, who was not involved in the work. Evolving regulatory landscape Under a federal framework that is being updated, researchers who receive federal funding are required to place orders with DNA synthesis companies that use biosecurity screening software. What worries many biosecurity experts is how the system still largely relies on voluntary compliance, and many gaps could allow people to make AI-designed toxins without anyone noticing. Not only can the screening software itself be tricked, as shown in the new study, but not all companies deploy the software. Another challenge is that not all synthesis occurs at large companies. Benchtop devices can be used to synthesize short strands of DNA, and these could be patched together to create proteins. And more fundamentally, while some proteins are toxic because they are similar to existing ones, people could also design entirely new kinds of toxins that could escape notice. A different approach, biosecurity experts say, is to ensure AI software itself is imbued with safeguards before digital ideas are at the cusp of being brought into labs for research and experimentation. Tessa Alexanian, a biosecurity expert at the International Biosecurity and Biosafety Initiative for Science, a Swiss nonprofit, said that 180 AI developers signed a series of commitments last year, including a vow to support the development of new strategies to add biosecurity screening earlier in the process, before proteins are being made. Some think the clearest path forward is a registry to deter bad actors. "The only surefire way to avoid problems is to log all DNA synthesis, so if there is a worrisome new virus or other biological agent, the sequence can be cross-referenced with the logged DNA database to see where it came from," David Baker, who shared the Nobel Prize in chemistry for his work on proteins, said in an email.

AI designs for dangerous DNA can slip past biosecurity measures, study shows

Major biotech companies that churn out made-to-order DNA for scientists have protections in place to keep dangerous biological material out of the hands of would-be evil-doers. They screen their orders to catch anyone trying to buy, say, smallpox or anthrax genes. But now, a new study in the journal Science has demonstrated how AI could be used to easily circumvent those biosafety processes. A team of AI researchers found that protein-design tools could be used to "paraphrase" the DNA codes of toxic proteins, "re-writing them in ways that could preserve their structure, and potentially their function," says Eric Horvitz, Microsoft's chief scientific officer. The computer scientists used an AI program to generate DNA codes for more than 75,000 variants of hazardous proteins - and the firewalls used by DNA manufacturers weren't consistently able to catch them. "To our concern," says Horvitz, "these reformulated sequences slipped past the biosecurity screening systems used worldwide by DNA synthesis companies to flag dangerous orders." A fix quickly got written that and slapped onto the biosecurity screening software. But it's not perfect -- it still wasn't able to detect a small fraction of the variants. And it's just the latest episode showing how AI is revving up long-standing concerns about the potential misuse of powerful biological tools. "AI-powered protein design is one of the most exciting frontiers in science. We're already seeing advances in medicine and public health," says Horvitz. "Yet like many powerful technologies, these same tools can often be misused." For years, biologists have worried that their ever-improving DNA tools might be harnessed to design potent biothreats, like more virulent viruses or easy-to-spread toxins. They've even debated whether it's really wise to openly publish certain experimental results, even though open discussion and independent replication has been the lifeblood of science. The researchers and the journal who published this new study decided to hold some of their information back, and will restrict who gets access to their data and software. They enlisted a third party, a non-profit called the International Biosecurity and Biosafety Initiative for Science, to make decisions about who has a legitimate need to know. "This is the first time such a model has been employed to manage risks of sharing hazardous information in a scientific publication," says Horvitz. Scientists who have been worried about future biosecurity threats for some time praised this work. "My overall reaction was favorable," says Arturo Casadevall, a microbiologist and immunologist at Johns Hopkins University. "Here we have a system in which we are identifying vulnerabilities. And what you're seeing is an attempt to correct the known vulnerabilities." The trouble is, says Casadevall, "what vulnerabilities don't we know about that will require future corrections?" He notes that this team did not do any lab work to actually generate any of the proteins designed by AI, to see if they would truly mimic the activity of the biological original threats. Such work would be an important reality check as society grapples with this kind of emerging threat from AI, says Casadevall, but would be tricky to do, as it might be precluded by international treaties prohibiting the development of biological weapons. This isn't the first time scientists have explored the potential for malevolent use of AI in a biological setting. For example, a few years ago, another team wondered if AI could be used to generate novel molecules that would have the same properties as nerve agents. In less than six hours, the AI tool dutifully concocted 40,000 molecules that met the requested criteria. It not only came up with known chemical warfare agents like the notorious one called VX, but also designed many unknown molecules that looked plausible and were predicted to be more toxic. "We had transformed our innocuous generative model from a helpful tool of medicine to a generator of likely deadly molecules," the researchers wrote. That team also didn't openly publish the chemical structures that the AI tool had devised, or create them in the lab, "because they thought they were way too dangerous," points out David Relman, a researcher at Stanford University. "They simply said, we're telling you all about this as a warning." Relman thinks this latest study, showing how AI could be used to evade security screening and finding a way to address that, is laudable. At the same time, he says, it just illustrates that there's an enormous problem brewing. "I think it leaves us dangling and wondering, 'Well, what exactly are we supposed to do?'" he says. "How do we get ahead of a freight train that is just evermore accelerating and racing down the tracks, in danger of careening off the tracks?" Despite concerns like these, some biosecurity experts see reasons to be reassured. Twist Bioscience is a major provider of made-to-order DNA, and in the past ten years, it's had to refer orders to law enforcement fewer than five times, says James Diggans, the head of policy and biosecurity at Twist Bioscience and chair of the board of directors at the International Gene Synthesis Consortium, an industry group. "This is an incredibly rare thing," he says. "In the cybersecurity world, you have a host of actors that are trying to access systems. That is not the case in biotech. The real number of people who are really trying to create misuse may be very close to zero. And so I think these systems are an important bulwark against that, but we should all find comfort in the fact that this is not a common scenario."

Microsoft on AI in Biology: Understanding the risks of zero-day threats

Zero-day vulnerabilities show urgent need for stronger AI biosecurity safeguards When Microsoft's chief scientific officer Eric Horvitz and his team describe a "zero-day" in biology, they are deliberately borrowing a term from cybersecurity. A zero-day vulnerability refers to an unknown flaw in software that hackers can exploit before anyone has time to patch it. But here, the flaw isn't in computer code - it's in the global biosecurity systems that are supposed to detect and prevent the misuse of synthetic DNA. And the exploit, as Microsoft researchers discovered, comes from AI. In a new study, Microsoft scientists revealed that artificial intelligence can help generate genetic sequences that evade current screening software. These systems, widely used by DNA synthesis companies and research labs, compare incoming orders against a database of known pathogens and toxins. The idea is simple: if someone tries to order a dangerous sequence - say, a segment of anthrax or a toxic protein - the system raises a red flag. But with the help of generative AI, the researchers showed that harmful designs could be rewritten in ways that still function biologically but no longer look suspicious to the software. Also read: Perplexity's Comet AI browser now free for all users, gets new Background Assistants feature The finding is being described as the first real "zero-day" in biosecurity. Much like cybercriminals who use new malware to slip past firewalls, AI was able to paraphrase dangerous code in protein form, creating sequences that existing screening methods failed to recognize. According to the researchers, this breakthrough isn't just theoretical: it demonstrates a fundamental weakness in how the world currently guards against biological misuse. While the Microsoft team quickly developed patches and proposed improvements to strengthen defenses, the deeper message is clear. As AI models become more powerful and more accessible, defensive systems will have to keep evolving just as quickly. What was once an unlikely scenario, AI accelerating the design of harmful biological agents. is now a tangible risk. For decades, biosecurity experts have relied on the assumption that creating bioweapons requires both advanced expertise and specialized equipment. The tacit knowledge needed to turn genetic code into a functional threat has acted as a natural barrier. But large AI models are starting to erode that barrier by guiding even non-specialists through steps that once demanded years of training. Also read: Vibe working explained: Microsoft's AI agent for Excel, Word, and PowerPoint At the same time, DNA synthesis is becoming faster, cheaper, and more distributed globally. If AI can help generate malicious code that evades standard filters, the result could be a dangerous widening of access to biothreat capabilities. This is especially concerning given that existing international safeguards remain voluntary and unevenly enforced. None of this means AI in biology is inherently bad. In fact, many of the same tools that can help design harmful sequences are revolutionizing drug discovery, protein engineering, and vaccine development. AI can speed up the search for cancer treatments, optimize enzymes for clean energy, and even predict the structure of proteins that were previously unsolvable puzzles. But the dual-use nature of the technology, equally capable of breakthroughs and biothreats, makes it uniquely challenging to regulate. What Microsoft's zero-day demonstration shows is that ignoring the problem is not an option. The tools are too powerful, and the stakes too high. Microsoft's researchers have urged for a "defense-in-depth" strategy: not just relying on sequence matching, but combining multiple approaches such as functional prediction, structure analysis, and even AI red-teaming to identify hidden threats. They also argue for stronger international coordination, noting that pathogens do not respect borders and neither do AI models. Governments and research institutions are beginning to take note. Discussions are underway on whether access to powerful biological design models should be gated, whether DNA synthesis should come with stricter oversight, and how to build rapid-response systems capable of spotting new threats. Just as the internet forced the world to invent cybersecurity, the rise of AI-assisted biology is pushing us toward a new field: bio-AI security. The Microsoft team's discovery may have closed one loophole, but it also underscored how many more may be waiting. The challenge now is not simply to react to each new exploit, but to build systems resilient enough to anticipate them. That means recognizing AI as both a catalyst for progress and a potential amplifier of risk. And it means preparing for a world where the next "zero-day" may not be in a line of computer code, but in the blueprint of life itself.