FDA's AI Tool 'Elsa' Raises Concerns Over Hallucinations and Accuracy in Drug Approval Process

FDA employees say the agency's Elsa generative AI hallucinates entire studies

"Anything that you don't have time to double-check is unreliable." Current and former members of the FDA told about issues with the Elsa generative AI tool by the federal agency last month. Three employees said that in practice, Elsa has hallucinated nonexistent studies or misrepresented real research. "Anything that you don't have time to double-check is unreliable," one source told the publication. "It hallucinates confidently." Which isn't exactly ideal for a tool that's supposed to be speeding up the clinical review process and aiding with making efficient, informed decisions to benefit patients. Leadership at the FDA appeared unfazed by the potential problems posed by Elsa. "I have not heard those specific concerns," FDA Commissioner Marty Makary told CNN. He also emphasized that using Elsa and participating in the training to use it are currently voluntary at the agency. The CNN investigation highlighting these flaws with the FDA's artificial intelligence arrived on the same day as the White House introduced an "." The program presented AI development as a technological arms race that the US should win at all costs, and it laid out plans to remove "red tape and onerous regulation" in the sector. It also demanded that AI be free of "ideological bias," or in other words, only following the biases of the current administration by removing mentions of climate change, misinformation, and diversity, equity and inclusion efforts. Considering each of has a documented impact on public health, the ability of tools like Elsa to provide genuine benefits to both the FDA and to US patients looks increasingly doubtful.

FDA's New Drug Approval AI Is Generating Fake Studies: Report

The AI, dubbed Elsa, is supposed to be making employees better at their jobs. Robert F. Kennedy Jr., the Secretary of Health and Human Services, has made a big push to get agencies like the Food and Drug Administration to use generative artificial intelligence tools. In fact, Kennedy recently told Tucker Carlson that AI will soon be used to approve new drugs "very, very quickly." But a new report from CNN confirms all our worst fears. Elsa, the FDA's AI tool, is spitting out fake studies. CNN spoke with six current and former employees at the FDA, three of whom have used Elsa for work that they described as helpful, like creating meeting notes and summaries. But three of those FDA employees told CNN that Elsa just makes up nonexistent studies, something commonly referred to in AI as "hallucinating." The AI will also misrepresent research, according to these employees. "Anything that you don't have time to double-check is unreliable. It hallucinates confidently," one unnamed FDA employee told CNN. And that's the big problem with all AI chatbots. They need to be double-checked for accuracy, often creating even more work for the human behind the computer if they care about the quality of their output at all. People who insist that AI actually saves them time are often fooling themselves, with one recent study of programmers showing that tasks took 20% longer with AI, even among people who were convinced they were more efficient. Kennedy's Make America Healthy Again (MAHA) commission issued a report back in May that was later found to be filled with citations for fake studies. An analysis from the nonprofit news outlet NOTUS found that at least seven studies cited didn't even exist, with many more misrepresenting what was actually said in a given study. We still don't know if the commission used Elsa to generate that report. FDA Commissioner Marty Makary initially deployed Elsa across the agency on June 2, and an internal slide leaked to Gizmodo bragged that the system was "cost-effective," only costing $12,000 in its first week. Makary said that Elsa was "ahead of schedule and under budget†when he first announced the AI rollout. But it seems like you get what you pay for. If you don't care about the accuracy of your work, Elsa sounds like a great tool for allowing you to get slop out the door faster, generating garbage studies that could potentially have real consequences for public health in the U.S. CNN notes that if an FDA employee asks Elsa to generate a one-paragraph summary of a 20-page paper on a new drug, there's no simple way to know if that summary is accurate. And even if the summary is more or less accurate, what if there's something within that 20-page report that would be a big red flag for any human with expertise? The only way to know for sure if something was missed or if the summary is accurate is to actually read the report. The FDA employees who spoke with CNN said they tested Elsa by asking basic questions like how many drugs of a certain class have been approved for children. Elsa confidently gave wrong answers, and while it apparently apologized when it was corrected, a robot being "sorry" doesn't really fix anything. We still don't know the workflow being deployed when Kennedy says AI will allow the FDA to approve new drugs, but he testified in June to a House subcommittee that it's already being used to "increase the speed of drug approvals." The secretary, whose extremist anti-vaccine beliefs didn't keep him from becoming a public health leader, seems intent on injecting unproven technologies into mainstream science. Kennedy also testified to Congress that he wants every American to be strapped with a wearable health device within the next four years. As it happens, President Trump's pick for Surgeon General, Casey Means, owns a wearables company called Levels that monitors glucose levels in people who aren't diabetic. There's absolutely no reason that people without diabetes need to constantly monitor their glucose levels, according to experts. Means, a close ally of Kennedy, has not yet been confirmed by the Senate. The FDA didn't respond to questions emailed on Wednesday about what the agency is doing to address Elsa's fake study problem. Makary acknowledged to CNN that Elsa could "potentially hallucinate," but that's "no different" from other large language models and generative AI. And he's not wrong on that. The problem is that AI is not fit for purpose when it's consistently just making things up. But that won't stop folks from continuing to believe that AI is somehow magic.

The FDA's new drug-approving AI chatbot is not helping

The Food and Drug Administration's new AI tool -- touted by Secretary of Health and Human Services Robert F. Kennedy, Jr. as a revolutionary solution for shortening drug approvals -- is initially causing more hallucinations than solutions. Known as Elsa, the AI chatbot was introduced to help FDA employees with daily tasks like meeting notes and emails, while simultaneously supporting quicker drug and device approval turnaround times by sorting through important application data. But, according to FDA insiders who spoke to CNN under anonymity, the chatbot is rife with hallucinations, often fabricating medical studies or misinterpreting important data. The tool has been sidelined by staffers, with sources saying it can't be used in reviews and does not have access to crucial internal documents employees were promised. "It hallucinates confidently," one FDA employee told CNN. According to the sources, the tool often provides incorrect answers on the FDA's research areas, drug labels, and can't link to third-party citations from external medical journals. Despite initial claims that the tool was already integrated into the clinical review protocol, FDA Commissioner Marty Makary told CNN that the tool was only being used for "organizational duties" and was not required of employees. The FDA's head of AI admitted to the publication that the tool was at risk of hallucinating, carrying the same risk as other LLMs. Both said they weren't surprised it made mistakes, and said further testing and training was needed. But not all LLM's have the job of approving life-saving medicine. The agency announced the new agentic tool in June, with Vinay Prasad, director of the FDA's Center for Biologics Evaluation and Research (CBER), and Makary writing that AI innovation was a leading priority for the agency in an accompanying Journal of the American Medical Association (JAMA) article. The tool, which examines device and drug applications, was pitched as a solution for lengthy and oft-criticized drug approval periods, following the FDA's launch of an AI-assisted scientific review pilot. The Trump administration has rallied government agencies behind an accelerated, "America-first" AI agenda, including recent federal guidance to establish FDA-backed AI Centers of Excellence for testing and deploying new AI tools, announced in the government's newly unveiled AI Action Plan. Many are worried that the aggressive push and deregulation efforts eschew necessary oversight of the new tech. "Many of America's most critical sectors, such as healthcare, are especially slow to adopt due to a variety of factors, including distrust or lack of understanding of the technology, a complex regulatory landscape, and a lack of clear governance and risk mitigation standards," the action plan reads. "A coordinated Federal effort would be beneficial in establishing a dynamic, 'try-first' culture for AI across American industry."

The FDA Is Using an AI to "Speed Up" Drug Approvals and Insiders Say It's Making Horrible Mistakes

Insiders at the Food and Drug Administration are ringing alarm bells over the agency's use of an AI to fast-track drug approvals. As CNN reports, six current and former FDA officials are warning that the AI, dubbed Elsa, which was unveiled weeks earlier, is "hallucinating" completely made-up studies. It's a terrifying reality that could, in a worst-case scenario, lead to potentially dangerous drugs mistakenly getting the stamp of approval from the FDA. It's part of a high-stakes and greatly accelerated effort by the US government to embrace deeply flawed AI tech. Elsa, much like other currently available AI chatbots, often makes stuff up. "Anything that you don't have time to double-check is unreliable," one FDA employee told CNN. "It hallucinates confidently." Health and Human Services secretary Robert Kennedy Jr, a noted figure of the anti-vaccine movement, who has no relevant credentials for the job and frequently furthers discredited conspiracy theories, lauded the administration's embrace of AI as a sign that the "AI revolution has arrived." "We are using this technology already at HHS to manage health care data, perfectly securely, and to increase the speed of drug approvals," he told Congress last month. But reality is rapidly catching up -- which shouldn't be a surprise to anybody who's used a large language model-based tool before. Given the tech's track record so far, the medical community's embrace of AI has already been mired in controversy, with critics pointing out the risks of overrelying on the tech. Instead of saving scientists time, Elsa is doing the exact opposite, highlighting a common refrain among companies' already backfiring attempts to shoehorn the tech into every aspect of their operations. "AI is supposed to save our time, but I guarantee you that I waste a lot of extra time just due to the heightened vigilance that I have to have," a second FDA employee told CNN. The insiders claim that Elsa can't really help them review drugs for approvals since it doesn't have access to relevant documentation. It can't even "answer basic questions," such as how many times a company filed for FDA approval, according to CNN. Worse yet, it often cites studies that don't exist, and when challenged, it ends up being "apologetic" and tells employees that its output needs to be verified. The damning claims fly in the face of the FDA's attempts to paint Elsa as a revolutionary tool that can significantly speed up drug evaluations. In a June statement, the FDA boasted that it was "already using Elsa to accelerate clinical protocol reviews, shorten the time needed for scientific evaluations, and identify high-priority inspection targets." Meanwhile, FDA's head of AI, Jeremy Walsh, told CNN that it's possible Esla "could potentially hallucinate," saying employees "don't have to use" the tool "if they don't find it to have value." In an attempt to reassure, Walsh said that Elsa's hallucinations could be mitigated by using more detailed questions. In many ways, Elsa couldn't have come at a worse time as Congress is racing to figure out how to approach AI regulation. Instead of implementing new rules to avoid a disaster, the Trump administration has remained far more focused on clearing a regulatory path as tens of billions of dollars continue to be poured into the industry. In other words, there doesn't appear to be much interest in reining in the FDA's use of unproven AI tech -- a concerning bet that could one day come home to roost.

FDA's Elsa AI Tool Hallucinates Studies, Sparks Concern Over Lack Of Oversight In High-Stakes Healthcare Decisions Where Accuracy Directly Impacts Public Safety And Trust

With artificial intelligence being adopted rapidly, varied domains are increasingly looking for ways to implement the technology in order to improve efficiency and save time. We see a vast use of AI assistants for smoother processes. It has not been long since the FDA rolled out Elsa, its generative AI tool that was meant to be a forward-thinking move, to help improve drug and medical device approval workflows, but it seems like the platform is not as smooth as we would hope it to be. Reports are coming forward from the company's former and current employees about the tool hallucinating and often misrepresenting real research. Just like the world at large is evolving, healthcare and regulatory tech are also inclined towards the adoption of generative AI to help streamline processes. The U.S. Food and Drug Administration introduced Elsa, a tool to help make drug and device approval seamless by cutting down on bottlenecks and bureaucratic delays and enhancing overall efficiency by reducing the administrative workloads. However, what was meant to be a technological leap turned out to be deeply concerning, raising questions regarding over-dependence on the technology and human negligence. While everything looked revolutionary on paper, given how efficiently life-saving treatments could be reviewed, the internal staff at the FDA laid out the true details about the tool. According to the report by CNN, many former and current employees have expressed disquiet about the generative AI tool hallucinating, generating inaccurate summaries, and even fabricating entire clinical studies, as references to these papers do not even exist. At least three employees have revealed the troubling tendency of the tool and how it would go to the extent of distorting actual research findings. This seems to be especially concerning when the assistant's very purpose is to save time and not have to supervise and double-check the work done. FDA leadership did not show much urgency when the issue surrounding Elsa was brought to their attention. They were also quick to point out that the staff is not required to use Elsa and that the implementation of the tool is completely voluntary. The staff are not made to attend the training sessions unless they want to. This means that no systems are in place to ensure the tool is being used properly and safely. While AI hallucinations are not rare occurrences, and these glitches sometimes can happen due to some updates or changes, if it is being used in high-stakes settings, then proper safeguards need to be in place. The issue is not with some flaws coming up in the AI tool but rather its use without enough human supervision, especially in a field where small errors can have serious consequences for people's health.