FDA's AI Tool 'Elsa' Raises Concerns Over Hallucinations and Accuracy in Drug Approval Process

FDA Introduces AI Tool 'Elsa' for Drug Approval Process

The Food and Drug Administration (FDA) recently unveiled a generative AI tool named Elsa, aimed at streamlining the drug and medical device approval process. Introduced by FDA Commissioner Marty Makary on June 2, Elsa was touted as a cost-effective solution, costing only $12,000 in its first week of operation 2

2

. The AI assistant was designed to help FDA employees with tasks such as creating meeting notes, summaries, and accelerating clinical protocol reviews 1

1

4

4

.

Concerns Raised by FDA Employees

However, current and former FDA employees have raised significant concerns about Elsa's performance. According to a CNN investigation, at least three employees reported that Elsa has been:

1. Hallucinating nonexistent studies
2. Misrepresenting real research
3. Providing incorrect answers to basic questions

One FDA employee stated, "Anything that you don't have time to double-check is unreliable. It hallucinates confidently" 1

1

2

2

. This raises serious questions about the tool's reliability in a field where accuracy is crucial for public health and safety.

Leadership Response and Implementation

Despite these concerns, FDA leadership appears to be downplaying the issues. FDA Commissioner Marty Makary told CNN that he had not heard of these specific concerns and emphasized that using Elsa and participating in training to use it are currently voluntary at the agency 1

1

. Jeremy Walsh, the FDA's head of AI, acknowledged that Elsa "could potentially hallucinate" but suggested that more detailed questions could mitigate this issue 4

4

.

Broader Context of AI in Healthcare

The introduction of Elsa is part of a larger push by the Trump administration to accelerate AI adoption in healthcare. Health and Human Services Secretary Robert F. Kennedy Jr. has been a vocal proponent of using AI to speed up drug approvals 2

2

3

3

. However, this aggressive push for AI implementation has raised concerns about potential oversight issues and the risks of relying on unproven technologies in critical healthcare decisions 3

3

5

5

.

Implications and Challenges

The reported issues with Elsa highlight several challenges:

1. Time Efficiency: Instead of saving time, some employees report spending extra time double-checking Elsa's output, contradicting claims of increased efficiency 4

   4

   .

2. Access to Information: Elsa reportedly lacks access to relevant documentation needed for drug reviews, limiting its usefulness in the approval process 4

   4

   .

3. Accuracy and Trust: The tool's tendency to cite nonexistent studies and provide incorrect information undermines trust in its capabilities 2

   2

   4

   4

   .

4. Regulatory Concerns: The rapid implementation of AI in high-stakes healthcare decisions without proper safeguards raises questions about regulatory oversight 3

   3

   5

   5

   .

Future Implications

The challenges faced by the FDA with Elsa serve as a cautionary tale for the implementation of AI in critical sectors like healthcare. As the U.S. government continues to push for AI adoption through initiatives like the AI Action Plan, the need for robust testing, proper oversight, and clear governance standards becomes increasingly apparent 3

3

5

5

.

The situation also underscores the importance of maintaining human expertise and judgment in processes that directly impact public health and safety. As AI tools continue to evolve, finding the right balance between technological innovation and reliable, human-centered decision-making remains a crucial challenge for regulatory agencies like the FDA.