AI Models Stumble on Medical Ethics Puzzles, Revealing Cognitive Biases

AI Models Struggle with Modified Medical Ethics Scenarios

A groundbreaking study conducted by researchers at the Icahn School of Medicine at Mount Sinai, in collaboration with Rabin Medical Center in Israel, has uncovered a significant flaw in advanced artificial intelligence (AI) models when faced with complex medical ethics scenarios. The study, published in NPJ Digital Medicine, reveals that even the most sophisticated large language models (LLMs) can make surprisingly simple mistakes when confronted with slightly modified versions of well-known ethical dilemmas 1

1

Source: Medical Xpress

Inspiration from Cognitive Psychology

The research team, inspired by Daniel Kahneman's book "Thinking, Fast and Slow," set out to test how well AI systems could navigate between fast, intuitive thinking and slower, analytical reasoning. They observed that LLMs often struggle when presented with subtle tweaks to classic lateral-thinking puzzles 2

2

Testing AI's Ethical Reasoning

To explore this phenomenon, the researchers tested several commercially available LLMs using a combination of creative lateral thinking puzzles and slightly modified versions of well-known medical ethics cases. One example involved adapting the classic "Surgeon's Dilemma," a puzzle that highlights implicit gender bias 1

1

In the modified version, where the researchers explicitly stated that the boy's father was the surgeon, some AI models still incorrectly responded that the surgeon must be the boy's mother. This error demonstrates how LLMs can cling to familiar patterns, even when contradicted by new information 3

3

Source: ScienceDaily

Implications for Healthcare

Dr. Eyal Klang, Chief of Generative AI in the Windreich Department of Artificial Intelligence and Human Health at Mount Sinai, emphasized the potential consequences of such errors in healthcare settings:

"AI can be very powerful and efficient, but our study showed that it may default to the most familiar or intuitive answer, even when that response overlooks critical details. In health care, where decisions often carry serious ethical and clinical implications, missing those nuances can have real consequences for patients." 1

1

Related Stories

AI Language Models Prioritize Helpfulness Over Accuracy in Medical Contexts, Study Reveals

AI Chatbots Vulnerable to Medical Misinformation, Study Reveals

AI in Medicine: Study Reveals Socioeconomic Bias in Treatment Recommendations

AI Diagnosis Outperforms Emergency Room Doctors in Harvard Study, Raises Questions on Patient Care

The Need for Human Oversight

The study's findings highlight the importance of human oversight in AI-assisted healthcare decision-making. Dr. Girish N. Nadkarni, Chair of the Windreich Department of Artificial Intelligence and Human Health at Mount Sinai, stressed that AI should be used as a complement to clinical expertise rather than a substitute, particularly in complex or high-stakes situations 2

2

Future Directions

The research team plans to expand their work by testing a wider range of clinical examples. They are also developing an "AI assurance lab" to systematically evaluate how well different models handle real-world medical complexity 3

3

Lead author Dr. Shelly Soffer emphasized the importance of these findings: "Simple tweaks to familiar cases exposed blind spots that clinicians can't afford. It underscores why human oversight must stay central when we deploy AI in patient care." 1

1

As AI continues to play an increasingly significant role in healthcare, this study serves as a crucial reminder of the need for careful integration and ongoing evaluation of these powerful tools in medical practice.