AI in Medicine: Study Reveals Socioeconomic Bias in Treatment Recommendations

AI Models Show Bias in Medical Recommendations

A groundbreaking study conducted by researchers at the Icahn School of Medicine at Mount Sinai has revealed that generative AI models may recommend different treatments for identical medical conditions based solely on a patient's socioeconomic and demographic background. The findings, published in the April 7, 2025 online issue of Nature Medicine, underscore the critical need for early detection and intervention to ensure AI-driven healthcare is safe, effective, and equitable for all patients 1

1

2

Comprehensive Stress Testing of AI Models

The research team, led by Dr. Eyal Klang and Dr. Girish N. Nadkarni, stress-tested nine large language models (LLMs) on 1,000 emergency department cases. Each case was replicated with 32 different patient backgrounds, generating over 1.7 million AI-generated medical recommendations. Despite identical clinical details, the AI models occasionally altered their decisions based on a patient's socioeconomic and demographic profile 1

1

Key Findings and Implications

The study revealed significant inconsistencies in AI-generated recommendations across several key areas:

1. Triage priority
2. Diagnostic testing
3. Treatment approach
4. Mental health evaluation

One of the most striking findings was the tendency of some AI models to escalate care recommendations, particularly for mental health evaluations, based on patient demographics rather than medical necessity. Additionally, high-income patients were more frequently recommended advanced diagnostic tests such as CT scans or MRI, while low-income patients were more often advised to undergo no further testing 1

1

2

Framework for AI Assurance in Healthcare

Dr. Klang emphasized that their research provides a framework for AI assurance, helping developers and healthcare institutions design fair and reliable AI tools. The team's rigorous validation process tests AI outputs against clinical standards and incorporates expert feedback to refine performance 1

1

Related Stories

Mount Sinai Researchers Develop AEquity: A Tool to Combat Bias in Healthcare AI

AI Models Stumble on Medical Ethics Puzzles, Revealing Cognitive Biases

AI Diagnosis Outperforms Emergency Room Doctors in Harvard Study, Raises Questions on Patient Care

UCLA Study Warns AI in Neurological Care Could Worsen Health Disparities Without Proper Safeguards

Future Directions and Implications

While the study offers critical insights, the researchers caution that it represents only a snapshot of AI behavior. Future research will continue to include assurance testing to evaluate how AI models perform in real-world clinical settings and whether different prompting techniques can reduce bias 2

2

Dr. Mahmud Omar, the first author of the study, stressed the importance of thoroughly evaluating AI's safety, reliability, and fairness as it becomes more integrated into clinical care. The team aims to work with other healthcare institutions to refine AI tools, ensuring they uphold the highest ethical standards and treat all patients fairly 1

1

2

Expanding the Research

The investigators plan to expand their work by:

1. Simulating multistep clinical conversations
2. Piloting AI models in hospital settings to measure real-world impact
3. Developing policies and best practices for AI assurance in healthcare

Dr. Nadkarni emphasized that while AI has the power to revolutionize healthcare, it must be developed and used responsibly. By implementing robust assurance protocols, the team aims to advance technology and build trust essential for transformative healthcare 1

1

2

This study marks a significant step towards establishing global best practices for AI assurance in healthcare, ensuring that these powerful tools improve care for all patients, regardless of their socioeconomic or demographic background.