Explainable AI in Medical Diagnostics: Non-Experts Trust AI Blindly While Clinicians Spot Errors

Explainable AI Shows Opposite Effects Across User Expertise Levels

A comprehensive study published in Nature Medicine1

1

reveals that explainable AI assistance produces starkly different outcomes in medical diagnostics depending on user expertise. Researchers from MIT, Stanford, and Columbia tested 623 non-experts and 153 primary care physicians on dermatological diagnosis tasks, uncovering critical insights about AI deference and automation bias in healthcare settings2

2

.

The research employed fairness-constrained deep learning models that achieved a weighted AUROC of 0.930 for melanoma vs. nevus classification, with substantially reduced skin tone disparities compared to baseline models. For non-experts, AI assistance improved diagnostic accuracy from 69.7% to 75.8% in melanoma detection tasks1

1

. However, this improvement came with a concerning caveat: non-experts demonstrated significant AI deference, trusting multimodal LLM textual explanations even when incorrect.

Non-Experts Fall Victim to AI Deference and Vague Explanations

The study revealed that general public participants found LLM-based explanations more convincing when they were vague or generic, regardless of accuracy. This pattern of automation bias represents a significant challenge for patient-facing diagnostic tools that increasingly rely on explainable AI methods2

2

. "Good AI systems can improve performance in some health settings, but this has to be balanced carefully with algorithmic deference that can lead to more error," explains Marzyeh Ghassemi, MIT associate professor and principal investigator at the Abdul Latif Jameel Clinic for Machine Learning in Health.

The research tested four distinct AI assistance methods: basic prediction with confidence levels, GradCAM heatmaps highlighting important image regions, content-based image retrieval showing similar cases, and multimodal LLM providing textual explanations. Non-experts benefited from AI assistance across methods but struggled to recognize diagnostic errors when AI predictions were incorrect1

1

.

Primary Care Physicians Perform Best With Minimal AI Explanations

In contrast to non-experts, primary care physicians demonstrated markedly different responses to explainable AI assistance in dermatological diagnosis. Clinicians were not misled by incorrect AI suggestions and performed optimally when given only the model's prediction without accompanying explanations. This finding challenges conventional assumptions about explainable AI benefits in clinical AI implementation2

2

.

The study employed a randomized between-subjects factorial design testing two human-AI decision-making paradigms: Human-First (users decide before reviewing AI) and AI-First (users review AI suggestions before deciding). Physicians evaluated 12 clinical images balanced by skin tone and pathology, focusing on four conditions with potential skin tone disparities: atopic dermatitis, pityriasis rosea, Lyme disease, and cutaneous T cell lymphoma1

1

.

Implications for Medical AI Design and Healthcare Equity

"These findings are important as patients increasingly turn to AI to help with their health care. Our findings show that those with the least medical knowledge are most likely to be led astray when explainable AI models give an erroneous output," notes Roxana Daneshjou, Stanford assistant professor of biomedical data science and dermatology2

2

.

The research underscores critical considerations for FDA-approved AI interfaces used in dermatological diagnosis and other medical applications. Lead author Orson Xu emphasizes: "Often the people who could benefit most from AI are the ones most likely to be led astray by it, so how we present a recommendation matters as much as whether it's correct." These results highlight the urgent need to develop explainability methods that encourage critical thinking rather than blind acceptance, particularly as AI-powered diagnostic tools become more prevalent in both clinical and consumer health settings2

2

.