AI Models Demonstrate Misleading Accuracy in Medical Imaging Study

AI thought knee X-rays could tell if you drink beer and eat refried beans

Some artificial intelligence models are struggling to learn the old principle, "Correlation does not equal causation." And while that's not a reason to abandon AI tools, a recent study should remind programmers that even reliable versions of the technology are still prone to bouts of weirdness -- like claiming knee X-rays can prove someone drinks beer or eats refried beans. Artificial intelligence models do much more than generate (occasionally accurate) text responses and (somewhat) realistic videos. Truly well-made tools are already helping medical researchers parse troves of datasets to discover new breakthroughs, accurately forecast weather patterns, and assess environmental conservation efforts. But according to a study published in the journal Scientific Reports, algorithmic "shortcut learning" continues to pose a problem of generating results that are simultaneously highly accurate and misinformed. Researchers at Dartmouth Health recently trained medical AI models on over 25,000 knee X-rays provided by the National Institutes of Health's Osteoarthritis Initiative. They then essentially worked backwards, tasking the deep learning programs to find commonalities that predicted nonsensical traits, such as which knee-owners clearly drank beer or ate refried beans -- which, as the study authors explain, is patently absurd. "The models are not uncovering a hidden truth about beans or beer hidden within our knees," they write. At the same time, however, the team explains these predictions aren't the result of "mere chance." The underlying issue is what's known as algorithmic shortcutting, in which deep learning models find connections through easily detectable -- but still irrelevant or misleading -- patterns. "Shortcutting makes it trivial to create models with surprisingly accurate predictions that lack all face validity," they warn. Variables identified by the algorithms, for example, included unrelated factors such as differences in X-ray machine models or the equipment's geographic locations. "These models can see patterns humans cannot, but not all patterns they identify are meaningful or reliable," Peter Schilling, an orthopaedic surgeon, Dartmouth Health assistant professor of orthopaedics, and study senior author added in a statement on December 9th. "It's crucial to recognize these risks to prevent misleading conclusions and ensure scientific integrity." An additional, ongoing problem is that there doesn't seem to be an easy fix to AI shortcut learning. Attempts to address these biases were only "marginally successful," according to Monday's announcement. "This goes beyond bias from clues of race or gender," said Brandon Hill, a machine learning scientist and study co-author. "We found the algorithm could even learn to predict the year an X-ray was taken. It's pernicious; when you prevent it from learning one of these elements, it will instead learn another it previously ignored." According to Hill, these problems can potentially lead human experts to trust "some really dodgy claims" made by AI models. To Schilling, Hill, and their colleagues, this means that although predictive deep learning programs have their uses, the burden of proof needs to be much more rigorous when using them in situations such as medical research. Hill likens working with the AI to dealing with an extraterrestrial lifeform while simultaneously attempting to anthropomorphize it. "It is incredibly easy to fall into the trap of presuming that the model 'sees' the same way we do," he says. "In the end it doesn't. It learned a way to solve the task given to it, but not necessarily how a person would. It doesn't have logic or reasoning as we typically understand it."

AI thought knee X-rays show if you drink beer -- they don't

Artificial intelligence can be a useful tool to health care professionals and researchers when it comes to interpreting diagnostic images. Where a radiologist can identify fractures and other abnormalities from an X-ray, AI models can see patterns humans cannot, offering the opportunity to expand the effectiveness of medical imaging. But a study in Scientific Reports highlights a hidden challenge of using AI in medical imaging research -- the phenomenon of highly accurate yet potentially misleading results known as "shortcut learning." The researchers analyzed more than 25,000 knee X-rays and found that AI models can "predict" unrelated and implausible traits such as whether patients abstained from eating refried beans or beer. While these predictions have no medical basis, the models achieved surprising levels of accuracy by exploiting subtle and unintended patterns in the data. "While AI has the potential to transform medical imaging, we must be cautious," says the study's senior author, Dr. Peter Schilling, an orthopaedic surgeon at Dartmouth Health's Dartmouth Hitchcock Medical Center and an assistant professor of orthopaedics in Dartmouth's Geisel School of Medicine. "These models can see patterns humans cannot, but not all patterns they identify are meaningful or reliable," Schilling says. "It's crucial to recognize these risks to prevent misleading conclusions and ensure scientific integrity." The researchers examined how AI algorithms often rely on confounding variables -- such as differences in X-ray equipment or clinical site markers -- to make predictions rather than medically meaningful features. Attempts to eliminate these biases were only marginally successful -- the AI models would just "learn" other hidden data patterns. "This goes beyond bias from clues of race or gender," says Brandon Hill, a co-author of the study and a machine learning scientist at Dartmouth Hitchcock. "We found the algorithm could even learn to predict the year an X-ray was taken. It's pernicious -- when you prevent it from learning one of these elements, it will instead learn another it previously ignored. This danger can lead to some really dodgy claims, and researchers need to be aware of how readily this happens when using this technique." The findings underscore the need for rigorous evaluation standards in AI-based medical research. Overreliance on standard algorithms without deeper scrutiny could lead to erroneous clinical insights and treatment pathways. "The burden of proof just goes way up when it comes to using models for the discovery of new patterns in medicine," Hill says. "Part of the problem is our own bias. It is incredibly easy to fall into the trap of presuming that the model 'sees' the same way we do. In the end, it doesn't." "AI is almost like dealing with an alien intelligence," Hill continues. "You want to say the model is 'cheating," but that anthropomorphizes the technology. It learned a way to solve the task given to it, but not necessarily how a person would. It doesn't have logic or reasoning as we typically understand it." Schilling, Hill, and study co-author Frances Koback, a third-year medical student in Dartmouth's Geisel School, conducted the study in collaboration with the Veterans Affairs Medical Center in White River Junction, Vt.

AI thought knee X-rays show if you drink beer -- they don't

Artificial intelligence can be a useful tool to health care professionals and researchers when it comes to interpreting diagnostic images. Where a radiologist can identify fractures and other abnormalities from an X-ray, AI models can see patterns humans cannot, offering the opportunity to expand the effectiveness of medical imaging. But a study in Scientific Reports highlights a hidden challenge of using AI in medical imaging research -- the phenomenon of highly accurate yet potentially misleading results known as "shortcut learning." The researchers analyzed more than 25,000 knee X-rays from the National Institutes of Health-funded Osteoarthritis Initiative and found that AI models can "predict" unrelated and implausible traits such as whether patients abstained from eating refried beans or beer. While these predictions have no medical basis, the models achieved surprising levels of accuracy by exploiting subtle and unintended patterns in the data. "While AI has the potential to transform medical imaging, we must be cautious," says the study's senior author, Dr. Peter Schilling, an orthopaedic surgeon at Dartmouth Health's Dartmouth Hitchcock Medical Center and an assistant professor of orthopaedics in Dartmouth's Geisel School of Medicine. "These models can see patterns humans cannot, but not all patterns they identify are meaningful or reliable," Schilling says. "It's crucial to recognize these risks to prevent misleading conclusions and ensure scientific integrity." The researchers examined how AI algorithms often rely on confounding variables -- such as differences in X-ray equipment or clinical site markers -- to make predictions rather than medically meaningful features. Attempts to eliminate these biases were only marginally successful -- the AI models would just "learn" other hidden data patterns. "This goes beyond bias from clues of race or gender," says Brandon Hill, a co-author of the study and a machine learning scientist at Dartmouth Hitchcock. "We found the algorithm could even learn to predict the year an X-ray was taken. It's pernicious -- when you prevent it from learning one of these elements, it will instead learn another it previously ignored. This danger can lead to some really dodgy claims, and researchers need to be aware of how readily this happens when using this technique." The findings underscore the need for rigorous evaluation standards in AI-based medical research. Overreliance on standard algorithms without deeper scrutiny could lead to erroneous clinical insights and treatment pathways. "The burden of proof just goes way up when it comes to using models for the discovery of new patterns in medicine," Hill says. "Part of the problem is our own bias. It is incredibly easy to fall into the trap of presuming that the model 'sees' the same way we do. In the end, it doesn't." "AI is almost like dealing with an alien intelligence," Hill continues. "You want to say the model is 'cheating,' but that anthropomorphizes the technology. It learned a way to solve the task given to it, but not necessarily how a person would. It doesn't have logic or reasoning as we typically understand it." Schilling, Hill, and study co-author Frances Koback, a third-year medical student in Dartmouth's Geisel School, conducted the study in collaboration with the Veterans Affairs Medical Center in White River Junction, Vt.