Mount Sinai Researchers Develop AEquity: A Tool to Combat Bias in Healthcare AI

Innovative approach advances equity in health care AI

Mount Sinai Health SystemSep 5 2025 A team of researchers at the Icahn School of Medicine at Mount Sinai has developed a new method to identify and reduce biases in datasets used to train machine-learning algorithms-addressing a critical issue that can affect diagnostic accuracy and treatment decisions. The findings were published in the September 4 online issue of the Journal of Medical Internet Research [DOI: 10.2196/71757]. To tackle the problem, the investigators developed AEquity, a tool that helps detect and correct bias in health care datasets before they are used to train artificial intelligence (AI) and machine-learning models. The investigators tested AEquity on different types of health data, including medical images, patient records, and a major public health survey, the National Health and Nutrition Examination Survey, using a variety of machine-learning models. The tool was able to spot both well-known and previously overlooked biases across these datasets. AI tools are increasingly used in health care to support decisions, ranging from diagnosis to cost prediction. But these tools are only as accurate as the data used to train them. Some demographic groups may not be proportionately represented in a dataset. In addition, many conditions may present differently or be overdiagnosed across groups, the investigators say. Machine-learning systems trained on such data can perpetuate and amplify inaccuracies, creating a feedback loop of suboptimal care, such as missed diagnoses and unintended outcomes. Our goal was to create a practical tool that could help developers and health systems identify whether bias exists in their data-and then take steps to mitigate it. We want to help ensure these tools work well for everyone, not just the groups most represented in the data." Faris Gulamali, MD, first author The research team reported that AEquity is adaptable to a wide range of machine-learning models, from simpler approaches to advanced systems like those powering large language models. It can be applied to both small and complex datasets and can assess not only the input data, such as lab results or medical images, but also the outputs, including predicted diagnoses and risk scores. The study's results further suggest that AEquity could be valuable for developers, researchers, and regulators alike. It may be used during algorithm development, in audits before deployment, or as part of broader efforts to improve fairness in health care AI. "Tools like AEquity are an important step toward building more equitable AI systems, but they're only part of the solution," says senior corresponding author Girish N. Nadkarni, MD, MPH, Chair of the Windreich Department of Artificial Intelligence and Human Health, Director of the Hasso Plattner Institute for Digital Health, and the Irene and Dr. Arthur M. Fishberg Professor of Medicine at the Icahn School of Medicine at Mount Sinai, and the Chief AI Officer of the Mount Sinai Health System. "If we want these technologies to truly serve all patients, we need to pair technical advances with broader changes in how data is collected, interpreted, and applied in health care. The foundation matters, and it starts with the data." "This research reflects a vital evolution in how we think about AI in health care-not just as a decision-making tool, but as an engine that improves health across the many communities we serve," says David L. Reich MD, Chief Clinical Officer of the Mount Sinai Health System and President of The Mount Sinai Hospital. "By identifying and correcting inherent bias at the dataset level, we're addressing the root of the problem before it impacts patient care. This is how we build broader community trust in AI and ensure that resulting innovations improve outcomes for all patients, not just those best represented in the data. It's a critical step in becoming a learning health system that continuously refines and adapts to improve health for all." The paper is titled "Detecting, Characterizing, and Mitigating Implicit and Explicit Racial Biases in Health Care Datasets With Subgroup Learnability: Algorithm Development and Validation Study." The study's authors, as listed in the journal, are Faris Gulamali, Ashwin Shreekant Sawant, Lora Liharska, Carol Horowitz, Lili Chan, Patricia Kovatch, Ira Hofer, Karandeep Singh, Lynne Richardson, Emmanuel Mensah, Alexander Charney, David Reich, Jianying Hu, and Girish Nadkarni. The study was funded by the National Center for Advancing Translational Sciences and the National Institutes of Health. Mount Sinai Health System Journal reference: Gulamali, F., et al. (2025). Detecting, Characterizing, and Mitigating Implicit and Explicit Racial Biases in Health Care Datasets With Subgroup Learnability: Algorithm Development and Validation Study. Journal of Medical Internet Research. doi.org/10.2196/71757

New AI tool addresses accuracy and fairness in data to improve health algorithms

A team of researchers at the Icahn School of Medicine at Mount Sinai has developed a new method to identify and reduce biases in datasets used to train machine-learning algorithms -- addressing a critical issue that can affect diagnostic accuracy and treatment decisions. The findings were published in the Journal of Medical Internet Research. The paper is titled "Detecting, Characterizing, and Mitigating Implicit and Explicit Racial Biases in Health Care Datasets With Subgroup Learnability: Algorithm Development and Validation Study." To tackle the problem, the investigators developed AEquity, a tool that helps detect and correct bias in health care datasets before they are used to train artificial intelligence (AI) and machine-learning models. The investigators tested AEquity on different types of health data, including medical images, patient records, and a major public health survey, the National Health and Nutrition Examination Survey, using a variety of machine-learning models. The tool was able to spot both well-known and previously overlooked biases across these datasets. AI tools are increasingly used in health care to support decisions, ranging from diagnosis to cost prediction. But these tools are only as accurate as the data used to train them. Some demographic groups may not be proportionately represented in a dataset. In addition, many conditions may present differently or be overdiagnosed across groups, the investigators say. Machine-learning systems trained on such data can perpetuate and amplify inaccuracies, creating a feedback loop of suboptimal care, such as missed diagnoses and unintended outcomes. "Our goal was to create a practical tool that could help developers and health systems identify whether bias exists in their data -- and then take steps to mitigate it," says first author Faris Gulamali, MD. "We want to help ensure these tools work well for everyone, not just the groups most represented in the data." The research team reported that AEquity is adaptable to a wide range of machine-learning models, from simpler approaches to advanced systems like those powering large language models. It can be applied to both small and complex datasets and can assess not only the input data, such as lab results or medical images, but also the outputs, including predicted diagnoses and risk scores. The study's results further suggest that AEquity could be valuable for developers, researchers, and regulators alike. It may be used during algorithm development, in audits before deployment, or as part of broader efforts to improve fairness in health care AI. "Tools like AEquity are an important step toward building more equitable AI systems, but they're only part of the solution," says senior corresponding author Girish N. Nadkarni, MD, MPH, Chair of the Windreich Department of Artificial Intelligence and Human Health, Director of the Hasso Plattner Institute for Digital Health, and the Irene and Dr. Arthur M. Fishberg Professor of Medicine at the Icahn School of Medicine at Mount Sinai, and the Chief AI Officer of the Mount Sinai Health System. "If we want these technologies to truly serve all patients, we need to pair technical advances with broader changes in how data is collected, interpreted, and applied in health care. The foundation matters, and it starts with the data." "This research reflects a vital evolution in how we think about AI in health care -- not just as a decision-making tool, but as an engine that improves health across the many communities we serve," says David L. Reich MD, Chief Clinical Officer of the Mount Sinai Health System and President of The Mount Sinai Hospital. "By identifying and correcting inherent bias at the dataset level, we're addressing the root of the problem before it impacts patient care. This is how we build broader community trust in AI and ensure that resulting innovations improve outcomes for all patients, not just those best represented in the data. It's a critical step in becoming a learning health system that continuously refines and adapts to improve health for all."

New AI Tool Addresses Accuracy and Fairness in Data to Improve Health Algorithms | Newswise

Newswise -- New York, NY [September 4, 2025] -- A team of researchers at the Icahn School of Medicine at Mount Sinai has developed a new method to identify and reduce biases in datasets used to train machine-learning algorithms -- addressing a critical issue that can affect diagnostic accuracy and treatment decisions. The findings were published in the September 4 online issue of the Journal of Medical Internet Research [DOI: 10.2196/71757]. To tackle the problem, the investigators developed AEquity, a tool that helps detect and correct bias in health care datasets before they are used to train artificial intelligence (AI) and machine-learning models. The investigators tested AEquity on different types of health data, including medical images, patient records, and a major public health survey, the National Health and Nutrition Examination Survey, using a variety of machine-learning models. The tool was able to spot both well-known and previously overlooked biases across these datasets. AI tools are increasingly used in health care to support decisions, ranging from diagnosis to cost prediction. But these tools are only as accurate as the data used to train them. Some demographic groups may not be proportionately represented in a dataset. In addition, many conditions may present differently or be overdiagnosed across groups, the investigators say. Machine-learning systems trained on such data can perpetuate and amplify inaccuracies, creating a feedback loop of suboptimal care, such as missed diagnoses and unintended outcomes. "Our goal was to create a practical tool that could help developers and health systems identify whether bias exists in their data -- and then take steps to mitigate it," says first author Faris Gulamali, MD. "We want to help ensure these tools work well for everyone, not just the groups most represented in the data." The research team reported that AEquity is adaptable to a wide range of machine-learning models, from simpler approaches to advanced systems like those powering large language models. It can be applied to both small and complex datasets and can assess not only the input data, such as lab results or medical images, but also the outputs, including predicted diagnoses and risk scores. The study's results further suggest that AEquity could be valuable for developers, researchers, and regulators alike. It may be used during algorithm development, in audits before deployment, or as part of broader efforts to improve fairness in health care AI. "Tools like AEquity are an important step toward building more equitable AI systems, but they're only part of the solution," says senior corresponding author Girish N. Nadkarni, MD, MPH, Chair of the Windreich Department of Artificial Intelligence and Human Health, Director of the Hasso Plattner Institute for Digital Health, and the Irene and Dr. Arthur M. Fishberg Professor of Medicine at the Icahn School of Medicine at Mount Sinai, and the Chief AI Officer of the Mount Sinai Health System. "If we want these technologies to truly serve all patients, we need to pair technical advances with broader changes in how data is collected, interpreted, and applied in health care. The foundation matters, and it starts with the data." "This research reflects a vital evolution in how we think about AI in health care -- not just as a decision-making tool, but as an engine that improves health across the many communities we serve," says David L. Reich MD, Chief Clinical Officer of the Mount Sinai Health System and President of The Mount Sinai Hospital. "By identifying and correcting inherent bias at the dataset level, we're addressing the root of the problem before it impacts patient care. This is how we build broader community trust in AI and ensure that resulting innovations improve outcomes for all patients, not just those best represented in the data. It's a critical step in becoming a learning health system that continuously refines and adapts to improve health for all." The paper is titled "Detecting, Characterizing, and Mitigating Implicit and Explicit Racial Biases in Health Care Datasets With Subgroup Learnability: Algorithm Development and Validation Study." The study's authors, as listed in the journal, are Faris Gulamali, Ashwin Shreekant Sawant, Lora Liharska, Carol Horowitz, Lili Chan, Patricia Kovatch, Ira Hofer, Karandeep Singh, Lynne Richardson, Emmanuel Mensah, Alexander Charney, David Reich, Jianying Hu, and Girish Nadkarni. The study was funded by the National Center for Advancing Translational Sciences and the National Institutes of Health. For more Mount Sinai artificial intelligence news, visit: https://icahn.mssm.edu/about/artificial-intelligence. -####- About Mount Sinai's Windreich Department of AI and Human Health Led by Girish N. Nadkarni, MD, MPH -- an international authority on the safe, effective, and ethical use of AI in health care -- Mount Sinai's Windreich Department of AI and Human Health is the first of its kind at a U.S. medical school, pioneering transformative advancements at the intersection of artificial intelligence and human health. The Department is committed to leveraging AI in a responsible, effective, ethical, and safe manner to transform research, clinical care, education, and operations. By bringing together world-class AI expertise, cutting-edge infrastructure, and unparalleled computational power, the department is advancing breakthroughs in multi-scale, multimodal data integration while streamlining pathways for rapid testing and translation into practice. The Department benefits from dynamic collaborations across Mount Sinai, including with the Hasso Plattner Institute for Digital Health at Mount Sinai -- a partnership between the Hasso Plattner Institute for Digital Engineering in Potsdam, Germany, and the Mount Sinai Health System -- which complements its mission by advancing data-driven approaches to improve patient care and health outcomes. At the heart of this innovation is the renowned Icahn School of Medicine at Mount Sinai, which serves as a central hub for learning and collaboration. This unique integration enables dynamic partnerships across institutes, academic departments, hospitals, and outpatient centers, driving progress in disease prevention, improving treatments for complex illnesses, and elevating quality of life on a global scale. In 2024, the Department's innovative NutriScan AI application, developed by the Mount Sinai Health System Clinical Data Science team in partnership with Department faculty, earned Mount Sinai Health System the prestigious Hearst Health Prize. NutriScan is designed to facilitate faster identification and treatment of malnutrition in hospitalized patients. This machine learning tool improves malnutrition diagnosis rates and resource utilization, demonstrating the impactful application of AI in health care. For more information on Mount Sinai's Windreich Department of AI and Human Health, visit: ai.mssm.edu About the Hasso Plattner Institute at Mount Sinai At the Hasso Plattner Institute for Digital Health at Mount Sinai, the tools of data science, biomedical and digital engineering, and medical expertise are used to improve and extend lives. The Institute represents a collaboration between the Hasso Plattner Institute for Digital Engineering in Potsdam, Germany, and the Mount Sinai Health System. Under the leadership of Girish Nadkarni, MD, MPH, who directs the Institute, and Professor Lothar Wieler, a globally recognized expert in public health and digital transformation, they jointly oversee the partnership, driving innovations that positively impact patient lives while transforming how people think about personal health and health systems. The Hasso Plattner Institute for Digital Health at Mount Sinai receives generous support from the Hasso Plattner Foundation. Current research programs and machine learning efforts focus on improving the ability to diagnose and treat patients. About the Icahn School of Medicine at Mount Sinai The Icahn School of Medicine at Mount Sinai is internationally renowned for its outstanding research, educational, and clinical care programs. It is the sole academic partner for the seven member hospitals* of the Mount Sinai Health System, one of the largest academic health systems in the United States, providing care to New York City's large and diverse patient population. The Icahn School of Medicine at Mount Sinai offers highly competitive MD, PhD, MD-PhD, and master's degree programs, with enrollment of more than 1,200 students. It has the largest graduate medical education program in the country, with more than 2,600 clinical residents and fellows training throughout the Health System. Its Graduate School of Biomedical Sciences offers 13 degree-granting programs, conducts innovative basic and translational research, and trains more than 560 postdoctoral research fellows. Ranked 11th nationwide in National Institutes of Health (NIH) funding, the Icahn School of Medicine at Mount Sinai is among the 99th percentile in research dollars per investigator according to the Association of American Medical Colleges. More than 4,500 scientists, educators, and clinicians work within and across dozens of academic departments and multidisciplinary institutes with an emphasis on translational research and therapeutics. Through Mount Sinai Innovation Partners (MSIP), the Health System facilitates the real-world application and commercialization of medical breakthroughs made at Mount Sinai. -------------------------------------------------------