AI-Enabled Wearable Cameras Detect Medication Errors in Clinical Settings

Wearable cameras allow AI to detect medication errors

A team of researchers says it has developed the first wearable camera system that, with the help of artificial intelligence, detects potential errors in medication delivery. In a test whose results were published Oct. 22 in npj Digital Medicine, the video system recognized and identified, with high proficiency, which medications were being drawn in busy clinical settings. The AI achieved 99.6% sensitivity and 98.8% specificity in detecting vial-swap errors. The system could become a critical safeguard, especially in operating rooms, intensive-care units and emergency-medicine settings, said co-lead author Dr. Kelly Michaelsen, an assistant professor of anesthesiology and pain medicine at the University of Washington School of Medicine. "The thought of being able to help patients in real time or to prevent a medication error before it happens is very powerful," she said. "One can hope for a 100% performance but even humans cannot achieve that. In a survey of more than 100 anesthesia providers, the majority desired the system to be more than 95% accurate, which is a goal we achieved." Drug administration errors are the most frequently reported critical incidents in anesthesia, and the most common cause of serious medical errors in intensive care. In the bigger picture, an estimated 5% to 10% of all drugs given are associated with errors. Adverse events associated with injectable medications are estimated to affect 1.2 million patients annually at a cost of $5.1 billion. Syringe and vial-swap errors most often occur during intravenous injections in which a clinician must transfer the medication from vial to syringe to the patient. About 20% of mistakes are substitution errors in which the wrong vial is selected or a syringe is mislabeled. Another 20% of errors occur when the drug is labeled correctly but administered in error. Safety measures, such as a barcode system that quickly reads and confirms a vial's contents, are in place to guard against such accidents. But practitioners might sometimes forget this check during high-stress situations because it is an extra step in their workflow. The researchers' aim was to build a deep-learning model that, paired with a GoPro camera, is sophisticated enough to recognize the contents of cylindrical vials and syringes, and to appropriately render a warning before the medication enters the patient. Training the model took months. The investigators collected 4K video of 418 drug draws by 13 anesthesiology providers in operating rooms where setups and lighting varied. The video captured clinicians managing vials and syringes of select medications. These video snippets were later logged and the contents of the syringes and vials denoted to train the model to recognize the contents and containers. The video system does not directly read the wording on each vial, but scans for other visual cues: vial and syringe size and shape, vial cap color, label print size. "It was particularly challenging, because the person in the OR is holding a syringe and a vial, and you don't see either of those objects completely. Some letters (on the syringe and vial) are covered by the hands. And the hands are moving fast. They are doing the job. They aren't posing for the camera," said Shyam Gollakota, a co-author of the paper and professor at the UW's Paul G. Allen School of Computer Science & Engineering. Further, the computational model had to be trained to focus on medications in the foreground of the frame and to ignore vials and syringes in the background. "AI is doing all that: detecting the specific syringe that the health care provider is picking up, and not detecting a syringe that is lying on the table," Gollakota said. This work shows that AI and deep learning have the potential to improve safety and efficiency across a number of health care practices. Researchers are just beginning to probe the potential, Michaelsen said. The study also included researchers from Carnegie Mellon University and Makerere University in Uganda. The Toyota Research Institute built and tested the system.

Wearable cameras allow AI to detect medication errors

In a test whose results were published today, the video system recognized and identified, with high proficiency, which medications were being drawn in busy clinical settings. The AI achieved 99.6% sensitivity and 98.8% specificity at detecting vial-swap errors. The findings are reported Oct. 22 in npj Digital Medicine. The system could become a critical safeguard, especially in operating rooms, intensive-care units and emergency-medicine settings, said co-lead author Dr. Kelly Michaelsen, an assistant professor of anesthesiology and pain medicine at the University of Washington School of Medicine. "The thought of being able to help patients in real time or to prevent a medication error before it happens is very powerful," she said. "One can hope for a 100% performance but even humans cannot achieve that. In a survey of more than 100 anesthesia providers, the majority desired the system to be more than 95% accurate, which is a goal we achieved." Drug administration errors are the most frequently reported critical incidents in anesthesia, and the most common cause of serious medical errors in intensive care. In the bigger picture, an estimated 5% to 10% of all drugs given are associated with errors. Adverse events associated with injectable medications are estimated to affect 1.2 million patients annually at a cost of $5.1 billion. Syringe and vial-swap errors most often occur during intravenous injections in which a clinician must transfer the medication from vial to syringe to the patient. About 20% of mistakes are substitution errors in which the wrong vial is selected or a syringe is mislabeled. Another 20% of errors occur when the drug is labeled correctly but administered in error. Safety measures, such as a barcode system that quickly reads and confirms a vial's contents, are in place to guard against such accidents. But practitioners might sometimes forget this check during high-stress situations because it is an extra step in their workflow. The researchers' aim was to build a deep-learning model that, paired with a GoPro camera, is sophisticated enough to recognize the contents of cylindrical vials and syringes, and to appropriately render a warning before the medication enters the patient. Training the model took months. The investigators collected 4K video of 418 drug draws by 13 anesthesiology providers in operating rooms where setups and lighting varied. The video captured clinicians managing vials and syringes of select medications. These video snippets were later logged and the contents of the syringes and vials denoted to train the model to recognize the contents and containers. The video system does not directly read the wording on each vial, but scans for other visual cues: vial and syringe size and shape, vial cap color, label print size. "It was particularly challenging, because the person in the OR is holding a syringe and a vial, and you don't see either of those objects completely. Some letters (on the syringe and vial) are covered by the hands. And the hands are moving fast. They are doing the job. They aren't posing for the camera," said Shyam Gollakota, a coauthor of the paper and professor at the UW's Paul G. Allen School of Computer Science & Engineering. Further, the computational model had to be trained to focus on medications in the foreground of the frame and to ignore vials and syringes in the background. "AI is doing all that: detecting the specific syringe that the healthcare provider is picking up, and not detecting a syringe that is lying on the table," Gollakota said. This work shows that AI and deep learning have potential to improve safety and efficiency across a number of healthcare practices. Researchers are just beginning to probe the potential, Michaelsen said. The study also included researchers from Carnegie Mellon University and Makerere University in Uganda. The Toyota Research Institute built and tested the system. The Washington Research Foundation, Foundation for Anesthesia Education and Research, and a National Institutes of Health grant (K08GM153069) funded the work.

First wearable camera system detects medication errors with AI

University of Washington School of Medicine/UW MedicineOct 22 2024 A team of researchers says it has developed the first wearable camera system that, with the help of artificial intelligence, detects potential errors in medication delivery. In a test whose results were published today, the video system recognized and identified, with high proficiency, which medications were being drawn in busy clinical settings. The AI achieved 99.6% sensitivity and 98.8% specificity at detecting vial-swap errors. The findings are reported Oct. 22 in npj Digital Medicine. The system could become a critical safeguard, especially in operating rooms, intensive-care units and emergency-medicine settings, said co-lead author Dr. Kelly Michaelsen, an assistant professor of anesthesiology and pain medicine at the University of Washington School of Medicine. The thought of being able to help patients in real time or to prevent a medication error before it happens is very powerful. One can hope for a 100% performance but even humans cannot achieve that. In a survey of more than 100 anesthesia providers, the majority desired the system to be more than 95% accurate, which is a goal we achieved." Dr. Kelly Michaelsen, assistant professor of anesthesiology and pain medicine, University of Washington School of Medicine Drug administration errors are the most frequently reported critical incidents in anesthesia, and the most common cause of serious medical errors in intensive care. In the bigger picture, an estimated 5% to 10% of all drugs given are associated with errors. Adverse events associated with injectable medications are estimated to affect 1.2 million patients annually at a cost of $5.1 billion. Syringe and vial-swap errors most often occur during intravenous injections in which a clinician must transfer the medication from vial to syringe to the patient. About 20% of mistakes are substitution errors in which the wrong vial is selected or a syringe is mislabeled. Another 20% of errors occur when the drug is labeled correctly but administered in error. Safety measures, such as a barcode system that quickly reads and confirms a vial's contents, are in place to guard against such accidents. But practitioners might sometimes forget this check during high-stress situations because it is an extra step in their workflow. The researchers' aim was to build a deep-learning model that, paired with a GoPro camera, is sophisticated enough to recognize the contents of cylindrical vials and syringes, and to appropriately render a warning before the medication enters the patient. Training the model took months. The investigators collected 4K video of 418 drug draws by 13 anesthesiology providers in operating rooms where setups and lighting varied. The video captured clinicians managing vials and syringes of select medications. These video snippets were later logged and the contents of the syringes and vials denoted to train the model to recognize the contents and containers. The video system does not directly read the wording on each vial, but scans for other visual cues: vial and syringe size and shape, vial cap color, label print size. "It was particularly challenging, because the person in the OR is holding a syringe and a vial, and you don't see either of those objects completely. Some letters (on the syringe and vial) are covered by the hands. And the hands are moving fast. They are doing the job. They aren't posing for the camera," said Shyam Gollakota, a coauthor of the paper and professor at the UW's Paul G. Allen School of Computer Science & Engineering. Further, the computational model had to be trained to focus on medications in the foreground of the frame and to ignore vials and syringes in the background. "AI is doing all that: detecting the specific syringe that the healthcare provider is picking up, and not detecting a syringe that is lying on the table," Gollakota said. This work shows that AI and deep learning have potential to improve safety and efficiency across a number of healthcare practices. Researchers are just beginning to probe the potential, Michaelsen said. The study also included researchers from Carnegie Mellon University and Makerere University in Uganda. The Toyota Research Institute built and tested the system. The Washington Research Foundation, Foundation for Anesthesia Education and Research, and a National Institutes of Health grant (K08GM153069) funded the work. University of Washington School of Medicine/UW Medicine Journal reference: Chan, J., et al. (2024) Detecting clinical medication errors with AI enabled wearable cameras. npj Digital Medicine. doi.org/10.1038/s41746-024-01295-2.

These wearable cameras use AI to detect and prevent medication errors in operating rooms

In the high-stress conditions of operating rooms, emergency rooms and intensive care units, medical providers can swap syringes and vials, delivering the wrong medications to patients. Now a wearable camera system developed by the University of Washington uses artificial intelligence to provide an extra set of digital eyes in clinical settings, double-checking that meds don't get mixed up. The UW researchers found that the technology had 99.6% sensitivity and 98.8% specificity at identifying vial mix ups. "The thought of being able to help patients in real time or to prevent a medication error before it happens is very powerful," said Dr. Kelly Michaelsen, an assistant professor of anesthesiology and pain medicine at the UW School of Medicine. "One can hope for a 100% performance but even humans cannot achieve that." The frequency of drug administration mistakes -- namely injected medications -- is troubling. Research shows that at least 1 in 20 patients experience a preventable error in a clinical setting, and drug delivery is a leading cause of the mistakes, which can cause harm or death. Across healthcare, an estimated 5% to 10% of all drugs given are associated with errors, impacting more than a million patients annually and costing billions of dollars. To address the problem, researchers used GoPro cameras to collect videos of anesthesiology providers working in operating rooms, performing 418 drug draws. They added data to the videos to identify the content of the vials and syringes, and used that information to train their model. "It was particularly challenging, because the person in the [operating room] is holding a syringe and a vial, and you don't see either of those objects completely," said Shyam Gollakota, a coauthor of the paper and professor at the UW's Paul G. Allen School of Computer Science & Engineering. Given those real-world difficulties, the system doesn't read the labels but can recognize the vials and syringes by their size and shape, vial cap color and label print size. The system could ultimately incorporate an audible or visual signal to alert a provider that they've made a mistake before the drug is administered. Michaelsen said the goal is to commercialize the technology, but more testing is needed prior to large scale deployment. Gollakota added that next steps will involve training the system to detect more subtle errors, such as drawing the wrong volume of medication. Another potential strategy would be to pair the technology with devices such as Meta smart glasses. Michaelsen, Gollokota and their coauthors published their study today in npj Digital Medicine. Researchers from Carnegie Mellon University and Makerere University in Uganda also participated in the work. The Toyota Research Institute built and tested the system.

Researchers Develop First AI-Enabled Wearable Camera To Detect Drug Errors

Every year, roughly 1.2 million patients experience adverse outcomes associated with injectable medications, and these errors are estimated to cost about $5 billion. A team including a researcher from Carnegie Mellon University has developed the first wearable camera that uses artificial intelligence to help prevent such errors. "The goal of our system is to catch these drug administration errors in real-time, before the injection, and provide an alert so the clinician has a chance to intervene before any patient harm," said Justin Chan(opens in new window), an assistant professor in the School of Computer Science's Software and Societal Systems Department(opens in new window) and the College of Engineering's Electrical and Computer Engineering Department(opens in new window). In addition to Chan, the team included researchers from the University of Washington, Makerere University and the Toyota Research Institute. The error rate for all drugs given in a hospital is about 5% to 10% and errors can happen at all levels of care. To design the wearable camera system, researchers focused on training deep learning algorithms that could detect errors when a clinician transfers a drug from a vial into a syringe. These could be vial swap errors, which occur when the wrong vial is used or the drug label on the syringe is incorrect; or syringe swaps, when the label is correct but the clinician administers the wrong drug. To prevent errors, hospitals use safeguards like requiring barcode scanning for syringes, but in high-pressure situations, clinicians can forget to scan the drug's barcode or manually record its contents. In their study, published today in npj Digital Medicine(opens in new window), researchers demonstrated how the AI-enabled wearable camera system could detect vial swap errors with a sensitivity of 99.6% and a specificity of 98.8%. Chan said creating the deep learning system to detect errors as they happen was challenging because syringes, vials and drug labels are small and clinicians can inadvertently obscure them when handling them. Researchers collected a large training dataset from different operating room environments with different backgrounds and lighting conditions. They collected footage using a small head-mounted camera strapped to physicians' foreheads, and the camera was tilted down to film the providers' hands. Over 55 days, they collected 4K video footage of drug preparation events from 13 anesthesiology providers and 17 operating rooms in two hospitals. "We designed the algorithm so that instead of reading the label text, which can be obscured, it only needs to catch a glimpse of visual cues like vial and syringe shape, label color, and font size for a short period of time to classify what the drug is," Chan said. Now that researchers have shown the accuracy of the system, their next step is incorporating the system into smart eyewear that can provide visual or auditory warnings to clinicians before a drug is delivered to a patient. "This work demonstrates how AI-enabled systems can serve as a second set of 'eyes' to improve health care practices and patient safety," Chan said. "When integrated into an electronic medical system, our system also opens up opportunities for automatic documentation of drug information and can reduce the overhead of manual record-keeping."