AI-Powered Real-Time American Sign Language Interpreter Breaks Communication Barriers

Engineers bring sign language to 'life' using AI to translate in real-time

For millions of deaf and hard-of-hearing individuals around the world, communication barriers can make everyday interactions challenging. Traditional solutions, like sign language interpreters, are often scarce, expensive and dependent on human availability. In an increasingly digital world, the demand for smart, assistive technologies that offer real-time, accurate and accessible communication solutions is growing, aiming to bridge this critical gap. American Sign Language (ASL) is one of the most widely used sign languages, consisting of distinct hand gestures that represent letters, words and phrases. Existing ASL recognition systems often struggle with real-time performance, accuracy and robustness across diverse environments. A major challenge in ASL systems lies in distinguishing visually similar gestures such as "A" and "T" or "M" and "N," which often leads to misclassifications. Additionally, the dataset quality presents significant obstacles, including poor image resolution, motion blur, inconsistent lighting, and variations in hand sizes, skin tones and backgrounds. These factors introduce bias and reduce the model's ability to generalize across different users and environments. To tackle these challenges, researchers from the College of Engineering and Computer Science at Florida Atlantic University have developed an innovative real-time ASL interpretation system. Combining the object detection power of YOLOv11 with MediaPipe's precise hand tracking, the system can accurately recognize ASL alphabet letters in real time. Using advanced deep learning and key hand point tracking, it translates ASL gestures into text, enabling users to interactively spell names, locations and more with remarkable accuracy. At its core, a built-in webcam serves as a contact-free sensor, capturing live visual data that is converted into digital frames for gesture analysis. MediaPipe identifies 21 keypoints on each hand to create a skeletal map, while YOLOv11 uses these points to detect and classify ASL letters with high precision. "What makes this system especially notable is that the entire recognition pipeline -- from capturing the gesture to classifying it -- operates seamlessly in real time, regardless of varying lighting conditions or backgrounds," said Bader Alsharif, the first author and a Ph.D. candidate in the FAU Department of Electrical Engineering and Computer Science. "And all of this is achieved using standard, off-the-shelf hardware. This underscores the system's practical potential as a highly accessible and scalable assistive technology, making it a viable solution for real-world applications." Results of the study, published in the journal Sensors, confirm the system's effectiveness, which achieved a 98.2% accuracy (mean Average Precision, [email protected]) with minimal latency. This finding highlights the system's ability to deliver high precision in real-time, making it an ideal solution for applications that require fast and reliable performance, such as live video processing and interactive technologies. With 130,000 images, the ASL Alphabet Hand Gesture Dataset includes a wide variety of hand gestures captured under different conditions to help models generalize better. These conditions cover diverse lighting environments (bright, dim and shadowed), a range of backgrounds (both outdoor and indoor scenes), and various hand angles and orientations to ensure robustness. Each image is carefully annotated with 21 keypoints, which highlight essential hand structures such as fingertips, knuckles and the wrist. These annotations provide a skeletal map of the hand, allowing models to distinguish between similar gestures with exceptional accuracy. "This project is a great example of how cutting-edge AI can be applied to serve humanity," said Imad Mahgoub, Ph.D., co-author and Tecore Professor in the FAU Department of Electrical Engineering and Computer Science. "By fusing deep learning with hand landmark detection, our team created a system that not only achieves high accuracy but also remains accessible and practical for everyday use. It's a strong step toward inclusive communication technologies." The deaf population in the U.S. is approximately 11 million, or 3.6% of the population, and about 15% of American adults (37.5 million) experience hearing difficulties. "The significance of this research lies in its potential to transform communication for the deaf community by providing an AI-driven tool that translates American Sign Language gestures into text, enabling smoother interactions across education, workplaces, health care and social settings," said Mohammad Ilyas, Ph.D., co-author and a professor in the FAU Department of Electrical Engineering and Computer Science. "By developing a robust and accessible ASL interpretation system, our study contributes to the advancement of assistive technologies to break down barriers for the deaf and hard of hearing population." Future work will focus on expanding the system's capabilities from recognizing individual ASL letters to interpreting full ASL sentences. This would enable more natural and fluid communication, allowing users to convey entire thoughts and phrases seamlessly. "This research highlights the transformative power of AI-driven assistive technologies in empowering the deaf community," said Stella Batalama, Ph.D., dean of the College of Engineering and Computer Science. "By bridging the communication gap through real-time ASL recognition, this system plays a key role in fostering a more inclusive society. It allows individuals with hearing impairments to interact more seamlessly with the world around them, whether they are introducing themselves, navigating their environment, or simply engaging in everyday conversations. This technology not only enhances accessibility but also supports greater social integration, helping create a more connected and empathetic community for everyone." Study co-authors are Easa Alalwany, Ph.D., a recent Ph.D. graduate of the FAU College of Engineering and Computer Science and an assistant professor at Taibah University in Saudi Arabia; Ali Ibrahim, Ph.D., a Ph.D. graduate of the FAU College of Engineering and Computer Science.

AI system translates American Sign Language in real-time - Earth.com

For millions of deaf and hard-of-hearing people, using sign language to communicate in a world built around spoken words can be exhausting. Whether it's ordering food, asking for directions, or taking part in a classroom discussion, barriers show up everywhere. While interpreters and captioning services can help, they're often limited, costly, or unavailable when most needed. As daily life becomes more digital, the need for smart tools that can translate sign language in real time is more urgent than ever. That's why researchers from Florida Atlantic University's College of Engineering and Computer Science have created a new system that could change the way we think about accessibility. They've developed a real-time American Sign Language (ASL) interpreter powered by artificial intelligence. This system uses deep learning and hand-tracking to convert ASL gestures into written text, all using a regular webcam and off-the-shelf hardware. ASL relies on precise hand shapes and movements to represent letters, words, and phrases. But current recognition tools often miss the mark - especially when signs look alike. For example, "A" and "T" or "M" and "N" can be hard to tell apart for machines. These tools also struggle in poor lighting, with motion blur, and with differences in hand shape or skin tone, all of which affect how accurate the machine interpretations are. To solve these issues, the FAU team combined two powerful tools: YOLOv11 for object detection and MediaPipe for detailed hand tracking. Together, they allow the system to detect and classify ASL alphabet letters with an accuracy of 98.2% (mean Average Precision at 0.5). The entire process works in real time and with very little delay. "What makes this system especially notable is that the entire recognition pipeline - from capturing the gesture to classifying it operates seamlessly in real time, regardless of varying lighting conditions or backgrounds," said Bader Alsharif, lead author on the study. "And all of this is achieved using standard, off-the-shelf hardware. This underscores the system's practical potential as a highly accessible and scalable assistive technology, making it a viable solution for real-world applications." At the center of the system is a basic webcam, which captures live video and turns it into digital frames. MediaPipe then pinpoints 21 key spots on each hand - fingertips, knuckles, and the wrist - to build a kind of skeleton map. These points help the system understand hand structure and motion. YOLOv11 uses this skeletal data to match hand gestures accurately to ASL letters. The researchers also built a massive dataset - 130,000 images strong - to train the model. These images include hands in a variety of lighting conditions, backgrounds, and angles. This diversity helps the system learn how to generalize across different people and environments, thus reducing the chance of bias. "This project is a great example of how cutting-edge AI can be applied to serve humanity," said Imad Mahgoub, a co-author on the publication. "By fusing deep learning with hand landmark detection, our team created a system that not only achieves high accuracy but also remains accessible and practical for everyday use. It's a strong step toward inclusive communication technologies." The deaf and hard-of-hearing community is large and diverse. In the U.S. alone, about 11 million people - roughly 3.6% of the population - are deaf or have significant hearing loss. Around 37.5 million adults experience some level of hearing difficulty. That's a lot of people who could benefit from better communication tools. "The significance of this research lies in its potential to transform communication for the deaf community by providing an AI-driven tool that translates American Sign Language gestures into text, enabling smoother interactions across education, workplaces, health care and social settings," commented Mohammad Ilyas, co-author of the research study. "By developing a robust and accessible ASL interpretation system, our study contributes to the advancement of assistive technologies to break down barriers for the deaf and hard-of-hearing population." While this new tool already shows strong results for recognizing the ASL alphabet, the team isn't stopping there. They're now working to expand the system to understand full ASL sentences. This would make communication even more natural and fluent, moving from spelling out words to sharing entire ideas. "This research highlights the transformative power of AI-driven assistive technologies in empowering the deaf community," said Stella Batalama, Dean of the Department of Electrical Engineering at FAU. "By bridging the communication gap through real-time ASL recognition, this system plays a key role in fostering a more inclusive society." "It allows individuals with hearing impairments to interact more seamlessly with the world around them, whether they are introducing themselves, navigating their environment, or simply engaging in everyday conversations." This technology enhances accessibility and supports greater social integration, helping create a more connected and empathetic community for everyone. With continued development, this AI-powered tool may soon become part of daily life, helping millions to communicate more freely - one gesture at a time. Like what you read? Subscribe to our newsletter for engaging articles, exclusive content, and the latest updates.

Engineers bring sign language to 'life' using AI to translate in real-time

For millions of deaf and hard-of-hearing individuals around the world, communication barriers can make everyday interactions challenging. Traditional solutions, like sign language interpreters, are often scarce, expensive and dependent on human availability. In an increasingly digital world, the demand for smart, assistive technologies that offer real-time, accurate and accessible communication solutions is growing, aiming to bridge this critical gap. American Sign Language (ASL) is one of the most widely used sign languages, consisting of distinct hand gestures that represent letters, words and phrases. Existing ASL recognition systems often struggle with real-time performance, accuracy and robustness across diverse environments. A major challenge in ASL systems lies in distinguishing between visually similar gestures such as "A" and "T" or "M" and "N," which often leads to misclassifications. Additionally, the dataset quality presents significant obstacles, including poor image resolution, motion blur, inconsistent lighting, and variations in hand sizes, skin tones and backgrounds. These factors introduce bias and reduce the model's ability to generalize across different users and environments. To tackle these challenges, researchers from the College of Engineering and Computer Science at Florida Atlantic University have developed an innovative real-time ASL interpretation system. Combining the object detection power of YOLOv11 with MediaPipe's precise hand tracking, the system can accurately recognize ASL alphabet letters in real time. Using advanced deep learning and key hand point tracking, it translates ASL gestures into text, enabling users to interactively spell names, locations and more with remarkable accuracy. At its core, a built-in webcam serves as a contact-free sensor, capturing live visual data that is converted into digital frames for gesture analysis. MediaPipe identifies 21 keypoints on each hand to create a skeletal map, while YOLOv11 uses these points to detect and classify ASL letters with high precision. "What makes this system especially notable is that the entire recognition pipeline -- from capturing the gesture to classifying it -- operates seamlessly in real time, regardless of varying lighting conditions or backgrounds," said Bader Alsharif, the first author and a Ph.D. candidate in the FAU Department of Electrical Engineering and Computer Science. "And all of this is achieved using standard, off-the-shelf hardware. This underscores the system's practical potential as a highly accessible and scalable assistive technology, making it a viable solution for real-world applications." Results of the study, published in the journal Sensors, confirm the system's effectiveness, which achieved a 98.2% accuracy (mean Average Precision, [email protected]) with minimal latency. This finding highlights the system's ability to deliver high precision in real-time, making it an ideal solution for applications that require fast and reliable performance, such as live video processing and interactive technologies. With 130,000 images, the ASL Alphabet Hand Gesture Dataset includes a wide variety of hand gestures captured under different conditions to help models generalize better. These conditions cover diverse lighting environments (bright, dim and shadowed), a range of backgrounds (both outdoor and indoor scenes), and various hand angles and orientations to ensure robustness. Each image is carefully annotated with 21 keypoints, which highlight essential hand structures such as fingertips, knuckles and the wrist. These annotations provide a skeletal map of the hand, allowing models to distinguish between similar gestures with exceptional accuracy. "This project is a great example of how cutting-edge AI can be applied to serve humanity," said Imad Mahgoub, Ph.D., co-author and Tecore Professor in the FAU Department of Electrical Engineering and Computer Science. "By fusing deep learning with hand landmark detection, our team created a system that not only achieves high accuracy but also remains accessible and practical for everyday use. It's a strong step toward inclusive communication technologies." The deaf population in the U.S. is approximately 11 million, or 3.6% of the population, and about 15% of American adults (37.5 million) experience hearing difficulties. "The significance of this research lies in its potential to transform communication for the deaf community by providing an AI-driven tool that translates American Sign Language gestures into text, enabling smoother interactions across education, workplaces, health care and social settings," said Mohammad Ilyas, Ph.D., co-author and a professor in the FAU Department of Electrical Engineering and Computer Science. "By developing a robust and accessible ASL interpretation system, our study contributes to the advancement of assistive technologies to break down barriers for the deaf and hard of hearing population." Future work will focus on expanding the system's capabilities from recognizing individual ASL letters to interpreting full ASL sentences. This would enable more natural and fluid communication, allowing users to convey entire thoughts and phrases seamlessly. "This research highlights the transformative power of AI-driven assistive technologies in empowering the deaf community," said Stella Batalama, Ph.D., dean of the College of Engineering and Computer Science. "By bridging the communication gap through real-time ASL recognition, this system plays a key role in fostering a more inclusive society. "It allows individuals with hearing impairments to interact more seamlessly with the world around them, whether they are introducing themselves, navigating their environment, or simply engaging in everyday conversations. This technology not only enhances accessibility but also supports greater social integration, helping create a more connected and empathetic community for everyone." Study co-authors are Easa Alalwany, Ph.D., a recent Ph.D. graduate of the FAU College of Engineering and Computer Science and an assistant professor at Taibah University in Saudi Arabia; Ali Ibrahim, Ph.D., a Ph.D. graduate of the FAU College of Engineering and Computer Science.

Engineers Bring Sign Language to 'Life' Using AI to Translate in Real-Time | Newswise

Newswise -- For millions of deaf and hard-of-hearing individuals around the world, communication barriers can make everyday interactions challenging. Traditional solutions, like sign language interpreters, are often scarce, expensive and dependent on human availability. In an increasingly digital world, the demand for smart, assistive technologies that offer real-time, accurate and accessible communication solutions is growing, aiming to bridge this critical gap. American Sign Language (ASL) is one of the most widely used sign languages, consisting of distinct hand gestures that represent letters, words and phrases. Existing ASL recognition systems often struggle with real-time performance, accuracy and robustness across diverse environments. A major challenge in ASL systems lies in distinguishing visually similar gestures such as "A" and "T" or "M" and "N," which often leads to misclassifications. Additionally, the dataset quality presents significant obstacles, including poor image resolution, motion blur, inconsistent lighting, and variations in hand sizes, skin tones and backgrounds. These factors introduce bias and reduce the model's ability to generalize across different users and environments. To tackle these challenges, researchers from the College of Engineering and Computer Science at Florida Atlantic University have developed an innovative real-time ASL interpretation system. Combining the object detection power of YOLOv11 with MediaPipe's precise hand tracking, the system can accurately recognize ASL alphabet letters in real time. Using advanced deep learning and key hand point tracking, it translates ASL gestures into text, enabling users to interactively spell names, locations and more with remarkable accuracy. At its core, a built-in webcam serves as a contact-free sensor, capturing live visual data that is converted into digital frames for gesture analysis. MediaPipe identifies 21 keypoints on each hand to create a skeletal map, while YOLOv11 uses these points to detect and classify ASL letters with high precision. "What makes this system especially notable is that the entire recognition pipeline - from capturing the gesture to classifying it - operates seamlessly in real time, regardless of varying lighting conditions or backgrounds," said Bader Alsharif, the first author and a Ph.D. candidate in the FAU Department of Electrical Engineering and Computer Science. "And all of this is achieved using standard, off-the-shelf hardware. This underscores the system's practical potential as a highly accessible and scalable assistive technology, making it a viable solution for real-world applications." Results of the study, published in the journal Sensors, confirm the system's effectiveness, which achieved a 98.2% accuracy (mean Average Precision, [email protected]) with minimal latency. This finding highlights the system's ability to deliver high precision in real-time, making it an ideal solution for applications that require fast and reliable performance, such as live video processing and interactive technologies. With 130,000 images, the ASL Alphabet Hand Gesture Dataset includes a wide variety of hand gestures captured under different conditions to help models generalize better. These conditions cover diverse lighting environments (bright, dim and shadowed), a range of backgrounds (both outdoor and indoor scenes), and various hand angles and orientations to ensure robustness. Each image is carefully annotated with 21 keypoints, which highlight essential hand structures such as fingertips, knuckles and the wrist. These annotations provide a skeletal map of the hand, allowing models to distinguish between similar gestures with exceptional accuracy. "This project is a great example of how cutting-edge AI can be applied to serve humanity," said Imad Mahgoub, Ph.D., co-author and Tecore Professor in the FAU Department of Electrical Engineering and Computer Science. "By fusing deep learning with hand landmark detection, our team created a system that not only achieves high accuracy but also remains accessible and practical for everyday use. It's a strong step toward inclusive communication technologies." The deaf population in the U.S. is approximately 11 million, or 3.6% of the population, and about 15% of American adults (37.5 million) experience hearing difficulties. "The significance of this research lies in its potential to transform communication for the deaf community by providing an AI-driven tool that translates American Sign Language gestures into text, enabling smoother interactions across education, workplaces, health care and social settings," said Mohammad Ilyas, Ph.D., co-author and a professor in the FAU Department of Electrical Engineering and Computer Science. "By developing a robust and accessible ASL interpretation system, our study contributes to the advancement of assistive technologies to break down barriers for the deaf and hard of hearing population." Future work will focus on expanding the system's capabilities from recognizing individual ASL letters to interpreting full ASL sentences. This would enable more natural and fluid communication, allowing users to convey entire thoughts and phrases seamlessly. "This research highlights the transformative power of AI-driven assistive technologies in empowering the deaf community," said Stella Batalama, Ph.D., dean of the College of Engineering and Computer Science. "By bridging the communication gap through real-time ASL recognition, this system plays a key role in fostering a more inclusive society. It allows individuals with hearing impairments to interact more seamlessly with the world around them, whether they are introducing themselves, navigating their environment, or simply engaging in everyday conversations. This technology not only enhances accessibility but also supports greater social integration, helping create a more connected and empathetic community for everyone." Study co-authors are Easa Alalwany, Ph.D., a recent Ph.D. graduate of the FAU College of Engineering and Computer Science and an assistant professor at Taibah University in Saudi Arabia; Ali Ibrahim, Ph.D., a Ph.D. graduate of the FAU College of Engineering and Computer Science. - FAU - About FAU's College of Engineering and Computer Science: The FAU College of Engineering and Computer Science is internationally recognized for cutting-edge research and education in the areas of computer science and artificial intelligence (AI), computer engineering, electrical engineering, biomedical engineering, civil, environmental and geomatics engineering, mechanical engineering, and ocean engineering. Research conducted by the faculty and their teams expose students to technology innovations that push the current state-of-the art of the disciplines. The College research efforts are supported by the National Science Foundation (NSF), the National Institutes of Health (NIH), the Department of Defense (DOD), the Department of Transportation (DOT), the Department of Education (DOEd), the State of Florida, and industry. The FAU College of Engineering and Computer Science offers degrees with a modern twist that bear specializations in areas of national priority such as AI, cybersecurity, internet-of-things, transportation and supply chain management, and data science. New degree programs include Master of Science in AI (first in Florida), Master of Science and Bachelor in Data Science and Analytics, and the new Professional Master of Science and Ph.D. in computer science for working professionals. For more information about the College, please visit eng.fau.edu.