Cornell Researchers Develop AI-Powered Ring for Sign Language Translation

This AI ring helps translate sign language.

The SpellRing was developed by Cornell researchers and uses AI plus micro-sonar tech to track real-time fingerspelling. Right now, it can be used to input text into computers and smartphones. It's a neat, given that the researchers say alternatives have been too bulky and impractical for deaf and hard-of-hearing communities. There are caveats. This isn't a consumer product yet and may never be. Plus, fingerspelling is only one aspect of American Sign Language. Still, you love to see people working on accessibility tech!

Wearable ring translates sign language into text

American Sign Language (ASL) has long enabled real-time conversations for English-speaking people who are deaf and hard-of-hearing. But discussions often face significant lags when one or more conversants aren't fluent in the language system. But by combining deep learning artificial intelligence and micro-sonar technologies, researchers at Cornell University are developing a new wearable to help overcome the communication barriers. With further refinement, SpellRing may one day facilitate entire conversations regardless of your ASL comprehension skills. ASL's earliest iterations developed in the early 18th century at the American School for the Deaf in Hartford, Connecticut. Today, around 400,000 people in the US utilize modern ASL, including a large number of children of deaf adults (CODA). Like any language, ASL often takes years of education and practice to reach fluency. Given that the majority of Americans don't regularly occupy spaces requiring it, however, the language still remains mostly relegated to populations that are deaf and hard-of-hearing. In the meantime, technological innovations haven't caught up with them. "Many other technologies that recognize fingerspelling in ASL have not been adopted by the deaf and hard-of-hearing community because the hardware is bulky and impractical," Hyunchul Lim, a Cornell information science doctoral student, said in a university profile on March 17. "We sought to develop a single ring to capture all of the subtle and complex finger movement in ASL." Lim and his colleagues previously worked on similar inventions through Cornell's Smart Computer Interfaces for Future Interactions (SciFi) Lab, including interpretational tools for facial expressions, virtual reality hand poses, and silent speech recognition. SpellRing builds off a previous iteration called Ring-a-Pose and relies on multiple inputs to analyze, interpret, and translate ASL fingerspelling gestures. The principle component is a quarter-sized 3D-printed ring casing that contains a small microphone and speaker, and is worn around the thumb. When the user begins fingerspelling, the microphone emits inaudible soundwaves that are subsequently detected by the microphone as a miniature gyroscope measures the hand motions. Meanwhile, a computer featuring a deep-learning algorithmic program analyzes and translates the resultant sonar images into individual letters in real-time on a computer screen. Researchers trained SpellRing with the help of 20 experienced and novice ASL signers as they spelled out over 20,000 words. Depending on length and difficulty, SpellRing's accuracy eventually ranged from 82-92 percent. "There's always a gap between the technical community who develop tools and the target community who use them. We've bridged some of that gap," said Cheng Zhang, an assistant professor of information science paper co-author. Despite the advances, SpellRing's designers know it is only an early phase. The wearable is currently limited to fingerspelling. ASL relies on a wider set of upper body movements, facial expressions, and other physicalities, and has more than 4,000 word signs. "Fingerspelling, while nuanced and challenging to track from a technical perspective, comprises but a fraction of ASL and is not representative of ASL as a language," said Jane Lu, study co-author and a linguistics doctoral student. "We still have a long way to go in developing comparable devices for full ASL recognition, but it's an exciting step in the right direction." Moving forward, the team plans to expand on SpellRing's capabilities to adapt the micro-sonar system for eyeglasses that assess a user's face and upper body.

AI ring tracks spelled words in American Sign Language

A Cornell University-led research team has developed an artificial intelligence-powered ring equipped with micro-sonar technology that can continuously and in real time track fingerspelling in American Sign Language (ASL). In its current form, SpellRing could be used to enter text into computers or smartphones via fingerspelling, which is used in ASL to spell out words without corresponding signs, such as proper nouns, names and technical terms. With further development, the device -- believed to be the first of its kind -- could revolutionize ASL translation by continuously tracking entire signed words and sentences. "Many other technologies that recognize fingerspelling in ASL have not been adopted by the deaf and hard-of-hearing community because the hardware is bulky and impractical," said Hyunchul Lim, a doctoral student in the field of information science. "We sought to develop a single ring to capture all of the subtle and complex finger movement in ASL." Lim is lead author of "SpellRing: Recognizing Continuous Fingerspelling in American Sign Language using a Ring," which will be presented at the Association of Computing Machinery's conference on Human Factors in Computing Systems (CHI), April 26-May 1 in Yokohama, Japan. SpellRing is worn on the thumb and equipped with a microphone and speaker. Together they send and receive inaudible sound waves that track the wearer's hand and finger movements, while a mini gyroscope tracks the hand's motion. A proprietary deep-learning algorithm then processes the sonar images and predicts the ASL fingerspelled letters in real time and with similar accuracy as many existing systems that require more hardware. Developers evaluated SpellRing with 20 experienced and novice ASL signers, having them naturally and continuously fingerspell a total of more than 20,000 words of varying lengths. SpellRing's accuracy rate was between 82% and 92%, depending on the difficulty of words. "There's always a gap between the technical community who develop tools and the target community who use them," said Cheng Zhang, assistant professor of information science and a paper co-author. "We've bridged some of that gap. We designed SpellRing for target users who evaluated it." Lim's future work will include integrating the micro-sonar system into eyeglasses to capture upper body movements and facial expressions, for a more comprehensive ASL translation system. "Deaf and hard-of-hearing people use more than their hands for ASL. They use facial expressions, upper body movements and head gestures," said Lim, who completed basic and intermediate ASL courses at Cornell as part of his SpellRing research. "ASL is a very complicated, complex visual language." This research was funded by the National Science Foundation.