MIT's ultrasound wristband lets users control robotic hands and play piano with 120ms delay

Ultrasound imaging turns a robot hand into a skillful mimic

By moving their own hand and fingers, users wearing a remote-control wristband can manipulate objects in a virtual environment or make a robot play the piano. Our hands are the nimblest parts of our bodies, coordinating 34 muscles, 27 joints, and over 100 tendons and ligaments to perform countless nuanced movements and gestures. So far, robots have been notoriously bad at mimicking that dexterity, in part because researchers struggle to capture what is actually going on under our skin in order to reproduce it. Now MIT researchers are pioneering a promising new approach. Mechanical engineering professor Xuanhe Zhao and colleagues at the Institute and the University of Southern California have designed a wristband equipped with an ultrasound "sticker" -- a miniaturized version of the transducers used in medical offices, paired with a hydrogel that can safely adhere to the skin. As the wearer's hand moves, the device produces ultrasound images of the wrist's muscles, tendons, and ligaments. Then an artificial-intelligence algorithm, trained on ultrasound images meticulously labeled by humans, continuously translates the images into the corresponding positions of the five fingers and the palm. "The tendons and muscles in your wrist are like strings pulling on puppets, which are your fingers," says Gengxi Lu, a former MIT postdoc and one of the lead authors of a paper on the work. "So the idea is: Each time you take a picture of the state of the strings, you'll know the state of the hand." In demonstrations, the team has shown that a person wearing the wristband can wirelessly control a robotic hand. As the person gestures or points, the robot does the same. In a sort of wireless marionette interaction, the wearer can manipulate the robot to play a simple tune on the piano and shoot a mini basketball into a desktop hoop. With the same wristband, a wearer can also manipulate objects on a computer screen -- for instance, pinching the fingers together to enlarge and minimize a virtual object. The researchers are planning to further miniaturize the wristband's hardware, which is currently similar in size to a cell phone. They also hope to train the AI software on movements from more volunteers with a wider variety of hand sizes, finger shapes, and gestures. They envision building a large data set of hand motions that can be plumbed, for instance, to train humanoid robots in delicate tasks such as surgical procedures. The ultrasound band could also be used to let people grasp, manipulate, and interact with objects in design applications, video games, or other virtual settings. Ultimately, the team is building toward a wearable hand tracker that anyone can use to wirelessly manipulate humanoid robots or virtual objects with high dexterity in real time. "We believe this is the most advanced way to track dexterous hand motion -- through wearable imaging of the wrist," Zhao says. "We think these wearable ultrasound bands can provide intuitive and versatile controls for virtual reality and robotic hands."

A wristband into a real-time hand reader, and the ultrasound system can make a robot play the piano, shoot mini basketballs, and mimic your fingers with surprising precision

Scientists have developed a revolutionary ultrasound wristband that tracks hand movements with remarkable accuracy. This wearable device, powered by AI, can decipher intricate finger and palm configurations in real-time, even translating them to control robotic hands. The technology promises to enhance virtual reality interactions and train robots for complex tasks, offering a glimpse into a future where human dexterity guides artificial limbs. Every day, your hands do something amazing, and most of us don't take a moment to think about it. Picking up a cup of coffee, threading a needle, typing on a keyboard: feats of biological engineering so complex that machines have struggled to copy them. Now a team at MIT may have found a promising new way forward, and it fits around your wrist. According to a study, 'Hand tracking using wearable wrist imaging,' published in Nature Electronics, researchers at MIT and the University of Southern California have created a fully integrated, wireless ultrasound imaging wristband that, when paired with an artificial intelligence algorithm, can continuously monitor arbitrary hand configurations of the five fingers and the palm in real time, with a delay of less than 120 milliseconds. The paper was led by MIT professor of mechanical engineering Xuanhe Zhao, and co-authored by Gengxi Lu, a former MIT postdoc, among others. Why your wrist holds the key The research is based on a beautiful insight. The tendons and muscles in your wrist act as control cables for your fingers. As lead co-author Gengxi Lu explained: "The tendons and muscles in your wrist are like strings pulling on puppets, which are your fingers. So the idea is, each time you take a picture of the state of the strings, you'll know the state of the hand." Sound waves through skin: how MIT turned a wristband into a motion capture studio. Image Credits: Melanie Gonick The wristband uses high-frequency sound waves to image the muscles, tendons, and ligaments in the wrist, providing a detailed, real-time view of the mechanisms that drive hand movement. The system tracks the full set of joint motions that allow for the fingers and palm to flex, rotate, and coordinate intricate activity, constantly monitoring all 22 degrees of freedom of the human hand. Then the ultrasound data is fed into an AI model to convert those images into continuous finger positions. What it can actually do In testing with eight volunteers of varying hand and wrist sizes, the wristband successfully tracked gestures, including all 26 letters of American Sign Language, as well as handling everyday objects such as a tennis ball, scissors, and a plastic bottle. During practical demonstrations, a person wearing the wristband showed how to guide a robotic hand to play a simple tune on a piano and shoot a miniature basketball into a desktop hoop. The same wristband allowed users to pinch, zoom, and manipulate objects on a computer screen smoothly and continuously. It's like a wireless marionette, but the strings are sound waves, and the puppet is a robotic hand. Why existing tools fall short The main methods for capturing hand motion today all have real limitations. Camera-based systems can track hands in 3D but require careful setup, can be blocked by other objects or people, and often have difficulty in different lighting conditions. Glove systems embed sensors in cloth, but the hardware can feel bulky, restrict natural movement, and deaden the sense of touch. An ultrasound wristband solves both problems by reading movement from the inside out, under the skin, where the real action is happening. Your fingers lead. The robot follows, with a 120-millisecond lag. Image Credits: Melanie Gonick Training the robots of tomorrow The implications go far beyond playing the piano or dropping through tiny hoops. According to the MIT News release, the team envisions collecting hand-motion data to build a dataset that could help train humanoid robots for dexterity tasks, such as performing certain surgical procedures. The ultrasound band could also be used to interact with objects in video games, design applications, or other virtual settings. What comes next Zhao's team hopes to shrink the hardware further and train the AI on data from more users. A future version could be lighter, longer-lasting, and easier to wear. The researchers also want the device to work across a wider range of hand sizes, without requiring lengthy personal calibration sessions. Zhao summed up both the scope and the ambition of the work: "We believe this is the most advanced way to track dexterous hand motion, through wearable imaging of the wrist. We think these wearable ultrasound bands can provide intuitive and versatile controls for virtual reality and robotic hands." For now, the tech is living in a lab. But what it hints at, a future where your own hands show robots how to move like humans, is starting to feel very real.