MIT researchers use AI and ultrasound wristband to turn hand gestures into robot training data

MIT researchers channel AI to turn hand gestures into robot training data

CAMBRIDGE, Mass. (AP) -- Humanoid robots struggling with tasks like grasping a cup have a new teacher -- a person wearing an ultrasound wristband that captures the movement of muscles, tendons and ligaments beneath the skin. Researchers at the Massachusetts Institute of Technology developed the tool to collect data of human hand motion that could eventually help robots achieve the dexterity that has been difficult for machines to master. "Imagine people doing housework," said Xuanhe Zhao, an MIT professor of mechanical engineering. "We can use the data obtained by our system to train a robot to do exactly (that) housework with this dexterous hand motion." As much of the tech world is still captivated with artificial intelligence assistants that are taking on computer-based tasks, Zhao is among the scientists trying to imbue AI with more sensory data from the physical world. Beyond housework, the technology could help with other tasks that require flexing fingers and hands, such as surgery. The wristband uses high-frequency sound waves to "see" through its wearer's skin. It relays images of the muscle and tendon movements to a computer that uses AI to enable a nearby robotic hand to mimic the gestures. An AI algorithm is trained to decode images generated by the device into what engineers call degrees of freedom - specific ways a joint can bend or rotate. The human hand has 22 of them. In earlier systems, tracking even a fraction of those movements was a significant challenge. In laboratory demonstrations with eight volunteers, developers showed the wristband could precisely mirror hand gestures - including all 26 letters in American Sign Language - within 120 milliseconds. The wristband can operate wirelessly, meaning the controlling person and the receiving robot need not be in the same room. Beyond remote control, the team sees a path toward using the wristband to build huge datasets of human motion that could eventually enable humanoids to learn dexterous tasks without human guidance. ___ AP Technology Writer Matt O'Brien contributed to this report.

MIT's ultrasound wristband could teach humanoid robots human hand skills

Humanoid robots may soon gain more human-like dexterity thanks to a new wearable device developed by researchers at the Massachusetts Institute of Technology. The ultrasound wristband uses high-frequency sound waves to capture the movement of muscles, tendons, and ligaments beneath the skin, creating detailed data on human hand motions. According to the team, the data collected by the system could be used to train robots to perform household tasks by replicating the dexterous hand movements used by humans. Last year, researchers at the University of California San Diego developed a wearable AI system that accurately interprets arm gestures and controls robots, even during running, vehicle motion, or rough ocean conditions. For robots to perform delicate tasks with human-like precision, they must first understand the complex movements of the human hand. Researchers have now developed a wearable ultrasound wristband that could help bridge that gap by capturing detailed muscle and tendon activity beneath the skin and translating it into robotic motion. Tracking hand and finger movements is a key challenge in fields such as robotics and spatial computing. Existing solutions have significant limitations. Camera-based systems can struggle with restricted viewing angles and visual obstructions, while wearable devices based on strain sensors, inertial sensors, or electromyography often limit movement or lack the resolution needed to accurately capture continuous finger motions, according to the team's study paper. The new wristband builds on a series of advances in wearable ultrasound technology over the past several years. Researchers have steadily expanded the capabilities of ultrasound wearables beyond gesture tracking and robotics. In 2022, scientists demonstrated a wearable ultrasound device capable of continuously monitoring multiple internal organs. Subsequent developments included wearable systems for continuous heart imaging and bladder volume monitoring. In 2024, researchers introduced a conformal ultrasound patch capable of tracking cerebral blood flow in three dimensions and an integrated wearable ultrasound system that monitored deep tissue activity with sub-millimeter resolution using a single transducer. More recently, flexible, disposable ultrasound patches were developed to reduce manufacturing costs. Now, in a new study, researchers led by Xuanhe Zhao at MIT introduced a wearable ultrasound imaging wristband capable of tracking arbitrary hand configurations. The device uses high-frequency sound waves to image muscles, tendons, and ligaments inside the wrist, providing a detailed view of the mechanisms that drive hand movement. The wristband could help robots achieve human-like hand dexterity by capturing the movements of muscles and tendons beneath the skin. The device incorporates a 256-channel wireless ultrasound imaging system that uses high-frequency sound waves to monitor structures inside the wrist responsible for controlling finger and palm motions. The ultrasound data is processed by a hybrid Transformer-ResNet AI model capable of interpreting subtle muscle activity. The system continuously tracks all 22 degrees of freedom of the human hand, covering the full range of joint movements that allow fingers and the palm to bend, rotate, and coordinate complex actions. Achieving this level of tracking has been a longstanding challenge for hand-motion sensing technologies, reported AP News. In laboratory tests involving eight volunteers, the wristband reproduced hand gestures with high accuracy and low latency, mirroring movements within 120 milliseconds. Researchers demonstrated the system's ability to recognize all 26 letters of American Sign Language, highlighting its precision in capturing intricate finger configurations. Because the wristband operates wirelessly, users can control robotic systems remotely without being physically connected or located in the same room. To showcase its capabilities, the team used the technology to manipulate three-dimensional objects in virtual reality and direct a robotic hand to play the piano. Beyond teleoperation, researchers believe the system could be used to generate large-scale datasets of human hand movements, providing valuable training data for future humanoid robots and helping them learn complex manipulation skills more autonomously.

MIT researchers channel AI to turn hand gestures into robot training data

CAMBRIDGE, Mass. (AP) -- Humanoid robots struggling with tasks like grasping a cup have a new teacher -- a person wearing an ultrasound wristband that captures the movement of muscles, tendons and ligaments beneath the skin. Researchers at the Massachusetts Institute of Technology developed the tool to collect data of human hand motion that could eventually help robots achieve the dexterity that has been difficult for machines to master. "Imagine people doing housework," said Xuanhe Zhao, an MIT professor of mechanical engineering. "We can use the data obtained by our system to train a robot to do exactly (that) housework with this dexterous hand motion." As much of the tech world is still captivated with artificial intelligence assistants that are taking on computer-based tasks, Zhao is among the scientists trying to imbue AI with more sensory data from the physical world. Beyond housework, the technology could help with other tasks that require flexing fingers and hands, such as surgery. The wristband uses high-frequency sound waves to "see" through its wearer's skin. It relays images of the muscle and tendon movements to a computer that uses AI to enable a nearby robotic hand to mimic the gestures. An AI algorithm is trained to decode images generated by the device into what engineers call degrees of freedom - specific ways a joint can bend or rotate. The human hand has 22 of them. In earlier systems, tracking even a fraction of those movements was a significant challenge. In laboratory demonstrations with eight volunteers, developers showed the wristband could precisely mirror hand gestures - including all 26 letters in American Sign Language - within 120 milliseconds. The wristband can operate wirelessly, meaning the controlling person and the receiving robot need not be in the same room. Beyond remote control, the team sees a path toward using the wristband to build huge datasets of human motion that could eventually enable humanoids to learn dexterous tasks without human guidance. ___ AP Technology Writer Matt O'Brien contributed to this report.

MIT Researchers Channel AI to Turn Hand Gestures Into Robot Training Data

CAMBRIDGE, Mass. (AP) -- Humanoid robots struggling with tasks like grasping a cup have a new teacher -- a person wearing an ultrasound wristband that captures the movement of muscles, tendons and ligaments beneath the skin. Researchers at the Massachusetts Institute of Technology developed the tool to collect data of human hand motion that could eventually help robots achieve the dexterity that has been difficult for machines to master. "Imagine people doing housework," said Xuanhe Zhao, an MIT professor of mechanical engineering. "We can use the data obtained by our system to train a robot to do exactly (that) housework with this dexterous hand motion." As much of the tech world is still captivated with artificial intelligence assistants that are taking on computer-based tasks, Zhao is among the scientists trying to imbue AI with more sensory data from the physical world. Beyond housework, the technology could help with other tasks that require flexing fingers and hands, such as surgery. The wristband uses high-frequency sound waves to "see" through its wearer's skin. It relays images of the muscle and tendon movements to a computer that uses AI to enable a nearby robotic hand to mimic the gestures. An AI algorithm is trained to decode images generated by the device into what engineers call degrees of freedom - specific ways a joint can bend or rotate. The human hand has 22 of them. In earlier systems, tracking even a fraction of those movements was a significant challenge. In laboratory demonstrations with eight volunteers, developers showed the wristband could precisely mirror hand gestures - including all 26 letters in American Sign Language - within 120 milliseconds. The wristband can operate wirelessly, meaning the controlling person and the receiving robot need not be in the same room. Beyond remote control, the team sees a path toward using the wristband to build huge datasets of human motion that could eventually enable humanoids to learn dexterous tasks without human guidance. ___ AP Technology Writer Matt O'Brien contributed to this report.

MIT researchers channel AI to turn hand gestures into robot training data

Robots are learning to grasp objects with help from a new ultrasound wristband. Developed at MIT, this device captures human muscle and tendon movements. This data trains robots to perform tasks requiring fine hand control. The technology could enable humanoids to master housework and even assist in surgery. Humanoid robots struggling with tasks like grasping a cup have a new teacher - a person wearing an ultrasound wristband that captures the movement of muscles, tendons and ligaments beneath the skin. Researchers at the Massachusetts Institute of Technology developed the tool to collect data of human hand motion that could eventually help robots achieve the dexterity that has been difficult for machines to master. "Imagine people doing housework," said Xuanhe Zhao, an MIT professor of mechanical engineering. "We can use the data obtained by our system to train a robot to do exactly (that) housework with this dexterous hand motion." As much of the tech world is still captivated with artificial intelligence assistants that are taking on computer-based tasks, Zhao is among the scientists trying to imbue AI with more sensory data from the physical world. Beyond housework, the technology could help with other tasks that require flexing fingers and hands, such as surgery. The wristband uses high-frequency sound waves to "see" through its wearer's skin. It relays images of the muscle and tendon movements to a computer that uses AI to enable a nearby robotic hand to mimic the gestures. An AI algorithm is trained to decode images generated by the device into what engineers call degrees of freedom - specific ways a joint can bend or rotate. The human hand has 22 of them. In earlier systems, tracking even a fraction of those movements was a significant challenge. In laboratory demonstrations with eight volunteers, developers showed the wristband could precisely mirror hand gestures - including all 26 letters in American Sign Language - within 120 milliseconds. The wristband can operate wirelessly, meaning the controlling person and the receiving robot need not be in the same room. Beyond remote control, the team sees a path toward using the wristband to build huge datasets of human motion that could eventually enable humanoids to learn dexterous tasks without human guidance.