MIT researchers use AI and ultrasound wristband to turn hand gestures into robot training data

MIT Researchers Develop Ultrasound Wristband to Capture Human Hand Movements

MIT researchers have created a wearable ultrasound wristband that captures the movement of muscles, tendons, and ligaments beneath the skin to turn hand gestures into robot training data1

1

. The device addresses a longstanding challenge in robotics: teaching humanoid robots the dexterity required for complex tasks like grasping a cup or performing housework. Led by Xuanhe Zhao, an MIT professor of mechanical engineering, the team developed this tool to collect detailed data on human hand motion that could help robots achieve precision that has been difficult for machines to master3

3

.

The wristband incorporates a 256-channel wireless ultrasound imaging system that uses high-frequency sound waves to "see" through the wearer's skin2

2

. It relays images of muscle and tendon movements to a computer that uses AI to enable a nearby robotic hand to mimic the gestures. As Zhao explained, "Imagine people doing housework. We can use the data obtained by our system to train a robot to do exactly that housework with this dexterous hand motion"1

1

.

AI Algorithm Tracks 22 Degrees of Freedom with Precision

The ultrasound data is processed by a hybrid Transformer-ResNet AI algorithm trained to decode images generated by the device into what engineers call degrees of freedom—specific ways a joint can bend or rotate4

4

. The human hand has 22 of them, covering the full range of joint movements that allow fingers and the palm to bend, rotate, and coordinate complex actions. In earlier systems, tracking even a fraction of those movements was a significant challenge1

1

.

In laboratory demonstrations with eight volunteers, developers showed the wristband could precisely mirror hand gestures—including all 26 letters in American Sign Language—within 120 milliseconds3

3

. The system continuously tracks subtle muscle activity, achieving a level of precision that highlights its capability in capturing intricate finger configurations2

2

.

Wireless Operation Enables Remote Robot Control and Virtual Reality Manipulation

The wristband can operate wirelessly, meaning the controlling person and the receiving robot need not be in the same room5

5

. To showcase its capabilities, the team used the technology to manipulate three-dimensional objects in virtual reality and direct a robotic hand to play the piano2

2

. This wireless functionality opens possibilities for remote control applications across various fields.

Beyond housework, the technology could assist with other tasks that require flexing fingers and hands, such as surgery1

1

. As much of the tech world remains captivated with artificial intelligence assistants taking on computer-based tasks, Zhao is among the scientists working to imbue AI with more sensory data from the physical world4

4

.

Building Large Datasets for Autonomous Robot Learning

Beyond remote control, the team sees a path toward using the wristband to build large datasets of human motion that could eventually enable humanoids to learn dexterous tasks without human guidance3

3

. Researchers believe the system could generate large-scale datasets of human hand movements, providing valuable robot training data for future humanoid robots and helping them learn complex manipulation skills more autonomously2

2

.

This development builds on advances in wearable ultrasound technology over recent years. In 2022, scientists demonstrated a wearable ultrasound device capable of continuously monitoring multiple internal organs, with subsequent developments including wearable systems for continuous heart imaging and bladder volume monitoring2

2

. The new wristband represents a significant step forward in addressing robot dexterity challenges, potentially bridging the gap between human capabilities and machine performance in tasks requiring fine motor control.