A smartwatch's speaker and mic can now track your whole body

Cornell's SciFi Lab, a research group led by Cheng Zhang that builds wearable interfaces, has built a system that tracks a person's full-body pose using only a smartwatch's built-in speaker and microphone. The watch emits ultrasonic waves and reads the echoes bouncing off the wearer's limbs, sonar-style. The team presented the work, called EchoMotion, at the International Symposium on Wearable Computers, part of the UbiComp conference series, in Shanghai on October 13, and it appears in the Association for Computing Machinery's Proceedings on Interactive, Mobile, Wearable and Ubiquitous Technologies 1

1

. Lead author Saif Mahmud, a Cornell doctoral student, calls it "a very important jump for body tracking" 1

1

.

Why a watch needs sonar

Today's wearables carry accelerometers and gyroscopes that sense only the watch's own movement, so tracking a whole body has traditionally meant cameras or a pile of extra devices such as phones, earbuds or virtual-reality headsets 1

1

. EchoMotion needs none of that. A deep learning model listens to the echoes and predicts where 12 joints sit, from the head down to the ankles 1

1

.

In a first test, 10 volunteers wore a 3D-printed prototype while doing 18 exercises, with poses checked against camera footage 1

1

. On a stock Samsung Galaxy Watch 6, 15 more participants averaged per-joint error under 4 inches before calibration 1

1

. A one-time 12-minute calibration cut that by more than a third, to about 2.4 inches 1

1

. The same model then ran without retraining on a Galaxy Watch 7 and a Google Pixel Watch 3, with only a slight dip in accuracy 2

2

.

Power is what makes this practical on a real watch. Sensing used roughly 12 milliwatts, about a fortieth of what a wearable camera draws 1

1

. A half-hour session on a Galaxy Watch 6 took 5 to 6% of the battery 1

1

. The whole software package is under 30MB and runs on Wear OS watches with no custom hardware, adding nothing to the bill of materials, per Digital Trends 2

2

.

Where it wobbles

The weak spot showed up in crowded spaces. In a narrow hallway, echoes bounced off other people and accuracy dipped slightly, although furniture and even noise from a nearby road caused no trouble 1

1

. Mahmud expects that training on thousands of users rather than the dozens tested so far will fix it 1

1

.

Fitness features come next

Fitness is the near-term target. In a demo reported by Digital Trends, the system recognized 20 gym workouts, from dumbbell lateral raises to squats, with 95.4% accuracy 2

2

. It kept working in real gyms with music playing at about 76 decibels 2

2

. The team is now building rep counting and form correction, something that today, per Digital Trends, means an in-person trainer or a camera-based remote service 2

2

. It is also exploring smart glasses that pair the sonar with a camera view to track objects in your hand 1

1

. Because the system captures joints and angles rather than just wrist motion, the researchers believe it could suit workouts that need precise feedback, like Pilates, tai chi or yoga 1

1

. They also see a role in at-home rehabilitation 1

1

.

Mahmud says the team worked hard to make the system run on ordinary commercial watches so that adoption would be easy 1

1

. "In the very near future, with wearables like smartwatches or smart glasses, a personal agent will write a description of your whole day -- what you did, what you ate," Mahmud said, framing body tracking as one part of that vision 1

1

.