Apple Pioneers New Training Method for Humanoid Robots Using Vision Pro and Human Demonstrations

Apple's Innovative Approach to Robot Training

In a groundbreaking study titled "Humanoid Policy ~ Human Policy," Apple researchers have introduced a novel method for training humanoid robots that could revolutionize the field of robotics 1

1

2

2

. The research, conducted in collaboration with MIT, Carnegie Mellon, the University of Washington, and UC San Diego, explores the use of first-person footage of human demonstrations to train general-purpose robot models.

The PH2D Dataset and HAT Model

At the heart of this innovation is the Physical Human-Humanoid Data (PH2D) dataset, comprising over 25,000 human demonstrations and 1,500 robot demonstrations 1

1

. This data is processed by a unified AI policy called the Human-humanoid Action Transformer (HAT), which can control a real humanoid robot in the physical world 2

2

.

The HAT model is designed to learn a single policy that generalizes across both human and robot bodies, making the system more flexible and data-efficient. This shared training approach has shown promising results, enabling robots to handle more challenging tasks, including ones they hadn't encountered before 1

1

.

Leveraging Apple Vision Pro for Data Collection

To collect the training data, the team developed an innovative application for the Apple Vision Pro 1

1

. The app captures video from the device's bottom-left camera and utilizes Apple's ARKit to track 3D head and hand motion 2

2

. This setup allows for high-quality demonstrations to be recorded in seconds, a significant improvement over traditional robot tele-operation methods.

Cost-Effective Alternatives

Recognizing the need for more affordable solutions, the researchers also explored using modified consumer products. They 3D-printed a mount to attach a ZED Mini Stereo camera to other headsets, such as the Meta Quest 3, offering similar 3D motion tracking capabilities at a lower cost 1

1

2

2

.

Overcoming Human-Robot Speed Differences

An interesting challenge the researchers faced was the speed disparity between human and robot movements. To address this, they slowed down the human demonstrations by a factor of four during training, allowing the robot to keep pace without requiring further adjustments 1

1

.

Improved Performance and Generalization

The study suggests that this combined training strategy offers significant benefits. Robots trained using this approach demonstrated better results in select tasks, such as vertical object grasping, compared to those trained exclusively with robot demonstrators 2

2

.

Future Implications

While Apple has only publicly demonstrated a robot-lamp prototype so far, rumors suggest the company is working on a mobile robot for consumers that could perform household chores and simple tasks 2

2

. This research could pave the way for more advanced and versatile humanoid robots in the future.

Conclusion

Apple's research represents a significant step forward in robotics training, potentially making the development of humanoid robots more scalable and cost-effective. By combining human demonstrations with traditional robot data, this approach could accelerate progress in the field and bring us closer to the reality of general-purpose humanoid robots in our daily lives.