Apple Pioneers New Training Method for Humanoid Robots Using Vision Pro and Human Demonstrations

New Apple study teaches robots how to act by watching humans - 9to5Mac

In a new paper called "Humanoid Policy ∼ Human Policy," Apple researchers propose an interesting way to train humanoid robots. And it involves wearing an Apple Vision Pro. The project is a collaboration between Apple, MIT, Carnegie Mellon, the University of Washington, and UC San Diego. It explores how first-person footage of people manipulating objects can be used to train general-purpose robot models. In total, the researchers gathered over 25,000 human demonstrations and 1,500 robot demonstrations (a dataset they called PH2D), and fed them into a unified AI policy that could then control a real humanoid robot in the physical world. As the authors explain: Training manipulation policies for humanoid robots with diverse data enhances their robustness and generalization across tasks and platforms. However, learning solely from robot demonstrations is labor-intensive and requires expensive teleoperated data collection, which is difficult to scale. This paper investigates a more scalable data source, egocentric human demonstrations, to serve as cross-embodiment training data for robot learning. Their solution? Let humans show the way. To collect the training data, the team developed an Apple Vision Pro app that captures video from the device's bottom-left camera, and uses Apple's ARKit to track 3D head and hand motion. However, to explore a more affordable solution, they also 3D-printed a mount to attach a ZED Mini Stereo camera to other headsets, like the Meta Quest 3, offering similar 3D motion tracking at a lower cost. The result was a setup that let them record high-quality demonstrations in seconds, a pretty big improvement over traditional robot tele-op methods, which are slower, more expensive, and harder to scale. And here's one last interesting detail: since people move way faster than robots, the researchers slowed down the human demos by a factor of four during training, just enough for the robot to keep up without needing further adjustments. The key to the whole study is the HAT, a model trained on both human and robot demonstrations in a shared format. Instead of splitting the data by source (humans v. robots), HAT learns a single policy that generalizes across both types of bodies, making the system more flexible and data-efficient. In some tests, this shared training approach helped the robot handle more challenging tasks, including ones that it hadn't seen before, when compared to more traditional methods. Overall, the study is pretty interesting and worth checking out if you are into robotics. Is the idea of a house humanoid robot scary, exciting, or pointless to you? Let us know in the comments.

Apple details combined training method for humanoid robots

Apple has published a research paper that details a new training method for humanoid robots. An Apple research paper suggests that humanoid robots can be more effectively trained with human instructors as well as robot demonstrators, which is part of a new combined approach the company calls "PH2D." On Wednesday, a week after the company revealed its Matrix3D and StreamBridge AI models, Apple published new research on robots and how to train them. The iPhone maker's previous robotics efforts included the creation of a robotic lamp, among other things, but Apple's latest study deals with humanoid robots specifically. The research paper, titled "Humanoid Policy ~ Human Policy," details the inadequacies of traditional robot-training methods and proposes a new solution that's both scalable and cost-effective. Rather than relying solely on robot demonstrators for humanoid robot training, a process the paper says is "labor-intensive" while also requiring "expensive teleoperated data collection," Apple's study suggests a combined approach. This involves the use of human instructors, alongside robot demonstrators, as part of the process. This aims to reduce training-related costs, as Apple's study explains that the company was able to produce training material for humanoid robots through the use of modified consumer products. Specifically, an Apple Vision Pro was modified to use only the lower left camera for visual observation, while Apple's ARKit was used to access 3D head and hand poses. The company also utilized a modified Meta Quest headset equipped with mini ZED Stereo cameras, which effectively made it a low-cost training option. The modified headsets were used to train humanoid robots' hand manipulation. Human instructors were told to sit upright and perform actions with their hands. This included grasping and lifting specific objects as well as pouring liquids, among other things, and audible instructions were provided as the actions were recorded. The resulting footage was then slowed down so that it could be used for humanoid robot training. Apple created a model that can process the training material created by human instructors, as well as that created by robotic demonstrators -- the paper calls this "Physical Human-Humanoid Data" or PH2D. The model that deals with the data, known as the "Human-humanoid Action Transformer" or HAT, is capable of processing input created by both humans and robots alike. The company's researchers were able to unify into a human and robot demonstration sources "generalizable policy framework." Apple's unique approach leads to "improved generalization and robustness compared to the counterpart trained using only real-robot data," the research paper says. Apple's study suggests that there are significant benefits to using this combined training strategy. Alongside its cost-effective nature, robots trained with the approach delivered better results compared to those where robot demonstrators were exclusively used. This only applies to select tasks, however, such as vertical object grasping. The company will likely implement this training method for future products. Though it has only demonstrated its robot-lamp prototype so far, Apple is said to be working on a mobile robot for end consumers that could perform chores and simple tasks.