Curated by THEOUTPOST
On Wed, 13 Nov, 12:01 AM UTC
2 Sources
[1]
Generative AI taught a robot dog to scramble around a new environment
Now, there's potentially a better option: a new system that uses generative AI models in conjunction with a physics simulator to develop virtual training grounds that more accurately mirror the physical world. Robots trained using this method worked with a higher success rate than those trained using more traditional techniques during real-world tests. Researchers used the system, called LucidSim, to train a robot dog in parkour, getting it to scramble over a box and climb stairs, despite never seeing any real world data. The approach demonstrates how helpful generative AI could be when it comes to teaching robots to do challenging tasks. It also raises the possibility that we could ultimately train them in entirely virtual worlds. The research was presented at the Conference on Robot Learning (CoRL) last week. "We're in the middle of an industrial revolution for robotics," says Ge Yang, a postdoc scholar at MIT CSAIL who worked on the project. "This is our attempt at understanding the impact of these [generative AI] models outside of their original intended purposes, with the hope that it will lead us to the next generation of tools and models." LucidSim uses a combination of generative AI models to create the visual training data. Firstly, the researchers generated thousands of prompts for ChatGPT, getting it to create descriptions of a range of environments that represent the conditions the robot will encounter in the real world, including different types of weather, times of day, and lighting conditions. For example, these included 'an ancient alley lined with tea houses and small, quaint shops, each displaying traditional ornaments and calligraphy' and 'the sun illuminates a somewhat unkempt lawn dotted with dry patches.' These descriptions were fed into a system which maps 3D geometry and physics data onto AI-generated images, creating short videos mapping the trajectory the robot will follow. The robot draws on this information to work out the height, width and depth of the things it has to navigate -- a box or a set of stairs, for example. The researchers tested LucidSim by instructing a four-legged robot equipped with a webcam to complete several tasks, including locating a traffic cone or soccer ball, climbing over a box and walking up and down stairs. The robot performed consistently better than when it ran a system trained on traditional simulations. Out of 20 trials to locate the cone, LucidSim had a 100% success rate, compared to 70% for systems trained on standard simulations. Similarly, LucidSim reached the soccer ball in another 20 trials 85% of the time, compared to just 35% for the other system. Finally, when the robot was running LucidSim, it successfully completed all 10 stair-climbing trials, compared to just 50% for the other system. These results are likely to improve even further in the future if LucidSim draws directly from sophisticated generative video models rather than a rigged-together combination of language, image and physics models, says Phillip Isola, an associate professor at MIT who worked on the research. The researchers' approach to using generative AI is a novel one that will pave the way for more interesting new research, says Mahi Shafiullah, a PhD student at New York University who is using AI models to train robots, and did not work on the project.
[2]
Virtual training uses generative AI to teach robots how to traverse real world terrain
A team of roboticists and engineers at MIT CSAIL, Institute for AI and Fundamental Interactions, has developed a generative AI approach to teaching robots how to traverse terrain and move around objects in the real world. The group has published a paper describing their work and possible uses for it on the arXiv preprint server. They also presented their ideas at the recent Conference on Robot Learning (CORL 2024), held in Munich Nov. 6-9. Getting robots to navigate in the real world at some point involves teaching them to learn on the fly, or by training them with videos of similar robots in a real-world environment. While such training has proven to be effective in limited environments, it tends to fail when a robot encounters something novel. In this new effort, the team at MIT developed virtual training that better translates to the real world. The work involved using generative AI and a physics simulator to allow a robot to navigate a virtual world as a means for learning to operate in the real world. They call the system LucidSim and have used it to train a robotic dog in parkour, a sport where players attempt to traverse obstacles in unknown territory as quickly as possible. The approach involves first prompting ChatGPT with thousands of queries designed to get the LLM to create descriptions of a wide range of environments, including outdoor weather. Next, the descriptions given by ChatGPT are fed to a 3D mapping system that uses them (along with AI generated images and physics simulators) to generate a video that also gives a trajectory for the robot to follow. The robot is then trained to make its way through the terrain in the virtual world and learn skills that it can use in a real environment. Robots trained using the system learned to clamber over boxes, climb stairs and deal with whatever they encountered. After virtual training, the robot was tested in the real world. The researchers tested their system using a small, four-legged robot equipped with a webcam. They found it performed better than a similar system trained the traditional way. The team suggests that improvements to their system could lead to a new approach to training robots in general.
Share
Share
Copy Link
MIT researchers develop LucidSim, a novel system using generative AI and physics simulators to train robots in virtual environments, significantly improving their real-world performance in navigation and obstacle traversal.
Researchers at MIT's Computer Science and Artificial Intelligence Laboratory (CSAIL) have introduced LucidSim, a groundbreaking system that leverages generative AI to enhance robot training for real-world applications. This innovative approach combines generative AI models with physics simulators to create virtual training environments that more accurately reflect real-world conditions 1.
LucidSim utilizes a multi-step process to generate comprehensive training data:
The researchers tested LucidSim using a four-legged robot equipped with a webcam. The robot was tasked with various challenges, including:
LucidSim-trained robots consistently outperformed those trained using traditional simulation methods:
Ge Yang, a postdoc scholar at MIT CSAIL, describes this development as part of an "industrial revolution for robotics." The research team believes that LucidSim could pave the way for training robots entirely in virtual worlds, potentially transforming the field of robotics 1.
Phillip Isola, an associate professor at MIT involved in the research, suggests that future iterations of LucidSim could achieve even better results by directly incorporating sophisticated generative video models 1.
Mahi Shafiullah, a PhD student at New York University specializing in AI-based robot training, commends the novel approach of LucidSim. Shafiullah, who was not involved in the project, believes this research will inspire further interesting developments in the field 1.
As generative AI continues to evolve, systems like LucidSim could revolutionize robot training methodologies, enabling machines to adapt more effectively to complex, real-world environments. This breakthrough has significant implications for various industries, from manufacturing and logistics to search and rescue operations.
Reference
[1]
MIT researchers have created a new method called Heterogeneous Pretrained Transformers (HPT) that uses generative AI to train robots for multiple tasks more efficiently, potentially revolutionizing the field of robotics.
6 Sources
6 Sources
The Genesis Project, an open-source simulation platform, is transforming robotics training by enabling ultra-fast, AI-powered virtual environments for robot learning and development.
6 Sources
6 Sources
Physical Intelligence, a San Francisco startup, has developed π0 (pi-zero), a generalist AI model for robotics that enables various robots to perform a wide range of household tasks with remarkable dexterity and adaptability.
2 Sources
2 Sources
NVIDIA introduces a three-computer solution to advance physical AI and robotics, combining training, simulation, and runtime systems to revolutionize industries from manufacturing to smart cities.
2 Sources
2 Sources
Google DeepMind unveils Gemini Robotics and Gemini Robotics-ER, advanced AI models designed to control robots with improved generalization, adaptability, and dexterity. These models, built on the Gemini 2.0 language model, aim to create more intuitive and capable robots for various tasks.
27 Sources
27 Sources
The Outpost is a comprehensive collection of curated artificial intelligence software tools that cater to the needs of small business owners, bloggers, artists, musicians, entrepreneurs, marketers, writers, and researchers.
© 2025 TheOutpost.AI All rights reserved