NVIDIA Unveils Advanced AI and Simulation Tools to Accelerate Robot Learning and Humanoid Development

NVIDIA Isaac Lab 1.2 Generally Available on GitHub, New Robot Learning Tools Out

In addition to Isaac Lab, six specialised workflows for their humanoid development platform, Project GR00T, were also released. NVIDIA has unveiled a range of advanced tools and workflows to expedite the development of robotics, including humanoids, at the Conference for Robot Learning (CoRL) in Munich, Germany. These new offerings, designed to enhance AI simulation and robotics, include the NVIDIA Isaac Lab's framework for robot learning, six specialised workflows for humanoid robot development under Project GR00T, and new video data processing tools, namely the NVIDIA Cosmos Tokenizer and NeMo Curator. This release marks the general availability of NVIDIA Isaac Lab, an open-source Omniverse-based framework tailored to train robots at scale across diverse forms, including humanoids and quadruped robots. Prominent robotics companies such as Boston Dynamics, Agility Robotics, and Swiss-Mile, have already adopted Isaac Lab for commercial and research purposes. This framework supports increasingly complex robotics tasks, enabling robots to perform intricate movements and interact effectively within environments. Project GR00T, NVIDIA's initiative for accelerating humanoid robot development, introduces six new workflows designed to address the core challenges of humanoid robotics. These include GR00T-Gen for creating AI-driven 3D environments, GR00T-Mimic for trajectory generation, GR00T-Dexterity for dexterous manipulation, GR00T-Control for body control, GR00T-Mobility for navigation, and GR00T-Perception for sensory processing. With NeMo Curator, an advanced video processing pipeline, NVIDIA plans to accelerate video data curation by up to 7x compared to conventional methods. Designed to manage extensive datasets, NeMo Curator incorporates automatic orchestration across multi-node, multi-GPU systems, handling over 100 petabytes of data. During CoRL, NVIDIA further solidified its leadership in robotics by presenting 23 research papers and conducting nine workshops focused on robot learning advancements. Among several other developments in robotics, NVIDIA recently rolled out HOVER, a 1.5-million-parameter neural network and DexMimicGen, a large-scale synthetic data generator. Earlier this year, NVIDIA, in partnership with Hugging Face, also launched LeRobot, an initiative to advance open-source robotics research using NVIDIA's Isaac Lab and Jetson platforms. Meta, also catching up in the open-source robotics race, recently released new research artefacts that allow robots to perceive touch. The race to open-sourcing robotic features is steadily expediting.

Nvidia advances robot learning and humanoid development with AI and simulation tools

Join our daily and weekly newsletters for the latest updates and exclusive content on industry-leading AI coverage. Learn More Nvidia revealed new AI and simulation tools that will advance robot learning and humanoid development. The world's biggest tech company by valuation (worth $3.432 trillion) said that the tools will enable robotics developers to greatly accelerate their work on AI-enabled robots, with tools revealed this week at the Conference for Robot Learning (CoRL) in Munich, Germany. The lineup includes the general availability of the Nvidia Isaac Lab robot learning framework; six new humanoid robot learning workflows for Project GR00T, an initiative to accelerate humanoid robot development; and new world-model development tools for video data curation and processing, including the Nvidia Cosmos tokenizer and Nvidia NeMo Curator for video processing. The open-source Cosmos tokenizer provides robotics developers superior visual tokenization by breaking down images and videos into high-quality tokens with exceptionally high compression rates. It runs up to 12 times faster than current tokenizers, while NeMo Curator provides video processing curation up to seven times faster than unoptimized pipelines. Also timed with CoRL, Nvidia released 23 papers and presented nine workshops related to robot learning, and also released training and workflow guides for developers. Further, Hugging Face and Nvidia announced they're collaborating to accelerate open-source robotics research with LeRobot, Nvidia Isaac Lab and Nvidia Jetson for the developer community. Accelerating robot development with Isaac Lab Nvidia Isaac Lab is an open-source, robot learning framework built on Nvidia Omniverse, a platform for developing OpenUSD applications for industrial digitalization and physical AI simulation. Developers can use Isaac Lab to train robot policies at scale. This open-source unified robot learning framework applies to any embodiment -- from humanoids to quadrupeds and collaborative robots -- to handle increasingly complex movements and interactions. Leading commercial robot makers, robotics application developers, and robotics research entities around the world are adopting Isaac Lab, including 1X, Agility Robotics, The AI Institute, Berkeley Humanoid, Boston Dynamics, Field AI, Fourier, Galbot, Mentee Robotics, Skild AI, Swiss-Mile, Unitree Robotics, and Xpeng Robotics. Project GR00T: Foundations for general-purpose humanoid robots The humanoids are coming. Building advanced humanoids is extremely difficult, demanding multilayer technological and interdisciplinary approaches to make the robots perceive, move and learn skills effectively for human-robot and robot-environment interactions. Project GR00T is an initiative to develop accelerated libraries, foundation models and data pipelines to accelerate the global humanoid robot developer ecosystem. Six new Project GR00T workflows provide humanoid developers with blueprints to realize the most challenging humanoid robot capabilities. They include things such as GR00T-Gen for building generative AI-powered, OpenUSD-based 3D environments and more. "Humanoid robots are the next wave of embodied AI," said Jim Fan, senior research manager of embodied AI at Nvidia, in a statement. "Nvidia research and engineering teams are collaborating across the company and our developer ecosystem to build Project GR00T to help advance the progress and development of global humanoid robot developers." New development tools for world model builders Today, robot developers are building world models -- AI representations of the world that can predict how objects and environments respond to a robot's actions. Building these world models is incredibly compute- and data-intensive with models requiring thousands of hours of real-world, curated image or video data. Nvidia Cosmos tokenizers provide efficient, high-quality encoding and decoding to simplify the development of these world models. They set a new standard of minimal distortion and temporal instability, enabling high-quality video and image reconstructions. Providing high-quality compression and up to 12 times faster visual reconstruction, the Cosmos tokenizer paves the path for scalable, robust and efficient development of generative applications across a broad spectrum of visual domains. 1X, a humanoid robot company, has updated the 1X World Model Challenge dataset to use the Cosmos tokenizer. "Nvidia Cosmos tokenizer achieves really high temporal and spatial compression of our data while still retaining visual fidelity," said Eric Jang, vice president of AI at 1X Technologies, in a statement. "This allows us to train world models with long horizon video generation in an even more compute-efficient manner." Other humanoid and general purpose robot developers including Xpeng Robotics and Hillbot are developing with the Nvidia Cosmos tokenizer to manage high-resolution images and videos. NeMo Curator NeMo Curator now includes a video processing pipeline. This enables robot developers to improve their world-model accuracy processing large-scale text, image and video data. Curating video data poses challenges due to its massive size, requiring scalable pipelines and efficient orchestration for load balancing across GPUs. Additionally, models for filtering, captioning and embedding need optimization to maximize throughput. NeMo Curator overcomes these challenges by streamlining data curation with automatic pipeline orchestration, reducing processing time significantly. It supports linear scaling across multi-node multi-GPU systems, efficiently handling over 100 petabytes of data. This simplifies AI development, reduces costs and accelerates time to market. Availability Nvidia Isaac Lab 1.2 is available now and is open source on GitHub. Nvidia Cosmos tokenizer is available now on GitHub and Hugging Face. NeMo Curator for video processing will be available at the end of the month. The new Nvidia Project GR00T workflows are coming soon to help robot companies build humanoid robot capabilities with greater ease. For researchers and developers learning to use Isaac Lab, new getting started developer guides and tutorials are now available, including an Isaac Gym to Isaac Lab migration guide.

NVIDIA Advances Robot Learning and Humanoid Development With New AI and Simulation Tools

New Project GR00T workflows and AI world model development technologies to accelerate robot dexterity, control, manipulation and mobility. Robotics developers can greatly accelerate their work on AI-enabled robots, including humanoids, using new AI and simulation tools and workflows that NVIDIA revealed this week at the Conference for Robot Learning (CoRL) in Munich, Germany. The lineup includes the general availability of the NVIDIA Isaac Lab robot learning framework; six new humanoid robot learning workflows for Project GR00T, an initiative to accelerate humanoid robot development; and new world-model development tools for video data curation and processing, including the NVIDIA Cosmos tokenizer and NVIDIA NeMo Curator for video processing. The open-source Cosmos tokenizer provides robotics developers superior visual tokenization by breaking down images and videos into high-quality tokens with exceptionally high compression rates. It runs up to 12x faster than current tokenizers, while NeMo Curator provides video processing curation up to 7x faster than unoptimized pipelines. Also timed with CoRL, NVIDIA presented 23 papers and nine workshops related to robot learning and released training and workflow guides for developers. Further, Hugging Face and NVIDIA announced they're collaborating to accelerate open-source robotics research with LeRobot, NVIDIA Isaac Lab and NVIDIA Jetson for the developer community. Developers can use Isaac Lab to train robot policies at scale. This open-source unified robot learning framework applies to any embodiment -- from humanoids to quadrupeds to collaborative robots -- to handle increasingly complex movements and interactions. Leading commercial robot makers, robotics application developers and robotics research entities around the world are adopting Isaac Lab, including 1X, Agility Robotics, The AI Institute, Berkeley Humanoid, Boston Dynamics, Field AI, Fourier, Galbot, Mentee Robotics, Skild AI, Swiss-Mile, Unitree Robotics and XPENG Robotics. Project GR00T: Foundations for General-Purpose Humanoid Robots Building advanced humanoids is extremely difficult, demanding multilayer technological and interdisciplinary approaches to make the robots perceive, move and learn skills effectively for human-robot and robot-environment interactions. Project GR00T is an initiative to develop accelerated libraries, foundation models and data pipelines to accelerate the global humanoid robot developer ecosystem. Six new Project GR00T workflows provide humanoid developers with blueprints to realize the most challenging humanoid robot capabilities. They include: "Humanoid robots are the next wave of embodied AI," said Jim Fan, senior research manager of embodied AI at NVIDIA. "NVIDIA research and engineering teams are collaborating across the company and our developer ecosystem to build Project GR00T to help advance the progress and development of global humanoid robot developers." New Development Tools for World Model Builders Today, robot developers are building world models -- AI representations of the world that can predict how objects and environments respond to a robot's actions. Building these world models is incredibly compute- and data-intensive, with models requiring thousands of hours of real-world, curated image or video data. NVIDIA Cosmos tokenizers provide efficient, high-quality encoding and decoding to simplify the development of these world models. They set a new standard of minimal distortion and temporal instability, enabling high-quality video and image reconstructions. Providing high-quality compression and up to 12x faster visual reconstruction, the Cosmos tokenizer paves the path for scalable, robust and efficient development of generative applications across a broad spectrum of visual domains. 1X, a humanoid robot company, has updated the 1X World Model Challenge dataset to use the Cosmos tokenizer. "NVIDIA Cosmos tokenizer achieves really high temporal and spatial compression of our data while still retaining visual fidelity," said Eric Jang, vice president of AI at 1X Technologies. "This allows us to train world models with long horizon video generation in an even more compute-efficient manner." Other humanoid and general-purpose robot developers, including XPENG Robotics and Hillbot, are developing with the NVIDIA Cosmos tokenizer to manage high-resolution images and videos. NeMo Curator now includes a video processing pipeline. This enables robot developers to improve their world-model accuracy by processing large-scale text, image and video data. Curating video data poses challenges due to its massive size, requiring scalable pipelines and efficient orchestration for load balancing across GPUs. Additionally, models for filtering, captioning and embedding need optimization to maximize throughput. NeMo Curator overcomes these challenges by streamlining data curation with automatic pipeline orchestration, reducing processing time significantly. It supports linear scaling across multi-node, multi-GPU systems, efficiently handling over 100 petabytes of data. This simplifies AI development, reduces costs and accelerates time to market. Advancing the Robot Learning Community at CoRL The nearly two dozen research papers the NVIDIA robotics team released with CoRL cover breakthroughs in integrating vision language models for improved environmental understanding and task execution, temporal robot navigation, developing long-horizon planning strategies for complex multistep tasks and using human demonstrations for skill acquisition. Groundbreaking papers for humanoid robot control and synthetic data generation include SkillGen, a system based on synthetic data generation for training robots with minimal human demonstrations, and HOVER, a robot foundation model for controlling humanoid robot locomotion and manipulation. NVIDIA researchers will also be participating in nine workshops at the conference. Learn more about the full schedule of events. Availability NVIDIA Isaac Lab 1.2 is available now and is open source on GitHub. NVIDIA Cosmos tokenizer is available now on GitHub and Hugging Face. NeMo Curator for video processing will be available at the end of the month. The new NVIDIA Project GR00T workflows are coming soon to help robot companies build humanoid robot capabilities with greater ease. Read more about the workflows on the NVIDIA Technical Blog. Researchers and developers learning to use Isaac Lab can now access developer guides and tutorials, including an Isaac Gym to Isaac Lab migration guide. Discover the latest in robot learning and simulation in an upcoming OpenUSD insider livestream on robot simulation and learning on Nov. 13, and attend the NVIDIA Isaac Lab office hours for hands-on support and insights. Developers can apply to join the NVIDIA Humanoid Robot Developer Program.

Nvidia announces new robot AI learning and simulation tools - SiliconANGLE

Nvidia announces new robot AI learning and simulation tools Nvidia Corp. today announced the release of new tools for developers working on artificial intelligence-enabled robots, including humanoids, that enable faster development cycles using simulation, blueprints and modeling. The company announced the new tools this week at the annual Conference for Robot Learning, a gathering focusing on the intersection of robotics and machine learning held in Munich, Germany. The tools included the general availability of Nvidia Isaac Lab, a robot learning framework; six new humanoid robot learning workflows for Project GR00T, to enable AI robot brain development; and new developer tools for video processing. Seeing and understanding the world is very important to robotics development. Video from cameras must be broken down so that AI models can process it. Nvidia announced the general availability of the open-source Cosmos tokenizer, which provides developers with high-quality tokens with exceptionally high compression rates that run 12x faster than current tokenizers. It couples with the NeMo Curator to optimize and understand inputs. This also allows developers to build better "world models," or AI representations of the world that can predict how objects and environments will respond when a robot performs actions. For example, what will happen when a robot gripper closes on a banana? A ripe banana is soft, so a robot gripper cannot close quickly or too hard, it will smash, deforming it and creating a mess. What about a piece of paper? It must be grabbed differently. Each of these situations involves high-quality encoding and decoding of video data. Eric Jang, vice-president of AI at 1X Technologies, a humanoid robot startup, explained the Cosmos tokenizer helps his company achieve high compression of data while still retaining extremely high visual quality. "This allows us to train world models with long horizon video generation in an even more compute-efficient manner," he said. Not all robot AI brains can be trained in the real world, so Nvidia released Isaac Lab, an open-source robot learning framework built on Omniverse, a digital twin simulation platform that allows developers to test and run robots in virtual worlds. Omniverse is a hyper-realistic real-time 3D graphics collaboration and simulation platform that allows artists, developers and enterprises to build 3D models and scenes of factories, cities and other spaces using fully actualized physics. This makes it a powerful tool for simulating virtual environments to train robots. Developers can use Isaac Lab to train robots and adjust policies at scale to understand performance and safety. The framework applies to any framework and robot embodiment, including arms, humanoids, quadrupeds, swarms and more. Nvidia said numerous commercial robot makers and research groups around the world have adopted Isaac Lab into their workflows including Agility Robots, Boston Dynamics, 1X, Galbot, Fourier, Mentee Robotics and Berkley Humanoid. Building and developing advanced humanoid robots is a tough challenge because what comes easily to humans -- walking, perceiving and taking action requires tremendous amounts of hardware engineering, AI training and AI compute to come together for robots to do even simple seemingly simple tasks. Project GR00T is an initiative from Nvidia that provides developers with AI foundation models for general-purpose humanoid robots, software libraries and data pipelines to help developers rapidly prototype and build faster. In an effort to provide developers with a leg-up building advanced humanoids, Nvidia announced six new Project GR00T workflow blueprints to help them build new capabilities into their robots. GR00T-Gen allows developers to create realistic simulated environments for training robots to move around in, manipulate objects, and perform other tasks. It uses large language models and 3D generative AI models to create visually diverse scenes and randomized scenes to help create robust training environments. GR00T-Mimic allows robots to learn from human teachers. Using this workflow, human demonstrators can teleoperate robots and perform actions in the same way that people would, such as walking around a warehouse, pulling boxes from shelves and placing them on carts. The idea is to allow the robot to mimic the same actions in the same environment. Nvidia said the approach uses a limited number of human demonstrations in the physical world using extended reality, such as Apple Vision Pro, and then scaling that motion data to help the robots produce more organic motion themselves. GR00T-Dexterity and GR00T-Control provide a suite of models and policies for fine-grained dexterous manipulation and broad body control for humanoid robots. Dexterity will help developers work with robots with highly dexterous hands that have actuators and knuckles and deal with missed grasps, grip force, and other grip motions. Control will help with motion planning for the entire body for walking, moving limbs or performing tasks. GR00T-Mobility provides developers with a set of models for helping humanoid robots walk and navigate around obstacles. Mobility is designed to allow for a learning-based approach that can generalize quickly to new environments. GR00T-Perception adds advanced software libraries and foundation models for human-robot interaction that help robots "remember" long histories of events. To do this, Nvidia added the aptly named ReMEmbR to Perception. This will give the robot a memory that personalizes human interactions and provides a context and spatial awareness to provide better perception, cognition and adaptability.