Toyota and Boston Dynamics Join Forces to Develop AI-Powered Humanoid Robots

Toyota joins with Hyundai's Boston Dynamics on AI-powered robots

Toyota Motor Corp.'s research unit and Hyundai Motor Co.'s Boston Dynamics are joining forces to speed up development of humanoid robots with artificial intelligence. The partnership will pair Toyota Research Institute's expertise in large behavior model learning for machines with Boston Dynamics' humanoid Atlas robot, they said Wednesday. Boston-based teams from TRI and Boston Dynamics will conduct research on use cases for AI-trained robots in areas such as human-robot interaction, they said. Toyota has said it's made a breakthrough with AI in teaching robots to learn and Boston Dynamics, which was bought by Hyundai in 2020, has had commercial success with its robotic guard dog and a mobile robotic arm for re-stocking warehouses. Their collaboration comes as a potential challenge to other smart bot programs such as the Optimus robot showcased last week by rival Tesla Inc. A number of humanoid robot startups are also attracting attention -- and billions of dollars in capital. The cooperation between Toyota and Boston Dynamics will focus on fundamental research with an eye toward eventual commercial use, but executives at the companies declined to specify a timeline or disclose their budgets for the project. "This kind of technology has tremendous promise for the future," Gill Pratt, Toyota's chief scientist, said in an interview. "The work that we're doing in generative AI can be a tremendous complement to the kind of work that Boston Dynamics has done." Pratt said that a goal is to eventually bring robots onto factory assembly lines and into homes for elder care. Carmakers have been at the vanguard of efforts to automate more assembly line processes to trim labor costs and bolster worker safety. The automotive industry deploys more robots in factories globally than any other industry, with about one-quarter of new installations in 2023, according to the Frankfurt-based International Federation of Robotics. The dexterity of Boston Dynamics robots paired with Toyota's behavioral know-how is a key differentiator with other companies' efforts to build smart robots, Aaron Saunders, Boston Dynamics' chief technology officer, told Bloomberg. "That's going to set the stage to deliver on the promise that a lot of people are making right now in bringing humanoid robots out to the world at scale," he said. Saunders said Boston Dynamics plans to deploy humanoid robots with narrowly-focused tasks in Hyundai factories "in the coming years" while it pursues research with Toyota on the longer-term objective of AI-enabled, multi-tasking robots. Hyundai, which already has a pilot program with Boston Dynamics technology at a Kia brand car plant in South Korea, bought a controlling 80% stake in the company from Japan's SoftBank Group Corp., which retained the rest. Generative AI holds the promise of allowing robots to adopt new skill sets more quickly based on learned behavior. Much like virtual assistance programs such as ChatGPT have evolved from large language models, large behavioral models are being used to program robots. Toyota Research Institute's advancement in what's known as diffusion policy has demonstrated a single AI technique can be used to teach a robot to do a wide range of individual tasks. Its next research focus involves creating large behavior models to allow a robot to learn many tasks simultaneously. The goal is for a robot programmed to beat eggs to also be able to handle any number of other things, like folding a shirt, said Russ Tedrake, an MIT professor and TRI's head of robotics research. "We're well on our way on that," Tedrake said. "It's something we're working very quickly on, very aggressively on, in trying to show that new capability." 2024 Bloomberg L.P. Distributed by Tribune Content Agency, LLC.

Toyota joins with Hyundai's Boston Dynamics on AI-powered robots

By Chester Dawson, Bloomberg News The Tribune Content Agency Toyota Motor Corp.'s research unit and Hyundai Motor Co.'s Boston Dynamics are joining forces to speed up development of humanoid robots with artificial intelligence. The partnership will pair Toyota Research Institute's expertise in large behavior model learning for machines with Boston Dynamics' humanoid Atlas robot, they said Wednesday. Boston-based teams from TRI and Boston Dynamics will conduct research on use cases for AI-trained robots in areas such as human-robot interaction, they said. Toyota has said it's made a breakthrough with AI in teaching robots to learn and Boston Dynamics, which was bought by Hyundai in 2020, has had commercial success with its robotic guard dog and a mobile robotic arm for re-stocking warehouses. Their collaboration comes as a potential challenge to other smart bot programs such as the Optimus robot showcased last week by rival Tesla Inc. A number of humanoid robot startups also are attracting attention - and billions of dollars in capital. The cooperation between Toyota and Boston Dynamics will focus on fundamental research with an eye toward eventual commercial use but executives at the companies declined to specify a timeline or disclose their budgets for the project. "This kind of technology has tremendous promise for the future," Gill Pratt, Toyota's chief scientist, said in an interview. "The work that we're doing in generative AI can be a tremendous complement to the kind of work that Boston Dynamics has done." Pratt said that a goal is to eventually bring robots onto factory assembly lines and into homes for elder care. Carmakers have been at the vanguard of efforts to automate more assembly line processes to trim labor costs and bolster worker safety. The automotive industry deploys more robots in factories globally than any other industry, with about one-quarter of new installations in 2023, according to the Frankfurt-based International Federation of Robotics. The dexterity of Boston Dynamics robots paired with Toyota's behavioral know-how is a key differentiator with other companies' efforts to build smart robots, Aaron Saunders, Boston Dynamics' chief technology officer, told Bloomberg. "That's going to set the stage to deliver on the promise that a lot of people are making right now in bringing humanoid robots out to the world at scale," he said. Saunders said Boston Dynamics plans to deploy humanoid robots with narrowly-focused tasks in Hyundai factories "in the coming years" while it pursues research with Toyota on the longer-term objective of AI-enabled, multi-tasking robots. Hyundai, which already has a pilot program with Boston Dynamics technology at a Kia brand car plant in South Korea, bought a controlling 80% stake in the company from Japan's SoftBank Group Corp., which retained the rest. Generative AI holds the promise of allowing robots to adopt new skill sets more quickly based on learned behavior. Much like virtual assistance programs such as ChatGPT have evolved from large language models, large behavioral models are being used to program robots. Toyota Research Institute's advancement in what's known as diffusion policy has demonstrated a single AI technique can be used to teach a robot to do a wide range of individual tasks. Its next research focus involves creating large behavior models to allow a robot to learn many tasks simultaneously. The goal is for a robot programmed to beat eggs to also be able to handle any number of other things, like folding a shirt, said Russ Tedrake, an MIT professor and TRI's head of robotics research. "We're well on our way on that," Tedrake said. "It's something we're working very quickly on, very aggressively on, in trying to show that new capability." (Updates from seventh paragraph with 2023 data on industrial robot installations, adds background on TRI research in 13th paragraph.)

Toyota, Hyundai's Boston Dynamics Join Forces for AI-Powered Robots

Toyota Motor Corp.'s research unit and Hyundai Motor Co.'s Boston Dynamics are joining forces to speed up development of humanoid robots with artificial intelligence. The partnership will pair Toyota Research Institute's expertise in large behavior model learning for machines with Boston Dynamics' humanoid Atlas robot, they said Wednesday. Boston-based teams from TRI and Boston Dynamics will conduct research on use cases for AI-trained robots in areas such as human-robot interaction, they said.

Boston Dynamics teams with TRI to bring AI smarts to Atlas humanoid robot

Boston Dynamics and Toyota Research Institute (TRI) Wednesday revealed plans to bring AI-based robotic intelligence to the electric Atlas humanoid robot. The collaboration will leverage the work that TRI has done around large behavior models (LBMs), which operate along similar lines as the more familiar large language models (LLMs) behind platforms like ChatGPT. Last September, TechCrunch paid a visit to TRI's Bay Area campus for a closer look at the institute's work on robot learning. In research revealed at last year's Disrupt conference, institute head Gill Pratt explained how the lab has been able to get robots to 90% accuracy when performing household tasks like flipping pancakes through overnight training. "In machine learning, up until quite recently there was a tradeoff, where it works, but you need millions of training cases," Pratt explained at the time. "When you're doing physical things, you don't have time for that many, and the machine will break down before you get to 10,000. Now it seems that we need dozens. The reason for the dozens is that we need to have some diversity in the training cases. But in some cases, it's less." Boston Dynamics is a good match for TRI on the hardware side. The Spot-maker has done its share on the software and AI front to power its own systems, but the manner of work required to teach robots to perform complex tasks with full autonomy is another beast altogether. "There has never been a more exciting time for the robotics industry, and we look forward to working with TRI to accelerate the development of general-purpose humanoids," Boston Dynamics CEO Robert Playter notes in a statement. "This partnership is an example of two companies with a strong research-and-development foundation coming together to work on many complex challenges and build useful robots that solve real-world problems." Boston Dynamics revealed its design for the electric Atlas in April, as it finally put to rest the humanoid's larger, hydraulic namesake. While we've seen very little of the robot since then, in August, TechCrunch managed to get its hands on a short video of the robot doing pushups. Like Atlas' initial video, the quick pushup demo was a good demonstration of the robot's remarkable strength. Boston Dynamics' chief competition in the humanoid robot space, including Agility, Figure, and Tesla, have primarily opted to build out their AI teams in-house. The Boston Dynamics-TRI deal is especially interesting given that the organizations are run by Hyundai and Toyota - direct competitors in the automotive space. Meanwhile, Boston Dynamics has its own research spinout, The AI Institute (formerly The Boston Dynamics AI Institute). Though run by Boston Dynamics founder and former CEO, Marc Raibert, the institute maintains independence from Boston Dynamics, proper. It's also a significantly younger organization still in the process of building out its team. TRI, for its part, has become less invested in the hardware side of the equation. The goal in all of this is a true general-purpose machine. That is to say, a system that is essentially capable of learning and doing all of the things a person can do - and, presumably, more. While we've seen robot hardware evolve closer to a point capable of that level of sophistication, something that approaches general intelligence is a much tougher nut to crack. Certainly, the advent of SDK for systems has helped dramatically increase the breadth of tasks that can be performed by robots like Boston Dynamics' Spot, true artificial general intelligence is further off - if we ever get there.

Coming soon: Toyota robots that get better with practice

Bots that learn to peel potatoes is a lot less scary than Black Mirror Boston Dynamics and Toyota Research Institute (TRI) announced on Wednesday they're partnering to combine the former's multi-jointed athletic humanoid, Atlas, with TRI's large behavior models (LBM). Boston Dynamics CEO Robert Playter enthused that he was looking forward to accelerating "the development of general-purpose humanoids," while TRI CEO Gill Pratt cheered that "recent advances in AI and machine learning hold tremendous potential for advancing physical intelligence." TRI's LBM work includes the generative AI technique known as diffusion policy, which allows robots to acquire new dexterous behaviors by having them demonstrated, rather than programmed. What that means, according to one video, [VIDEO] is that robots can learn to do tricky things like peel potatoes and flip pancakes. Diffusion policies help a robot learn how to perform fine motor skills by generating small, sequential actions that gradually build up to a more complex behavior. Instead of predicting a single, definitive action in one step, the diffusion policy predicts an array of possible actions and gradually narrows them down over time - allowing the robot to become more accurate at handling objects. The LBMs are based on combining the skills created via the diffusion policy efforts. "This allows us to teach robots skills faster and with significantly fewer demonstrations," explained TRI VP Russ Tedrake. Before diffusion policies, most robotic object manipulation focused on "pick and place tasks," which limited robots to simple objects and rearranging, commented TRI's Ben Burchfiel. Diffusion policies can make the most of hardware capabilities such as touch sensors without modifying any code or explicitly programming any new skills, he added. One action can be taught to a machine by a human in the afternoon, then the robot is left alone to practice the action and by the next morning, the machine has learned the action. Importantly, once one robot has mastered a skill, that knowledge can be deployed to a fleet of robots at once. TRI has referred to its training as a "kindergarten for robots." "It's just amazing to see the tasks that the robots can perform. Even a year ago, I never would have expected that robots would become this skilled," remarked Tedrake. Tedrake co-leads the partnership along with Boston Dyanmics' senior director Scott Kuindersma. "The physical capabilities of the new electric Atlas robot, coupled with the ability to programmatically command and teleoperate a broad range of whole-body bimanual manipulation behaviors, will allow research teams to deploy the robot across a range of tasks and collect data on its performance," stated TRI. That data, said the company, will go back into training more advanced LBMs. The joint team's future research will then focus on answering "fundamental training questions for humanoid robots, the ability of research models to leverage whole-body sensing, and understanding human-robot interaction and safety/assurance cases to support these new capabilities." Here's hoping for a focus on the safety part, as humanity is currently in an era where similar AI-empowered robots have been seen toting machine guns. There have been expressions of concern from researchers on efforts to equip robots with AI - including this February when computer scientists at the University of Maryland (UMD) warned that "it is easy to manipulate or misguide the robot's actions, leading to safety hazards." ®

Boston Dynamics and Toyota Research to develop 'game-changing' robotics

Combining TRI's AI technology with Boston Dynamics' hardware is 'game-changing', said TRI CEO Gill Pratt. Boston Dynamics and Toyota Research Institute (TRI), a subsidiary of automotive giant Toyota Motor Corporation, have announced a partnership to accelerate the development of humanoid robots by utilising TRI's large behaviour models (LBM) and Boston Dynamics' iconic humanoid Atlas robot. The latest generation of Boston Dynamic's humanoid electric robot model Atlas, released earlier this year, is the "ideal platform" for advancing the science of AI-based manipulation skills, the US robotics manufacturer said - technology that TRI's rapidly advancing LBMs are recognised for. In the announcement made yesterday (16 October), the partnership, designed to leverage each partner's strength equally, will couple the physical capabilities of the Atlas robot with technology that enables robot operation through computer programmes, allowing the researchers to support the training and advancement of LBMs. It will be co-led by Scott Kuindersma, Boston Dynamic's senior director of robotics research and Russ Tedrake, the vice-president of robotics research at TRI. TRI's work includes a new approach that allows a robot to acquire dexterous behaviours quickly and easily from a generative AI approach called Diffusion Policy. TRI also helped develop OpenVLA, an open-source vision-language-action model and DROID, a large scale robot manipulation dataset. "Recent advances in AI and machine learning hold tremendous potential for advancing physical intelligence," said Gill Pratt, TRI's CEO and Toyota's chief scientist. "The opportunity to implement TRI's state-of-the-art AI technology on Boston Dynamics' hardware is game-changing for each of our organisations as we work to amplify people and improve quality of life." Robert Playter, the CEO of Boston Dynamics, said "there has never been a more exciting time for the robotics industry". "This partnership is an example of two companies with a strong research-and-development foundation coming together to work on many complex challenges and build useful robots that solve real-world problems." Don't miss out on the knowledge you need to succeed. Sign up for the Daily Brief, Silicon Republic's digest of need-to-know sci-tech news.

Boston Dynamics gave its Atlas robot an AI brain | Digital Trends

Boston Dynamics and Toyota Research Institute (TRI) announced on Tuesday that they are partnering to develop general-purpose humanoid robots. Boston Dynamics will contribute its new electric Atlas robot to the task, while TRI will utilize its industry-leading Large Behavior Models. Boston Dynamics, which launched in 1992 as an offshoot from the Massachusetts Institute of Technology (MIT), has been at the forefront of robotics development for more than 30 years. It burst into the mainstream in 2009 with the BigDog and LittleDog quadrupedal systems and debuted the first iteration of its bipedal Atlas platform in 2013. Atlas' capabilities have undergone a steady evolution in the past decade, enabling the robot to perform increasingly difficult acrobatics and dexterity tasks, from dancing and doing back flips to to conquering parkour courses and navigating simulated construction sites. In April 2024, the company retired its long-standing, hydraulic Atlas platform in favor of a new generation driven by electric servos. The company describes the electric Atlas as "one of the most advanced humanoid robots ever built," one that is "able to move in ways that exceed human capabilities." Recommended Videos TRI, on the other hand, stands at the forefront of Large Behavior Model (LBM) development. LBMs are to robotics as LLMs are to chatbots. Just as LLM's are trained on massive multimodal datasets to respond to a human like a human would, LBMs are trained on enormous corpora of human behaviors, enabling robots to move and act in a humanlike manner. They also help robots learn new behaviors and generalize across tasks. Per the announcement blog, "TRI's work on LBMs aims to achieve multitask, vision-and-language-conditioned foundation models for dexterous manipulation." "Recent advances in AI and machine learning hold tremendous potential for advancing physical intelligence," Gill Pratt, chief scientist for Toyota and CEO of TRI, said in a statement. "The opportunity to implement TRI's state-of-the-art AI technology on Boston Dynamics' hardware is game-changing for each of our organizations as we work to amplify people and improve quality of life." This news comes amid an increasingly crowded field of companies looking to incorporate robots into the future workforce. Agility Robotics' Digit and Figure's 01 and 02 models, for example, are already being tested in industrial settings such as BMW's Spartanburg plant in South Carolina and a Spanx production facility in Flowery Branch, Georgia. Tesla's Optimus is ostensibly also in the running, though even the most recent models still need to be remotely operated when performing more than the most basic tasks.

Boston Dynamics and Toyota team up: Teaching Atlas how to learn

The old Atlas: a broadly flamboyant and highly gymnastic fellow The Toyota Research Institute has been doing some incredible work teaching robots to rapidly learn and perform tasks autonomously - now, it's bringing its Large Behavior Model tech to the extraordinary Atlas humanoid in partnership with Boston Dynamics. Humanoid robot hardware, believe it or not, is probably good enough already. More than a decade's worth of work at Boston Dynamics has resulted not only in an incredibly athletic and capable hydraulic Atlas robot, but a slew of emerging commercial competitors from Tesla, Figure, Agility, Sanctuary, Fourier and many others. These remarkable robot bodies will continue to improve, but they're already good enough to perform all manner of useful work. The software is the problem. If you need a coding team to teach a robot a new behavior, it's scarcely better than today's conventional production robots. But developing a general-purpose humanoid robot that understands the world and how to interact with it in flexible and adaptable ways is an enormous task. The answer is AI, of course, like the answer will soon be for everything - but AI needs to be trained on lots of data. ChatGPT, Grok, Llama and Claude all benefit from the insane quantity of (largely written) data humanity has accumulated over the centuries. Large Language Models (LLMs) have advanced so far so quickly, because language is a highly compressed representation of reality, crunched down to such small file sizes that vast amounts of it can be processed. There's much less data available to help robots learn the basics of movement - other than video, which doesn't tell the whole story about why somebody made what movement. And they really do need to learn things from the ground up. Hence the idea of a 'Large Behavior Model,' or LBM - a way for robots to slowly build up the basic movements they can use to interact with the world, and combine them into more complex movements in service of a task or goal, in a similar way to how LLMs have developed an 'understanding' of human language and learned to interact with us. If you haven't seen the LBM work the Toyota Research Institute (TRI) was doing last year, take a moment to catch up: In essence, The TRI team developed a telepresence system allowing human pilots to 'drive' robotic arms, looking through VR goggles fed from the robot's cameras to see exactly what the robot sees, and equipped with haptic gloves to let them also feel what the robot's tactile sensors can feel. Then, with the human pilot limited to the exact set of 'senses' that the robot has, they set about doing a bunch of tasks, a lot of them in a kitchen setting. They'd spend a couple of hours doing the same task over and over from different starting points, correcting their mistakes if they made them, and marking each attempt as a success or failure. From there, the robots would spend some time 'thinking' about the problem, effectively running millions of different simulations of the task while adding random variables and starting points, grading their own performance each time according to their own understanding of success and failure modes. And it worked. The TRI team had taught its robot arms more than 60 complex behaviors by September last year when we first saw the video above. Researchers reported that they'd often spend an afternoon doing the piloted training, then go home as the behavior learning system ran its simulations overnight, then they'd turn up in the morning to find that the robots were now able to do the task by themselves, and in a fairly flexible manner. It was remarkable stuff, and we're fascinated to learn how far it's come in the last 12 months, given how astonishingly quickly things are progressing across all fields of AI. But it was also fairly limited research, done using pairs of robot arms rather than whole bodies. Well, that's about to change. Boston Dynamics is the absolute gold standard in robotics research, and has been for decades. The old, hydraulic Atlas humanoid will go down as one of the most groundbreaking and significant machines in the history of robotics. And the fully-electric new Atlas, which you may have seen on New Atlas, made its public debut just five months ago. This remarkable evolution lost some of the explosive power that made the original Atlas such an extraordinary gymnast - but made up for it with fully swivel-capable joints all over its body, allowing free rotation at the hips, shoulders, waist, neck, biceps and thighs, so any given section of its body can face any direction. From gymnast to contortionist ... Check it out: It's a truly remarkable looking robot, already scratched, dented and looking very second-hand in typical Boston Dynamics fashion, but we've seen curiously little of the new Atlas in the last five months to determine exactly where it's at and what its current capabilities are. Well, other than this: we know it can rip out a set of push-ups. So today's news is very exciting; the undisputed grandmasters of humanoid robot hardware, teaming up with a leading team in AI LBM development in order to advance the useful capabilities of humanoid robots. "There has never been a more exciting time for the robotics industry, and we look forward to working with TRI to accelerate the development of general-purpose humanoids," says Robert Playter, CEO of Boston Dynamics, in a press release. "This partnership is an example of two companies with a strong research-and-development foundation coming together to work on many complex challenges and build useful robots that solve real-world problems." "Recent advances in AI and machine learning hold tremendous potential for advancing physical intelligence," adds Gill Pratt, chief scientist for Toyota and CEO of TRI. "The opportunity to implement TRI's state-of-the-art AI technology on Boston Dynamics' hardware is game-changing for each of our organizations as we work to amplify people and improve quality of life." The partnership aims to rapidly develop whole-body behavior models for the Atlas robot, but also for other humanoid platforms that TRI may go on to work with. It'll be interesting to see what different kinds of telepresence training hardware are brought to bear on the problem, since Atlas is so much more complex than the simple bimanual setups TRI was working with originally. Ultimately, though, it's still unclear whether Boston intends to scale Atlas into a commercial product. And scale may be crucial here; companies like Tesla and Figure are designing their humanoids with mass manufacturing in mind, aiming to deploy hundreds, then thousands of them out in the world doing small, simple, useful tasks. There, they'll see an extraordinary range of things happening, collect great scads of real-world data, and use that data to drive swarm-based learning. That's the approach Tesla claims makes it a world leader in autonomous cars; there's millions of these things on the road already, constantly watching and contributing to the knowledge of the whole. AI is a big-data game, and whoever gathers the most data and uses it most efficiently will win, according to this model. And the prize, according to people like Elon Musk, is possibly the biggest product in history, a transformational labor-replacing machine that could eventually take over basically any physical job. While Boston Dynamics has been miles ahead of everyone on humanoids for at least a decade, Atlas has been specifically designated a research platform. The company has restricted its commercial activities to smaller, practical quadrupeds like its Spot platform, and its single-arm, heavy-lift Stretch box handler. Perhaps it's telling that this pioneering company doesn't seem to think humanoids are ready to get to work just yet, and will need a few more years in the lab, painstakingly putting together the building blocks of physical behavior. The promise of general-purpose humanoids is so massive, and the challenge so great, that there's sure to be plenty of surprises on the path. This does feel like a field where we're watching the history of the future happen in real time.

Boston Dynamics and Toyota Research Team Up on Robots

A new partnership with Toyota Research should bring new capabilities to Boston Dynamics' Atlas robot. Today, Boston Dynamics and the Toyota Research Institute (TRI) announced a new partnership "to accelerate the development of general-purpose humanoid robots utilizing TRI's Large Behavior Models and Boston Dynamics' Atlas robot." Committing to working towards a general purpose robot may make this partnership sound like a every other commercial humanoid company right now, but that's not at all that's going on here: BD and TRI are talking about fundamental robotics research, focusing on hard problems, and (most importantly) sharing the results. The broader context here is that Boston Dynamics has an exceptionally capable humanoid platform capable of advanced and occasionally painful-looking whole-body motion behaviors along with some relatively basic and brute force-y manipulation. Meanwhile, TRI has been working for quite a while on developing AI-based learning techniques to tackle a variety of complicated manipulation challenges. TRI is working toward what they're calling large behavior models (LBMs), which you can think of as analogous to large language models (LLMs), except for robots doing useful stuff in the physical world. The appeal of this partnership is pretty clear: Boston Dynamics gets new useful capabilities for Atlas, while TRI gets Atlas to explore new useful capabilities on. Here's a bit more from the press release: The project is designed to leverage the strengths and expertise of each partner equally. The physical capabilities of the new electric Atlas robot, coupled with the ability to programmatically command and teleoperate a broad range of whole-body bimanual manipulation behaviors, will allow research teams to deploy the robot across a range of tasks and collect data on its performance. This data will, in turn, be used to support the training of advanced LBMs, utilizing rigorous hardware and simulation evaluation to demonstrate that large, pre-trained models can enable the rapid acquisition of new robust, dexterous, whole-body skills. The joint team will also conduct research to answer fundamental training questions for humanoid robots, the ability of research models to leverage whole-body sensing, and understanding human-robot interaction and safety/assurance cases to support these new capabilities. Russ Tedrake: We have a ton of respect for the Boston Dynamics team and what they've done, not only in terms of the hardware, but also the controller on Atlas. They've been growing their machine learning effort as we've been working more and more on the machine learning side. On TRI's side, we're seeing the limits of what you can do in tabletop manipulation, and we want to explore beyond that. Scott Kuindersma: The combination skills and tools that TRI brings the table with the existing platform capabilities we have at Boston Dynamics, in addition to the machine learning teams we've been building up for the last couple years, put us in a really great position to hit the ground running together and do some pretty amazing stuff with Atlas. What will your approach be to communicating your work, especially in the context of all the craziness around humanoids right now? Tedrake: There's a ton of pressure right now to do something new and incredible every six months or so. In some ways, it's healthy for the field to have that much energy and enthusiasm and ambition. But I also think that there are people in the field that are coming around to appreciate the slightly longer and deeper view of understanding what works and what doesn't, so we do have to balance that. The other thing that I'd say is that there's so much hype out there. I am incredibly excited about the promise of all this new capability; I just want to make sure that as we're pushing the science forward, we're being also honest and transparent about how well it's working. Kuindersma: It's not lost on either of our organizations that this is maybe one of the most exciting points in the history of robotics, but there's still a tremendous amount of work to do. What are some of the challenges that your partnership will be uniquely capable of solving? Kuindersma: One of the things that we're both really excited about is the scope of behaviors that are possible with humanoids -- a humanoid robot is much more than a pair of grippers on a mobile base. I think the opportunity to explore the full behavioral capability space of humanoids is probably something that we're uniquely positioned to do right now because of the historical work that we've done at Boston Dynamics. Atlas is a very physically capable robot -- the most capable humanoid we've ever built. And the platform software that we have allows for things like data collection for whole body manipulation to be about as easy as it is anywhere in the world. Tedrake: In my mind, we really have opened up a brand new science -- there's a new set of basic questions that need answering. Robotics has come into this era of big science where it takes a big team and a big budget and strong collaborators to basically build the massive data sets and train the models to be in a position to ask these fundamental questions. Tedrake: Nobody has the beginnings of an idea of what the right training mixture is for humanoids. Like, we want to do pre-training with language, that's way better, but how early do we introduce vision? How early do we introduce actions? Nobody knows. What's the right curriculum of tasks? Do we want some easy tasks where we get greater than zero performance right out of the box? Probably. Do we also want some really complicated tasks? Probably. We want to be just in the home? Just in the factory? What's the right mixture? Do we want backflips? I don't know. We have to figure it out. There are more questions too, like whether we have enough data on the Internet to train robots, and how we could mix and transfer capabilities from Internet data sets into robotics. Is robot data fundamentally different than other data? Should we expect the same scaling laws? Should we expect the same long-term capabilities? The other big one that you'll hear the experts talk about is evaluation, which is a major bottleneck. If you look at some of these papers that show incredible results, the statistical strength of their results section is very weak and consequently we're making a lot of claims about things that we don't really have a lot of basis for. It will take a lot of engineering work to carefully build up empirical strength in our results. I think evaluation doesn't get enough attention. What has changed in robotics research in the last year or so that you think has enabled the kind of progress that you're hoping to achieve? Kuindersma: From my perspective, there are two high-level things that have changed how I've thought about work in this space. One is the convergence of the field around repeatable processes for training manipulation skills through demonstrations. The pioneering work of diffusion policy (which TRI was a big part of) is a really powerful thing -- it takes the process of generating manipulation skills that previously were basically unfathomable, and turned it into something where you just collect a bunch of data, you train it on an architecture that's more or less stable at this point, and you get a result. The second thing is everything that's happened in robotics-adjacent areas of AI showing that data scale and diversity are really the keys to generalizable behavior. We expect that to also be true for robotics. And so taking these two things together, it makes the path really clear, but I still think there are a ton of open research challenges and questions that we need to answer. Do you think that simulation is an effective way of scaling data for robotics? Tedrake: I think generally people underestimate simulation. The work we've been doing has made me very optimistic about the capabilities of simulation as long as you use it wisely. Focusing on a specific robot doing a specific task is asking the wrong question; you need to get the distribution of tasks and performance in simulation to be predictive of the distribution of tasks and performance in the real world. There are some things that are still hard to simulate well, but even when it comes to frictional contact and stuff like that, I think we're getting pretty good at this point. Is there a commercial future for this partnership that you're able to talk about? Kuindersma: For Boston Dynamics, clearly we think there's long-term commercial value in this work, and that's one of the main reasons why we want to invest in it. But the purpose of this collaboration is really about fundamental research -- making sure that we do the work, advance the science, and do it in a rigorous enough way so that we actually understand and trust the results and we can communicate that out to the world. So yes, we see tremendous value in this commercially. Yes, we are commercializing Atlas, but this project is really about fundamental research. What happens next? Tedrake: There are questions at the intersection of things that BD has done and things that TRI has done that we need to do together to start, and that'll get things going. And then we have big ambitions -- getting a generalist capability that we're calling LBM (large behavior models) running on Atlas is the goal. In the first year we're trying to focus on these fundamental questions, push boundaries, and write and publish papers. I want people to be excited about watching for our results, and I want people to trust our results when they see them. For me, that's the most important message for the robotics community: Through this partnership we're trying to take a longer view that balances our extreme optimism with being critical in our approach.