Waymo taps Genie 3 to simulate tornadoes, elephants, and extreme scenarios for self-driving cars

Waymo leverages Genie 3 to create a world model for self-driving cars

Google-spinoff Waymo is in the midst of expanding its self-driving car fleet into new regions. Waymo touts more than 200 million miles of driving that informs how the vehicles navigate roads, but the company's AI has also driven billions of miles virtually, and there's a lot more to come with the new Waymo World Model. Based on Google DeepMind's Genie 3, Waymo says the model can create "hyper-realistic" simulated environments that train the AI on situations that are rarely (or never) encountered in real life -- like snow on the Golden Gate Bridge. Until recently, the autonomous driving industry relied entirely on training data collected from real cars and real situations. That means rare, potentially dangerous events are not well represented in training data. The Waymo World Model aims to address that by allowing engineers to create simulations with simple prompts and driving inputs. Google revealed Genie 3 last year, positioning it as a significant upgrade over other world models by virtue of its long-horizon memory. In Google's world model, you can wander away from a given object, and when you look back, the model will still "remember" how that object is supposed to look. In earlier attempts at world models, the simulation would lose that context almost immediately. With Genie 3, the model can remember details for several minutes. Autoregressive world models like Genie don't actually create 3D spaces, but instead render video quickly enough that it feels like an explorable world. Naturally, video games are cited as a prime application for world models, so much so that gaming company stocks dropped when Google recently expanded access to the technology as Project Genie. However, the latency and still rather short memory of Genie make gaming uses far from a certainty. Nevertheless, Waymo says Genie 3 is actually ideal for simulating the kind of data it needs to train self-driving cars.

What happens when Waymo runs into a tornado? Or an elephant?

An autonomous vehicle drives down a lonely stretch of highway. Suddenly, a massive tornado appears in the distance. What does the driverless vehicle do next? This is just one of the scenarios that Waymo can simulate in the "hyper realistic" virtual world that it has just created with help from Google's DeepMind. Waymo's World Model is built using Genie 3, Google's new AI world model that can generate virtual interactive spaces with text or images as prompts. But Genie 3 isn't just for creating bad knockoffs of Nintendo games; it can also build photorealistic and interactive 3D environments "adapted for the rigors of the driving domain," Waymo says. Simulation is a critical component in autonomous vehicle development, enabling developers to test their vehicles in a variety of settings and scenarios, many of which may only come up in the rarest of occasions -- without any physical risk of harming passengers or pedestrians. AV companies use these virtual environments to run through a battery of tests, racking up millions -- or even billions -- of miles in the process, in the hopes of better training their vehicles for any possible "edge case" that they may encounter in the real world. What kind of edge cases is Waymo testing? In addition to the aforementioned tornado, the company can also simulate a snow-covered Golden Gate Bridge, a flooded suburban cul-de-sac with floating furniture, a neighborhood engulfed in flames, or even an encounter with a rogue elephant. In each scenario, the Waymo robotaxi's lidar sensors generate a 3D rendering of the surrounding environment, including the obstacle in the road. "The Waymo World Model can generate virtually any scene -- from regular, day-to-day driving to rare, long-tail scenarios -- across multiple sensor modalities," the company says in a blog post. Waymo says Genie 3 is ideal for creating virtual worlds for its robotaxis, citing three unique mechanisms: driving action control, scene layout control, and language control. Driving action control allows developers to simulate "what if" counterfactuals, while scene layout control enables customization of the road layouts, like traffic signals and other road user behavior. Waymo describes language control as its "most flexible tool" that allows for time-of-day and weather condition adjustment. This is especially helpful if developers are trying to simulate low-light or high-glare conditions, in which the vehicle's various sensors may have difficulty seeing the road ahead. The Waymo World Model can also take real-world dashcam footage and transform it into a simulated environment, for the "highest degree of realism and factuality" in virtual testing, the company says. And it can create longer simulated scenes, such as ones that run at 4X speed playback, without sacrificing image quality or computer processing. "By simulating the 'impossible,' we proactively prepare the Waymo Driver for some of the most rare and complex scenarios," the company says in its blog post. This isn't the first time that Waymo has leaned on Google's vast AI resources to improve its autonomous driving techniques. Waymo's EMMA (End-to-End Multimodal Model for Autonomous Driving) training model was built using Google's Gemini. Waymo is reportedly working on a Gemini-based in-car voice assistant. And DeepMind, the company's AI lab, has provided solutions to Waymo to help reduce its rate of "false positives" in its sensor data.

Waymo Says Genie 3 Simulations Can Help Boost Robotaxi Rollout

The collaboration with DeepMind will help the expansion of Waymo self-driving services across more markets, according to the company, and aid in making autonomous vehicle systems more reliable in uncommon scenarios. Alphabet Inc.'s Waymo said it is using DeepMind's Genie 3 AI model to create realistic, digital worlds for its autonomous driving technology to train on edge-case scenarios. The self-driving tech developer published a blog post on Friday about a new Waymo World Model, built atop Genie 3's ability to construct virtual environments based on text prompts. The collaboration with another segment of Google's tech ecosystem will help the expansion of Waymo self-driving services across more markets, according to the company. Genie 3 drew widespread renown last week with demonstrations of its so-called world-building capabilities, triggering a selloff in companies that provide game development and graphics creation tools. Waymo said it has customized it to generate synthetic driving footage and depth perception data as if they were captured by cameras and lidar sensors on vehicles. Waymo's new system can also take real-life dashcam datasets and convert them into scenes and depth maps for vehicle simulations, the company said. The combination will aid in making autonomous vehicle systems more reliable in uncommon scenarios. "Traditional AV simulation models are constrained by the on-road data they collect," a spokeswoman for Waymo said. The new world model "allows us to explore situations that were never directly observed by our fleet." "This will enhance Waymo's ability to safely scale our service across more places and new driving environments," she added. Robotaxi operators and artificial intelligence companies more broadly have been seeking more sources of data in the race to advance their models. Nvidia Corp., which supplies chips and AI models for developers of self-driving technology, has partnered with rideshare leader Uber Technologies Inc. to collect millions of hours of robotaxi-specific driving data to fuel driverless model training and validation. SoftBank Group Corp.-backed Wayve Technologies Ltd., which plans to trial robotaxis on the Uber platform in the UK this year, has announced its own world model to generate synthetic driving data. And Tesla Inc. has also said it built a similar simulator. Having bigger training data sets will be crucial for Waymo as it faces safety probes from US authorities after a series of software mishaps in recent months. The National Highway Traffic Safety Administration and the National Transportation Safety Board are looking into several incidents where Waymo failed to stop for parked school buses in Austin, violations that also prompted Waymo to file a voluntary software recall. The Waymo spokesperson declined to comment on whether it has conducted simulations on stopped school bus encounters, or mass power outages like the one that disrupted its San Francisco operations last December. But she said the Waymo World Model "can simulate virtually any scene." Incorporating Google's Genie 3 model could be a boon for Waymo's plans to expand to about a dozen cities this year. Simulation is just one tool that it uses to prepare its autonomous systems for certain situations and to validate its safety, the spokesperson said. In December, the company said that there is "no substitute" for its experience driving in the real world. It surpassed more than 20 million autonomous trips that months, Alphabet said on Wednesday, more than any other provider in the US.

Waymo Catches World Model Fever, and the Only Prescription Is More World Models

Waymo vehicles have reportedly racked up more than 200 million miles of autonomous driving on public roads. But it's yet to run into a tornado or an elephant, and odds are that it'd respond poorly if it did. To try to help with those once-in-a-billion-miles scenarios, Waymo announced Friday that it is introducing Waymo World Model, a generative AI model that it will use to run near-endless situations to try to make sure its cars are prepared for the unpredictable, which also just happens to fit into the latest trend in the AI space. To be clear, Waymo's world model makes about as much sense as any use case for the technology. The company has a ton of high-definition data that it has collected from its time on the road that it can use to generate realistic re-creations of roads. But, the company said, instead of building a model based only on that information, it's going to use Google's Genie 3 model to put its cars in simulated situations that extend beyond what is already in its data set collected from cameras and lidar sensors. Google made a splash last month when it released a beta version of Genie 3 to the public, allowing a subset of paid subscribers to generate 3D worlds with realistic physics. Unlike a large language model (LLM)â€"the underlying technology that powers most AI tools including Google’s own Geminiâ€"which use the vast amount of training data they are given to predict the most likely next part of a sequence, world models are trained on the dynamics of the real world, including physics and spatial properties, to create a simulation of how physical environments operate. Waymo plans to tap into that to put its cars through a gauntlet of scenarios that they likely wouldn't find themselves in until it's too late. That includes extreme weather conditions and natural disasters, so the cars can figure out how to navigate a tornado or flood waters; sudden safety emergencies like falling tree branches or an accident with lots of debris; and run-ins with the unexpected, like an elephant on the road. “By simulating the â€~impossible,’ we proactively prepare the Waymo Driver for some of the most rare and complex scenarios,†the company said. The theory is certainly sound, though world models aren't without their drawbacks. The early feedback on the consumer version of Genie 3 was a bit spotty, and world models are still susceptible to hallucinations. We're still in the earliest stages of seeing these models deployed, and they have lots of room to iterate. And Waymos have definitely had their issues in edge-case scenarios in the real world. Late last year, a Waymo ran over a beloved bogeda cat named Kit Kat, and last month, one ran into a kid in a school zone. Those interactions aren't even particularly rare for a driver to find themselves in, so hopefully Waymo can refine its responses in those scenarios on top of prepping for the most unlikely situations.