Waymo taps Google's Genie 3 to simulate tornadoes, elephants, and extreme edge cases

Waymo leverages Genie 3 to create a world model for self-driving cars

Google-spinoff Waymo is in the midst of expanding its self-driving car fleet into new regions. Waymo touts more than 200 million miles of driving that informs how the vehicles navigate roads, but the company's AI has also driven billions of miles virtually, and there's a lot more to come with the new Waymo World Model. Based on Google DeepMind's Genie 3, Waymo says the model can create "hyper-realistic" simulated environments that train the AI on situations that are rarely (or never) encountered in real life -- like snow on the Golden Gate Bridge. Until recently, the autonomous driving industry relied entirely on training data collected from real cars and real situations. That means rare, potentially dangerous events are not well represented in training data. The Waymo World Model aims to address that by allowing engineers to create simulations with simple prompts and driving inputs. Google revealed Genie 3 last year, positioning it as a significant upgrade over other world models by virtue of its long-horizon memory. In Google's world model, you can wander away from a given object, and when you look back, the model will still "remember" how that object is supposed to look. In earlier attempts at world models, the simulation would lose that context almost immediately. With Genie 3, the model can remember details for several minutes. Autoregressive world models like Genie don't actually create 3D spaces, but instead render video quickly enough that it feels like an explorable world. Naturally, video games are cited as a prime application for world models, so much so that gaming company stocks dropped when Google recently expanded access to the technology as Project Genie. However, the latency and still rather short memory of Genie make gaming uses far from a certainty. Nevertheless, Waymo says Genie 3 is actually ideal for simulating the kind of data it needs to train self-driving cars.

What happens when Waymo runs into a tornado? Or an elephant?

An autonomous vehicle drives down a lonely stretch of highway. Suddenly, a massive tornado appears in the distance. What does the driverless vehicle do next? This is just one of the scenarios that Waymo can simulate in the "hyper realistic" virtual world that it has just created with help from Google's DeepMind. Waymo's World Model is built using Genie 3, Google's new AI world model that can generate virtual interactive spaces with text or images as prompts. But Genie 3 isn't just for creating bad knockoffs of Nintendo games; it can also build photorealistic and interactive 3D environments "adapted for the rigors of the driving domain," Waymo says. Simulation is a critical component in autonomous vehicle development, enabling developers to test their vehicles in a variety of settings and scenarios, many of which may only come up in the rarest of occasions -- without any physical risk of harming passengers or pedestrians. AV companies use these virtual environments to run through a battery of tests, racking up millions -- or even billions -- of miles in the process, in the hopes of better training their vehicles for any possible "edge case" that they may encounter in the real world. What kind of edge cases is Waymo testing? In addition to the aforementioned tornado, the company can also simulate a snow-covered Golden Gate Bridge, a flooded suburban cul-de-sac with floating furniture, a neighborhood engulfed in flames, or even an encounter with a rogue elephant. In each scenario, the Waymo robotaxi's lidar sensors generate a 3D rendering of the surrounding environment, including the obstacle in the road. "The Waymo World Model can generate virtually any scene -- from regular, day-to-day driving to rare, long-tail scenarios -- across multiple sensor modalities," the company says in a blog post. Waymo says Genie 3 is ideal for creating virtual worlds for its robotaxis, citing three unique mechanisms: driving action control, scene layout control, and language control. Driving action control allows developers to simulate "what if" counterfactuals, while scene layout control enables customization of the road layouts, like traffic signals and other road user behavior. Waymo describes language control as its "most flexible tool" that allows for time-of-day and weather condition adjustment. This is especially helpful if developers are trying to simulate low-light or high-glare conditions, in which the vehicle's various sensors may have difficulty seeing the road ahead. The Waymo World Model can also take real-world dashcam footage and transform it into a simulated environment, for the "highest degree of realism and factuality" in virtual testing, the company says. And it can create longer simulated scenes, such as ones that run at 4X speed playback, without sacrificing image quality or computer processing. "By simulating the 'impossible,' we proactively prepare the Waymo Driver for some of the most rare and complex scenarios," the company says in its blog post. This isn't the first time that Waymo has leaned on Google's vast AI resources to improve its autonomous driving techniques. Waymo's EMMA (End-to-End Multimodal Model for Autonomous Driving) training model was built using Google's Gemini. Waymo is reportedly working on a Gemini-based in-car voice assistant. And DeepMind, the company's AI lab, has provided solutions to Waymo to help reduce its rate of "false positives" in its sensor data.

Waymo Is Using Google's Genie 3 AI to Practice Handling Tornadoes, Elephants

Waymo is looking to improve how its self-driving vehicles react when faced with unique scenarios, and it's leveraging Google's new Genie 3 world engine AI model to do it. It's testing everything from sudden tornadoes and heavy snow conditions to deep flood waters and wild animal encounters. When Google launched its Genie 3 world engine for Gemini Ultra Plan subscribers last week, it was quickly picked up as a tool to shortcut game prototype development. But one of the more realistic ways to use world engines is in training robots and other physical AI devices to understand the world around them. So, Waymo (a subsidiary of Google parent company Alphabet) is using this fast world-building system to test out its driving AI in the kind of scenarios that are hard to plan for in the real world. "By simulating the 'impossible,' we proactively prepare the Waymo Driver for some of the most rare and complex scenarios," Waymo said in a blog post. "This creates a more rigorous safety benchmark, ensuring the Waymo Driver can navigate long-tail challenges long before it encounters them in the real world." The Waymo World Model is based on Genie 3, but designed to be more realistic than the Zelda clones people have been producing. Waymo uses it to create interactive driving environments, then converts the 2D video output into 3D LiDAR that can be applied to Waymo's hardware. Waymo even figured out how get around Genie 3's limited long-term stability. The model tends to fall down and lose its consistency and object permanence after a minute or so, but Waymo found that by speeding up the footage derived from it by 4x, it could create much longer scenarios. They'd be horrible to play as a game, but are invaluable for an AI training simulation. Waymo had its robotaxis encounter animals like elephants, car accidents, wildfires, snowstorms, and heavy flooding. It also trialed different scenes, like heavy traffic and busy crosswalks, a semi blocking the road, and even an enormous tumbleweed the size of a car. While some of these situations are ridiculous, and many of them extremely unlikely to ever occur, they give Waymo an injection of additional data that's close enough to the real thing that its robotaxis should be better equipped should they ever encounter these odd situations.

Waymo Says Genie 3 Simulations Can Help Boost Robotaxi Rollout

The collaboration with DeepMind will help the expansion of Waymo self-driving services across more markets, according to the company, and aid in making autonomous vehicle systems more reliable in uncommon scenarios. Alphabet Inc.'s Waymo said it is using DeepMind's Genie 3 AI model to create realistic, digital worlds for its autonomous driving technology to train on edge-case scenarios. The self-driving tech developer published a blog post on Friday about a new Waymo World Model, built atop Genie 3's ability to construct virtual environments based on text prompts. The collaboration with another segment of Google's tech ecosystem will help the expansion of Waymo self-driving services across more markets, according to the company. Genie 3 drew widespread renown last week with demonstrations of its so-called world-building capabilities, triggering a selloff in companies that provide game development and graphics creation tools. Waymo said it has customized it to generate synthetic driving footage and depth perception data as if they were captured by cameras and lidar sensors on vehicles. Waymo's new system can also take real-life dashcam datasets and convert them into scenes and depth maps for vehicle simulations, the company said. The combination will aid in making autonomous vehicle systems more reliable in uncommon scenarios. "Traditional AV simulation models are constrained by the on-road data they collect," a spokeswoman for Waymo said. The new world model "allows us to explore situations that were never directly observed by our fleet." "This will enhance Waymo's ability to safely scale our service across more places and new driving environments," she added. Robotaxi operators and artificial intelligence companies more broadly have been seeking more sources of data in the race to advance their models. Nvidia Corp., which supplies chips and AI models for developers of self-driving technology, has partnered with rideshare leader Uber Technologies Inc. to collect millions of hours of robotaxi-specific driving data to fuel driverless model training and validation. SoftBank Group Corp.-backed Wayve Technologies Ltd., which plans to trial robotaxis on the Uber platform in the UK this year, has announced its own world model to generate synthetic driving data. And Tesla Inc. has also said it built a similar simulator. Having bigger training data sets will be crucial for Waymo as it faces safety probes from US authorities after a series of software mishaps in recent months. The National Highway Traffic Safety Administration and the National Transportation Safety Board are looking into several incidents where Waymo failed to stop for parked school buses in Austin, violations that also prompted Waymo to file a voluntary software recall. The Waymo spokesperson declined to comment on whether it has conducted simulations on stopped school bus encounters, or mass power outages like the one that disrupted its San Francisco operations last December. But she said the Waymo World Model "can simulate virtually any scene." Incorporating Google's Genie 3 model could be a boon for Waymo's plans to expand to about a dozen cities this year. Simulation is just one tool that it uses to prepare its autonomous systems for certain situations and to validate its safety, the spokesperson said. In December, the company said that there is "no substitute" for its experience driving in the real world. It surpassed more than 20 million autonomous trips that months, Alphabet said on Wednesday, more than any other provider in the US.

Waymo Catches World Model Fever, and the Only Prescription Is More World Models

Waymo vehicles have reportedly racked up more than 200 million miles of autonomous driving on public roads. But it's yet to run into a tornado or an elephant, and odds are that it'd respond poorly if it did. To try to help with those once-in-a-billion-miles scenarios, Waymo announced Friday that it is introducing Waymo World Model, a generative AI model that it will use to run near-endless situations to try to make sure its cars are prepared for the unpredictable, which also just happens to fit into the latest trend in the AI space. To be clear, Waymo's world model makes about as much sense as any use case for the technology. The company has a ton of high-definition data that it has collected from its time on the road that it can use to generate realistic re-creations of roads. But, the company said, instead of building a model based only on that information, it's going to use Google's Genie 3 model to put its cars in simulated situations that extend beyond what is already in its data set collected from cameras and lidar sensors. Google made a splash last month when it released a beta version of Genie 3 to the public, allowing a subset of paid subscribers to generate 3D worlds with realistic physics. Unlike a large language model (LLM)â€"the underlying technology that powers most AI tools including Google’s own Geminiâ€"which use the vast amount of training data they are given to predict the most likely next part of a sequence, world models are trained on the dynamics of the real world, including physics and spatial properties, to create a simulation of how physical environments operate. Waymo plans to tap into that to put its cars through a gauntlet of scenarios that they likely wouldn't find themselves in until it's too late. That includes extreme weather conditions and natural disasters, so the cars can figure out how to navigate a tornado or flood waters; sudden safety emergencies like falling tree branches or an accident with lots of debris; and run-ins with the unexpected, like an elephant on the road. “By simulating the â€~impossible,’ we proactively prepare the Waymo Driver for some of the most rare and complex scenarios,†the company said. The theory is certainly sound, though world models aren't without their drawbacks. The early feedback on the consumer version of Genie 3 was a bit spotty, and world models are still susceptible to hallucinations. We're still in the earliest stages of seeing these models deployed, and they have lots of room to iterate. And Waymos have definitely had their issues in edge-case scenarios in the real world. Late last year, a Waymo ran over a beloved bogeda cat named Kit Kat, and last month, one ran into a kid in a school zone. Those interactions aren't even particularly rare for a driver to find themselves in, so hopefully Waymo can refine its responses in those scenarios on top of prepping for the most unlikely situations.

Google World Model AI Accelerates Waymo Robotaxi Expansion | PYMNTS.com

The company's newly introduced Waymo World Model is built on Google DeepMind's general purpose world model Genie 3, which Waymo then adapted for autonomous driving simulation, Waymo said in a Friday blog post. With Genie 3's world knowledge, Waymo World Model can simulate a wider range of events, including extreme weather conditions, natural disasters, and rare and safety-critical events, according to the post. Waymo World Model also enables engineers to use simple language prompts, driving inputs and scene layouts to modify simulations, per the post. "This combination of broad world knowledge, fine-grained controllability and multi-modal realism enhances Waymo's ability to safely scale our service across more places and new driving environments," Waymo said in the post. PYMNTS reported in November that world models are systems designed to understand how the world behaves rather than just how it looks. These models integrate perception, simulation, spatial reasoning and prediction so that machines build an internal model of cause and effect. Genie 3 is one of these models, and it can generate 3D environments governed by physics, where AI agents learn by exploring virtual worlds rather than static datasets. Google DeepMind introduced an experimental research prototype powered by Genie 3 on Jan. 29, saying it uses the latest world model AI to generate and explore interactive virtual environments. The company opened access to this prototype, Project Genie, for Google AI Ultra subscribers in the United States, allowing them to experiment with the world-generation features. It was reported Jan. 31 that Wall Street reacted to the unveiling of Genie 3 by erasing billions in market value throughout the video game industry due to concerns that the AI system can generate video games from scratch. Meanwhile, Waymo announced Monday (Feb. 2) that it raised $16 billion in an investment round that valued the company at $126 billion post-money. The company said Google parent company Alphabet continued to support Waymo as the company's majority investor.