Waymo taps Google's Genie 3 to simulate tornadoes, elephants, and extreme edge cases

Waymo World Model Built on Google's Genie 3 Technology

Waymo has unveiled a new training approach for self-driving cars that leverages Google DeepMind's Genie 3 to create hyper-realistic simulated environments. The Waymo World Model addresses a fundamental challenge in autonomous vehicle training: rare, potentially dangerous events are not well represented in real-world data 1

1

. While the Alphabet subsidiary touts more than 200 million miles of autonomous driving on public roads, its AI has now driven billions of miles virtually through digital simulations 1

1

.

The generative AI model enables engineers to create simulations with simple prompts and driving inputs, testing scenarios that would be impossible or extremely rare to encounter in real life. These include snow on the Golden Gate Bridge, a flooded suburban cul-de-sac with floating furniture, neighborhoods engulfed in flames, and even encounters with rogue elephants 2

2

. By simulating the "impossible," Waymo proactively prepares its vehicles for some of the most rare and complex scenarios, creating a more rigorous safety benchmark 3

3

.

How Genie 3 Enables Advanced Autonomous Vehicle Training

Genie 3 represents a significant upgrade over earlier world models through its long-horizon memory capability. The model can remember details for several minutes, maintaining context even when the virtual camera moves away from objects 1

1

. Autoregressive world models like Genie don't actually create 3D spaces but instead render video quickly enough that it feels like an explorable world.

Waymo has customized Genie 3 to generate synthetic driving footage and depth perception data as if captured by cameras and LiDAR sensors on vehicles 4

4

. The system offers three unique mechanisms: driving action control for "what if" counterfactuals, scene layout control for customizing road layouts and traffic signals, and language control for adjusting time-of-day and weather conditions 2

2

. This language control proves especially helpful when simulating low-light or high-glare conditions where sensors may struggle.

Extending Simulation Duration and Realism

Waymo engineers discovered an innovative workaround for Genie 3's limited long-term stability. While the model typically loses consistency after about a minute, Waymo found that by speeding up footage by 4X, it could create much longer scenarios without sacrificing image quality or computer processing 3

3

. The system can also take real-world dashcam footage and transform it into simulated environments, converting datasets into scenes and depth maps for the "highest degree of realism and factuality" in virtual testing 2

2

.

In each scenario, the robotaxi's LiDAR sensors generate a 3D rendering of the surrounding environment, including obstacles in the road. Traditional AV simulation models are constrained by on-road data they collect, but the new world model allows Waymo to explore situations never directly observed by its fleet 4

4

.

Implications for Robotaxi Rollout and Safety

The collaboration with DeepMind arrives at a critical time for Waymo's expansion plans. The company aims to expand to about a dozen cities this year and surpassed more than 20 million autonomous trips as of December 4

4

. Having bigger training data sets will prove crucial as Waymo faces safety probes from US authorities after a series of software mishaps in recent months. The National Highway Traffic Safety Administration and National Transportation Safety Board are investigating several incidents where Waymo failed to stop for parked school buses in Austin, violations that prompted a voluntary software recall 4

4

.

Robotaxi operators and AI companies more broadly have been seeking additional data sources in the race to advance their models. Nvidia has partnered with Uber Technologies to collect millions of hours of robotaxi-specific driving data, while SoftBank-backed Wayve Technologies announced its own world model to generate synthetic driving footage 4

4

. Tesla has also built a similar simulator. This isn't the first time Waymo has leaned on Google's vast AI resources—its EMMA training model was built using Gemini, and DeepMind has provided solutions to reduce false positives in sensor data 2

2

.

While the theory behind using world models for autonomous vehicles appears sound, questions remain about real-world performance. Waymo vehicles have had issues in edge-case scenarios, including recent incidents involving a cat and a child in a school zone 5

5

. Whether digital simulations of tornadoes and natural disasters will translate to better handling of everyday edge cases remains to be seen as the technology matures.