Google's GameNGen AI Simulates DOOM in Real-Time Without a Game Engine

Google reaches grim AI milestone, replicates playable version of Doom entirely by algorithm

Doom has been ported to dozens of devices, but it's never been playable quite like this. Google researchers have now generated an AI version of the retro first-person shooter classic entirely via neural network, based on ingested video clips of gameplay. It's a milestone, if a grim one, recorded in a paper published this week entitled "Diffusion models are real-time game engines" (thanks, VentureBeat). This documents how a small team from Google were able to "interactively simulate" a version of Doom, with only a "slightly better than random chance" of humans being able to tell the difference. Humans are still (currently) required to play Doom first, to provide the video clips of gameplay that are then fed into GameNGen, the research team's game engine which is "powered entirely by a neural model". It's the same principle as the now-commonplace ability for AI to learn from and then generate static images, based on ingesting huge amouts of dubiously-sourced data online. GameNGen then produces sequential frames based on its learnings of 'watching' that gameplay, which are then output at 20fps with a visual quality "comparable to the original". Here's how it looks: "Can a neural model running in real-time simulate a complex game at high quality?" the paper asks. "In this work we demonstrate that the answer is yes. While not an exact simulation, the neural model is able to perform complex game state updates, such as tallying health and ammo, attacking enemies, damaging objects, opening doors, and persist the game state over long trajectories. "GameNGen answers one of the important questions on the road towards a new paradigm for game engines, one where games are automatically generated, similarly to how images and videos are generated by neural models in recent years. Key questions remain, such as how these neural game engines would be trained and how games would be effectively created in the first place, including how to best leverage human inputs. We are nevertheless extremely excited for the possibilities of this new paradigm." Earlier this year, a major Eurogamer investigation looked at how AI was already changing video game development forever - with positive and negative results. A report from Unity in March claimed that more than 60 percent of game developers were already using AI at some stage in the development process.

This AI Model Can Simulate the PC Game Doom in Real-Time

I've been with PCMag since October 2017, covering a wide range of topics, including consumer electronics, cybersecurity, social media, networking, and gaming. Prior to working at PCMag, I was a foreign correspondent in Beijing for over five years, covering the tech scene in Asia. We've all seen how AI image generators can churn out pictures of whatever you'd like. But what if you took the same technology and applied it to generating stills for a playable game? Researchers at Google recently used this concept to develop an AI model that's capable of simulating the 1993 classic PC shooter Doom -- but without using computer code from the game itself. Instead, the researchers' model works by pumping out stills for the game like an AI image generator does, except it can do so in real-time at over 20 frames per second for a playable experience. The model is called GameNGen, and it's the subject of a new paper from researchers at Google and Tel Aviv University. "Can a neural model running in real-time simulate a complex game at high quality? In this work we demonstrate that the answer is yes," they write. "Specifically, we show that a complex video game, the iconic game Doom, can be run on a neural network." In the paper, the researchers note a computer game fundamentally works like this: the player makes an action or input, the game state updates accordingly, and then it renders the result on the screen. This so-called "game loop" creates the illusion that you're in an interactive virtual world, even though your computer is just showing you changing pictures on the screen. The researchers used Stable Diffusion version 1.4, an open-source AI image generator. They also developed a separate AI model to play the real Doom game while recording the footage for a total of 900 million frames. The resulting training data is then used by Stable Diffusion to pump out game images, adapting them as it receives inputs from the player. The team posted several clips of GameNGen rendering Doom, including footage of human players trying it out. The results show the AI model is able to accurately simulate the classic PC shooter both visually and on a gameplay level. For example, the model can simulate a door opening as the player approaches and a fireball hitting the player, taking away some health. However, GameNGen also contains some major limitations. "The model only has access to a little over 3 seconds of history," the researchers wrote. As a result, enemies and objects can sometimes pop in of nowhere and then disappear seconds later. Nevertheless, GameNGen is able to create the illusion it can remember the game world because each rendered image allows the model to infer the player's ammo, health status, weapons, and location. The other issue is that a traditional computer game can be quite complex. In addition to rendering pixels on a screen, a game can contain dialogue, numerous characters, along with story and game mechanics that can happen off-screen. But despite the limitations, the researchers say GameNGen shows how generative AI could transform game development, potentially leading to AI-created games, which Nvidia's CEO has also predicted could occur in the next five to 10 years. "For example, we might be able to convert a set of frames into a new playable level or create a new character just based on example images, without having to author code," the researchers wrote in their paper while adding: "Today, video games are programmed by humans. GameNGen is a proof-of-concept for one part of a new paradigm where games are weights of a neural model, not lines of code."

Google's GameNGen: AI breaks new ground by simulating Doom without a game engine

Join our daily and weekly newsletters for the latest updates and exclusive content on industry-leading AI coverage. Learn More Google researchers have reached a major milestone in artificial intelligence by creating a neural network that can generate real-time gameplay for the classic shooter Doom -- without using a traditional game engine. This system, called GameNGen, marks a significant step forward in AI, producing playable gameplay at 20 frames-per-second on a single chip, with each frame predicted by a diffusion model. "We present GameNGen, the first game engine powered entirely by a neural model that enables real-time interaction with a complex environment over long trajectories at high quality," the researchers state in their paper, published on the preprint server arXiv. This achievement marks the first time an AI has fully simulated a complex video game with high-quality graphics and interactivity. Running on a single Tensor Processing Unit (TPU) -- Google's custom-built AI accelerator chip -- GameNGen handles Doom's intricate 3D environments and fast-paced action with remarkable efficiency, all without the usual components of a game engine. AI game engines: A game-changer for the $200 billion gaming industry Doom has long been a technological benchmark since its 1993 release, ported to an astonishing array of platforms -- from microwaves to digital cameras. However, GameNGen transcends these earlier adaptations. Unlike traditional game engines that rely on painstakingly coded software to manage game states and render visuals, GameNGen autonomously simulates the entire game environment using an AI-driven generative diffusion model. The transition from traditional game engines to AI-driven systems like GameNGen could transform the $200 billion global gaming industry. By eliminating the need for manually programmed game logic, AI-powered engines have the potential to significantly reduce both development time and costs. This technological shift could democratize game creation, enabling smaller studios and even individual creators to produce complex, interactive experiences that were previously unimaginable. Beyond cost and time savings, AI-driven game engines could open the door to entirely new genres of games, where the environment, narrative, and gameplay mechanics dynamically evolve based on player actions. This innovation could reshape the gaming landscape, moving the industry away from a blockbuster-centric model toward a more diverse and varied ecosystem. From video games to autonomous vehicles: Broader implications of AI-driven simulations The potential applications for GameNGen extend far beyond gaming. Its capabilities suggest transformative possibilities in industries such as virtual reality, autonomous vehicles, and smart cities, where real-time simulations are essential for training, testing, and operational management. For instance, autonomous vehicles require the ability to simulate countless driving scenarios to safely navigate complex environments -- a task that an AI-driven engine like GameNGen could perform with high fidelity and real-time processing. In the realm of virtual and augmented reality, AI-driven engines could create fully immersive, interactive worlds that adapt in real-time to user inputs. This could revolutionize sectors like education, healthcare, and remote work, where interactive simulations can provide more effective and engaging experiences. The future of gaming: When AI dreams of virtual worlds While GameNGen represents a significant leap forward, it also presents challenges. Although it can run Doom at interactive speeds, more graphically intensive modern games would likely require much greater computational power. Additionally, the current system is tailored to a specific game (i.e. Doom), and developing a more general-purpose AI game engine capable of running multiple titles remains a tough challenge. Nevertheless, GameNGen is a crucial step toward a new era in game engines -- one where games are not just played by AI but also created and powered by it. As AI continues to advance, we may be on the cusp of a future where our favorite games are born not from lines of code, but from the boundless creativity of machines. This development also opens up exciting possibilities for game creation and interaction. Future games could adapt in real-time to player actions, generating new content on the fly. AI-powered game engines might also dramatically reduce development time and costs, potentially democratizing game creation. As we stand on the brink of this new era in gaming, one thing is clear: the lines between human creativity and machine intelligence are blurring, promising a future of digital entertainment we can scarcely imagine. With GameNGen, Google researchers have given us an exciting glimpse of that future -- a world where the only limit to our virtual experiences is the imagination of AI.

AI creates a playable version of the original Doom, generating each frame in real-time

Google's research scientists have published a paper on its new GameNGen technology, an AI game engine that generates each new frame in real-time based on player input. It kind of sounds like Frame Generation gone mad in that everything is generated by AI, including visual effects, enemy movement, and more. AI generating an entire game in real-time is impressive, even more so when GameNGen uses its tech to recreate a playable version of id Software's iconic Doom. This makes sense when you realize that getting Doom to run on lo-fi devices, high-tech gadgets, and even organic material is a right of passage. Seeing it in action, you can see some of the issues when it comes to AI generating everything (random artifacts, weird animation), but it's important to remember that everything you see is being generated and built around you in real-time as you move, strafe, and fire shotgun blasts at demons. As expected, the underlying AI model was trained on Doom and played repeatedly by AI agents trained to play the game, simulating various skills and playstyles. The result is impressive, to be sure. However, the game runs at 20 FPS, so there are still latency and performance improvements before GameNGen could be considered a viable option for playing a game. What makes this an essential breakthrough for generative AI is how the image stays consistent between frames, something AI has struggled with when animating physical objects and characters. Each frame is separate without any underlying physics calculations or physical rendering. GameNGen presents a notable improvement thanks to Google Research extending the training on new frames with preceding frames and user input information. Here's the official description of what it does and how it works.

AI makes Doom in its own game engine -- Google's GameNGen project uses Stable Diffusion to simulate gameplay

Google Research scientists have released their paper on GameNGen, an AI-based game engine that generates original Doom gameplay on a neural network. Using Stable Diffusion, scientists Dani Valevski, Yaniv Leviathan, Moab Arar, and Shlomi Fruchter designed GameNGen to process its previous frames and the current input from the player to generate new frames in the world with surprising visual fidelity and cohesion. AI-generating a complete game engine with consistent logic is a unique achievement. GameNGen's Doom can be played like an actual video game, with turning and strafing, firing weapons, and accurate damage from enemies and environmental hazards. An actual level is built around you in real-time as you explore it. It even keeps a mostly precise tally of your pistol's ammo. According to the study, the game runs at 20 FPS and is difficult to distinguish in short clips from actual Doom gameplay. To obtain all of the training data necessary for GameNGento to accurately model its own Doom levels, the Google team trained its agent AI to play Doom at all difficulties and simulate a range of player skill levels. Actions like collecting power-ups and completing levels were rewarded. At the same time, player damage or death was punished, creating agents that could play Doom and providing hundreds of hours of visual training data for the GameNGen model to reference and recreate. A significant innovation in the study is how the scientists maintained cohesion between frames while using Stable Diffusion over long periods. Stable Diffusion is a ubiquitous generative AI model that generates images from image or text prompts and has been used for animated projects since its release in 2022. Stable Diffusion's two most significant weaknesses for animation are its lack of cohesion from frame to frame and its eventual regression in visual fidelity over time. As seen in Corridor's Anime Rock Paper Scissors short film, Stable Diffusion can create convincing still images but suffers from flickering effects as the model outputs consecutive frames (notice how the shadows seem to jump all across the faces of the actors from frame to frame). The flickering can be solved by feeding Stable Diffusion its output and training it using the image it created to ensure frames match one another. However, after several hundred frames, the image generation becomes less and less accurate, similar to the effect of photocopying a photocopy many times. Google Research solved this problem by training new frames with a more extended sequence of user inputs and frames that preceded them -- rather than just a single prompt image -- and corrupting these context frames using Gaussian noise. Now, a separate but connected neural network fixes its context frames, ensuring a constantly self-correcting image and high levels of visual stability that remain for long periods. The examples of GameNGen seen so far are, admittedly, less than perfect. Blobs and blurs pop up on-screen at random times. Dead enemies become blurry mounds after death. Doomguy on the HUD is constantly flickering his eyebrows up and down like he's The Rock on Monday Night Raw. And, of course, the levels generated are inconsistent at best; the embedded YouTube video above ends in a poison pit where Doomguy suddenly stops taking damage at 4% and completely changes its layout after turning around 360 degrees inside it. While the result is not a winnable video game, GameNGen produces an impressive simulacrum of the Doom we love. Somewhere between tech demos and thought experiments on the future of AI, Google's GameNGen will become a crucial part of future AI game development if the field continues. Paired with Caltech's research on using Minecraft to teach AI models consistent map generation, AI-baked video game engines could be coming to a computer near you sooner than we'd thought.

AI Is Hallucinating DOOM

Can it hallucinate DOOM? Google Research and Tel Aviv University have successfully simulated DOOM within a neural learning model named GameNGen. It's been a big year for the "Can It Run DOOM" scene. We got DOOM running on poop germs, for example, and a mad scientist taught a lab-grown rat brain to play DOOM. But Google Research and Tel Aviv University have flipped the script with their GameNGen project -- they aren't just running DOOM in a neural model, they're simulating DOOM without utilizing any traditional code, visual assets, or game engines. Metaphorically speaking, we now have a machine that can "think" DOOM into existence. The simulated DOOM is fully interactive and immediately recognizable. It runs in full color at 20 FPS on a single TPU (tensor processing unit), meaning that "neural game engines" like GameNGen can be relatively lightweight. While this is not the first AI simulation of DOOM, it is by far the most impressive and accurate. GameNGen's training was achieved through a two-phase process. First, a reinforcement learning model (a reward-seeking AI, kind of like a lab rat) was taught to play DOOM. Its gaming sessions were recorded and passed on to a diffusion model (an AI that's comparable to the predictive text algorithm in your smartphone keyboard), which learned to predict and generate in-game visuals. The models were not exposed to DOOM's source code or visual asset library. "A complex video game, the iconic game DOOM, can be run on a neural network (an augmented version of the open Stable Diffusion v1.4, in real-time, while achieving a visual quality comparable to that of the original game. While not an exact simulation, the neural model is able to perform complex game state updates, such as tallying health and ammo, attacking enemies, damaging objects, opening doors, and persist the game state over long trajectories." While the AI DOOM simulation is obviously very impressive, it's not perfect. Many of the "complex game state updates" simulated by the AI are affected by tell-tale visual artifacts. The health and ammo tickers at the bottom of the screen regularly flick between numbers, and movement is often subject to the kind of smudginess that we often see in generative video. Still, GameNGen runs DOOM at a better quality and frame rate than most PCs did in the mid-90s. And this is without the elegant DOOM Engine (or any conventional game engine, for that matter). Google Research also found that, when viewing short clips between 1.6 seconds and 3.2 seconds, humans had a lot of trouble differentiating the fake DOOM from the real DOOM (their success rate was 58% to 60%, depending on video length). The image is often perfect; it just fails to be consistently perfect. As for how this research will be used in the future -- it's anyone's guess. Google Research and Tel Aviv University have proven that an interactive game can run within the paradigm of a neural model. But they did not create a game from scratch. The arduous process of simulating a game within a neural model has no practical or economic benefit as of 2024. GameNGen, in its current form, is just a proof of concept. However, this research may lead to the development of a neural model that can generate unique games. If generative game development can be achieved at a lower cost than traditional game development (while also providing a fun experience for gamers), something like GameNGen could become a viable product. But training may prove to be the biggest hurdle here, as the AI would need a decent understanding of how games work (GameNGen appears to lean heavily on visual observations), and, importantly, it would need a massive dataset containing a diverse array of existing, copyrighted games. While I've tried my best to explain this research, I suggest reading the Diffusion Models Are Real-Time Game Engines whitepaper and visiting the GameNGen Github page. Source: GameNGen

New AI model can hallucinate a game of 1993's Doom in real time

"Why write rules for software by hand when AI can just think every pixel for you?" On Tuesday, researchers from Google and Tel Aviv University unveiled GameNGen, a new AI model that can interactively simulate the classic 1993 first-person shooter game Doom in real time using AI image generation techniques borrowed from Stable Diffusion. It's a neural network system that can function as a limited game engine, potentially opening new possibilities for real-time video game synthesis in the future. For example, instead of drawing graphical video frames using traditional techniques, future games could potentially use an AI engine to "imagine" or hallucinate graphics in real time as a prediction task. Further Reading "The potential here is absurd," wrote app developer Nick Dobos in reaction to the news. "Why write complex rules for software by hand when the AI can just think every pixel for you?" GameNGen can reportedly generate new frames of Doom gameplay at over 20 frames per second using a single tensor processing unit (TPU), a type of specialized processor similar to a GPU that is optimized for machine learning tasks. In tests, the researchers say that ten human raters sometimes failed to distinguish between short clips (1.6 seconds and 3.2 seconds) of actual Doom game footage and outputs generated by GameNGen, identifying the true gameplay footage 58 percent or 60 percent of the time. Real-time video game synthesis using what might be called "neural rendering" is not a completely novel idea. Nvida CEO Jensen Huang predicted during an interview in March, perhaps somewhat boldly, that most video game graphics could be generated by AI in real time within five to ten years. GameNGen also builds on previous work in the field, cited in the GameNGen paper, that includes World Models in 2018, GameGAN in 2020, and Google's own Genie in March. And a group of University researchers trained an AI model (called "DIAMOND") to simulate vintage Atari video games using a diffusion model earlier this year. Also, ongoing research into "world models" or "world simulators," commonly associated with AI video synthesis models like Runway's Gen-3 Alpha and OpenAI's Sora, is leaning toward a similar direction. For example, during the debut of Sora, OpenAI showed demo videos of the AI generator simulating Minecraft. Diffusion is key In a preprint research paper titled "Diffusion Models Are Real-Time Game Engines," authors Dani Valevski, Yaniv Leviathan, Moab Arar, and Shlomi Fruchter explain how GameNGen works. Their system uses a modified version of Stable Diffusion 1.4, an image synthesis diffusion model released in 2022 that people use to produce AI-generated images. "Turns out the answer to "can it run DOOM?" is yes for diffusion models," wrote Stability AI Research Director Tanishq Mathew Abraham, who was not involved with the research project. While being directed by player input, the diffusion model predicts the next gaming state from previous ones after having been trained on extensive footage of Doom in action. The development of GameNGen involved a two-phase training process. Initially, the researchers trained a reinforcement learning agent to play Doom, with its gameplay sessions recorded to create an automatically generated training dataset -- that footage we mentioned. They then used that data to train the custom Stable Diffusion model. However, using Stable Diffusion introduces some graphical glitches, as the researchers note in their abstract: "The pre-trained auto-encoder of Stable Diffusion v1.4, which compresses 8x8 pixel patches into 4 latent channels, results in meaningful artifacts when predicting game frames, which affect small details and particularly the bottom bar HUD." And that's not the only challenge. Keeping the images visually clear and consistent over time (often called "temporal coherency" in the AI video space) can be a challenge. GameNGen researchers say that "interactive world simulation is more than just very fast video generation," as they write in their paper. "The requirement to condition on a stream of input actions that is only available throughout the generation breaks some assumptions of existing diffusion model architectures," including repeatedly generating new frames based on previous ones (called "autoregression"), which can lead to instability and a rapid decline in the quality of the generated world over time.