Google DeepMind's Dreamer AI Masters Minecraft: A Leap Towards Self-Improving AI

Google DeepMind Taught Itself to Play Minecraft

The Dreamer AI system of Google's DeepMind reached the milestone of mastering Minecraft by 'imagining' the future impact of possible decisions An artificial intelligence (AI) system has for the first time figured out how to collect diamonds in the hugely popular video game Minecraft -- a difficult task requiring multiple steps -- without being shown how to play. Its creators say the system, called Dreamer, is a step towards machines that can generalize knowledge learn in one domain to new situations, a major goal of AI. "Dreamer marks a significant step towards general AI systems," says Danijar Hafner, a computer scientist at Google DeepMind in San Francisco, California. "It allows AI to understand its physical environment and also to self-improve over time, without a human having to tell it exactly what to do." Hafner and his colleagues describe Dreamer in a study in Nature published on 2 April. In Minecraft, players explore a virtual 3D world containing a variety of terrains, including forests, mountains, deserts and swamps. Players use the world's resources to create objects, such as chests, fences and swords -- and collect items, among the most prized of which are diamonds. If you're enjoying this article, consider supporting our award-winning journalism by subscribing. By purchasing a subscription you are helping to ensure the future of impactful stories about the discoveries and ideas shaping our world today. Importantly, says Hafner, no two experiences are the same. "Every time you play Minecraft, it's a new, randomly generated world," he says. This makes it useful for challenging an AI system that researchers want to be able to generalize from one situation to the next. "You have to really understand what's in front of you; you can't just memorize a specific strategy," he says. Collecting a diamond is "a very hard task," says computer scientist Jeff Clune at the University of British Columbia in Vancouver, Canada, who was part of a separate team that trained a program to find diamonds using videos of human play. "There is no question this represents a major step forward for the field." AI researchers have focused on finding diamonds, says Hafner, because it requires a series of complicated steps, including finding trees and breaking them down to gather wood, which players can use to build a crafting table. This, together with more wood, can be used to make a wooden pickaxe -- and so on, until players have assembled the correct tools to collect a diamond, which is buried deep underground. "There's a long chain of these milestones, and so, it requires very deep exploration," he says. Previous attempts to get AI systems to collect diamonds relied on using videos of human play or researchers leading systems through the steps. By contrast, Dreamer explores everything about the game on its own, using a trial-and-error technique called reinforcement learning -- it identifies actions that are likely to beget rewards, repeats them and discards others. Reinforcement learning underpins some major advances in AI. But previous programs were specialists -- they could not apply knowledge in new domains from scratch. Key to Dreamer's success, says Hafner, is that it builds a model of its surroundings and uses this 'world model' to 'imagine' future scenarios and guide decision-making. Rather like our own abstract thoughts, the world model is not an exact replica of its surroundings. But it allows the Dreamer agent to try things out and predict the potential rewards of different actions using less computation than would be needed to complete those actions in Minecraft. "The world model really equips the AI system with the ability to imagine the future," says Hafner. This ability could also help to create robots that can learn to interact in the real world -- where the costs of trial and error are much higher than in a video game, says Hafner. Testing Dreamer on the diamond challenge was an afterthought. "We built this whole algorithm without that in mind," says Hafner. But it occurred to the team that it was the ideal way to test whether its algorithm could work, out of the box, on an unfamiliar task. In Minecraft, the team used a protocol that gave Dreamer a 'plus one' reward every time it completed one of 12 progressive steps involved in diamond collection -- including creating planks and a furnace, mining iron and forging an iron pickaxe. These intermediate rewards prompted Dreamer to select actions that were more likely to lead to a diamond. The team reset the game every 30 minutes so that Dreamer did not become accustomed to one particular configuration -- but rather learnt general rules for gaining rewards. Under this set-up, it takes around nine days of continuous play for Dreamer to find at least one diamond, says Hafner. Expert human players will take 20-30 minutes to find a diamond, whereas novices take longer. "This paper is about training a single algorithm to perform well across diverse reinforcement-learning tasks," says computer scientist Keyon Vafa at Harvard University in Boston, Massachusetts. "This is a notoriously hard problem and the results are fantastic." An even bigger target for AI, says Clune, is the ultimate challenge for Minecraft players: killing the Ender Dragon, the virtual world's most fearsome creature.

This Google AI learned to mine diamonds in Minecraft in 9 days flat

While the movie is passable at best and strictly for superfans or kids, Google's AI breakthrough is anything but trivial. The tech giant's DeepMind team has managed to train an AI called Dreamer (not to be confused with Dream, a streamer who also plays Minecraft) to master Minecraft entirely on its own in just nine days, marking a significant milestone in AI self-improvement. Unlike previous AI models trained with hours of human gameplay footage, Dreamer learned Minecraft without prior exposure. Researchers at Google DeepMind and the University of Toronto designed Dreamer with a unique reinforcement learning system that rewarded it for collecting diamonds, one of the game's most valuable resources. The AI was not given step-by-step instructions but was instead encouraged to explore and optimize its approach through trial and error. Every 30 minutes, researchers reset the Minecraft world, forcing Dreamer to adapt to a new, randomly generated environment. Despite these constant changes, the AI rapidly improved.

Google's AI Dreamer learns how to self-improve over time by mastering Minecraft

A trio of AI researchers at Google's Google DeepMind, working with a colleague from the University of Toronto, report that the AI algorithm Dreamer can learn to self-improve by mastering Minecraft in a short amount of time. In their study published in the journal Nature, Danijar Hafner, Jurgis Pasukonis, Timothy Lillicrap and Jimmy Ba programmed the AI app to play Minecraft without being trained and to achieve an expert level in just nine days. Over the past several years, computer scientists have learned a lot about how deep learning can be used to train AI applications to conduct seemingly intelligent activities such as answering questions. Researchers have also found that AI apps can be trained to play games and perform better than humans. That research has extended into video game playing, which may seem to be redundant, because what could you get from a computer playing another computer? In this new study, the researchers found that it can produce advances such as helping an AI app learn to improve its abilities over a short period of time, which could give robots the tools they need to perform well in the real world. In this effort, the researchers programmed Dreamer to play the popular video game Minecraft by building a system of rewards, specifically rewards for finding diamonds. With this approach, the app did not need to be taught how to play the game; it just needed to know the parameters within which it could work, one of which included envisioning a virtual future world. Once the algorithm had learned to play Minecraft, the researchers added a new twist -- they only allowed it to play under a given scenario for 30 minutes at a time. At that point, the game would be restarted with a whole new virtual universe. Using this approach, the researchers found that the algorithm improved quickly, achieving expert status after playing the game for just nine days. The research team suggests that the algorithm's ability to imagine a future where all its goals have been achieved enabled it to remain focused on only those tasks that led to the desired goal, and then to use them in each new virtual world it encountered. This result could eventually be used to help robots teach themselves how to achieve predefined goals in the real world.

AI Just Passed a Key Intelligence Threshold: Obtaining Minecraft Diamonds

Google DeepMind's Dreamer AI has learned to mine diamonds in Minecraft, a key milestone for self-learning AI systems. | Credit: Andrew Chin / Getty Images. In efforts to build more capable, agentic AI systems, researchers have turned to games as an ideal learning environment. Following this strategy, Google DeepMind's AI platform, Dreamer, recently mastered a key task researchers have used to assess AI capabilities -- obtaining diamonds in Minecraft. For AI systems that may one day be called on to interact with physical environments, the game provides a more lifelike alternative to traditional performance benchmarks. While relatively easy for human players, obtaining diamonds in Minecraft is a complex task for AI. To mine diamond ore, users must make a strong enough pickaxe, which requires multiple steps. OpenAI managed to achieve the feat in 2022 by training a neural network on 70,000 hours of labeled gameplay video. In contrast, Dreamer's recent breakthrough is noteworthy because the AI is dropped into the game without prior experience and must learn skills independently through trial and error. Games as AI Benchmarks As interest in reinforcement learning and real-world AI applications grows, Minecraft isn't the only game researchers use to test and train AI. Previously, Dreamer learned to perform actions on classic Atari games and DeepMind's DM Lab, a suite of 3D navigation and puzzle-solving tasks. Chatbot Showdown While reinforcement learning agents like Dreamer are specifically designed to operate in virtual environments, in an era of rising AI computer use, even general-purpose chatbots can learn to play games. In February, a Twitch channel called ClaudePlaysPokemon started streaming videos of Anthropic's chatbot playing Pokemon Red and Blue. The project leverages Claude's multimodal capabilities, relying on computer vision to interpret screenshots of the game and linguistic reasoning to determine its course of action. Using the same technique, other popular consumer AI platforms can also learn to play games.