Microsoft Unveils Muse: A Groundbreaking AI Model for Game Development and Simulation

Xbox's new Muse generative AI may be compatible with user-created mods

TL;DR: Microsoft's new generative AI, Muse, aims to create customized gaming experiences by generating procedural gameplay from user inputs. It will be trained using first-party games, with developers having the option to opt out. Muse could integrate with user-created mods, enhancing game diversity and persistence. Microsoft is experimenting with Muse using data from the game Bleeding Edge. Microsoft's new generative AI tech could be compatible with user-generated content and pave the way to a new kind of customized gaming experience. A bit ago, Microsoft made waves by announcing Muse, a generative AI gaming world model that essentially creates procedural gameplay from user inputs. Microsoft also said that it will start using its first-party games to train Muse, but devs can opt out if they choose. There's still a lot of unknowns about Muse's practical applications, at least in the sense how a gamer could utilize it. In a recent interview with Dwarkesh Patel, Microsoft CEO Satya Nadella suggests that content created with Muse could mesh well with user-created mods, and as someone who grew up playing Bethesda games on PC, this immediately piqued my interest. "We're going to call it Muse. It's going to be the model of this world action, or human action model. "This is very cool. One of the things is that obviously, Dall-E and Sora have been unbelievable in what they've been able to do in terms of generative models. "One thing that we wanted to go after was using gameplay data. Can you actually generate games that are both consistent and then have the ability to generate the diversity of what that game represents, and then are persistent to user mods?" Immediately after this quote, Nadella goes on to say that Microsoft will continue experimenting and utilizing Muse for various game projects. To collect the data for Muse, Microsoft recorded seven years' worth of gameplay matches in an unsuspecting live service game called Bleeding Edge. "The cool thing is what I'm excited about is bringing--we're going to have a catalog of games soon that we will start using these models, or we're going to train these models to generate, and then start playing them," Nadella finished. The Muse AI video has stayed unlisted on the Xbox channel and has a negatively-skewed like-dislike ratio.

Xbox's Muse AI Model Is A One-of-a-Kind Model That Can Generate Gameplay

It has long been speculated when the time would come for generative AI models to be widely used in game development to actually create gameplay, and it seems like that time is coming with Muse, an Xbox one-of-a-kind AI model that was unveiled today. Presented today on Nature, Muse is a generative AI model that can essentially generate gameplay based on visuals or players' controller actions, understanding a 3D game world and physics and reacting to how players interact with a game. The model was trained on human gameplay data from Ninja Theory's Bleeding Edge, giving it access to the equivalent of seven years of human gameplay, a close collaboration that gave Microsoft Research the chance to understand what needs to be in place for the model and how to access a large amount of gameplay data. However, Xbox's Muse AI model is still in the early stages of development. It can only generate gameplay visuals at a very low resolution of 300 x 180, so it will be some time before it can be used for anything practical. At the same time, Microsoft stresses that it will never be used to replace actual game developers but only empower them, as it can be used, for example, in the early stages of development. Something that is really hard to believe, given the sad state of the video games industry. In addition to aiding in new game development, Xbox's Muse AI model could also be used to improve classic games and bring them to modern consoles, according to Phil Spencer. These models and their ability to learn how a game plays without the necessity of the original engine running on original hardware will provide new opportunities for game preservation, according to the Microsoft Gaming CEO. More in-depth information on the Xbox Muse AI model can be found on Microsoft's official website.

Microsoft's Muse AI Set to Transform Xbox Game Development

Microsoft is taking a bold step into the future of gaming. CEO Satya Nadella has confirmed that Xbox will soon have a catalog of games built with generative AI. The company's latest AI model, Muse, is set to transform how games are created and played, marking a major shift in the industry. Muse can generate game visuals, control actions, or even both. This technology promises to make gaming worlds more dynamic, adaptable, and immersive. Nadella compared its impact to the first time he saw like ChatGPT in action, calling it a "massive moment of wow. Microsoft has already tested using gameplay data from Ninja Theory's Bleeding Edge. The AI model's ability to generate new experiences within a game has sparked both excitement and debate. While some see this as a breakthrough, others worry about its effect on traditional game development and jobs in the industry.

Microsoft's new AI for game development called Muse can generate entire gameplay sequences

In brief: Despite the controversy surrounding generative AI, some game designers use the technology to brainstorm concepts during development. Microsoft's new open-source model aims to make this practice feasible for interactive content. It focuses on maintaining consistency across many frames while recognizing and integrating player input. Microsoft researchers recently introduced Muse, a generative AI model designed to extrapolate interactive video game scenarios from images, clips, and recorded player input. The tool aims to streamline game development while upholding ethical training practices. The weights, sample data, and interactive interface for Muse, which Microsoft calls a "World and Human Action Model," are now available on Azure AI Foundry. The developers discussed details of the technology on the company's research blog and in a newly published study in Nature. Following training on one million updates, Muse can faithfully predict up to half a minute of gameplay based on one second of actual footage and nine seconds of player input. Unlike prior models, where game details disappear or become distorted as soon as they leave the visible frame, Muse remembers details like terrain, characters, and game mechanics. n a video interview with Xbox head Phil Spencer, Dom Matthews from Ninja Theory explained that the company doesn't plan to use Muse to build content for end users. Instead, the studio plans to use it to help draft and iterate ideas by rapidly generating snapshots of a cohesive vision. It remains unclear whether other developers would follow a similar philosophy, but Microsoft won't mandate the use of its models across its studios. Some developers immediately expressed disinterest. David Goldfarb, who worked on Battlefield, Mass Effect, and Killzone, offered a brief, dismissive response. Marc Burrage from Creative Assembly told Wired that even restricting generative AI to the prototyping stage can harm game development by robbing employees of hands-on experience. Spencer optimistically speculated that technologies like Muse might facilitate game preservation by helping developers port older titles to modern devices. Microsoft has frequently promised to preserve its customers' digital libraries across hardware generations, but it remains to be seen whether regenerating them with AI can produce satisfactory results. Frank Cifaldi from the Video Game History Foundation isn't as optimistic, comparing the idea to a photocopy of a painting. Microsoft also faced harsh criticism regarding Muse's training data. Much of the harsh controversy facing AI models stems from companies using material from across the internet without the original creators' consent. Microsoft sidestepped the issue by training Muse on games it already owns. The initial demonstration used player telemetry data from Ninja Theory's multiplayer shooter, Bleeding Edge. Users who agreed to the game's EULA have already consented to help train Muse. Theoretically, the company could extend the practice to popular titles like Minecraft, Call of Duty, Halo, Overwatch, StarCraft, Doom, or Forza.

Microsoft: Our 'Muse' Generative AI Can Simulate Video Games

Microsoft researchers and Xbox staffers have created a generative AI model that can simulate a playable video game. The prototype program, called "Muse," can churn out the in-game visuals and corresponding actions based on the controller input. The 1.6 billion parameter Muse model was trained "on more than 1 billion images and controller actions" from a 2020 multiplayer title called Bleeding Edge, which comes from the Xbox studio Ninja Theory "What's groundbreaking about Muse is its detailed understanding of the 3D game world, including game physics and how the game reacts to players' controller actions," Microsoft's Xbox team wrote in a blog post on Wednesday. The company describes Muse as the first of a new class of AI programs called "World and Human Action Model." The result can not only achieve "consistent and diverse gameplay sequences" but also ensure that the game state persists even as the player makes modifications to their character or to the game world, Microsoft's team stated in a research paper. Others have also shown that generative AI programs are capable of simulating classic games, including Doom and Super Mario Bros, but in a limited format. In Muse's case, the program can only render gameplay sequences at a low 300-by-180 resolution over several minutes, resulting in some garbled graphics. The other limitation is that the generated gameplay isn't always accurate to the player's controller actions or to the game it was trained on. Despite the limitations, Microsoft views Muse as a promising way to improve video game development and preservation. This includes using AI models to revive classic games no longer supported on current hardware. "You could imagine a world where, from gameplay data and video, that a model could learn old games and really make them portable to any platform," said Xbox chief Phil Spencer in a video. The company also sees potential to use AI models to help human software developers prototype game experiences on the fly. "We are already using Muse to develop a real-time playable AI model trained on other first-party games," the Xbox team added, without offering specifics. Still, Microsoft's debut of Muse is bound to also spark fears and criticism about AI replacing human developers and destroying the creative spirit of video gaming. "Every time I think Xbox can't get worse, Phil [Spencer] proves me wrong," wrote one user on YouTube. But in response to such concerns, Microsoft's announcement noted: "For Xbox, game creators will always be the center of our overall AI efforts. We believe there is space for traditional game development and future generative AI technologies that serve as an extension of creative work and offer novel experiences. As part of this, we have empowered creative leaders here at Xbox to decide on the use of generative AI." In the meantime, other companies, including Netflix and EA, have also been embracing generative AI tools, seeing them as a way to streamline game development.

Microsoft and Hellblade Developer Ninja Theory Built an AI Called Muse Designed to Generate Ideas for a Game's Design -- and It's Already Making 'Complex' Gameplay Sequences - IGN

Microsoft has announced a new generative AI model designed for gameplay ideation. The company detailed what it calls the first World and Human Action Model (WHAM). The WHAM, Katja Hofmann, Senior Principal Research Manager and lead of the Microsoft Research Game Intelligence team, said in a blog post, is a generative AI model of a video game that can generate game visuals, controller actions, or both. Microsoft calls this generative AI model Muse, which was developed by the Microsoft Research Game Intelligence and Teachable AI Experiences (Tai X) teams in collaboration with Hellblade developer Ninja Theory. It's open sourcing the weights and sample data and making the executable available for the WHAM Demonstrator -- a concept prototype that provides a visual interface for interacting with WHAM models and multiple ways of prompting the models. The company provided a number of gameplay clips showing what Muse is capable of. Currently, the model can generate "complex gameplay sequences that are consistent over several minutes" just by prompting the model with 10 initial frames (one second) of human gameplay and the controller actions of the whole play sequence. The game used to train Muse was Ninja Theory's 2020 multiplayer game Bleeding Edge. "We worked closely with our colleagues at Ninja Theory and with Microsoft compliance teams to ensure that the data was collected ethically and used responsibly for research purposes," Hofmann insisted. "It's been amazing to see the variety of ways Microsoft Research has used the Bleeding Edge environment and data to explore novel techniques in a rapidly moving AI industry," said Gavin Costello, technical director at Ninja Theory. "From the hackathon that started it all, where we first integrated AI into Bleeding Edge, to building AI agents that could behave more like human players, to the World and Human Action Model being able to dream up entirely new sequences of Bleeding Edge gameplay under human guidance, it's been eye-opening to see the potential this type of technology has." Muse is used in "world model mode" meaning it is used to predict how the game will evolve from the initial prompt sequence. The more closely the generated gameplay sequence resembles the actual game, the more accurately Muse has captured the dynamics of that game. Generative AI is one of the hottest and most controversial topics in the creative industries. As video game development costs rise, publishers are increasingly looking to AI tools to speed up work and cut costs. Call of Duty reportedly sold an "AI-generated cosmetic" for Call of Duty: Modern Warfare 3 in late 2023, and fans accused Activision of using generative AI again for a loading screen last year. EA said in September that AI was "the very core" of its business, and just last month, Capcom said it was experimenting with generative AI to create the "hundreds of thousands" of ideas needed for in-game environments. Head of PlayStation Productions and head of product at PlayStation Studios Asad Qizilbash weighed in on AI to say its use in video games is important to Gen Z and Gen Alpha gamers who seek "personalization across everything." "For instance, non-player characters in games could interact with players based on their actions, making it feel more personal," Qizilbash said. "This is important for the younger Gen Z and Gen Alpha audiences, who are the first generations that grew up digitally and are looking for personalization across everything, as well as looking for experiences to have more meaning." Microsoft is all-in on generative AI, as anyone with even a cursory knowledge of ChatGPT and OpenAI will know, and so this development with Ninja Theory comes as little surprise. Still, the company will face tough questions from some within the video game development community who are worried models like Muse will put them out of a job. In the blog post, Hofmann insisted that Microsoft's team "focus on exploring the capabilities that models like Muse need to effectively support human creatives." "I look forward to seeing the many ways in which the community will explore these models and build on our research," Hofmann continued. "I cannot wait to see all the ways that these models and subsequent research will help shape and increase our understanding of how generative AI models of human gameplay may support gameplay ideation and pave the way for future, novel, AI-based game experiences, including the use cases that our colleagues at Xbox have already started to explore."

Microsoft's Muse AI wants to change how games are made - but can it work?

Ninja Theory's generative AI can replicate gameplay and assets. Muse is Microsoft's new generative AI for game development that promises to be a gameplay concept tool - it will ideate playable ideas, building game visuals, controller inputs and more. It sounds groundbreaking but Muse feels like a tech experiment rather than the game development revolution Microsoft is pitching. Unlike some generative AI models for game devs, like Tencent's GameGen-O that promises to be a AI game engine and creator platform, Muse isn't pitched as a replacement for the traditional game dev pipeline. Instead, it can 'extend' a gameplay demo with an AI-created simulation, for a few seconds. Co-developed with Ninja Theory, the same team that has pioneered mocap and the use of Unreal Engine's MetaHuman on games such as Senua's Saga: Hellblade II, Muse's standout feature is the ability to duplicate in-game props and enemies. The trick is, Muse analyses the vast game data and these copied assets feature the object's animation, design and interactive functions. Muse is built to understand how game engines and interactive objects work, ensuring dev teams can A developer generate 3D game environments that respond to actions - it sounds like a speedy way to iterate on a prototype, hone gameplay ideas and mechanics, and even remake retro games onto new engines. (Read more on the Microsoft blog.) We've seen similar AI driven tech, for example Nvidia RTX Remix enables artists to capture and remake game assets, updating them with AI-enhanced textures. The difference is in scope, Muse teases the ability to recreate a world and its actions, though it comes at a huge cost. Muse was trained using a cluster of 100 Nvidia H100 GPUs, leveraging around 1 million training updates to stretch one second of gameplay into an additional nine seconds of AI-simulated action. Ninja Theory used its Xbox game Bleeding Edge as the basis for its experiment. The AI model learned from this existing multiplayer gameplay footage, meaning it isn't conjuring mechanics from scratch but rather extrapolating based on real in-game moments. The output is rendered at 300x180 pixels, meaning it's not a final use-case, but it can be used to internally test ideas. To this end, Microsoft is pitching its AI as a companion, not a replacement of human creativity; a tool to aid traditional game design not replace it. In this context Muse joins a long list of generative AI experiments that have small real world uses. While it's fascinating to see a new AI tool emerge that can mimic ideas already in play, Muse feels like another AI looking for a purpose. Of course, this is all early days. And the need to open up game development can only be a good thing, it's why the best game development software like Unity and Unreal Engine have been adopting more visual scripting processes, and why no-code game dev apps like Godot and GameMaker are become more popular (read our best no-code game dev engines guide for more). Could future versions of Muse offer more compelling use cases? Possibly. Highly likely, even. But for now Ninja Theory's generative AI remains a proof of concept rather than a dev tool to shake up the industry.

Microsoft's new 'breakthrough' generative AI model is designed to 'create consistent and diverse gameplay' and could be used to preserve classic games

Muse could also preserve classic games for future generation Microsoft has unveiled its new generative AI project called Muse which it says will be used to help studios create gameplay and preserve classic titles. In partnership with Xbox Game Studios' Ninja Theory and Microsoft Research, Muse was trained on the developer's multiplayer battle arena game, Bleeding Edge, and is said to have a "detailed understanding of the 3D game world" that is aware of game physics and players' controller actions. "This allows the model to create consistent and diverse gameplay rendered by AI, demonstrating a major step toward generative AI models that can empower game creators," Microsoft said in its announcement post. The company is already using Muse to develop a real-time playable AI model that is being trained on other first-party games and provided footage showing the model's capabilities in the blog post. Microsoft also believes it can one day benefit players and developers and even preserve classic games. "Today, countless classic games tied to aging hardware are no longer playable by most people," it said. "Thanks to this breakthrough, we are exploring the potential for Muse to take older back catalog games from our studios and optimize them for any device. "We believe this could radically change how we preserve and experience classic games in the future and make them accessible to more players." Xbox boss Phil Spencer expanded on what this means in the attached announcement video, saying that Muse has the potential to "learn about older games" from older hardware. "...I think about an opportunity to have models learn about older games, games that were maybe tied to unique pieces of hardware where that engine on that hardware..." Spencer said. "Time will erode the amount of hardware that's out there that can actually play a game. Spencer also suggested that AI could make older games "portable to any platform where these models could run." "...We've talked about game preservation as an activity for us, and these models and their ability to learn completely how a game plays, without the necessity of the original engine running on the original hardware, I think opens up a ton of opportunity." It's still early days, according to Microsoft, but more information about Muse is expected at the Game Developers Conference (GDC) 2025.

Microsoft's Muse AI Model Will Generate Gameplay Visuals and Actions

However, MS has clarified that the AI isn't meant to replace human development and instead supplement it. In a world where AI is practically everywhere, you would expect almost every industry to capitalize on it. The gaming industry seems to be following suit, but recent developments have made it take a huge turn. To that effect, Microsoft has announced Muse, an AI model made in partnership with Ninja Theory that can change how developers make video games. MS Muse AI, made public through the science journal Nature, is the first World and Human Action Model (WHAM). This is a generative AI model of a game that can create both game visuals and controller actions. Muse AI can understand a game's 3D world and react to player actions to create visuals and effects. Microsoft's gaming AI model's data comes from collaborating with the Ninja Theory studio. The model used data from the studio's 4v4 shooter, Bleeding Edge. Using gameplay performed by human players, Muse was able to learn from over seven years' worth of data. This lets the AI generate visuals capable of up to 300×180 pixels for now. While not close to FHD, it's still very impressive. The Muse model hands players the power to generate gameplay and load visual elements through prompting. While the training above lets the Muse AI create artificial agents and visuals, Microsoft has clarified that the model isn't meant to replace the human development process. Instead, it is meant to enhance creativity and unlock new possibilities. As video game production costs continue to rise, a move like this by Microsoft is sure to make development easier. That said, what do you think about the Muse AI? Drop your thoughts in the comments below.

Xbox Will Have Catalogue of Games That Use Muse AI Model: Satya Nadella

Satya Nadella said games that utilise Muse will be playable soon Microsoft plans to have catalogue of games that utilise its recently revealed artificial intelligence (AI) model that can generate 3D gameplay environments, CEO Satya Nadella has said. The Xbox parent announced Muse, an AI model that can generate game visuals and controller actions to aid in game development, last week. Over the past year, Microsoft has been integrating AI to its gaming arm. The company had released an AI-powered chatbot for the Xbox platform in November 2024. Nadella talked about the Muse AI model in a podcast appearance last week and said that Microsoft would have a catalogue of games generated by the model that would be ready to play soon. "The cool thing is what I'm excited about is bringing... we're going to have a catalogue of games soon that we will start using these models, or we're going to train these models to generate, and then start playing them," Nadella said on an episode of the Dwarkesh Podcast. "In fact, when Phil Spencer first showed it to me, where he had an Xbox controller, and this model basically took the input and generated the output based on the input and it was consistent with the game," he added. According to him, the release of the Muse model was a "massive" moment, one that matched the releases of OpenAI's ChatGPT or Sora. "This is kind of one such moment," Nadella said. The Microsoft chief said that with Muse, the company wanted to generate games with gameplay data that play consistently and support persistent user mods. Microsoft unveiled its World and Human Action Model (WHAM), or Muse, last week. The model has been developed by the company's Research Game Intelligence and Teachable AI Experiences (Tai X) teams in collaboration with Xbox Games Studios' Ninja Theory, developer of Senua's Saga: Hellblade 2. According to the tech giant, the generative AI model can help game developers in the ideation process and help generate game visuals and controller actions. Muse was trained on a large amount of human gameplay data taken from the Ninja Theory's Bleeding Edge, a MOBA title released in 2020.

Microsoft Muse generative AI model for gaming unveiled

Microsoft on Wednesday introduced Muse, its first generative AI model for video game creation. Developed by the Microsoft Research Game Intelligence and Teachable AI Experiences (Tai X) teams with Xbox Games Studios' Ninja Theory, Muse introduces the World and Human Action Model (WHAM). This AI, detailed in a Nature paper by Katja Hofmann, Senior Principal Research Manager and lead of the Game Intelligence team, can generate game visuals and controller actions based on a 3D understanding of game worlds. Muse, using WHAM-1.6B with 1.6 billion parameters, was trained on over 1 billion images and controller actions -- equivalent to more than 7 years of continuous human gameplay -- from Ninja Theory's Bleeding Edge. Muse emerged from a partnership between Ninja Theory and Microsoft Research, both based in Cambridge, UK. Trained on Bleeding Edge, a 4v4 online game from 2020, the model uses ethically collected gameplay data (visuals and controller actions at 300×180 px resolution) with player consent via the End User License Agreement. Katja Hofmann noted, "I'm incredibly proud of our teams and the milestone we have achieved," highlighting Muse's ability to learn rich game structures. To support further research, Microsoft is open-sourcing Muse's weights, sample data, and the WHAM Demonstrator -- a visual interface for interacting with WHAM models -- available on Azure AI Foundry. Muse excels in generating complex gameplay sequences lasting up to 2 minutes. In demos, it's prompted with 10 frames (1 second) of human gameplay and controller actions, then predicts game evolution in "world model mode." The more its output matches Bleeding Edge dynamics, the better it captures the game. Examples show behavioral diversity (e.g., camera shifts, path choices) and visual diversity (e.g., different hoverboards). Evaluated for consistency, diversity, and persistency, Muse ensures gameplay respects physics, offers variety, and adapts user changes -- like adding a character into a scene, which it then integrates naturally. The project began in December 2022, sparked by ChatGPT's release, which inspired Hofmann to explore transformer-based models for gaming. With years of Bleeding Edge data from Ninja Theory, the team scaled training from a V100 cluster (100 GPUs) to H100s. Early demos, like one by researcher Tim Pearce, showed Muse improving over 10k, 100k, and 1M training updates -- progressing from basic recognition to mastering mechanics like flying after 1M updates. Gavin Costello, Ninja Theory's technical director, said, "It's been eye-opening to see the potential this technology has," from AI agents to new gameplay sequences. Muse pushes AI-driven game design forward. Fatima Kardar, Corporate Vice President of Gaming AI, said, "We are already using Muse to develop a real-time playable AI model trained on other first-party games." It could revive classic games tied to old hardware, optimizing them for any device. Kardar added, "This could radically change how we preserve and experience classic games." Developers might also use Muse to prototype ideas or expand existing titles, with short AI game demos soon available on Copilot Labs. Microsoft interviewed 27 game creators globally -- from indies to AAA studios -- to align Muse with real needs. Linda Wen, a design researcher, emphasized including underrepresented voices to ensure broad benefits. During a hackathon, teams built the WHAM Demonstrator, letting users prompt Muse with visuals (e.g., a Bleeding Edge promo image) and tweak outcomes using controllers. Cecily Morrison, Senior Principal Research Manager, noted, "It was a great opportunity to shape model capabilities to suit the needs of creatives right from the start." Xbox has long used AI, and Kardar said, "We're now seeing an acceleration of generative AI research." Muse reflects a creator-first approach, with usage guided by each team's vision. Looking ahead, Xbox aims to reduce barriers in gaming via AI. Kardar explained, "We'll share our AI product innovation earlier on," inviting players and creators to co-build features. Guided by principles like fairness and transparency, Xbox plans more AI updates in 2025 to enhance tools and experiences across devices. Muse is available now for researchers and developers. Its model weights, sample data, and WHAM Demonstrator can be accessed on Azure AI Foundry. Short interactive AI game demos will soon be testable on Copilot Labs, with broader updates expected in 2025.

Microsoft reveals new gaming-focused generative AI model 'Muse' that could revive classic games

In a new blog post on Wednesday, Microsoft's research department debuted a new generative AI model called Muse that it describes as a "breakthrough" for gameplay ideation. Muse is a World and Human Action Model, or WHAM, built by the Microsoft Research Lab in Cambridge in conjunction with the British game developer and Xbox subsidiary Ninja Theory. For years, Microsoft Research has used Ninja Theory's multiplayer shooter Bleeding Edge as a testbed for experiments in how to create more human-like CPU opponents: bots that act and perform more like humans would. With Muse, it's used the equivalent of roughly seven years of continuous human gameplay as training data, in order to create a new type of model that possessed a detailed understanding of the 3D game world. "The impressive abilities we first witnessed with ChatGPT and GPT-4 to learn human language are now being matched by AI's abilities to learn the mechanics of how things work, in effect developing a practical understanding of interactions in the world," wrote Peter Lee, Microsoft Research president, in a post on the official Microsoft blog. This allows Muse to generate virtual gameplay sequences of up to two minutes based upon preprogrammed criteria such as controller actions. According to Fatima Kardar, Microsoft's corporate vice president of gaming AI, Muse is already being used in-house to develop a "real-time playable AI model" using training data from other first-party games from Xbox Game Studios. In addition, Kardar suggests that Muse could be used as a method of preserving and updating classic games. "Today, countless classic games tied to aging hardware are no longer playable by most people," Kardar wrote in a post on Xbox Wire. "Thanks to this breakthrough, we are exploring the potential for Muse to take older back catalog games from our studios and optimize them for any device." Kardar continued, "We believe this could radically change how we preserve and experience classic games in the future and make them accessible to more players." Lee also notes that Muse's ability to visualize and navigate 3D spaces could lead to future breakthroughs in fields such as interior design or architectural modeling. The details behind Muse are further explored in a post on the official Microsoft Research blog, as well as a new paper in Nature, "World and Human Action Models towards gameplay ideation." Hoffman further announced that the weights and sample data are being made open source. Interested researchers can test Microsoft Research's WHAM model now on Azure AI Foundry.

Microsoft's Xbox Unveils Generative AI Model for Gameplay

Microsoft is introducing a generative artificial-intelligence model for gameplay on the Xbox. The company said Wednesday on its Xbox blog that the model, called Muse, can create consistent and diverse gameplay rendered by AI. Microsoft is using Muse to develop a real-time playable AI model trained on other first-party games. It also believes Muse will make it possible for classic games tied to aging hardware to become playable again. Muse was developed by Microsoft Research in partnership with the Xbox studio Ninja Theory, and trained on Ninja Theory's multiplayer arena game Bleeding Age. Microsoft said it expects to unveil new developments this year about how AI is being used to improve player experiences and empower game creators.

Xbox's AI initiative with Muse is an attempt to read the tea leaves, not the room

Join our daily and weekly newsletters for the latest updates and exclusive content on industry-leading AI coverage. Learn More Earlier today, the heads of Microsoft's Xbox division revealed Muse, a generative AI model that intends to create both visuals and gameplay for games. The model, which was trained on the largely forgotten Ninja Theory multiplayer game Bleeding Edge, is not a shocking leap for Microsoft's Xbox division. The company as a whole, from CEO Satya Nadella down, has wholeheartedly embraced generative AI in all its initiatives. It was only a matter of time until the Xbox division would have to follow suit. But it does come after a recent game developer survey of 1500 developers in a State of the Industry report statin that 30% of developers have a negative opinion of generative AI. It can be argued back and forth whether generative AI is here to stay or not, the developers making games seem to have an increasingly hostile reactions to generative AI replacing ideation processes. But Microsoft is not alone in pursuing this, either. Capcom recently spoke about using generative AI for the idea phase of game development, arguing that the thousands of small decisions that go into games can be automated so as to phase out the tedium of game development and free up time and bandwidth to focus on the creativity of it. It does stand to reason that a model like Muse, which absorbed hundreds of hours of a game created by people, still needs people to create those ideas for anything to work at all. It also begs the question: should Microsoft even care what developers think right now? As both a platform holder and arguably the largest third-party publisher in the industry, tackling modern design complexity and trying -- like Capcom -- to make it both cheaper and more efficient is likely of paramount importance for them in the coming years. As an example, the latest Fable project from Xbox Games Studio supposedly began work in 2018 and, as of writing, still does not have an announced date. Would generative AI fix that or make it faster? Unknown. But games are taking longer and longer to develop and there is only so long that can be tenable for Xbox head Phil Spencer. For Xbox's part, "preservation" is a key part of their argument for Muse. In the initial statement, Fatima Kardar, corporate vice president of gaming AI at Microsoft, argued that Muse could make older games accessible to modern audiences without much work regardless of hardware advancements. This would, under Kardar's reasoning, make the opportunity cost spent on remasters and backwards compatibility engineering moot. But nothing Xbox showed regarding Muse today bared that out. In coming out publicly with such a big marketing push for Muse, it seems Xbox is trying to serve the two masters of shareholders and futurists. That's all well and good, but it is important to remember that Muse had to train off hundreds of hours of a game created by people in Cambridge that the audience mostly did not connect with. If all of the elements in that equation are not happy, what can the end product be called except a compromise?

New open-source AI for game developers interprets footage and player input

In brief: Despite the controversy surrounding generative AI, some game designers use the technology to brainstorm concepts during development. Microsoft's new open-source model aims to make this practice feasible for interactive content. It focuses on maintaining consistency across many frames while recognizing and integrating player input. Microsoft researchers recently introduced Muse, a generative AI model designed to extrapolate interactive video game scenarios from images, clips, and recorded player input. The tool aims to streamline game development while upholding ethical training practices. The weights, sample data, and interactive interface for Muse, which Microsoft calls a "World and Human Action Model," are now available on Azure AI Foundry. The developers discussed details of the technology on the company's research blog and in a newly published study in Nature. Following training on one million updates, Muse can faithfully predict up to half a minute of gameplay based on one second of actual footage and nine seconds of player input. Unlike prior models, where game details disappear or become distorted as soon as they leave the visible frame, Muse remembers details like terrain, characters, and game mechanics. n a video interview with Xbox head Phil Spencer, Dom Matthews from Ninja Theory explained that the company doesn't plan to use Muse to build content for end users. Instead, the studio plans to use it to help draft and iterate ideas by rapidly generating snapshots of a cohesive vision. It remains unclear whether other developers would follow a similar philosophy, but Microsoft won't mandate the use of its models across its studios. Some developers immediately expressed disinterest. David Goldfarb, who worked on Battlefield, Mass Effect, and Killzone, offered a brief, dismissive response. Marc Burrage from Creative Assembly told Wired that even restricting generative AI to the prototyping stage can harm game development by robbing employees of hands-on experience. Spencer optimistically speculated that technologies like Muse might facilitate game preservation by helping developers port older titles to modern devices. Microsoft has frequently promised to preserve its customers' digital libraries across hardware generations, but it remains to be seen whether regenerating them with AI can produce satisfactory results. Frank Cifaldi from the Video Game History Foundation isn't as optimistic, comparing the idea to a photocopy of a painting. Microsoft also faced harsh criticism regarding Muse's training data. Much of the harsh controversy facing AI models stems from companies using material from across the internet without the original creators' consent. Microsoft sidestepped the issue by training Muse on games it already owns. The initial demonstration used player telemetry data from Ninja Theory's multiplayer shooter, Bleeding Edge. Users who agreed to the game's EULA have already consented to help train Muse. Theoretically, the company could extend the practice to popular titles like Minecraft, Call of Duty, Halo, Overwatch, StarCraft, Doom, or Forza.

Microsoft trained an AI to play (and maybe develop) games

Today, Microsoft announced its first World and Human Action Model, or WHAM, a generative AI capable of generating game visuals and controller actions. Nicknamed "Muse," this AI was built in collaboration with Microsoft Research Game Intelligence and Teachable AI Experiences alongside the game studio Ninja Theory. Muse is designed "to effectively support human creatives," according to Microsoft. The AI was trained on Bleeding Edge, a multiplayer arena battler, but the intended applications go far beyond just creating an AI-controlled character (although that is one possible outcome.) Recommended Videos "What's groundbreaking about Muse is its detailed understanding of the 3D game world, including game physics and how the game reacts to players' controller actions. This allows the model to create consistent and diverse gameplay rendered by AI, demonstrating a major step toward generative AI models that can empower game creators," says Fatima Kardar, Corporate VP for Gaming AI. Please enable Javascript to view this content https://www.microsoft.com/en-us/research/uploads/prod/2025/02/diversity2new.mp4 Credit: Microsoft Research Blog The team has great expectations for this new AI, even though it's still in a relatively early stage of development. Muse could make it possible to bring back older games that are no longer playable on modern hardware by optimizing the game -- or potentially rebuilding it from the ground up. "To imagine that beloved games lost to time and hardware advancement could one day be played on any screen with Xbox is an exciting possibility for us," Kardar said. Taking that a step further, Kardar suggests Muse has the potential to create new content for existing games, and perhaps one day inject it on the fly. As AI models like Muse develop, they could have a massive impact on not only how games are made, but also how quickly. Microsoft is touching on the kind of AI explored in popular culture, all the way from anime like Sword Art Online to popular novels like Ready Player One. And it isn't so far away that you won't be able to experience this for yourself. "We'll create opportunities for people to participate in this exploration, starting with short interactive AI game experiences for you to try on CoPilot Labs very soon." Despite the potential, the use of AI raises concerns. The Muse team also addressed this, promising that it will "continue to be built on our commitment to Responsible AI" and that it would be guided by six principles: "fairness, reliability and safety, privacy and security, inclusiveness, transparency, and accountability."

Xbox announces 'a generative AI model for gameplay ideation' called Muse, but don't get too excited: Machines aren't about to make games for you just yet

"It's still early," but Microsoft has big ambitions for generative AI in game creation. "In nearly every corner of our lives, the buzz about AI is impossible to ignore," Microsoft said in today's announcement of its new generative AI model called Muse, and that is absolutely true. "It's destined to revolutionize how we work, learn, and play," it continued, and well, I'm not so sure about that part. But Microsoft seems determined to roll with it, saying, "At Xbox, we're all about using AI to make things better (and more fun!) for players and game creators." Created in a partnership between Microsoft Research and Xbox studio Ninja Theory, developer of the Hellblade games, Muse is a "world and human action model (WHAM)" trained on Ninja Theory's Bleeding Edge. "What's groundbreaking about Muse is its detailed understanding of the 3D game world, including game physics and how the game reacts to players' controller actions," Microsoft corporate vice president of gaming AI Fatima Kardar wrote. "This allows the model to create consistent and diverse gameplay rendered by AI, demonstrating a major step toward generative AI models that can empower game creators." Microsoft acknowledged that "it's still early," but predicted that Muse could be used for "faster creative ideation," presumably by making it easier for developers to chuck in new stuff for the machine to play to see how it works. Interestingly, though, an even greater focus was put on the system's potential for game preservation. "Today, countless classic games tied to aging hardware are no longer playable by most people," Kardar wrote. "Thanks to this breakthrough, we are exploring the potential for Muse to take older back catalog games from our studios and optimize them for any device. We believe this could radically change how we preserve and experience classic games in the future and make them accessible to more players." Xbox boss Phil Spencer dove more deeply into that aspect of Microsoft's ambitions for Muse in a supplementary video, saying the potential impact on game preservation is "one of the things that I get excited about." "Time will erode the amount of hardware that's out there that can actually play a game," Spencer said. "But you could imagine a world where, from gameplay data and video, that a model could learn old games and really make them portable to any platform where these models could run. "I think that's really exciting. We've talked about game preservation as an activity for us, and these models and their ability to learn completely how a game plays without the necessity of the original engine running on the original hardware I think opens up a ton of opportunity." Amidst all the excitement about what its generative AI systems can do, Spencer and company predictably took pains to explain that its embrace of AI isn't meant to replace human beings, but to -- as mentioned earlier -- "empower" them, saying that "game creators will always be the center of our overall AI efforts." Ninja Theory studio chief Dom Matthews echoed that in his own comments. "It's not about using AI to generate content, but it's actually about creating workflows and approaches that allow our team here of 100 creative experts to do more, to go further, to iterate quicker, to ideate quicker, to bring the ideas in their heads to life in a tangible form for others to see," Matthews said. "Although this is technology that we haven't used in the creation of our games and we don't intend to use this technology for the creation of content, I think the interesting aspect for us that is exciting is how can we use technology like this to make the process of making games quicker and easier for our talented team so that they can really focus on the thing that I think is really special about games, which is that human creativity." For a closer look at what Muse actually is, you're probably better off diving into the full report published on Nature. It's dry but detailed, and also more clear that this is all preliminary work -- we're not yet at a point where we can tell a computer to make Arx Fatalis 2 and then come back in an hour and it's done. From the conclusion: As we navigate the unfolding role of generative AI in the creative industries, there are ways to direct its development to ensure human agency over the creative process. We have presented a user study with diverse game creatives through which we identified three model capabilities that should be given priority when developing AI systems that aim to support creative ideation through iterative practice and divergent thinking: consistency, diversity and persistency. We have also shown that it is possible to develop generative AI models that exhibit these capabilities when trained on appropriate datasets. Microsoft CEO Satya Nadella was a little more forward-looking on the whole thing, however: Dr. Michael Cook, an AI researcher, game designer, and senior lecturer at King's College London, threw some cold water on the Muse hype in a blog post, writing that the research paper "is not really about 'generating gameplay' or 'ideas'," but rather "a study of how human designers think about working with generative AI tools." Cook also took issue with the idea that Muse could serve as a preservation tool: While AI models could have uses for reconstruction (with some limitations), there's more to game preservation than simply mimicking what's on the screen at any given moment. Cook clarified, however, that he's not against using AI as a design tool -- in fact, he thinks it's cool. "I think systems like this can inspire people to make interesting stuff, I think they can make alien games that humans would not, I think they can be fun art pieces and interesting engineering challenges," he wrote. "I think when we feel we have ownership over technology, that we can build things ourselves on small scales where we feel in control of what it does, and in charge of its scope and potential harms, then we feel a lot happier opening ourselves up to ideas like 'what if I used algorithms to generate game levels for me'." I do have to wonder who this Microsoft report is for. It's clearly high-level research, and it seems equally clear to me that we're a long way from seeing this sort of thing deployed for any sort of practical use. I have no doubt that the continued growth of generative AI will have a major impact on the videogame industry, but that's always been the way of it: Technology advances, tools evolve, and games get -- well, not necessarily better, but certainly bigger and prettier. Game development also becomes more accessible, which is a big deal in its own right. I can't shake the feeling that Microsoft is trumpeting its AI advances because everybody is trumpeting their AI advances these days, and there's literally billions of dollars at stake. But there are real advancements underpinning the hype, and Microsoft said we'll be seeing more "about how we are using AI to empower game creators and provide the best possible player experience across all devices" in future announcements coming this year. "New immersive AI gaming experiences" powered by Muse are also "coming soon to Copilot Labs."

Microsoft's Muse AI can design video game worlds after watching you play

Join our daily and weekly newsletters for the latest updates and exclusive content on industry-leading AI coverage. Learn More Microsoft researchers have achieved what many in artificial intelligence considered a distant goal: teaching AI to understand and interact with three-dimensional spaces the way humans do. The breakthrough comes in the form of Muse, an AI model that can comprehend and generate complex gameplay sequences while maintaining consistent physics and character behaviors. The model, detailed in a paper published in Nature, learned entirely from observing human gameplay data -- over seven years worth -- from the Xbox game Bleeding Edge. Unlike traditional AI models that work with text or static images, Muse develops what researchers call a "practical understanding" of how objects, characters, and environments interact in three-dimensional space over time. How Microsoft's Muse AI sees, learns, and plays like a human "The model architecture is agnostic to the game; the only requirement is access to an appropriate data set," said Katja Hofmann, Senior Principal Research Manager at Microsoft Research, in an exclusive interview with VentureBeat. "We designed the model to use the most general data format, which we call the 'human interface' of visuals and controller actions." This approach allows Muse to generate consistent gameplay sequences lasting up to two minutes -- a significant technical achievement in maintaining coherent 3D world interactions over extended periods. The system can take just one second of game visuals as input and generate complex scenarios that respect game physics and character behaviors. However, current limitations exist. "Image resolution is fixed to 300×180 pixels," Hofmann told VentureBeat. "There is a trade-off between model size and speed, meaning that our largest and most consistent models are also slowest at inference time." Beyond gaming: How Muse could shape architecture, retail, and manufacturing The development of Muse was shaped by extensive input from game creators. Microsoft researchers interviewed 27 game developers globally, including studios from both developed and developing nations, to ensure the technology would serve real creative needs. Beyond gaming, Microsoft sees broader applications for the technology. Peter Lee, President of Microsoft Research, highlighted in a blog post potential uses in architecture, retail, and manufacturing: "From reconfiguring the kitchen in your home to redesigning a retail space to building a digital twin of a factory floor to test and explore different scenarios. All these things are just now becoming possible with AI." "The main limitation for applications beyond gaming is access to high quality data," Hofmann told VentureBeat. "Gaming is an excellent application area for driving advances, because large amounts of high quality data can typically be collected more easily than in other 3D environments." Preserving gaming history and empowering future creators For the gaming industry specifically, Xbox is exploring how this technology could help preserve classic games. "Thanks to this breakthrough, we are exploring the potential for Muse to take older back catalog games from our studios and optimize them for any device," said Fatima Kardar, Corporate Vice President of Gaming AI at Microsoft, in a blog post. The model achieves three key technical innovations: maintaining coherent physics and game mechanics over extended sequences, generating multiple varied but plausible continuations from the same starting point, and allowing users to modify generated content while maintaining those changes consistently. "I am personally fascinated by Muse's ability to learn a detailed understanding of a complex 3D environment purely from observing human gameplay data," Hofmann said. "Our research demonstrates an exciting step towards novel interactive experiences crafted by creatives that are hyper-personalized to and by their players." Microsoft is releasing the model weights and a demonstrator tool to researchers and creatives under a Microsoft Research License, though this is not yet an enterprise customer offering. This release aims to encourage further research and exploration of the technology's capabilities. The development signals a broader shift in AI capabilities - from understanding static content like text and images to comprehending dynamic 3D environments and human interactions. This could have far-reaching implications for how we design and interact with virtual spaces across industries. As Microsoft moves to productize this research, they emphasize that human creativity remains central. The technology is positioned as an assistive tool rather than a replacement for human game designers, aiming to augment rather than automate the creative process.

Muse, Microsoft's new AI engine, can imagine gameplay of video games that don't exist

With just an image and a dream, Microsoft's Muse imagines up to two minutes of gameplay. This GIF version of a video shared by the researchers is taken from the original gameplay data, and an image of an additional character is edited into the image. The generated gameplay sequence shows how the character is adapted into the generated gameplay sequence. (Image credit: Microsoft Research) If you need more proof that AI and gaming are on a collision course, Microsoft has you covered. On Wednesday, Microsoft unveiled its Muse AI, which can generate chunks of games based on an image or just a few frames totaling as little as one second. That means, with a very minimal prompt, you can create game visuals and even brainstorm potential game inputs that Microsoft hopes will propel an idea from concept to reality. Muse -- which was outlined Wednesday in a research paper published in the journal Nature and jointly developed by Microsoft Research Game Intelligence and Xbox Games Studios' Ninja Theory -- is what Microsoft is calling the first "World and Human Action Model," or WHAM for short. In theory, the generative AI model works like a Copilot or ChatGPT for game creation that Microsoft hopes will accelerate the development of both AAA and indie games. "What's groundbreaking about Muse is its detailed understanding of the 3D game world, including game physics and how the game reacts to players' controller actions," Microsoft says in its statement. "This allows the model to create consistent and diverse gameplay rendered by AI, demonstrating a major step toward generative AI models that can empower game creators." And based on initial demos, it's hard to argue with that synopsis. If demos are to be believed, Muse appears capable of generating a fairly convincing-looking game with very little to go off of. In one demo, Microsoft shows how Muse took a single promotional image for the game Bleeding Edge -- the game that Muse is trained with -- and is able to generate multiple different continuations from that image. Once the demo is created, users are able to actually explore the generated sequence -- using a game controller to move a character, for example -- and then tweak the outputs using text prompts. While Microsoft says it's already using Muse to develop a "real-time playable AI model trained on other first-party games" there's also potential for Muse to help port old games over to modern-day experiences. "We see potential for this work to one day benefit both players and game creators: from allowing us to revive nostalgic games to faster creative ideation," Microsoft says in a statement. Alternatively, Microsoft says it's also exploring how Muse might be able to augment games that already exist. For example, a designer working on new Fortnite experiences might be able to simulate or generate new content and test it out in real time before actually beginning development. While the initial results of Muse are impressive, there's obviously still a lot of room for growth. For one, the resolution of Muse's outputs is still a work in progress. According to Microsoft, its current models generate outputs at a resolution of 300 x 180 pixels which was improved from the 128 x 128 resolution of the initial model. That's good enough for a general idea, of course, but not exactly high res. Currently, the model can generate consistent gameplay for up to two minutes, but with enough power and finesses, there's no reason why Muse couldn't improve to extend beyond that. There's no clear indication of when Muse will be ready for public or private consumption, but Microsoft says that it plans to "create opportunities for people to participate in this exploration, starting with short interactive AI game experiences" in Copilot Labs "very soon." It also says that it plans to share its experiments with AI and gaming in earlier stages to help accelerate new AI products' growth and generate more feedback early on. In any event, it's clear AI is coming for pretty much all aspects of gaming, whether it's helping players beat a boss or improve their skills, or actually making the games that we play. Study abstract: "Generative artificial intelligence (AI) has the potential to transform creative industries through supporting human creative ideation -- the generation of new ideas. However, limitations in model capabilities raise key challenges in integrating these technologies more fully into creative practices. Iterative tweaking and divergent thinking remain key to enabling creativity support using technology, yet these practices are insufficiently supported by state-of-the-art generative AI models. Using game development as a lens, we demonstrate that we can make use of an understanding of user needs to drive the development and evaluation of generative AI models in a way that aligns with these creative practices. Concretely, we introduce a state-of-the-art generative model, the World and Human Action Model (WHAM), and show that it can generate consistent and diverse gameplay sequences and persist user modifications -- three capabilities that we identify as being critical for this alignment. In contrast to previous approaches to creativity support tools that required manually defining or extracting structure for relatively narrow domains, generative AI models can learn relevant structure from available data, opening the potential for a much broader range of applications."

Microsoft Introduces Muse, a GenAI Model for Gameplay Ideation

Muse was trained on human gameplay data from Bleeding Edge, a 4v4 online game by Ninja Theory. Microsoft has introduced Muse, a generative AI model designed for gameplay ideation. The model, built on the World and Human Action Model (WHAM), can generate game visuals, controller actions, or both. The research, published in Nature, was developed by the Microsoft Research Game Intelligence and Teachable AI Experiences (Tai X) teams in collaboration with Xbox Game Studios' Ninja Theory. The research aims to refine AI-generated gameplay for game development and interactive storytelling. Microsoft has open-sourced the model's weights, sample data, and the WHAM Demonstrator, a concept prototype for interacting with WHAM models. These resources are available on Azure AI Foundry. "I'm incredibly proud of our teams and the milestone we have achieved, not only by showing the rich structure of the game world that a model like Muse can learn but also by demonstrating how to develop research insights to support creative uses of generative AI models," said Katja Hofmann, senior principal research manager at Microsoft Research. Muse was trained on human gameplay data from Bleeding Edge, a 4v4 online game by Ninja Theory. The dataset includes visuals and controller actions recorded with user consent. The model has been trained on over 1 billion images and actions, representing more than seven years of continuous gameplay. Gavin Costello, technical director at Ninja Theory, said, "It's been amazing to see the variety of ways Microsoft Research has used the Bleeding Edge environment and data to explore novel techniques in a rapidly moving AI industry." The research was motivated by the release of ChatGPT in 2022. Microsoft scaled the model's training from a V100 GPU cluster to H100s, refining its representation of controller actions and images. Early versions struggled with consistency, but iterative training improved the model's ability to predict accurate game dynamics. Comparing Muse's generated visuals with actual gameplay, researchers assessed key capabilities such as consistency, diversity, and persistency. Consistency measures whether generated sequences adhere to game dynamics. On the other hand, diversity evaluates how gameplay variations evolve from the same prompt. Persistency determines if introduced elements are maintained in subsequent sequences. Cecily Morrison, senior principal research manager at Microsoft, highlighted the importance of involving game creators from the outset. "It was a great opportunity to join forces at this early stage to shape model capabilities to suit the needs of creatives right from the start, rather than try to retrofit an already developed technology." Meanwhile, xAI chief Elon Musk recently announced that the company is launching a game studio to reshape the gaming industry. While announcing xAI's latest model, Grok-3, Musk said, "We're launching an AI gaming studio at xAI. If you're interested in joining us and building AI games, please join xAI."

Microsoft used an Xbox game to train its new generative AI tools

TL;DR: Microsoft announced Muse, a generative AI toolset that was trained using player inputs from the Xbox game Bleeding Edge. Microsoft's new generative AI toolset, Muse, was trained using real-world player inputs from an Xbox game, the company today announced. Remember when Xbox announced its generative AI plans? These strategies had been nebulous for some time, but the mystery has been lifted with a research update that gives a better idea of what Microsoft is up to. At the time, Xbox president Sarah Bond said that gaming AI would be focused on three areas: Xbox's AI tech is called Muse, and it was actually trained using data from Ninja Theory's failed live game Bleeding Edge. Microsoft Research supplied the AI team with key player data. According to Nature, Microsoft used anonymized captures from 500,000 play sesssions. For years, we had collaborated with Xbox Game Studios' Ninja Theory (based in Cambridge, UK, just like our research team) to collect gameplay data from Bleeding Edge, their 2020 Xbox game. Bleeding Edge is a 4-versus-4 game where all games are played online, and matches are recorded if the player agrees to the End User License Agreement (EULA). We worked closely with our colleagues at Ninja Theory and with Microsoft compliance teams to ensure that the data was collected ethically and used responsibly for research purposes. Microsoft also says that it is using its catalog of first-party games to power its AI ambitions, and that Muse is already being used to make a playable AI model: "We are already using Muse to develop a real-time playable AI model trained on other first-party games, and we see potential for this work to one day benefit both players and game creators: from allowing us to revive nostalgic games to faster creative ideation." More details about Muse and how developers are using AI tech will be discussed at GDC 2025.

Microsoft debuts generative AI that can create video game scenes

Microsoft unveiled artificial intelligence tools that can create video game scenes that would normally have to be programmed and animated by a human, a model it built using data collected from Xbox gamers and their controllers. Called Muse, the model is the first of its kind, according to Microsoft. A machine learning research team surveyed game developers to discern how generative AI could be helpful and what they'd need for such tools to be effective, said Katja Hofmann, a senior principal research manager at Microsoft. Then, to train the AI model, Hofmann's team collected seven years' worth of gameplay data from Bleeding Edge, a 2020 multiplayer battle game from Xbox's Ninja Theory studio. "If you haven't seen any of this generated gameplay footage before, you might think these are just clips from a video game," Hofmann said in a demonstration for reporters. The Muse model was announced Wednesday in the journal Nature. Microsoft's Xbox team and other industry players are eager to use AI to help reduce the hundreds of millions of dollars studios typically spend on blockbusters -- costs that have skyrocketed in recent years even as sales growth has slowed. It's controversial because players are suspicious that generative AI could hurt game quality. And studio employees, already hit with a series of layoffs, fret that the technology could further reduce their ranks. Bleeding Edge pits two teams of four against each another and is played entirely online. All games are recorded with players' permission. That let the researchers amass more than a billion game images paired with corresponding controller usage. The result is what Microsoft calls a "World and Human Action Model." Like most generative AI models, it creates something after being prompted -- in this case a sequence of visuals after being shown a game scene created by a human. "The model is able to generate something that is really consistent, that demonstrates a very accurate understanding of the actual game world," Hofmann said. The AI-made elements include interactions between characters and other elements of the game. Outside researchers will be able to build on Microsoft's work with prototype software the company plans to release.

Microsoft Xbox showcases an incredibly impractical AI model for "gameplay ideation"

Microsoft's Muse AI being used to simulate inserting extra props and enemies into an existing scene from the game Bleeding Edge -- these soon become fully interactable as expected based on existing training data. (Image credit: Microsoft, Ninja Theory) Yesterday, Microsoft Xbox debuted Muse, "a generative AI model designed for gameplay ideation," alongside an open-access Nature.com article and corresponding blog post with a YouTube video. Not sure what "gameplay ideation" means? While Microsoft defines it as generating "game visuals, controller actions, or both," its actual functional purposes are pretty limited and certainly don't do anything like skipping an actual game development pipeline. That said, some of the data is still interesting. The training was done at scale on H100 GPUs. It required about 1 million training updates to extend just one second of real gameplay into an additional 9 seconds of responsive, engine-accurate simulated gameplay. Training data was primarily extracted from existing multiplayer gameplay sessions, though. Instead of just running the game on a single PC, Microsoft had to train this model on a cluster of 100 Nvidia H100 GPUs, which is an order of magnitude more expensive and power-consuming while still only producing an output resolution of 300x180 pixels for about nine more seconds of extrapolated gameplay. The most interesting thing the team demonstrated with Muse was duplicating existing props and enemies within the environment and having Muse replicate their functionality. All this hardware cost, electricity toll, and AI training instead of, you know, using development tools to spawn enemies or props instead? While it's interesting how Muse eventually maintained proper object permanence and duplicated the original game's behavior, its final use cases seem downright wasteful compared to the already effective, traditional video game development pipeline. While future developments of Muse may be capable of pulling off more interesting feats, it ultimately ends up on a long list of other projects trying to simulate gameplay entirely within AI. While it's nice to see some degree of engine accuracy and object permanence still present here, it's such a sub-optimal way to develop, test, or play a video game that I genuinely don't understand why anyone would want to use this, even after hours of poring over the relevant material.

Muse is Xbox's generative AI model for gameplay ideation

Join our daily and weekly newsletters for the latest updates and exclusive content on industry-leading AI coverage. Learn More Microsoft Research unveiled Muse, the company's first generative AI model designed for gameplay ideation. Fatima Kardar, corporate vice president of gaming AI at Microsoft, said in a blog post that we hear about how AI is transforming our lives daily. How will it show up in gaming? For Xbox, the principles that drive our approach to AI are to create more value for players and game creators, bring more games to more people around the world, and recognize that the development of a great game will always be grounded in the creator's vision and artistry, Karder said. "We believe generative AI has the potential to enhance this creativity and unlock new possibilities. We're excited to announce a generative AI breakthrough, published today in the journal Nature and announced by Microsoft Research, that demonstrates this potential -- including the opportunity to make older games accessible to future generations of players across new devices and in new ways. In partnership with Xbox Games Studios' Ninja Theory, Microsoft Research developed a world and human action model (WHAM) named Muse: a first of its kind generative AI model trained on Ninja Theory's multiplayer battle arena game Bleeding Edge. Ninja Theory and Microsoft Research are both based in Cambridge, which made collaboration easy and led to the use of Bleeding Edge for this research. What's groundbreaking about Muse is its detailed understanding of the 3D game world, including game physics and how the game reacts to players' controller actions. This allows the model to create consistent and diverse gameplay rendered by AI, demonstrating a major step toward generative AI models that can empower game creators. Microsoft Research initially set out to explore how generative AI models can support new experiences, focusing on developing AI capabilities that could allow game developers to ideate and expand upon their work in new ways. To accomplish that, they interviewed 27 global game creators -- from indies to AAA game studios -- to make sure the research was shaped by the people who would use it. To allow other research teams and game developers to explore the model and experiment with its capabilities, Muse, its model weights, sample data, and interactive interface are now all available on Azure AI Foundry. What Muse Means for Gaming Although it's still early, this model research is pushing the boundaries of what the team thought was possible. "We are already using Muse to develop a real-time playable AI model trained on other first-party games, and we see potential for this work to one day benefit both players and game creators: from allowing us to revive nostalgic games to faster creative ideation," Karder said. Today, countless classic games tied to aging hardware are no longer playable by most people. Thanks to this breakthrough, Microsoft is exploring the potential for Muse to take older back catalog games from studios and optimize them for any device. "We believe this could radically change how we preserve and experience classic games in the future and make them accessible to more players. To imagine that beloved games lost to time and hardware advancement could one day be played on any screen with Xbox is an exciting possibility for us," Karder said. Karder added, "Another opportunity we are exploring is how Muse can help game teams prototype new gameplay experiences during the creative process and introduce new content -- taking games players already love and enabling our developers to inject new experiences for them to enjoy, or even enable you to participate in the creation process. We'll create opportunities for people to participate in this exploration, starting with short interactive AI game experiences for you to try on Copilot Labs very soon. There's still so much for us to discover, and we can't wait to share more." Xbox's approach to Generative AI Xbox has been innovating with AI and machine learning for decades, including in partnership with Microsoft Research -- constantly working to find ways to delight players, and harness new computing capabilities to bring creative visions to life. While AI isn't new to gaming, Microsoft is now seeing an acceleration of generative AI research, both within Microsoft and across the broader technology community. "We believe it's important to shape how these new generative AI breakthroughs can support our industry and game creation community in a collaborative and responsible way," Karder said. For Xbox, game creators will always be the center of overall AI efforts, she said. Microsoft believes there is space for traditional game development and future generative AI technologies that serve as an extension of creative work and offer novel experiences, she said. "As part of this, we have empowered creative leaders here at Xbox to decide on the use of generative AI. There isn't going to be a single solution for every game or project, and the approach will be based on the creative vision and goals of each team," Karder said. What's Ahead? Microsoft is focused on how AI can address the barriers and frictions to playing and developing games. "This means that we'll share our AI product innovation earlier on, providing opportunities for players and creators to experiment with and co-build new AI features and capabilities with us. This allows us to make sure that our AI innovations address real problems and add new value to creating or playing with Xbox, Karder said. To develop tools that are used in ways that benefit everyone, Karder said the innovation will continue to be built on a commitment to Responsible AI and dedication to developing AI solutions guided by six principles: fairness, reliability and safety, privacy and security, inclusiveness, transparency, and accountability. "You can expect more exciting announcements this year about how we are using AI to empower game creators and provide the best possible player experience across all devices. Meanwhile, Katja Hofmann, senior principal research manager and lead of the Microsoft Research Game Intelligence team, said in a blog post that the journal Nature published the company's research on Muse, which is a generative AI model of a video game that can generate game visuals, controller actions or both. The company says it is the first World and Human Action Model (WHAM). Microsoft Research Game Intelligence(opens in new tab) and Teachable AI Experiences teams created Muse in collaboration with Xbox Games Studios' Ninja Theory, the maker of the Hellblade games. To help other researchers explore these models and build on our work, Microsoft is open sourcing the weights and sample data and making the executable available for the WHAM Demonstrator -- a concept prototype that provides a visual interface for interacting with WHAM models and multiple ways of prompting the models. Developers can learn and experiment with the weights, sample data, and WHAM Demonstrator on Azure AI Foundry. In the research, the team focused on exploring the capabilities that models like Muse need to effectively support human creatives. "I'm incredibly proud of our teams and the milestone we have achieved, not only by showing the rich structure of the game world that a model like Muse can learn, as you see in the video demo below, but also, and even more importantly, by demonstrating how to develop research insights to support creative uses of generative AI models," Hofmann said.

Microsoft will train its gaming AI on a "catalogue" of titles soon, Satya Nadella says, "and then start playing them"

Microsoft boss Satya Nadella has said the company plans to train its new gaming AI on a "catalogue of games" soon, so it can then "start playing them". Speaking on the Dwarkesh Patel podcast, Nadella described seeing Microsoft's gaming AI Muse generate gameplay videos that reflected a user's input as a "massive, massive moment of 'wow'". Microsoft lifted the lid on Muse last week, and described the technology as something that would help game ideation and preservation - though not everyone was convinced. In short, Muse has so far been trained based on years of gameplay footage from Ninja Theory's arena shooter Bleeding Edge. After ingesting this gameplay archive, Muse can then generate new gameplay footage and estimate how it might change based on a specific prompt suggested by the user. For example, Muse could generate gameplay footage with an additional jump pad in the level, if commanded to do so. As yet, this is not an AI able to start coding its own games, or dreaming up its own concepts. Rather, it's a tool that could eventually become a shortcut for providing generated gameplay footage to speed up the process of development. "Can you actually generate games that are both consistent and then have the ability to generate the diversity of what that game represents, and then are persistent to user mods?" Nadella said. "That's what this is. "What I'm excited about is bringing - we're going to have a catalogue of games soon that we will start using these models, or we're going to train these models to generate, and then start playing them." Nadella did not elaborate on how users would be able to play the generated footage Muse provides. "When Phil Spencer first showed it to me, he had an Xbox controller and this model basically took the input and generated the output based on the input. And it was consistent with the game," Nadella continued. "That to me is a massive, massive moment of 'wow'. It's kind of like the first time we saw ChatGPT complete sentences, or Dall-E draw, or Sora. This is one such moment." Writing in response to Microsoft's announcement of Muse last week, gaming AI expert Michael Cook described the advancement as "impressive" but "not a practical process". "It's impressive that it can do this using visual information because things like lighting, camera angles, user interface and so on are a lot for an AI model to handle. But ultimately, even with all of this data, all the time spent annotating datasets, and so on, it was still only just about able to generate footage predicting player behaviour. "The research team behind this probably believe it will get more efficient over time, which might make it more affordable or tractable for small developers," Cook continued. "However, it still raises the question of how we get video footage of people playing our game in the first place."

What to Expect from Muse: Microsoft's New AI for Video Games

Microsoft's Muse AI: Revolutionizing Video Game Development Microsoft presents its generative AI model Muse, which brings revolutionary changes to the video game development process. Museums can produce visual aspects for games and forecast how players interact with them, providing an innovative method for producing and conserving games. This development emerged through Microsoft Research teaming up with Ninja Theoryto create tools that enhance production speed and future potential for iconic video games.

Microsoft wants to use generative AI tool to help make video games

Using AI to produce footage of video games with a consistent world and rules could prove useful to game designers An artificial intelligence model from Microsoft can recreate realistic video game footage that the company says could help designers make games, but experts are unconvinced that the tool will be useful for most game developers. Neural networks that can produce coherent and accurate footage from video games are not new. A recent Google-created AI generated a fully playable version of the classic computer game Doom without access to the underlying game engine. The original Doom, however, was released in 1993; more modern games are far more complex, with sophisticated physics and computationally intensive graphics, which have proved trickier for AIs to faithfully recreate. Now, Katja Hofmann at Microsoft Research and her colleagues have developed an AI model called Muse, which can recreate full sequences of the multiplayer online battle game Bleeding Edge. These sequences appear to obey the game's underlying physics and keep players and in-game objects consistent over time, which implies that the model has grasped a deep understanding of the game, says Hofmann. Muse is trained on seven years of human gameplay data, including both controller and video footage, provided by Bleeding Edge's Microsoft-owned developer, Ninja Studios. It works similarly to large language models like ChatGPT; when given an input, in the form of a video game frame and its associated controller actions, it is tasked with predicting the gameplay that might come next. "It's really quite mind-boggling, even to me now, that purely from training models to predict what's going to appear next... it learns a sophisticated, deep understanding of this complex 3D environment," says Hofmann. To understand how people might use an AI tool like Muse, the team also surveyed game developers to learn what features they would find useful. As a result, the researchers added the capability to iteratively adjust to changes made on the fly, such as a player's character changing or new objects entering a scene. This could be useful for coming up with new ideas and trying out what-if scenarios for developers, says Hofmann. But Muse is still limited to generating sequences within the bounds of the original Bleeding Edge game -- it can't come up with new concepts or designs. And it is unclear if this is an inherent limitation of the model, or something that could be overcome with more training data from other games, says Mike Cook at King's College London. "This is a long, long way away from the idea that AI systems can design games on their own." While the ability to generate consistent gameplay sequences is impressive, developers might prefer to have greater control, says Cook. "If you build a tool that is actually testing your game, running the game code itself, you don't need to worry about persistency or consistency, because it's running the actual game. So these are solving problems that generative AI has itself introduced." It's promising that the model is designed with developers in mind, says Georgios Yannakakis at the Institute of Digital Games at the University of Malta, but it might not be feasible for most developers who don't have so much training data. "It comes down to the question of is it worth doing?" says Yannakakis. "Microsoft spent seven years collecting data and training these models to demonstrate that you can actually do it. But would an actual game studio afford [to do] this?" Even Microsoft itself is equivocal over whether AI-designed games could be on the horizon: when asked if developers in its Xbox gaming division might use the tool, the company declined to comment. While Hofmann and her team are hopeful that future versions of Muse will be able to generalise beyond their training data - coming up with new scenarios and levels for games on which they are trained, as well as working for different games - this will be a significant challenge, says Cook, because modern games are so complex. "One of the ways a game distinguishes itself is by changing systems and introducing new conceptual level ideas. That makes it very hard for machine learning systems to get outside of their training data and innovate and invent beyond what they've seen," he says.

Microsoft's Latest Muse AI Model Can Generate Game Environments

Muse was trained on human gameplay data from the Xbox game Bleeding Edge Microsoft researchers introduced a new artificial intelligence (AI) model on Wednesday that can generate 3D gameplay environments. Dubbed the World and Human Action Model (WHAM) or Muse, the new AI model was developed by the tech giant's Research Game Intelligence and Teachable AI Experiences (Tai X) teams in collaboration with Xbox Games Studios' Ninja Theory. The company said that the large language model (LLM) can help game designers in the ideation process, as well as help generate game visuals and controller actions to help creatives in game development. In a blog post, the Redmond-based tech giant detailed the Muse AI model. This is a research product currently, although the company said that it is open-sourcing the weights and sample data of the model for the WHAM Demonstrator (a concept prototype of a visual interface to interact with the AI model). Developers can try out the model on Azure AI Foundry. A paper detailing the technical aspects of the model is published in the Nature journal. To train a model on such a complex area is a difficult proposition. Microsoft researchers collected a large amount of human gameplay data from the 2020 game Bleeding Edge, a game published by Ninja Theory. The LLM was trained on a billion image action pairs, which is equivalent to seven years of human gameplay. The data is said to be collected ethically and is used only for research purposes. The researchers said that scaling up the model training was a major challenge. Initially, Muse was trained on a cluster of Nvidia V100 GPUs, but then it was scaled to multiple Nvidia H100 GPUs. Coming to the functionality, the Muse AI model accepts text prompts as well as visual inputs. Additionally, once a game environment is generated, it can be further enhanced using controller actions. The AI responds to the movements made by the user to render new environments aligned with the initial prompt, and consistent with the rest of the gameplay. Due to being a unique AI model, typical benchmark tests cannot properly evaluate its capabilities. The researchers highlighted that they have internally tested the LLM on metrics such as consistency, diversity, and persistence. Since it is a research-focused model, the outputs have been limited to just 300x180p resolution.

Microsoft thinks Xbox's new generative AI could save classic games, future-proof game libraries

TL;DR: Microsoft believes its new generative AI tech could one day be used to future-proof games libraries and access classic tough-to-find games. Microsoft hopes to use its new AI technology to preserve older games, potentially future-proofing backwards compatibility. Microsoft is all-in on artificial intelligence, and Xbox has briefly discussed its AI plans. First, Xbox signed a multi-year deal with Inworld to use its generative AI for narrative/quest design in games. Now, Microsoft has announced Muse, its own tech that is able to generate gameplay sequences. Muse was trained using real-world inputs from players in Ninja Theory's Bleeding Edge game. The implications for Muse are unclear, and Microsoft tells developers they don't have to use generative AI for their games (Ninja Theory has already opted out, for instance). The announcement page, blog post, and accompanying YouTube video with Phil Spencer only go so far. What can we expect Muse to be used for? The written posts have some interesting theoretical applications, especially given Microsoft's business model. It's possible that Microsoft could use Muse to basically preserve its old games; train the AI with the game and it might be able to just...recreate the game itself. This is probably a long ways down the line, but it's nonetheless compelling. We do have to wonder if this AI technology will impact jobs in an industry that is already being slammed with layoffs. Here's what Microsoft says about Muse as it pertains to games preservation:

Microsoft trained an AI model on a game no one played

World models -- AI algorithms capable of generating simulated environments -- represent one forefront of machine learning. Today, Microsoft published new research in the journal Nature detailing Muse, a model capable of generating game visuals and controller inputs. Unexpectedly, it was born out of a training set Microsoft built from Bleeding Edge. If, like me, you had completely erased that game from your memory (or never knew it existed in the first place), Bleeding Edge is a 4 vs. 4 brawler developed by Ninja Theory, the studio better known for its work on the Hellblade series. Ninja Theory stopped updating Bleeding Edge less than a year after release, but Microsoft included a clause in the game's EULA that gave it permission to record games people played online. So if you were one of the few people who played Bleeding Edge, congratulations, I guess: you helped the company make something out of a commercial flop. So what's Muse good for anyway? Say a game designer at Blizzard wants to test an idea for a new hero in Overwatch 2. Rather than recruiting a team of programmers and artists to create code and assets that the studio may eventually scrap, they could instead use Muse to do the prototyping. Iteration is often the most time-consuming (and expensive) part of making a video game, so it's easy to see why Microsoft would be interested in using AI to augment the process; it offers a way for the company to control runaway development costs. That's because, according to Microsoft, Muse excels at a capability of world models the company calls persistency. "Persistency refers to a model's ability to incorporate (or 'persist') user modifications into generated gameplay sequences, such as a character that is copy-pasted into a game visual," says Katya Hofmann, senior principal research manager at Microsoft Research. Put another way, Muse can quickly adapt to new gameplay elements as they're introduced in real-time. In one of the examples Microsoft shared, you can see the "player" character immediately react as two power-ups are introduced next to them. The model seemingly knows that the pickups are valuable and something players would go out of their way to obtain. So the simulation reflects that, in the process creating a convincing facsimile of a real Bleeding Edge match. According to Fatima Kardar, corporate vice president of gaming AI at Microsoft, the company is already using Muse to create a "real-time playable AI model trained on other first-party games," and exploring how the technology might help it bring old games stuck on aging hardware to new audiences. Microsoft says Muse is a "first-of-its-kind" generative AI model, but that's not quite right. World models aren't new; in fact, Muse isn't even the first one trained on a Microsoft game. In October, the company Decart debuted Oasis, which is capable of generating Minecraft levels. What Muse does show is how quickly these models are evolving. That said, there's a long way for this technology to go, and Muse has some clear limitations. For one, the model generates visuals at a resolution of 300 x 180 pixels and about 10 frames per second. For now, the company is releasing Muse's weights and sample data, and a way for researchers to see what the system is capable of.

Microsoft Debuts Generative AI That Can Create Video-Game Scenes

Microsoft Corp. unveiled artificial intelligence tools that can create video-game scenes that would normally have to be programmed and animated by a human, a model it built using data collected from Xbox gamers and their controllers. Called Muse, the model is the first of its kind, according to Microsoft. A machine learning research team surveyed game developers to discern how generative AI could be helpful and what they'd need for such tools to be effective, said Katja Hofmann, a senior principal research manager at Microsoft. Then, to train the AI model, Hofmann's team collected seven years' worth of gameplay data from Bleeding Edge, a 2020 multiplayer battle game from Xbox's Ninja Theory studio.

Satya Nadella on the use of AI in Xbox games: "Very soon they will start using these models"

It seems that lately all the talk inside and outside the tech world has to do with AI and its good or bad use, and what everyone interprets that to mean. There is a lot of discussion about whether the use of AI should be restricted to optimising mechanical processes that contribute nothing beyond wasting time, or whether so-called generative AI should be used to create new entertainment experiences, such as movies, books, TV series or video games. Elon Musk has already said he is launching a game development studio on this premise, and now Microsoft CEO Satya Nadella says Xbox will follow suit. In an interview with Dwarkesh Patel's YouTube channel, Nadella claims that Xbox "will soon have a catalogue of games that will start using these [AI/Muse] models". "(...)we're going to train these models to generate, and then we'll start playing with them." "In fact, when Phil Spencer first showed it to me, he had an Xbox controller and this model basically took the input and generated the output based on the input. And it was consistent with the game. To me, that's an incredible moment. It's like the first time we saw ChatGPT complete sentences, or Dall-E draw, or Sora. This is one of those moments." What do you think - do you find it interesting that Xbox is interested in creating new games based on generative AI, or do you think this will lose some of the essence of the games?

Microsoft's Plans to Make Games With AI Don't Sound Great

Microsoft CEO Satya Nadella has recently touted how great the company's recently introduced technology called Muse is. While this model is a big advancement, it's nowhere near as good as the company makes it seem. While speaking on the Dwarkesh Podcast, Nadella claimed that Microsoft would eventually use this technology to make games. Something like a prompt system equivalent to ChatGPT or Dolly. "It's just very cool... one thing that we wanted to go after was, using gameplay data, can you actually generate games that are both consistent and then have the ability to generate the diversity of what that game represents," Nadella said on the podcast. The Muse page announcing the technology shows what the model is capable of doing. Microsoft likely took the best outputs from Muse to show off, and it's not much. The output is 180p (that is not a typo), it is choppy, it stutters, and it looks like it has a blurred filter on. It's nowhere near close enough to make a game that players would want to see, let alone play. Then there is the gameplay itself, which works more like Minecraft AI than ChatGPT. The game seems to randomly add things into the game, even in a moment, just because the AI thinks that's what belongs. It doesn't feel like the AI is adapting to what should be there; it feels like it is guessing what the gameplay should have. Muse AI is nowhere near the point where it can make games. The closest that Microsoft is likely going to get within a few years is the ability to remaster a game. That is only if the company can somehow get Muse to understand every element of a given game, which seems improbable with current technology. From what we've been shown, it feels like the CEO is just saying what he needs to for stockholders to believe in Xbox's work on Muse. It feels more like a gimmick that isn't usable in any real sense than a tool that will help gaming move forward -- let alone make games. Companies don't expect AI use to be an issue at all, and maybe it won't be in a smaller sense, but it definitely will be in a grander one. Plenty of games likely have AI tools used to help make them. Procedural generation may be way further than we expect and arguably could be considered AI, but that is a strict tool. If it were possible to generate an entire game using AI, and it clearly is not, it likely would do a lot of damage to the publisher. There has yet to be a change in the idea that AI replacing creativity is bad, so it would come with a lot of backlash and boycotting. It's hard enough to generate a profit on a game with a lot of good publicity, let alone one that would be attacked from every direction because AI made it. Nadella really tried to sell Muse AI in the podcast, saying that seeing it in action was like "a massive, massive moment of, 'wow.' It's kind of like the first time we saw ChatGPT complete sentences or Dolly draw or Sora, this is kind of one such moment." So, it shows he wants the technology to get somewhere close. Even still, his idea of Microsoft's studios making "a catalog of games soon that we will start using these models, or we're going to train these models to generate, and then start playing them," is not feasible for what Musa AI is. If anything, it feels like it's only been half-baked and was shown to the public too soon. 180p should not feel good enough to show off when most AI video demonstrations have much better quality than that. Source: Dwarkesh Podcast

Xbox first-party developers have a choice whether or not to use generative AI, Microsoft says

TL;DR: Microsoft emphasizes that AI will not replace traditional development and empowers creative leaders to decide its use. Ninja Theory opts not to use Muse, focusing on enhancing human creativity. Following heavy layoffs in the games industry, including its own video games division, Microsoft today made a bunch of announcements regarding its generative AI technology that will be used for games development. Today Microsoft revealed Muse, its new generative AI tech, and talked more about its plans for AI in games. The company is going full steam ahead on a "real-time playable AI model trained on other first-party games," and its experimental Muse technology is able to create generated instances of gameplay at low-level resolution. Using AI poses a potential ethical dilemma in the creative space, especially with so many fellow developers having been laid off. In a blog post, Microsoft says that first-party devs won't be made to use AI tech. "For Xbox, game creators will always be the center of our overall AI efforts. We believe there is space for traditional game development and future generative AI technologies that serve as an extension of creative work and offer novel experiences. "As part of this, we have empowered creative leaders here at Xbox to decide on the use of generative AI. There isn't going to be a single solution for every game or project, and the approach will be based on the creative vision and goals of each team." One team has already pledged not to use Muse. Ninja Theory Studio Head Dom Matthews says that his studio won't be using the tech for its games: So, although this is technology that we haven't used in the creation of our games, and we don't intend to use this technology for the creation of content, I think the interesting aspect for us that is exciting is how can we use technology like this to make the process of making games quicker and easier for our talented team so that they can really focus on the thing that I think is really special about games, which is that human creativity. Games to me are really creators using the medium of games to talk to their audience, to communicate through the medium of games. And I think that's a really important aspect of interactivity, and that is core to game creation for me. So what's important with this is that we have the human creativity, but through this technology we can empower them to do more and push their own dreams further than they have been before.

Xbox has officially opened the generative AI floodgates, and Phil Spencer wants to use it for game preservation and save titles that "were maybe tied to unique pieces of hardware"

Xbox has unveiled its generative AI project Muse, a "generative AI model of a video game that can generate game visuals, controller actions, or both." The publisher is not being specific about this model's exact uses, but has outlined some potential applications in its announcement. One of those proposed uses is game preservation. "Today, countless classic games tied to aging hardware are no longer playable by most people," Xbox says in its announcement post. "Thanks to this breakthrough, we are exploring the potential for Muse to take older back catalog games from our studios and optimize them for any device. We believe this could radically change how we preserve and experience classic games in the future and make them accessible to more players." "One of the things we care a lot about at Xbox is game preservation," Phil Spencer says in a video accompanying the announcement. "And I think about an opportunity to have models learn about older games, games that were maybe tied to unique pieces of hardware where that engine on that hardware... Time will erode the amount of hardware that's out there that can actually play a game." Spencer suggests that "gameplay data and video" could help build an AI model which "could learn old games and really make them portable to any platform where these models could run. I think that's really exciting. We've talked about game preservation as an activity for us, and these models and their ability to learn completely how a game plays, without the necessity of the original engine running on the original hardware, I think opens up a ton of opportunity." The current model Muse is built on the 2020 multiplayer game Bleeding Edge, in part due to developer Ninja Theory's close proximity to Microsoft Research in Cambridge. A few tiny bits of footage in the blog linked above suggest that Muse can at least create a reasonable facsimile of the game, but whether it can genuinely preserve any game remains to be seen, and gaming historians are more than a little skeptical. "I actually do think an AI model could observe and replicate game logic," Video Game History Foundation founder Frank Cifaldi says on Bluesky, "and that might even be a useful tool for development! But this is like saying a photocopy of a painting is 'preservation.' It's misleading and an insult to the thankless work we archivists do. I'm offended and sickened by this." VGHF library director Phil Salvador puts it even more simply: "Generative AI video is a great way to preserve video games, in the sense that mirages are a great source of water." Whatever form Muse and the AI models that follow end up taking remains to be seen, and this technology is admittedly still in its infancy - maybe someday it really will be able to watch a bunch of Super Mario Bros. footage and pop out a perfect replica of a classic. But that's a dream about the what the future of technology might be able to do, as is the case with much of generative AI.

Xbox Pushes Ahead With New Generative AI. Developers Say 'Nobody Will Want This'

One developer responds to the release of Muse: "We are implicitly empowering a class of people who own these tools and don't give a fuck about how they reshape our lives." Microsoft is wading deeper into generative artificial intelligence for gaming with Muse, a new AI model announced today. The model, which was trained on Ninja Theory's multiplayer game Bleeding Edge, can help Xbox game developers build parts of games, Microsoft says. Muse can understand the physics and 3D environment inside a game and generate visuals and reactions to players' movements. Among the various use cases for Muse that Microsoft outlines in its announcement, perhaps the most intriguing involves game preservation. The company says Muse AI can study games from its vast back catalog of classic titles and optimize them for modern hardware. Fatima Kardar, Microsoft's corporate vice president for Gaming AI wrote in the company's press release: "To imagine that beloved games lost to time and hardware advancement could one day be played on any screen with Xbox is an exciting possibility for us." The company says it will continue to explore generative AI, including how to help game teams prototype their projects. In its announcement, Microsoft says the Xbox team interviewed 27 game creators globally "to make sure the research was shaped by the people who would use it." The response from developers and the larger community online, however, has been swift, with Muse being poorly received. As longtime game developer and founder of the development studio The Outsiders David Goldfarb said in response to the news: "Fuck this shit." While executives continue to grow more interested in generative AI, the technology is becoming less popular with the people who actually make games. In a direct message, Goldfarb says he doesn't believe generative AI is good for video games, "because the people who are promoting it are doing it to reduce capital expenditure and whether they intend to do it or not, are effectively disenfranchising and devaluing millions of collective years of aesthetic effort by game devs and artists." "The primary issue is that we are losing craft," Goldfarb says. "When we rely on this stuff we are implicitly empowering a class of people who own these tools and don't give a fuck about how they reshape our lives." A WIRED investigation found that AI is pushing human workers out of the work of creating video games at the same time the games industry is undergoing massive constriction. Thousands of developers have been laid off over the past few years, and that trend is continuing in 2025. While some developers believe AI cannot replace creativity in games, others are still concerned about their job security in an industry that is spinning up new tools that obviate the need for their skills. "It's the classic issue of Xbox bleeding talent but also so heavily invested in GenAI that they can't see the forest for the trees," said a AAA developer who asked to remain anonymous because they are not allowed to talk publicly about Muse. "They don't see that nobody will want this. They don't CARE that nobody will want this ... internal discussions about these sorts of things are quiet because EVERYONE fears being against this and losing their jobs due to the tumultuous time in our industry."

'A massive, massive moment of wow.' Microsoft CEO predicts AI-generated games are a 'CGI moment' for the industry

Microsoft's new AI-powered game generation platform Muse is akin to the impact that CGI had on the film industry. So, says CEO Satya Nadella in a wide ranging interview covering everything from AI and quantum computing, to the future of economic growth, while slightly ignoring the fact that Microsoft's AI-generated games are only running at a resolution of 300 by 180 pixels. Hold that thought. Aside from such trivialities as quantum computing and economic growth, what does one of the world's most powerful CEOs think about the impact of AI on PC gaming? "That to me is a massive, massive moment of wow. It's like the first time we saw ChatGPT complete sentences," says Nadella of Muse in the interview. What's Muse? Why it's the the first World and Human Action Model (WHAM), a generative AI model of a video game that can generate game visuals, controller actions, or both. You can read Microsoft's own explanation of what Muse is here. But it seems to boil down to that borderline dystopian notion of sauntering up to your keyboard, spooling up your AI buddy and asking, "hey, I wanna play a first-person shooter with aliens set in ancient Rome, me as India Jones, and a comedy narrative inspired by Monty Python," hitting enter and having it actually happen there and then. Actually, the main impediment to a scenario like that at this stage might well be IP and copyright, though when has that ever gotten in the way of AI progress... but I digress. Although Muse is just a research platform for now, Nadella thinks its impact on real-world gaming is just around the corner. "What I'm excited about is bringing a catalogue of games soon that we are going to train these models to generate and then start playing them," he says, adding, "it's kinda like the CGI moment even for gaming long term." He also emphasised just how special a status gaming has at Microsoft. "We didn't invest in gaming to build models. Here's the interesting thing, we built our first game before we built Windows (Microsoft Flight Simulator was 1982), Flight Simulator was a product long before we even built Windows. Gaming has got a long history at the company and we want to be in gaming for gaming's sake," Nadella explained. He also explained quite a few other things, such as how he will measure the success and impact of AI not on some floaty notion of AI becoming self aware or achieving AGI or Aritificail General Intelligence, but on the impact it has on economics and therefore people's lives, in short solving what he sees as the world's current "growth challenge". "We get a little ahead of ourselves with this AGI hype. The first thing we have to observe is GDP growth," he says. "The developed world is what, 2% growth, and if you adjust for inflation it's zero. "Let's have that industrial revolution type of growth, that means to me 10%, 7% growth, the developed world inflation adjusted growing at 5%, that's the real marker. It can't just be supply side." It's also interesting to note that Nadella doesn't seem to have monopolistic instincts when he talks about Microsoft's business. For instance, he doesn't see AI generally as a winner-takes-all industry, where a single player will come to dominate, perhaps as Google has in search. Instead, he says he likes to enter categories that are so big, there's less risk of winner-takes-all happening, the kind of "big market that can accommodate a couple of winners, and you're one of them." Anyway, exactly when we might expect to be able to, well muse on what imagined game we might like to play next to an AI agent, only for that exact game to fire up is unclear. While it seems like that or something close to it is on its way, as we pointed out earlier in the week, some observers think Muse is being over hyped. Dr. Michael Cook, an AI researcher, game designer, and senior lecturer at King's College London, for instance, says that Muse, "is not really about 'generating gameplay' or 'ideas'," but rather, "a study of how human designers think about working with generative AI tools." Moreover, Muse games are currently being generated at a resolution of just 300 by 180 pixels. So, it's not exactly threatening that hand-coded 4K experience, yet. And yet, the very idea that an AI might dream up an entire game is a rather astounding prospect. Personally, I have no idea what to think about it all. It's kind of thrilling, but it's also thoroughly unsettling. Let us know what you think below.

Microsoft wants to preserve your games for the future using AI

Allowing artificial intelligence to interfere with our games is usually a hot topic. Some fear that it will come at the expense of imagination and creativity because AI largely bases its work on what already exists. In addition, developers, voice actors and others are not super keen on losing their jobs. Microsoft has now developed a tool they call Muse AI, which can be used for game development, and on Xbox Wire we can read: "Another opportunity we are exploring is how Muse can help game teams prototype new gameplay experiences during the creative process and introduce new content-taking games players already love and enabling our developers to inject new experiences for them to enjoy, or even enable you to participate in the creation process." However, Muse AI has more benefits than that, and looks set to be a powerful solution that will allow older titles to be played in the future: "Today, countless classic games tied to aging hardware are no longer playable by most people. Thanks to this breakthrough, we are exploring the potential for Muse to take older back catalog games from our studios and optimize them for any device. We believe this could radically change how we preserve and experience classic games in the future and make them accessible to more players. To imagine that beloved games lost to time and hardware advancement could one day be played on any screen with Xbox is an exciting possibility for us." And that sounds like a way of using AI in game development that quite a few of us can appreciate anyway, right? Check out the video below for more information.

Microsoft develops AI model for videogames

The model, called Muse, was made in collaboration with Ninja Theory, a developer under Xbox Game Studios - Microsoft's gaming division.Microsoft has developed an artificial intelligence model of a videogame which can help generate visuals and actions, the company announced on Wednesday. The model, called Muse, was made in collaboration with Ninja Theory, a developer under Xbox Game Studios - Microsoft's gaming division. The push towards AI comes as videogame development costs continue to rise. Spending on new games has also been weak as consumers stick to proven titles due to economic uncertainty. "We are already using Muse to develop a real-time playable AI model trained on other first-party games, and we see potential for this work to one day benefit both players and game creators," said Fatima Kardar, corporate vice president for gaming AI.

Gaming Data is to Microsoft What YouTube is to Google: Nadella

"It could serve as both a general action model and a world model." Microsoft recently launched Muse, a generative AI model for gameplay ideation. Built on the World and Human Action Model (WHAM), the model can generate game visuals, controller actions, or both. In a recent interview with Dwarkesh Patel, Microsoft chief Satya Nadella expressed his excitement about the future of gaming. He revealed that Microsoft will soon have a catalogue of games wherein AI models will play a key role. He explained that these models will either be trained to generate game content or will be directly integrated into gameplay. "We're going to have a catalogue of games soon that we will start using these models for, or we're going to train these models to generate and then start playing them," he said. When Xbox chief Phil Spencer first demoed Muse for him, Nadella saw the model take inputs from an Xbox controller and generate outputs that perfectly matched the game. "It was a massive moment of 'wow'. It's kind of like the first time we saw ChatGPT complete sentences, or DALL-E draw, or Sora (sic)." Nadella shared an interesting fact about Microsoft's history, mentioning that the company developed a game before even creating Windows. "Flight Simulator was a Microsoft product long before we even built Windows. Gaming has a long history at the company, and we want to be in gaming for gaming's sake," he said. Nadella described gaming data as more than just a resource for the gaming industry, suggesting it could serve as both a general action model and a world model. "It's fantastic," he said, drawing a comparison: "I think about gaming data as perhaps, you know, what YouTube is to Google, gaming data is to Microsoft." Google uses YouTube data to train its models, such as its video generation model Veo 2. On the other hand, Muse was trained on human gameplay data from Bleeding Edge, a 4v4 online game by Ninja Theory. The dataset includes visuals and controller actions recorded with user consent. The model has been trained on over 1 billion images and actions, representing over seven years of continuous gameplay. On October 13, 2023, Microsoft acquired Activision Blizzard for $75.4 billion. The acquisition included franchises such as Call of Duty, Warcraft, Diablo, Overwatch, and Candy Crush. In the recent quarter, Xbox content and services reported a 2% increase in revenue. Speaking about Microsoft's substantial investment in gaming, Nadella said cloud gaming is a natural area to invest in as it expands the TAM and allows people to play games everywhere. He added that the combination of AI and gaming could be the 'CGI moment' for the gaming industry. Reportedly, Microsoft is not the only one betting on AI gaming. Elon Musk's xAI has similar plans. While introducing xAI's latest model, Grok-3, Musk said, "We're launching an AI gaming studio at xAI. If you're interested in joining us and building AI games, please join xAI." Since the launch of Grok-3, many developers have used it to create games. "Grok-3 is an incredible AI coding assistant. After a few hours and over 1,000 lines of generated code, I now have a fully functional 2D vertical jumping game," said Alvaro Cintas-Canto, assistant professor of cybersecurity at Marymount University. He added that the game features different heroes, monsters, platforms, difficulty levels, and lives. "Grok 3 is so good at programming that it makes game creation feel more like an art project. I generated this themed, endless-runner arcade game in < 30 mins," said Mickey Friedman, co-founder of Flair.ai. Besides xAI, Google DeepMind last year introduced Genie 2, a large-scale foundation world model capable of generating diverse playable 3D environments. It enables the development of embodied AI agents by transforming a single image into interactive virtual worlds, which can be explored by humans or AI using standard keyboard and mouse controls. "The world model is taking shape," said Google DeepMind chief Demis Hassabis while announcing it. Building on this trend, Decart AI launched Oasis, the world's first real-time, generative AI-based playable world model. This fully interactive game generates each frame through a Transformer model that responds instantly to keyboard and mouse inputs, simulating physics, game mechanics, and graphics. Meanwhile, Netflix recently appointed Mike Verdu as the VP of GenAI for games. "I am working on driving a once-in-a-generation inflexion point for game development and player experiences using generative AI. This transformational technology will accelerate the velocity of development and unlock truly novel game experiences that will surprise, delight, and inspire players," said Verdu in a LinkedIn post.

Microsoft is replacing human gamers (and even games) with AI

AI is being used to predict how human gamers will react, in order to fine-tune how video games play and feel. In the future, Microsoft suggests, you may be playing AI. No, not on the battlefield, but on games that actually use AI to simulate the entire game itself. As a first step, Microsoft has developed an AI model, called WHAM, that "beta tests" games early in the development cycle using AI instead of human players. Gamers know that realistic AI can turn a good game into something great, like how the older F.E.A.R. games would realistically model how soldiers might react to a hostile, armed player. Microsoft's World and Human Action Model (WHAM) takes the opposite approach -- it tries to figure out how human players will react in a given situation, right down to a specific frame or setup within the existing game world. Microsoft calls this WHAM by the name "Muse." The point of Muse's WHAM, Microsoft said, wasn't to improve the way NPCs or in-game monsters necessarily reacted to players. Instead, WHAM was developed to make a game "feel right" -- not too hard, not too easy, with interactions that felt realistic. That's something that normally takes hours upon hours of beta testing and evaluating how gamers interact with the environment. WHAM was designed to help automate that, the company said. Microsoft said Wednesday that it has released the WHAM model to huggingface.com, alongside a "WHAM Demonstrator" to essentially place the AI player in a specific spawn location, and then test and evaluate what would happen if the AI made different decisions. Microsoft also published a paper describing WHAM to the Nature scientific journal, which was made available to PCWorld before publication. To develop the model, Microsoft used about 500,000 anonymized gaming sessions (over all seven of the game's maps) from Ninja Theory's Bleeding Edge, a 4×4 multiplayer combat game that Ninja Theory released in 2020 but halted development on less than a year later. Each frame of the session was reduced to 300×180 resolution, then encoded into 540 AI tokens. Likewise, each motion on the Xbox controller, including the buttons, was reduced to 16 different inputs based on the stick direction and button input. Microsoft said that the GIF below was generated by the Muse WHAM. Microsoft encoded all of this gameplay into a 1.6-billion parameter model, condensing essentially seven entire years of gameplay into a single transformer. The company also developed smaller models based upon a single map, Skygarden, with 128×128 images used instead, with parameters ranging from 15 million to 894 million. (In AI, a larger number of parameters usually generates more realistic outcomes, at the cost of additional computing resources.) Microsoft then built a concept prototype, known as the "WHAM Demonstrator" -- sort of the AI chatbot based upon the WHAM model. In this case, the user was able to "place" the AI player upon a map, in relation to various objects around it. When enabled, the WHAM Demonstrator then sketched out how the "human" player was likely to respond. In this case, the developer could run and then re-run the Demonstrator to see various outcomes, then select an outcome to continue to see how the AI "human" would respond. From its training, Demonstrator understood the gameplay rules and physics, though it took more training iterations to understand that some players could achieve flight, depending upon game conditions. The idea is that the WHAM Demonstrator could be used to run different scenarios from the same starting point. In the Nature paper, Microsoft showed how WHAM, beginning with the same eight frames, could produce 16 widely divergent endpoints, based on the AI decisions that WHAM made. Even more interestingly, WHAM was developed so that users could add additional enemies or objects, and the AI would react accordingly. Draw a line through WHAM/Muse into the future, and you arrive at a "game" which is generated more and more in real time using AI. According to Microsoft's vice president of gaming AI, Fatima Kardar, that's where Microsoft hopes to go -- apparently following Google, which has already demonstrated consistent game worlds from a prompt. "Today, countless classic games tied to aging hardware are no longer playable by most people," Kardar said in a statement. "Thanks to this breakthrough, we are exploring the potential for Muse to take older back catalog games from our studios and optimize them for any device. We believe this could radically change how we preserve and experience classic games in the future and make them accessible to more players. To imagine that beloved games lost to time and hardware advancement could one day be played on any screen with Xbox is an exciting possibility for us." Microsoft is also exploring the idea of "modding" games using AI, and making those early experiences available to players via Copilot Labs. Microsoft said, however, that it does not necessarily plan on using AI as part of game development. That will be up to the company's creative leaders, Kardar said, and any AI work will be shared "earlier on" with players and creators.

Microsoft develops AI model for videogames

(Reuters) - Microsoft has developed an artificial intelligence model of a videogame which can help generate visuals and actions, the company announced on Wednesday. The model, called Muse, was made in collaboration with Ninja Theory, a developer under Xbox Game Studios - Microsoft's gaming division. The push towards AI comes as videogame development costs continue to rise. Spending on new games has also been weak as consumers stick to proven titles due to economic uncertainty. "We are already using Muse to develop a real-time playable AI model trained on other first-party games, and we see potential for this work to one day benefit both players and game creators," said Fatima Kardar, corporate vice president for gaming AI. (Reporting by Zaheer Kachwala in Bengaluru; Editing by Maju Samuel)

Microsoft's generative AI model Muse isn't creating games - and it's certainly not going to solve game preservation, expert says

Last night, Microsoft trumpeted the announcement of Muse, a new "generative AI breakthrough" designed to aid "gameplay ideation". The company also published some grainy-looking gifs of AI-generated gameplay footage, based on Xbox studio Ninja Theory's multiplayer game Bleeding Edge. (The images were miniature in size, presumably to avoid highlighting some of the wonkiness AI is known for.) Finally, Microsoft made the claim that Muse would "radically change how we preserve and experience classic games in the future", and that the algorithm could be used to make older games compatible with "any device". Reaction to Microsoft's announcement was swift, with social media awash with posts pointing out that, yes, Microsoft was absolutely jumping on the tech industry's AI buzzword bandwagon - but also the suggesting that Xbox was about to start using Muse to pump out AI slop. Thankfully, someone has done a better job than Microsoft itself of explaining what Muse is actually doing - and that person is AI researcher and game designer Dr Michael Cook. For background, Cook is an expert on the subject of AI in games - he's the guy who built an artificial intelligence to see if it could win a game jam a full decade ago, whose work Eurogamer has covered on several occasions. He's also a senior lecturer at King's College London, and has published and spoken extensively on the subject of AI. As Cook lays out in an extended blog post on Muse, the AI model is not generating gameplay or creating its own original ideas. In short, Muse was fed seven years of video footage of people playing a singular game - in this case, Bleeding Edge - to see if it could then generate further gameplay footage of it. (If this all sounds familiar, it's similar to the process Google used to generate footage of classic first-person shooter Doom last year.) So, what's the point of all this? Well, as Cook writes, it's so Microsoft researcher could ask Muse to predict what might come next if changes to a game were made. "They made a tool that let game developers edit a game level using existing game concepts like adding in a jump pad to a place where there wasn't one before," Cook explains. "They then gave this new level to their model, and asked it to show what it thought the footage of a player playing from this new position would look like." In other words, the idea is Muse could be used as a shortcut tool for predicting and visualising how gameplay might adapt to a particular input by a developer. And, crucially, that developer is still a human. Microsoft's research paper on Muse says the AI model is required to understand persistency, consistency, and diversity in order to succeed. In other words, if a human input is provided, the AI needs to ensure the effects of that input remain, whatever else is going on, and that the effects remain similar while adapting to a range of player behaviours. For example, say a human adds a jump pad to a game's level. The paper says Muse must predict gameplay footage to reflect this which also ensures the jump pad persists (no deleting it!), that it reacts the same each time (it should always make players jump), regardless of what else is happening (this should apply even if a different player activates it). "The paper is not really about 'generating gameplay' or 'ideas'," Cook continued. "It's about these researchers thinking about the implications of how people will work with these tools." Notably, there's nothing in the paper that discusses the implications of Muse being added to the development processes of teams with a strong aversion to using AI, Cook notes. That said, in Xbox's blog post on Muse yesterday, Microsoft gaming AI exec Fatima Kardar stated that the company had simply "empowered creative leaders here at Xbox to decide on the use of generative AI" - suggesting there was no specific mandate from Microsoft to use AI, for now. But even with Muse being just used for a limited purpose, Cook has further questions around whether it will ever be fully viable. "This is not a practical process," Cook writes. "It's impressive that it can do this using visual information because things like lighting, camera angles, user interface and so on are a lot for an AI model to handle. "But ultimately, even with all of this data, all the time spent annotating datasets, and so on, it was still only just about able to generate footage predicting player behaviour." Such a system currently lacks any practical use, is enormously expensive, and requires any developer already have a vast vault of gameplay footage lying around for Muse to ingest in the first place. "The research team behind this probably believe it will get more efficient over time, which might make it more affordable or tractable for small developers," Cook continues. "However, it still raises the question of how we get video footage of people playing our game in the first place. "If you've been in development for a couple of months then you won't have enough footage, and even if we make the systems able to run on less input data there must be a minimum level required to understand the full game logic. So I think there is a question here not just of whether it makes sense as a tool now, but whether it can ever make sense." Lastly, on game preservation. "You could imagine a world where, from gameplay data and video, a model could learn old games and really make them portable to any platform where these models could run," Xbox chief Phil Spencer said yesterday in a video on Muse. "I feel that's really exciting." Bluntly, Cook calls Spencer's comments "idiotic". "I mean, in a sense anything is a preservation tool," Cook writes. "I could ask my friend's five-year-old son to draw a crayon picture of what he thinks the ending cutscene of Final Fantasy 8 looks like and that would still count as game preservation of a certain sort." Despite a decade of AI growth, Cook says, there's no method yet to measure what exactly an AI model has captured and what it has not. Muse is able to provide grainy gifs of one fairly simple video game based on seven years of footage, but it is not a solution for holding everything about a game or every possible outcome of what players could do. "This is absolutely not a solution for game preservation," Cook concludes, citing a report by gaming archeologist Florence Smith Nicholls about the archiving of digital games. "What does it mean to preserve a gameplay experience? Even if this model was a perfect replication of the original executable software, this is not the be-all and end-all of game preservation. A generative model of what game footage maybe looked like once might be a nice curio on the side of a real preservation process, but it is always going to be inferior to other ways we approach the problem."

Microsoft builds AI that creates 'impressive' video-game worlds

Generative artificial intelligence (AI) tools have upended creative industries from music and film to scientific publishing. Now they're upending the world of video games, too. In a study published in Nature on 19 February, a team of researchers reveals a generative AI engine that can create coherent, immersive video-game worlds that respond to player inputs. "I knew that this could be done, but it's still impressive," says Julian Togelius, a researcher at New York University specializing in AI in video games. Last year, researchers at Google showed that an AI engine could simulate the 1990s video game DOOM. But the ability for generative AI tools to generate fresh ideas in complex worlds has been limited. When a team led by Microsoft game-intelligence researcher Katja Hoffman interviewed game developers to find out what they wanted in AI tools, they suggested engines that can generate gameplay sequences that remain consistent with the rules and physics of the game. They also wanted the ability to tweak outputs throughout development. The team built and trained WHAM, a generative AI system focusing on these capabilities. The researchers took the online multiplayer game Bleeding Edge, developed by a Microsoft-owned studio named Ninja Theory, and extracted video frames and controller inputs from 500,000 anonymized game sessions, which equated to more than seven years of continuous play. They trained the engine on one-second slices of gameplay involving up to 1.6 billion parameters. The team showed that WHAM could perform well in three crucial areas. It could act within the rules of the original game (consistency) and generate a range of gameplay slices (diversity), with anything that was added to the world by the user remaining there (persistency). Together, these features demonstrate WHAM's potential for creative ideation, the authors say. Finally, they built a concept prototype, the WHAM Demonstrator, which provides an interface for users to prompt the AI engine. In one example, the researchers showed that they could feed a single promotional image for Bleeding Edge into the Demonstrator and it would generate plausible gameplay and controller inputs. They also added a function that allows developers to drag and drop imagery into the Demonstrator, which the AI engine then incorporates directly into the generated game world. Togelius says this is the most impressive thing to emerge from the paper. Kazjon Grace, a computational designer at the University of Sydney, Australia, says the work is notable for its focus on creative conceptualization. "I'm happy that people are looking into these questions of AI for ideation rather than just AI for productivity," he says. Grace and Togelius say applications for WHAM are limited because the AI engine is built for one video game. "Do I think WHAM is a useful tool? No -- but it's a way to push the field forward," says Grace. To create engines with broader applications, studios will need access to a wide variety of player data and sessions. Grace adds that there are several concerns around the use of generative AI, including its large energy consumption and where the training data are sourced. Some generative AI models have been accused of using copy-righted material to train their systems. In the case of WHAM, training data was acquired through end-user licence agreements. But he worries that deploying generative AI systems in game development might enable studios to lay off a large number of staff or create lots of low-quality, AI-generated content. "It's up to us as researchers, as consumers, as designers, as game-studio owners, to really focus on trying -- in a world that makes this pretty tricky -- to make a better future," he says.

GRTV News - Xbox is using AI to preserve older games

Muse AI is here to help with game development and bring extra life to deceased or unavailable projects. "Hello everyone and welcome back to another episode of GRTV News. Today we're going to be talking a little bit about Microsoft again. Because yesterday they announced something that was really quite surprising. Whether it's a good thing or a bad thing is kind of up to you I would guess. But it's definitely surprising and I think it's definitely a sign of what the future holds, especially for video game development. So essentially what Microsoft has done is they've created or they've partnered up with an AI and intend to incorporate it more into their video game world. That's going to include sort of streamlining development but it's also going to include an attempt to sort of preserve games for the future." "So titles that may have once been shut down because maybe they didn't have the player base or the interest or maybe it was too costly to keep them running. Now there's a way to do that using this AI system. So quite a lot of things it can do. Let's dive on in and take a look. Microsoft wants to preserve your games for the future using AI. The explicit goal is that beloved games lost to time and hardware advancement could one day be played on any screen with Xbox. So allowing artificial intelligence to interfere with our games is usually a hot topic. Some fear that it will come at the expense of imagination and creativity because AI largely bases its work on what already exists. In addition, developers, voice actors and others are not super keen on losing their jobs. Microsoft has now developed a tool they call Muse AI which can be used for game development and on Xbox Y we can read." "Another opportunity we're exploring is how Muse can help game teams prototype new gameplay experiences during the creative process and introduce new content, taking games players already love and enabling our developers to inject new experiences for them to enjoy or even enable you to participate in the creation process. However, Muse AI has more benefits than that and looks set to be a powerful solution that will allow older titles to be played in the future. Today, countless classic games tied to ageing hardware are no longer playable by most people. Thanks to this breakthrough, we are exploring the potential for Muse to take older back catalogue games from our studios and optimise them for any device. We believe this can radically change how we preserve and experience classic games in the future and make them accessible to more players. To imagine that beloved games lost to time and hardware advancement could one day be played on any screen with Xbox is an exciting possibility for us. And that sounds like a way of using AI in game development that quite a few of us can appreciate anyway, right? Check out the video below for more information." "So while you can look at it and say, yes, it's nice that they can preserve games, the problem is when you give an AI a job to do, it does it in its own way. And the concern obviously is going to be that it takes these sort of classic games or these sort of ageing games and it preserves them in its own way. We see a lot when you ask an AI, even just like an image generator, to generate a relatively straightforward image and it does it and you look at it and go, well, that person has six fingers or something daft and silly like that." "And that's the problem with using AI at the moment like this. So it has to be controlled and it has to be managed by a human counterpart. Now, the question is, is whether that's going to be something that happens because these old games are probably not going to draw much revenue for Microsoft, if at all. So is Microsoft going to spend money employing people to act as quality control and again, sort of like management for this AI or are they just going to let it run wild? It's hard to tell. Tech companies like to do things that us regular folk don't particularly like. Then the other thing is obviously the streamlining development."

GRTV News - Satya Nadella on the use of AI in Xbox games: "Very soon they will start using these models"

"Hello there, welcome back to GRTV News, I'm Alex as usual, going through the afternoons latest and greatest when it comes to gaming, technology, entertainment, whatever you like, whatever you love, we always have it here for you at GRTV News and in the wider Gamereactor network as a whole, so if you like what you see and you want to see more be sure to check out Gamereactor wherever you get it from for more gaming news, previews, interviews, exclusives, movie reviews, series reviews as well and so much more, but without further ado, today we're talking a bit of AI, those two magical letters that pop up no matter where you go, no matter what industry you're in, it does seem like more and more this technology is being pushed and for good or for bad that's entirely up to you, but it seems that Xbox is sort of going full force ahead, full steam ahead with AI, at least that's according to the Microsoft CEO Satya Nadella who has been recently talking about how AI can use, Xbox can use this sort of muse AI, you might have seen the clips online of these sort of generated gameplay moments that have been made entirely through AI, so we've had you know Nadella claiming that Xbox will have a catalogue of games very soon apparently that will use these AI muse models, he says we're going to train these models to generate and then we'll start playing with them, in fact he continues when Phil Spencer first showed it to me he had an Xbox controller and this model basically took the input and generated the output based on the input and it was consistent with the game, to me that's an incredible moment, it's like the first time we saw ChatGPT complete sentences or Dali draw or Sora, this is one of those moments, now this isn't to say that next week Xbox will unveil its catalogue of AI games and we'll be able to play something that didn't have a human hand on it at all, but it is sort of that moment where Nadella is hoping that in the future they'll be able to be games entirely made through this technology, now again AI and your opinions on it are obviously going to be very subjective, personally I'll say right now just to get that out of the way that I'm not really a fan of AI in creative mediums, if you've read articles on Game Reactor from me you probably know why, recently I talked about Elon Musk sort of saying that he's going to make an AI game studio, I don't think that AI is quite there yet when it comes to making games, but then again it's worth me saying that the technology I think can be impressive and it can be very interesting to see what it could do, it's just that often right now it seems to be used by the wrong people for the wrong means, in any case my personal views aside, this is about the news, this is quite interesting to see because we've not really seen a big platform owner like Microsoft come in and say yes we're going to completely embrace AI, now Xbox has a lot of studios, a lot of studios with a lot of people employed under its banner, so it sort of does, it is a bit of a head scratcher as to why you would want technology to do the job for you, but then again with games taking much longer than they're used to nowadays, with games costing much more than they used to nowadays, it's perhaps not that we'll see games entirely generated by this Muse model, but very heavily supported by the Muse model so that we could see yearly releases more commonly in a lot of big studios, but we'll have to wait and see, again this is still in very very early days and a lot of the time a lot of tech people talk about this and yet we're still probably years away from anything really substantial being coming out from this, but let me know what you think of Xbox's use of AI and Muse, whether you think it's going to be promising for the industry, whether you think it's a bit concerning for the industry, let me know all that and more and I'll see you tomorrow for some more GLTV news, goodbye!"

Why would I ever want to play a video game created by AI?

The generative AI craze has come for video games. While companies like Square Enix and Ubisoft have been casually experimenting with the tech for years, Microsoft jumped into the race this week. The company revealed its new AI model, nicknamed Muse, which it describes as "the first World and Human Action Model." The short of it is that Microsoft collaborated with Ninja Theory to research a new generative game development model that was trained on Bleeding Edge. The end result is a model that can generate "gameplay sequences," as shown in a series of low-res videos. Microsoft has big plans for the tool, saying that it can be used to assist developers, preserve lost games, or even create content for existing games and insert it on the fly. As is always the case with AI, there's a gulf between the potentially useful things Muse can do versus the aspirational talk from executives trying to hype up a big investment. The latter is great sales pitch for investors hungry to see the countless dollars that's been poured into the tech pay off, but I'm left with one question as a player: Who on Earth beyond that small group actually wants any of this? Recommended Videos That question has long been the sticking point for plenty of recent tech trends, from NFTs to the ill-fated Humane AI Pin, but it's especially pressing when it comes to video games. The growing desire to bring more generative AI into the development process comes from a total misunderstanding of what makes gaming so special. Companies like Microsoft seem to be betting big on the idea that players don't actually care too much about the human touches that go into crafting a game. It's a gamble that's bound to leave overeager adopters leaving the table with empty pockets. The human touch Any criticism of AI's theoretical role in game development is tricky to approach. Companies like Microsoft have chosen their words very carefully when talking about the tech, always stressing that generative models would act in support of real people, not replace them. At last year's Game Developers Conference, I tried out a demo built with Nvidia Ace that had me naturally talking to responsive NPCs. Nvidia introduced me to a few writers who helped create the demo, who stressed that all of the generated responses I saw were the product of their own work. They explained that they wrote extensive character biographies and fed them into the AI model, which shaped the characters around that writing. The recurring argument from companies embracing AI is that it will be used in this way, giving creators a new tool akin to Photoshop. You can take that at face value, but the line of thinking begs skepticism. Last year, Microsoft laid off over 1,500 workers from its gaming division. The move even saw it shuttering studios like Tango Gameworks entirely, one year after it released the critically acclaimed Hi-Fi Rush. Now, we've learned that Microsoft was quietly training a new AI model amid those layoffs that it hopes will be capable of game development, in some form, one day. Those events were likely unrelated, but it doesn't take a mathematician to see where the two points could inevitably intersect. It's time to stop playing stupid: Generative AI will replace jobs. Anyone saying otherwise is lying to you. When overpromising what Muse could do, Microsoft's executives are not describing additive task that no one else is currently doing. It is someone's job to generate worlds, preserve old games, and create content for existing ones. Why build a machine capable of doing those tasks just to make a glorified assistant? The only person in games who is being honest about the endgame is Split Fiction director Josef Fares, who recently told VGC that the tech will likely come with layoffs. "We need to adapt to it," Fares told VGC when asked about generative AI's role in game development. "If it's part of the industry we should see how to implement it to see how we get better games. I can understand the fact that some people could lose their jobs but that goes for every new technology." To realistically discuss the future of generative AI in video games, we must reject damage controlling assurances from executives and accept that models like Muse could indeed remove some human hands from the process. That doesn't mean that all games will be entirely built by AI models going forward, but it does mean that you'll see more instances where anything from a piece of art to an entire NPC conversation might be credited to an AI tool. Were Muse to reach its full potential beyond generating rough gameplay snippets, you might even wind up walking around a level entirely sculpted by AI or playing a machine's approximation of a classic game -- at least if higher ups who don't quite understand the realistic applications of the tech have their way. If you love video games, that reality should make your stomach churn. There's a fundamental flaw in the entire AI game development boom. Its loudest advocates seem to believe that players will simply play anything. If you stick a machine-imagined Bleeding Edge clone in front of their face, they will gobble it up the exact same way they would a meticulously designed passion project from a celebrated studio with something to say. There is no mental separation between those two ideas for some because we're currently living in an age where "content" has replaced art. It's the same thinking that has turned Netflix into a cesspool of reality shows and hastily assembled true crime documentaries. Why overthink it? A game is just a game, right? It's that surface level thought that is bound to backfire for those who look to AI as a cost-cutting crutch rather than a tool. All of the things players love about video games come from the humans behind them. Do you love the way Kratos' Leviathan Axe feels in God of War: Ragnarok? That's because a group of people sat down and discussed the weight of the weapon and how the way it is swung communicates something about Kratos' mental state. Are you enjoying Avowed's story so far? You only get that from a team that experienced a real pandemic and organized their thoughts into a narrative that explores how authoritarianism preys on vulnerable societies. Do you like the way The Legend of Zelda: Breath of the Wild creates such a distinct atmosphere through minimalistic piano ambiance? That didn't pop out of a toaster. Generative AI may be the future, but the tech is tethered to the past. Video games are a creative medium and creativity can only come from human beings. An AI model can simulate creativity by throwing human art into a blender and spitting up some approximation of the real thing, but the final result is always devoid of meaning. It can animate Kratos swinging an axe, but it won't be thinking of what his body language says about the lifetime of violence he's desperately trying to hold back from his son. It can write a story about a pandemic, but it can't actually tell you its original thoughts on what it means. It can put piano notes next to one another, but it doesn't have a vision of Hyrule. At best, generative AI is an energy-consuming photocopier that can only spit out grainy, black and white scans of your butt cheeks. What is the endgame of an industry built on photocopies? So, let's say a machine could study Bleeding Edge and spit out something that looks and plays like Bleeding Edge. What exactly is the appeal of that? I understand why it's exciting for an investor looking to reduce headcounts, but how does that benefit the actual person playing these games? Today's gamers are already resistant towards templatized titles that try too hard to ape what's popular in a hollow, businesslike manner. Take a walk through the graveyard of dead battle royale games that barely survived a year. A machine trained on Zelda games would never have thought to generate Wind Waker. No amount of digital ideation could have birthed Balatro, because nothing like it existed in a data set. And no machine could possibly preserve either game correctly without understanding the creative decisions that guided how they were built. Virtually no video game you enjoy today would exist in a world where AI models are too ingrained into the creative process. Generative AI may be the future, but the tech is tethered to the past: It can only ever remix what came before. Naturally, the tech will improve. I'm sure Muse will be able to spit out something more than a series of the ugliest 480p gifs you've ever seen in your life eventually. Optimistically, maybe it will even be implemented in the right way, finding its place as an assistive tool that can help developers implement their ideas more effectively (that is, if developers really want that at all). AI, in a broader form, has been a useful part of game development for a long time after all, with a range of great use cases to be found in everything from Final Fantasy XVI to Immortality. There is a place for the tech if it is not abused. But generative AI will never truly be a full substitute for human touch in a medium defined by it. It will only ever be able to finger paint recreations of the Mona Lisa with its six gnarled digits.

Xbox wants to tackle video game preservation with AI - Softonic

Everyone thinks it's a terrible idea and Phil Spencer does what he does best: pretends not to notice Xbox has presented its new generative artificial intelligence project, called Muse, which has the potential to transform the way classic video games are preserved. According to the company, Muse is a model capable of generating visuals of games and control actions, which opens the door to optimizing old titles to make them accessible on different devices. "Nowadays, countless classic games linked to obsolete hardware are no longer playable by most people," Xbox stated in its announcement. Phil Spencer, director of Xbox, highlighted that the preservation of video games is a topic of great interest for the company. Spencer suggests that Muse could learn about old games and make them portable to modern platforms without the need for the original engine to run on the original hardware. However, the community of video game historians is skeptical about Muse's ability to achieve true preservation. Frank Cifaldi, founder of the Video Game History Foundation, compares the proposal to a simple photocopy of a work of art, expressing his concern about the meaning of "preservation". In addition to skepticism, the preservation process faces significant legal challenges. Publishers fear that preserved games will be used for recreational purposes, which has led the U.S. Copyright Office to reject a major effort for video game preservation in 2024. Although the technology behind Muse is in its early stages, experts are divided on its future potential. One day Muse may be able to perfectly replicate iconic titles, but for now, its viability remains a topic of debate.

GenAI tool promises jump forward for game devs

Researchers have produced a generative AI tool they say can create a three-dimensional game world to help developers design and tweak gameplay. A paper in Nature said a research team from Microsoft was able to demonstrate the model generating complex 3D video game sequences that were consistent with the training game's specific mechanics, but with diverse designs. It also allowed developers to tweak the output iteratively. The World and Human Action Model (WHAM) was developed by Katja Hofmann, Microsoft senior principal research manager, and colleagues who started out interviewing 27 video game developers about their creative needs. The model was trained on around seven years of human gameplay from Bleeding Edge, a multiplayer online battle arena game developed by Ninja Theory and published by Microsoft's Xbox Game Studios. The paper said the model could "generate consistent and diverse gameplay sequences and persist user modifications." "In contrast to previous approaches to creativity support tools that required manually defining or extracting structure for relatively narrow domains, generative AI models can learn relevant structure from available data, opening the potential for a much broader range of applications," it said. The team also produced WHAM Demonstrator as a visual interface to allow developers to engage with and customize the outputs of WHAM. It can be found at Hugging Face, along with WHAM's weights and evaluation dataset. "The key novelty that generative AI models such as WHAM contribute is that they remove the need for handcrafting or learning domain-specific models for individual domains, making it likely that model innovations such as these will broaden creativity support to other domains, such as music59 or video60," the researchers said. "Extrapolating from our use case focusing on a single 3D video game, we can also get a first sense of how powerful future models will be in allowing teams of human creators to craft complex new experiences." In the meantime, there has been a kickback against the use of generative AI in game development. Indie developer Polygon Treehouse, which produced Röki and Mythwrecked, has proposed a "No Gen AI" seal for games. "The issue is that these generative technologies are trained on existing works by human artists who have not given their permission, or been compensated, for their work being utilized," the company said. ®

Microsoft shows progress toward real-time AI-generated game worlds

For a while now, many AI researchers have been working to integrate a so-called "world model" into their systems. Ideally, these models could infer a simulated understanding of how in-game objects and characters should behave based on video footage alone, then create fully interactive video that instantly simulates new playable worlds based on that understanding. Microsoft Research's new World and Human Action Model (WHAM), revealed today in a paper published in the journal Nature, shows how quickly those models have advanced in a short time. But it also shows how much further we have to go before the dream of AI crafting complete, playable gameplay footage from just some basic prompts and sample video footage becomes a reality. Much like Google's Genie model before it, WHAM starts by training on "ground truth" gameplay video and input data provided by actual players. In this case, that data comes from Bleeding Edge, a four-on-four online brawler released in 2020 by Microsoft subsidiary Ninja Theory. By collecting actual player footage since launch (as allowed under the game's user agreement), Microsoft gathered the equivalent of seven player-years' worth of gameplay video paired with real player inputs. Early in that training process, Microsoft Research's Katja Hoffman said the model would get easily confused, generating inconsistent clips that would "deteriorate [into] these blocks of color." After 1 million training updates, though, the WHAM model started showing basic understanding of complex gameplay interactions, such as a power cell item exploding after three hits from the player or the movements of a specific character's flight abilities. The results continued to improve as the researchers threw more computing resources and larger models at the problem, according to the Nature paper. To see just how well the WHAM model generated new gameplay sequences, Microsoft tested the model by giving it up to one second's worth of real gameplay footage and asking it to generate what subsequent frames would look like based on new simulated inputs. To test the model's consistency, Microsoft used actual human input strings to generate up to two minutes of new AI-generated footage, which was then compared to actual gameplay results using the Frechet Video Distance metric.