Meet Muse: Microsoft’s AI That Creates Gameplay Sequences

Microsoft has announced a new generative AI model, which is designed for gameplay ideation. It’s a first-of-its-kind model and is named Muse. It’s part of the company’s World and Human Action Model (WHAM) and can generate both game visuals and controller actions. The research behind Muse has been published in Nature and was developed in collaboration with Xbox Game Studios’ Ninja Theory. So, let’s take a closer look at Microsoft’s latest AI research. 

How Muse Works

Muse is trained on large datasets of human gameplay, particularly from Ninja Theory’s 2020 online multiplayer game Bleeding Edge. By analyzing gameplay sequences, Muse can generate new gameplay scenarios, simulating how a game might evolve based on different player inputs. 

However, it’s not all perfect. According to a VentureBeat report, “Image resolution is fixed at 300×180 pixels.” The report also highlights a trade-off between model size and speed, stating that the largest and most consistent models have the slowest inference times.

According to Microsoft, the model has three key capabilities:  

• Consistency: The model makes sure that the generated gameplay follows game mechanics, such as movement rules and physics.
• Diversity: It produces a variety of gameplay sequences based on the same prompt.
• Persistency: The model retains modifications made by users in generated outputs.

Muse Potential Applications in Other Businesses 

Currently, the Muse is focused on game development needs. Microsoft sees way broader applications for this technology in the future. The official blog post also highlights the potential use cases of Muse in industries like architecture, retail, and manufacturing. 

From reconfiguring the kitchen in your home to redesigning a retail space to building a digital twin of a factory floor to test and explore different scenarios. All these things are just now becoming possible with AI.

Peter Lee, president of Microsoft Research

Muse Open Source Release and Developer Access 

The tech giant is open-sourcing the model’s weights and providing sample data. Developers can access Muse via Azure AI Foundry. The company has released the WHAM Demonstrator, an interactive prototype that allows users to experiment with the model’s capabilities.