GameNGen

A 2024 Google project that uses a diffusion model to generate playable DOOM footage in real time, replacing the game engine.

GameNGen is a paper released in August 2024 by researchers at Google Research and Google DeepMind, demonstrating that a neural network can simulate a complex game and be played in real time. The game chosen was the classic first-person shooter DOOM. The method has two steps: first, train a reinforcement-learning agent to play the game, recording footage and key presses; then adapt Stable Diffusion 1.4 into a diffusion model that predicts the next frame conditioned on a number of past frames and the player's actions. Autoregressive generation — using self-generated frames to generate the next ones — tends to get blurrier and blurrier over time, so the authors added Gaussian noise to history frames during training, teaching the model to correct itself, which lets it run stably for several minutes; they also separately fine-tuned the image decoder to reduce compression artifacts. It reaches about 20 frames per second on a single TPU. This work drew wide attention to interactive world models, and is often discussed alongside DIAMOND and Genie 2.

ExampleThe player presses keys to shoot, open doors, and pick up health packs, and the diffusion model generates the footage frame by frame, including the on-screen health and ammo counts; the generated frames reach a PSNR of 29.4, comparable to lossy JPEG compression, and human raters could barely do better than chance at telling short real and generated clips apart.

Also called

Neural Game Engine, Diffusion Models Are Real-Time Game Engines

Related

Interactive World Model · Diffusion Model · Autoregressive Video Generation · DIAMOND · Genie 2 · World Model

Sources

Diffusion Models Are Real-Time Game Engines (arXiv 2408.14837)
GameNGen 项目主页 (Chinese)

As of

2024-08