GameWM: Toward Next-Generation Video Games with Interaction-Centric World Models
Abstract
For more than half a century, video games have pursued better graphics, more natural interaction, and more engaging gameplay. Generative video models (VMs) now bear on all three. Neural networks already draw most of the pixels in shipped games, and recent world models let a lightweight game engine keep the world state while a VM turns the engine's coarse proxy video into final frames. These systems, however, treat the VM as a renderer that only restyles motion the engine has already authored, which leaves most of its generative capacity unused and limits the diversity a game can offer. We present GameWM, a design for next-generation games built on a clear division of labor. The GameWM-engine keeps what must be exact: rules, state, camera, and the layout of the world, which also spares the video model from having to remember it. The GameWM-VM generates what is expensive to author by hand: the look of the world and the detail of the player's interactions. One game mechanic can then be realized in many visually and physically distinct ways, while its outcome stays exactly as the engine computed it. To realize this division of labor, we build GameWM as a complete system: a data engine centered on interactions rather than exploration, an interface through which authored character sheets, scene art, and skill descriptions enter the model, and a model family from 5B to 33B parameters with a real-time causal variant for play. Together they give every player a fresh version of the same game: each player's own character, world, and skills turn the same engine mechanics into a new experience, while every event keeps the timing and outcome the GameWM-engine computed.
Pipeline
A character sheet, a first frame, and a skill tree are authored once. At runtime the GameWM-engine keeps the rules: it renders a low-poly proxy and resolves the event, including who is hit and when. The GameWM-VM then fills in the look of that interaction from the first frame, the sheet, the proxy, and the event text.

From authored assets to play. Left: a first frame, a character sheet, and a skill tree that fuses an attribute such as fire with each mechanic. Middle: the engine renders the proxy and resolves the event. Right: the video model turns those four inputs into the finished frames.
Dataset
The data engine is built around interactions rather than exploration. Unpacked game buffers and screen recordings are turned into aligned pairs: a coarse proxy, an event caption, and many visual realizations of the same clip, screened by a vision-language model.

The interaction-centric data engine. Both source paths yield proxies and an event caption. One clip is then expanded into low-poly and high-fidelity targets, so each proxy can pair with many looks of the same event.

One training pair per engine event. Each strip shows the capsule re-render, which carries camera, layout, and timing, above the source recording. The proxy does not draw limbs, faces, or the final look of the action.
BibTeX
@article{he2026gamewm,
title={GameWM: Toward Next-Generation Video Games with Interaction-Centric World Models},
author={He, Bing and Wang, Runze and Zha, Fei and Wu, Zhongyuan and Xu, Ke and Zhuang, Tao and Xie, Yuxuan and Liu, Zhongyuan},
journal={arXiv preprint arXiv:2611.19247},
year={2026}
}