arXiv · 2611.11136
From Game Generation to Game Playing: Fully Automated Ability Training for Vision-Language-Action Models
Abstract
Training vision-language-action (VLA) models to follow gameplay instructions across diverse games is limited by the cost of collecting demonstrations. We introduce a framework that automatically generates game environments and expert demonstrations. An LLM-powered agent uses game-building tools to construct executable environments and tasks targeting specified skills. Programmatic experts then use privileged simulator state to generate aligned instructions, rendered observations, and keyboard and mouse actions. We train our game policy, Script2Play, by post-training a 4B vision-language model with behavior cloning on 150.3 hours of synthetic Minecraft and Roblox gameplay, without human gameplay demonstrations or reinforcement learning. Evaluation on multiple public game benchmarks demonstrates that Script2Play follows gameplay instructions effectively and generalizes across game engines, highlighting the effectiveness of targeted synthetic supervision.
Explore related subjects
Keep this discovery
Explore connections, maps & timelines
Changran Liu, Qi Wu, Wenchao Ma, Sharon X. Huang, Haomiao Jiang. 2026-11-08. From Game Generation to Game Playing: Fully Automated Ability Training for Vision-Language-Action Models. https://searcharxiv.com/abs/2611.11136
Citations retain the original authors and DOI and include this site’s record URL. If this discovery service contributed to your research, you can also cite Searcharxiv by Indiciae: BibTeX · RIS.