Gamer trains one vision-language-action model to play 2D and 3D games in real time
A 27-author paper describes Gamer, an agent pre-trained on tens of thousands of hours of keyboard and mouse trajectories, then tuned with sparse thinking and multi-turn reinforcement learning for long-horizon play.
Most game-playing agents are built for one game, or for one kind of interface. A paper posted to arXiv on 2026-11-13 describes Gamer, a generalist vision-language-action foundation model meant for long-horizon tasks in complex interactive environments. It targets both conventional 2D graphical interfaces and complex 3D simulated worlds, with one shared pipeline for training and deployment. The paper lists 27 authors and links a code repository.
How the model is built
According to the abstract, Gamer is an end-to-end architecture that combines perception, memory management and multi-step reasoning. Training happens in stages. The model first continues pre-training on a dataset built from tens of thousands of hours of device-level keyboard and mouse interaction. Those trajectories span simulated games and embodied environments, so the model sees the dynamics of physical and virtual domains in the same data.
Next comes what the authors call sparse-thinking instruction tuning. The stated aim is to cut redundant computation while keeping the reasoning that complex decisions still need, which matters for an agent that has to act inside a running game instead of answering at its own pace.
The policy for long-horizon interaction is then trained with a multi-turn reinforcement learning method that the authors describe as stable. It is designed around three problems the abstract names: finite context windows, errors that accumulate over a rollout, and sparse rewards.
Serving it in real time
Real-time play is handled by a cloud-edge inference engine. The abstract says it serves numerous simulated game environments concurrently, supports unbounded-context execution and keeps online inference latency low. No latency figure or environment count is given at the abstract level.
What the authors report
Across multiple challenging 3D open-world games, the authors report solid generalization and operational efficiency comparable to skilled players, along with competitive results on a range of conventional 2D game benchmarks. Their reading of the results is that even repetitive keyboard and mouse trajectories can support substantial gains when data and learning algorithms are scaled, and that training on them helps the model learn complex skills more steadily in environments with both physical and digital properties.
These claims come from the abstract, not from an independent evaluation. The abstract does not name the 3D games or the 2D benchmarks, does not say how skilled players were defined or measured, and gives no model size. The paper frames the approach as a path toward generalist agents that can interact across digital and physical worlds; how far the physical side of that claim reaches beyond simulated and embodied environments is a question for the full paper and the linked repository.