ROAR: Benchmarking LMM Agents under Real-Time Constraints in Multiplayer Action-Shooter Games

Dawei Wang; Di Zhao; Jianran Yu; Chengming Zhou; Marci Chi Ma; Xiaoyang Liu; Beining Zhang; Shuliang Chu; Yun Sing Koh; Gary Ushaw; Richard Davison

arXiv · 2610.24282

ROAR: Benchmarking LMM Agents under Real-Time Constraints in Multiplayer Action-Shooter Games

Dawei Wang·Di Zhao·Jianran Yu·Chengming Zhou·Marci Chi Ma·Xiaoyang Liu·Beining Zhang·Shuliang Chu·Yun Sing Koh·Gary Ushaw·Richard Davison

View original source ↗Read source document ↗Export metadata ↓

Abstract

Large multimodal models (LMMs) are increasingly deployed as agents that perceive, reason, and act in interactive multi-agent environments. Yet most existing benchmarks for LMM-based agents in multi-agent settings remain turn-based: the environment and other agents pause while an agent performs inference. Such protocols obscure a fundamental challenge of real-time constraints, where the world continues to evolve and delayed decisions may become outdated before execution. Motivated by this limitation, we introduce ROAR, a multiplayer action-shooter game environment for evaluating LMM-based agents in multi-agent settings under real-time constraints. ROAR covers cooperative, competitive, mixed cooperative-competitive, and basic single-agent scenarios. We further propose a Multi-Agent Asynchronous Gym interface that decouples environment evolution from agent reasoning by running the environment and each agent in separate threads, enabling the environment to advance at fixed time intervals without waiting for LMM inference and allowing agents to reason independently without blocking one another. ROAR also features convenient deployment, multimodal observations, and customizable levels. ROAR offers a practical testbed for advancing LMM-based agents beyond static, turn-based evaluation toward real-time multi-agent decision-making.

Explore related subjects

cs.MA

Keep this discovery

Explore connections, maps & timelines

BibTeX RIS

Dawei Wang, Di Zhao, Jianran Yu, Chengming Zhou, Marci Chi Ma, Xiaoyang Liu, Beining Zhang, Shuliang Chu, Yun Sing Koh, Gary Ushaw, Richard Davison. 2026-10-16. ROAR: Benchmarking LMM Agents under Real-Time Constraints in Multiplayer Action-Shooter Games. https://arxiv.org/abs/2610.24282

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

CARVS: A Congestion-Aware Algorithm for Online Multi-Agent Pickup-and-Delivery in Human-Collaborative Warehouses

Intelligent warehouse systems have transformed logistics by integrating robots with minimal infrastructure changes, boosting efficiency and optimizing workflows compared to traditional human-only operations. However, human-robot collaboration introduces new challenges, particularly due to human uncertainty, which leads to dynamic interactions and unpredictable behavior. To address these complexities, this paper presents the Multi-Agent Pickup-and-Delivery with Human Collaboration (MAPDwHC) problem, an extension of the Multi-Agent Pickup-and-Delivery (MAPD) problem that incorporates human unpredictability. This problem adds key constraints, such as dynamic obstacles caused by robots temporarily blocking paths when interacting with humans, and the need for real-time responsiveness. We formalize the MAPDwHC problem with additional constraints and properties. Based on this problem setting, we propose the CARVS algorithm, which improves task assignment and path planning by integrating task swapping, validation checks, and random initialization. CARVS reduces congestion, enhances adaptability to human-robot interactions, and ensures real-time responsiveness. Simulation results show that CARVS significantly reduces timeouts of path planning in large-scale scenarios where traditional MAPD algorithms fail, improving both efficiency and scalability.

cs.MA↗

arXiv2026

Coordination Matters: Process-Aware Evaluation of Cooperative Multi-Agent Reinforcement Learning

Cooperative multi-agent reinforcement learning (MARL) benchmarks commonly emphasize aggregate outcomes such as return, success rate, or completion time. While essential, these metrics can obscure how agents coordinate and fail to distinguish policies that achieve similar outcomes through different interaction processes. We propose a process-aware evaluation framework that complements aggregate performance with diagnostics of interference, coordinated task completion, participation, and coordination efficiency. We evaluate the framework in STAT, a controlled spatial task-allocation testbed, and examine its broader applicability in two established cooperative MARL benchmarks. Across diverse value-based and actor-critic methods, we find policies with nearly identical return can exhibit substantial differences in coordination behavior. Controlled scaling in STAT reveals distinct process signatures for spatial, task, and agent scaling, while a reward-shaping ablation shows that explicitly penalizing assignment conflicts can alter coordination behavior with little change in task return. Results in the established benchmarks further show that the framework remains informative under different coordination mechanisms. These findings support process-aware evaluation as a complement to outcome-based benchmarking. The framework provides a common structure for selecting and interpreting environment-specific diagnostics that characterize not only what cooperative policies achieve, but how they coordinate to achieve it.

cs.MA↗

arXiv2026

Do Multi-Agent LLMs Explore Each Other?

Exploration is essential for reliable autonomy in multi-agent systems, yet it remains unclear whether large language model (LLM) agents can explore effectively when interacting with one another. We show that modern LLM agents often fail to do so, exhibiting myopic and polarized interaction patterns that lead to suboptimal coordination and increased regret. We formalize this challenge as the Multi-Agent Exploration problem, modeling it as a partially observable stochastic game (POSG), in which agents must probe peers to infer their capabilities and identify effective interaction strategies. To address this, we introduce Multi-Agent Contextual Exploration (MACE), a lightweight framework that explicitly promotes exploration through structured peer selection. Across both contextual and parametric diversity settings, exploration behavior and downstream task performance substantially improve when MACE is applied. We further provide theoretical insight that MACE achieves a sublinear expected-regret bound. Overall, our results highlight a fundamental limitation of current LLM agents and underscore the importance of explicitly guided exploration for reliable multi-agent autonomy.

cs.MA↗