Reflex First, Reflect Later: Latency-Aware Embodied LLM Agents for Dynamic Response

Abstract

Large language models (LLMs) have substantially improved the planning capabilities of embodied agents, enabling their deployment in dynamic and safety-critical environments. However, these settings expose a critical limitation: inference latency. Delayed LLM responses can weaken real-time responsiveness and misalign agent reasoning with rapidly changing environmental states. This paper systematically studies the impact of inference latency on LLM-based embodied agents in dynamic environments. We introduce an FPS-based Time Conversion Mechanism (TCM) that maps inference time to elapsed simulation time, allowing computational delays to directly affect environmental evolution and agent outcomes. We instantiate this protocol in HAZARD and introduce Response Latency (RL) and Latency-to-Action Ratio (LAR) to evaluate agent responsiveness. Building on this framework, we propose the Rapid-Reflex Async-Reflect Agent (RRARA), which integrates rapid reflexive actions with asynchronous LLM reflection to mitigate latency-induced errors. We further introduce an LLM-based PrePlanner that generates cached object-centric subgoals, reducing repeated LLM calls while retaining the model's high-level reasoning capability. Experiments show that accounting for inference latency substantially changes embodied-agent performance and that RRARA achieves a stronger balance between decision quality and responsiveness.