ChatBEV: Empowering Traffic Scene Understanding and Simulation via Vision-Language Model

Abstract

Comprehensive traffic scene understanding is a foundational capability for Intelligent Transportation Systems (ITS) underpinning applications such as traffic simulation. While VisionLanguage Models (VLMs) have demonstrated strong reasoning potential, their application to Bird's-Eye View (BEV) maps in traffic contexts remains limited by narrow task definitions and scarce annotated data. We introduce ChatBEV-QA, a large-scale BEV VQA benchmark of 137K+ QA pairs, designed to evaluate global scene understanding, vehicle-lane interactions, and vehiclevehicle interactions within complex traffic environments. Building on this, we fine-tune ChatBEV, a specialized VLM that accurately interprets diverse scene understanding queries from BEV maps. To demonstrate downstream utility in ITS applications, we integrate ChatBEV into a language-guided traffic simulation framework. Its global understanding and navigation reasoning provide crucial context-aware guidance, reducing trajectory displacement error by up to 20.9% and scenario collision rates by up to 37.9% over text-only baselines.