From Representational Complementarity to Dual Systems: Synergizing VLM and Vision-Only Backbones for End-to-End Driving

Abstract

Vision-Language-Action (VLA) driving augments end-to-end (E2E) planning with language-enabled visual backbones, yet it remains unclear how vision-language models (VLMs) differ from standard vision-only encoders and whether these differences survive policy learning. We study this question under a unified VLM-hidden + diffusion-policy paradigm, comparing multiple VLM families/scales (e.g., InternVL3 and Qwen3VL) with vision-only encoders (e.g., ResNet, ViT, and EVA-CLIP). We ask three questions: how similar are their representations, do residual differences induce meaningful behavioral differences, and how can they improve accuracy-cost trade-offs? We find that VLM and vision-only policies share a substantial common subspace after policy learning, yet both retain non-transferable residual subspaces. Using a Shared-Unique SAE, we show that these residual factors are behaviorally relevant: vision-only encoders are stronger in simple, geometry-dominant scenarios, whereas VLMs are stronger in long-tail, semantically complex, and interaction-heavy cases. The two policy families also exhibit distinct driving styles, with vision-only models being more conservative on average and VLMs more assertive. Exploiting the complementarity between a VLM branch and a ViT branch yields an oracle upper bound of 93.58 PDMS on NAVSIM. We introduce HybridDriveVLA, which runs both branches and uses a learned trajectory scorer for selection, improving PDMS to 92.10 (+1.30 over the VLM baseline), and DualDriveVLA, a fast-slow variant that invokes the VLM in only 15% of scenarios, achieving 91.00 PDMS (+0.20) with about a 1.9x latency speedup over the VLM baseline. Code will be released.