arXiv subjects

Letian Wang

Publications and source records attributed to Letian Wang.

At least 19 recordsLinked to original sources

arXiv2026

VBVR-Pro: A Scalable and Verifiable Suite for Native Visual Reasoning

Native visual reasoning treats visual generation as the medium of reasoning itself: visual states (i.e. images and videos) are not merely inputs to be understood or outputs to be rendered, but first-class substrates for problem solving beyond language. Yet progress remains bottlenecked by the lack of scalable training tasks, reliable feedback, and controlled comparisons across generative substrates. In this work, we introduce VBVR-Pro, a closed-loop testbed that makes native visual reasoning through generation trainable, verifiable, optimizable, and experimentally controllable. 1) Task scaling. VBVR-Pro turns visual reasoning into a controlled task space of 300 procedurally generated tasks. Models trained on VBVR-Pro show strong transfer beyond the proposed suite across seven external visual reasoning benchmarks such as RISE-Video, MME-CoF-Pro, and BabyVision. 2) Verifiable rewards. VBVR-Pro provides verifiable reward scorers for task-grounded evaluation. Through a systematic study of leading MLLMs as judges, we identify recurring failure modes of the prevalent VLM-as-a-judge paradigm. In contrast, the proposed scorers are grounded in deterministic, task-specific rules, achieve fine-grained alignment with human judgments. Importantly, they serve as reliable reward signals for large-scale multi-task reinforcement learning and demonstrate stronger post-RL performance across visual reasoning tasks. 3) Mechanism study. VBVR-Pro enables controlled modality studies across more than 30 image, video, and interleaved generators. Our analysis shows that video generation remains strongest for tasks requiring persistent spatiotemporal state tracking, while interleaved generation provides a compute-efficient alternative. Critically, ablations and probing suggest the presence of vision-native trajectories that are crucial to visual reasoning. We release all data, models, scorers, and code.

cs.CV↗

arXiv2026

Video Generation Models are General-Purpose Vision Learners

Driven by next-token prediction, NLP shifted from task-specific models into powerful generalist foundation models. What, then, is the equivalent catalyst needed to achieve a general-purpose model in computer vision? In this paper, we contend that large-scale text-to-video generation serves as a strong pre-training paradigm for computer vision, providing the necessary spatiotemporal priors, vision-language alignment, and scalability required for general visual intelligence. We introduce GenCeption, which leverages a pre-trained video generative diffusion backbone to define a feed-forward perception model, capable of performing various vision tasks steered by text instructions. Empirical results demonstrate that GenCeption achieves state-of-the-art performance across a diverse suite of tasks, including depth, surface normal, and camera pose estimation, expression-referring segmentation, and 3D keypoint prediction, often matching or surpassing specialized models (e.g. DepthAnything3, SAM3, D4RT, VGGT-Omega, Sapiens, David, Genmo, and Lotus-2). Furthermore, the video generative pretrained backbone outperforms alternative pretraining paradigms (e.g., V-JEPA, and Video MAE) under comparable settings. Importantly, GenCeption exhibits preliminary data and model scaling properties along with exceptional data efficiency, where it achieves comparable performance with leading models like D4RT and VGGT-Omega with 7 to 500 less training data. Finally, GenCeption also exhibits intriguing emergent behaviors: a model trained exclusively on synthetic human videos generalizes to real-world footage and out-of-distribution object categories (e.g., animals and robots). These findings suggest that video generation is not merely a synthesis tool, but a foundational path toward generalist vision intelligence for the physical world. Project page: https://genception.github.io

cs.CV↗

arXiv2026

BeyondSight: Object Permanence for End-to-End Autonomous Driving

Autonomous driving operates in partially observable environments where actors may become fully occluded by other vehicles or infrastructure. Most end-to-end driving systems implicitly couple actor existence to instantaneous observations, causing actor hypotheses to degrade or disappear during prolonged occlusion and removing potentially critical agents from downstream prediction and planning. We introduce BeyondSight, a permanence-aware end-to-end driving framework that decouples actor existence from observability by maintaining persistent actor hypotheses over time. BeyondSight propagates actor queries temporally and updates them with observation-conditioned evidence, enabling joint perception, prediction, and planning to reason about actors even when they are temporarily unobservable. To enable principled training and evaluation of persistence-aware models, we further introduce nuScenes-Permanence, an extension of nuScenes that provides supervision and observability-conditioned evaluation for unobservable actors. Experiments show that BeyondSight substantially improves reasoning under occlusion, increasing detection performance for unobservable actors from 0 to 0.249 mAP while reducing planning error from 0.61 to 0.54 L2avg. These results highlight object permanence as an important modeling principle for robust end-to-end autonomous driving.

cs.RO↗

arXiv2026

Grounded World Model for Semantically Generalizable Planning

In Model Predictive Control (MPC), world models predict the future outcomes of various action proposals, which are then scored to guide the selection of the optimal action. For visuomotor MPC, the score function is a distance metric between a predicted image and a goal image, measured in the latent space of a pretrained vision encoder like DINO and JEPA. However, it is challenging to obtain the goal image in advance of the task execution, particularly in new environments. Additionally, conveying the goal through an image offers limited interactivity compared with natural language. In this work, we propose to learn a Grounded World Model (GWM) in a vision-language-aligned latent space. As a result, each proposed action is scored based on how close its future outcome is to the task instruction, reflected by the similarity of embeddings. This approach transforms the visuomotor MPC to a VLA that surpasses VLM-based VLAs in semantic generalization. On the proposed WISER benchmark, GWM-MPC achieves a 87% success rate on the test set comprising 288 tasks that feature unseen visual signals and referring expressions, yet remain solvable with motions demonstrated during training. In contrast, traditional VLAs achieve an average success rate of 22%, even though they overfit the training set with a 90% success rate.

cs.RO↗

arXiv2026

LMGenDrive: Bridging Multimodal Understanding and Generative World Modeling for End-to-End Driving

Recent years have seen remarkable progress in autonomous driving, yet generalization to long-tail and open-world scenarios remains a major bottleneck for large-scale deployment. To address this challenge, some works use LLMs and VLMs for vision-language understanding and reasoning, enabling vehicles to interpret rare and safety-critical situations when generating actions. Others study generative world models to capture the spatio-temporal evolution of driving scenes, allowing agents to imagine possible futures before acting. Inspired by human intelligence, which unifies understanding and imagination, we explore a unified model for autonomous driving. We present LMGenDrive, the first framework that combines LLM-based multimodal understanding with generative world models for end-to-end closed-loop driving. Given multi-view camera inputs and natural-language instructions, LMGenDrive generates both future driving videos and control signals. This design provides complementary benefits: video prediction improves spatio-temporal scene modeling, while the LLM contributes strong semantic priors and instruction grounding from large-scale pretraining. We further propose a progressive three-stage training strategy, from vision pretraining to multi-step long-horizon driving, to improve stability and performance. LMGenDrive supports both low-latency online planning and autoregressive offline video generation. Experiments show that it significantly outperforms prior methods on challenging closed-loop benchmarks, with clear gains in instruction following, spatio-temporal understanding, and robustness to rare scenarios. These results suggest that unifying multimodal understanding and generation is a promising direction for more generalizable and robust embodied decision-making systems.

cs.CV↗

arXiv2026

DriveDreamer-Policy: A Geometry-Grounded World-Action Model for Unified Generation and Planning

Recently, world-action models (WAM) have emerged to bridge vision-language-action (VLA) models and world models, unifying their reasoning and instruction-following capabilities and spatio-temporal world modeling. However, existing WAM approaches often focus on modeling 2D appearance or latent representations, with limited geometric grounding-an essential element for embodied systems operating in the physical world. We present DriveDreamer-Policy, a unified driving world-action model that integrates depth generation, future video generation, and motion planning within a single modular architecture. The model employs a large language model to process language instructions, multi-view images, and actions, followed by three lightweight generators that produce depth, future video, and actions. By learning a geometry-aware world representation and using it to guide both future prediction and planning within a unified framework, the proposed model produces more coherent imagined futures and more informed driving actions, while maintaining modularity and controllable latency. Experiments on the Navsim v1 and v2 benchmarks demonstrate that DriveDreamer-Policy achieves strong performance on both closed-loop planning and world generation tasks. In particular, our model reaches 89.2 PDMS on Navsim v1 and 88.7 EPDMS on Navsim v2, outperforming existing world-model-based approaches while producing higher-quality future video and depth predictions. Ablation studies further show that explicit depth learning provides complementary benefits to video imagination and improves planning robustness.

cs.CV↗

arXiv2026

THFM: A Unified Video Foundation Model for 4D Human Perception and Beyond

We present THFM, a unified video foundation model for human-centric perception that jointly addresses dense tasks (depth, normals, segmentation, dense pose) and sparse tasks (2d/3d keypoint estimation) within a single architecture. THFM is derived from a pretrained text-to-video diffusion model, repurposed as a single-forward-pass perception model and augmented with learnable tokens for sparse predictions. Modulated by the text prompt, our single unified model is capable of performing various perception tasks. Crucially, our model is on-par or surpassing state-of-the-art specialized models on a variety of benchmarks despite being trained exclusively on synthetic data (i.e.~without training on real-world or benchmark specific data). We further highlight intriguing emergent properties of our model, which we attribute to the underlying diffusion-based video representation. For example, our model trained on videos with a single human in the scene generalizes to multiple humans and other object classes such as anthropomorphic characters and animals -- a capability that hasn't been demonstrated in the past.

cs.CV↗

arXiv2026

SCATR: Mitigating New Instance Suppression in LiDAR-based Tracking-by-Attention via Second Chance Assignment and Track Query Dropout

LiDAR-based tracking-by-attention (TBA) frameworks inherently suffer from high false negative errors, leading to a significant performance gap compared to traditional LiDAR-based tracking-by-detection (TBD) methods. This paper introduces SCATR, a novel LiDAR-based TBA model designed to address this fundamental challenge systematically. SCATR leverages recent progress in vision-based tracking and incorporates targeted training strategies specifically adapted for LiDAR. Our work's core innovations are two architecture-agnostic training strategies for TBA methods: Second Chance Assignment and Track Query Dropout. Second Chance Assignment is a novel ground truth assignment that concatenates unassigned track queries to the proposal queries before bipartite matching, giving these track queries a second chance to be assigned to a ground truth object and effectively mitigating the conflict between detection and tracking tasks inherent in tracking-by-attention. Track Query Dropout is a training method that diversifies supervised object query configurations to efficiently train the decoder to handle different track query sets, enhancing robustness to missing or newborn tracks. Experiments on the nuScenes tracking benchmark demonstrate that SCATR achieves state-of-the-art performance among LiDAR-based TBA methods, outperforming previous works by 7.6\% AMOTA and successfully bridging the long-standing performance gap between LiDAR-based TBA and TBD methods. Ablation studies further validate the effectiveness and generalization of Second Chance Assignment and Track Query Dropout. Code can be found at the following link: \href{https://github.com/TRAILab/SCATR}{https://github.com/TRAILab/SCATR}

cs.CV↗

arXiv2026

DrivingGen: A Comprehensive Benchmark for Generative Video World Models in Autonomous Driving

Video generation models, as one form of world models, have emerged as one of the most exciting frontiers in AI, promising agents the ability to imagine the future by modeling the temporal evolution of complex scenes. In autonomous driving, this vision gives rise to driving world models: generative simulators that imagine ego and agent futures, enabling scalable simulation, safe testing of corner cases, and rich synthetic data generation. Yet, despite fast-growing research activity, the field lacks a rigorous benchmark to measure progress and guide priorities. Existing evaluations remain limited: generic video metrics overlook safety-critical imaging factors; trajectory plausibility is rarely quantified; temporal and agent-level consistency is neglected; and controllability with respect to ego conditioning is ignored. Moreover, current datasets fail to cover the diversity of conditions required for real-world deployment. To address these gaps, we present DrivingGen, the first comprehensive benchmark for generative driving world models. DrivingGen combines a diverse evaluation dataset curated from both driving datasets and internet-scale video sources, spanning varied weather, time of day, geographic regions, and complex maneuvers, with a suite of new metrics that jointly assess visual realism, trajectory plausibility, temporal coherence, and controllability. Benchmarking 14 state-of-the-art models reveals clear trade-offs: general models look better but break physics, while driving-specific ones capture motion realistically but lag in visual quality. DrivingGen offers a unified evaluation framework to foster reliable, controllable, and deployable driving world models, enabling scalable simulation, planning, and data-driven decision-making.

cs.CV↗

arXiv2026

SN 2024abvb: A Type Icn Supernova in the Outskirts of its Host Galaxy

We present multiband photometric and spectroscopic observations of supernova (SN) 2024abvb, which exhibits early-time prominent photoionized narrow emission lines of C II superposed on a blue continuum. The absence of Balmer features indicates that the SN exploded within hydrogen-poor circumstellar matter (CSM). Together with the lack of explicit evidence of helium signatures, we tentatively identify SN 2024abvb as a Type Icn SN (SN Icn). After correcting for extinction, we estimate an r-band peak absolute magnitude of -19.7, placing SN 2024abvb in the luminous regime of SNe Icn. We adopted a hybrid model that accounts for both the energy released by the ejecta-CSM interaction and the radioactive decay of nickel synthesized in the SN ejecta to fit the light curve of SN 2024abvb. The best-fit model to the multiband light curves within the first ~ 40 days after explosion suggests that the CSM, radioactive nickel, and ejecta masses to be 0.28 Msun, < 3.8 * 10^-2 Msun, and 0.12 Msun, respectively. Such a low ejecta mass indicates that the progenitor star of SN 2024abvb experienced a significant mass-stripping process, consistent with the hydrogen-poor and helium-poor spectral features. SN 2024abvb provides important insights into the physical origins of the rare subclass of SNe Icn.

astro-ph.HE↗

arXiv2025

OmniGen: Unified Multimodal Sensor Generation for Autonomous Driving

Autonomous driving has seen remarkable advancements, largely driven by extensive real-world data collection. However, acquiring diverse and corner-case data remains costly and inefficient. Generative models have emerged as a promising solution by synthesizing realistic sensor data. However, existing approaches primarily focus on single-modality generation, leading to inefficiencies and misalignment in multimodal sensor data. To address these challenges, we propose OminiGen, which generates aligned multimodal sensor data in a unified framework. Our approach leverages a shared Bird\u2019s Eye View (BEV) space to unify multimodal features and designs a novel generalizable multimodal reconstruction method, UAE, to jointly decode LiDAR and multi-view camera data. UAE achieves multimodal sensor decoding through volume rendering, enabling accurate and flexible reconstruction. Furthermore, we incorporate a Diffusion Transformer (DiT) with a ControlNet branch to enable controllable multimodal sensor generation. Our comprehensive experiments demonstrate that OminiGen achieves desired performances in unified multimodal sensor data generation with multimodal consistency and flexible sensor adjustments.

cs.CV↗

arXiv2025

EP250827b/SN 2025wkm: An X-ray Flash-Supernova Powered by a Central Engine and Circumstellar Interaction

We present the discovery of EP250827b/SN 2025wkm, an X-ray Flash (XRF) discovered by the Einstein Probe (EP), accompanied by a broad-line Type Ic supernova (SN Ic-BL) at $z = 0.1194$. EP250827b possesses a prompt X-ray luminosity of $\sim 10^{45} \, \rm{erg \, s^{-1}}$, lasts over 1000 seconds, and has a peak energy $E_{\rm{p}} < 1.5$ keV at 90\% confidence. SN 2025wkm possesses a double-peaked optical light curve (LC), though its bolometric luminosity plateaus after its initial peak for $\sim 20$ days, consistent with a central engine injecting additional energy into the explosion. Its spectrum transitions from a blue to red continuum with clear blueshifted broad absorption features consistent with a SN Ic-BL classification. We do not detect any transient radio emission and rule out the existence of an on-axis, energetic jet $\gtrsim 10^{50}~$erg assuming a typical LGRB circumburst constant density ($n \approx 10^{-3}$--$10^{-1}~{\rm cm}^{-3}$) and microphysical parameters ($\epsilon_{\rm e} = 0.1$ and $\epsilon_{\rm B} = 0.01$). In the model we invoke, the collapse gives rise to a long-lived magnetar, potentially surrounded by an accretion disk. Magnetically--driven winds from the magnetar and the disk mix together and break out with a velocity $\sim 0.35c$ and interact with an extended circumstellar medium with radius $\sim 10^{13}$ cm, generating X-ray breakout emission through non-thermal free-free processes. The disk outflows and magnetar winds power blackbody photospheric emission as they cool adiabatically and thermalize, producing the first SN peak. The spin-down luminosity of the magnetar and radioactive decay of $^{56}$Ni powers the late-time emission. We end by discussing the landscape of XRF-SNe within the context of EP's recent discoveries.

astro-ph.HE↗

arXiv2025

MiroThinker: Pushing the Performance Boundaries of Open-Source Research Agents via Model, Context, and Interactive Scaling

We present MiroThinker v1.0, an open-source research agent designed to advance tool-augmented reasoning and information-seeking capabilities. Unlike previous agents that only scale up model size or context length, MiroThinker explores interaction scaling at the model level, systematically training the model to handle deeper and more frequent agent-environment interactions as a third dimension of performance improvement. Unlike LLM test-time scaling, which operates in isolation and risks degradation with longer reasoning chains, interactive scaling leverages environment feedback and external information acquisition to correct errors and refine trajectories. Through reinforcement learning, the model achieves efficient interaction scaling: with a 256K context window, it can perform up to 600 tool calls per task, enabling sustained multi-turn reasoning and complex real-world research workflows. Across four representative benchmarks-GAIA, HLE, BrowseComp, and BrowseComp-ZH-the 72B variant achieves up to 81.9%, 37.7%, 47.1%, and 55.6% accuracy respectively, surpassing previous open-source agents and approaching commercial counterparts such as GPT-5-high. Our analysis reveals that MiroThinker benefits from interactive scaling consistently: research performance improves predictably as the model engages in deeper and more frequent agent-environment interactions, demonstrating that interaction depth exhibits scaling behaviors analogous to model size and context length. These findings establish interaction scaling as a third critical dimension for building next-generation open research agents, complementing model capacity and context windows.

cs.CL↗

arXiv2025

SN 2024iss: A Double-peaked Type IIb Supernova with Evidence of Circumstellar Interaction

We present optical, ultraviolet, and X-ray observations of supernova (SN) 2024iss, a Type IIb SN that shows a prominent double-peaked light curve. We modeled the first peak with a semianalytical shock-cooling model and the X-ray emission with a free-free model. We compare the envelope radius and mass-loss rate with other Type IIb SNe to explore the relationships between the progenitor envelope and the circumstellar material (CSM). The shock-cooling peak in the $V$-band light curve reached $M_V = -17.33\pm 0.26$mag, while the $^{56}$Ni-powered second peak attained $M_V = -17.43\pm 0.26$mag. Early spectra show an photospheric velocity of $\sim19,400\,km\,s^{-1}$ at 3.82days from the H$\alpha$ P~Cygni profile. The Balmer lines persist at least +87 days after the explosion, characterizing hydrogen-rich ejecta. Modeling the first light-curve peak suggests an extended envelope with a mass of $0.11\pm0.04\,M_{\odot}$ and a radius of $244\pm43~R_{\odot}$. Fitting the second light-curve peak with an Arnett-like model indicates a typical $^{56}$Ni mass of $ 0.117\pm0.013~M_{\odot}$ and a relatively low ejecta mass of $1.272\pm0.343\,M_{\odot}$. X-ray observations reveal bright thermal bremsstrahlung emission and indicate a mass-loss rate of $1.6\times10^{-5}\ M_{\odot} \ \rm{yr}^{-1}$. SN 2024iss occupies a transitional position between the two subclasses of extended (eIIb) and compact (cIIb) Type IIb SNe. Its envelope radius and pre-explosion mass-loss rate appear to be correlated as theoretically predicted. The observational properties of SN 2024iss are compatible with a binary interaction scenario being the dominant mechanism for envelope stripping. Furthermore, the low column density of neutral hydrogen suggests a compact CSM with an outer radius of $\lesssim1.3\times10^{14}$ cm, indicating that the progenitor star experienced eruptive mass loss within $\sim4\,yr$ of its terminal explosion.

astro-ph.HE↗

arXiv2025

Practice on Long Behavior Sequence Modeling in Tencent Advertising

Long-sequence modeling has become an indispensable frontier in recommendation systems for capturing users' long-term preferences. However, user behaviors within advertising domains are inherently sparse, posing a significant barrier to constructing long behavioral sequences using data from a single advertising domain alone. This motivates us to collect users' behaviors not only across diverse advertising scenarios, but also beyond the boundaries of the advertising domain into content domains-thereby constructing unified commercial behavior trajectories. This cross-domain or cross-scenario integration gives rise to the following challenges: (1) feature taxonomy gaps between distinct scenarios and domains, (2) inter-field interference arising from irrelevant feature field pairs, and (3) target-wise interference in temporal and semantic patterns when optimizing for different advertising targets. To address these challenges, we propose several practical approaches within the two-stage framework for long-sequence modeling. In the first (search) stage, we design a hierarchical hard search method for handling complex feature taxonomy hierarchies, alongside a decoupled embedding-based soft search to alleviate conflicts between attention mechanisms and feature representation. In the second (sequence modeling) stage, we introduce: (a) Decoupled Side Information Temporal Interest Networks (TIN) to mitigate inter-field conflicts; (b) Target-Decoupled Positional Encoding and Target-Decoupled SASRec to address target-wise interference; and (c) Stacked TIN to model high-order behavioral correlations. Deployed in production on Tencent's large-scale advertising platforms, our innovations delivered significant performance gains: an overall 4.22% GMV lift in WeChat Channels and an overall 1.96% GMV increase in WeChat Moments.

cs.IR↗

arXiv2025

Seeing and Meteorological Analysis at the North-1 and North-2 Points of Muztagh-Ata Site on the Pamir Plateau

To support the selection of large optical/infrared telescope sites in western China, long-term monitoring of atmospheric conditions and astronomical seeing has been conducted at the Muztagh-Ata site on the Pamir Plateau since 2017. With the monitoring focus gradually shifting northward, three stations were established: the South Point, North-1 point, and North-2 point. The North-1 point,selected as the site for the Muztagh-Ata 1.93 m Synergy Telescope (MOST), has recorded seeing and meteorological parameters since late 2018. In 2023,the North-2 point was established approximately 1.5 km northeast of North-1 point as a candidate location for a future large-aperture telescope. A 10m DIMM tower and a PC-4A environmental monitoring system were installed to evaluate site quality. This study presents a comparative analysis of data from the North-1 and North-2 points during 2018-2024.The median seeing is 0.89 arcsecs at North-1 and 0.78 arcsecs at North-2. Both points show clear seasonal and diurnal variations,with winter nights offering optimal observing conditions.On average, about 64% of the nighttime duration per year is suitable for astronomical observations. Nighttime temperature variation is low :2.03 at North-1 and 2.10 at North-2 .Median wind speeds are 5-6 m/s, with dominant directions between 210 and 300, contributing to stable airflow. Moderate wind suppresses turbulence, while strong shear and rapid fluctuations degrade image quality. These findings confirm that both the North-1 and North-2 points offer high-quality atmospheric conditions and serve as promising sites for future ground-based optical/infrared telescopes in western China.

astro-ph.IM↗

arXiv2025

ForeSight: Multi-View Streaming Joint Object Detection and Trajectory Forecasting

We introduce ForeSight, a novel joint detection and forecasting framework for vision-based 3D perception in autonomous vehicles. Traditional approaches treat detection and forecasting as separate sequential tasks, limiting their ability to leverage temporal cues. ForeSight addresses this limitation with a multi-task streaming and bidirectional learning approach, allowing detection and forecasting to share query memory and propagate information seamlessly. The forecast-aware detection transformer enhances spatial reasoning by integrating trajectory predictions from a multiple hypothesis forecast memory queue, while the streaming forecast transformer improves temporal consistency using past forecasts and refined detections. Unlike tracking-based methods, ForeSight eliminates the need for explicit object association, reducing error propagation with a tracking-free model that efficiently scales across multi-frame sequences. Experiments on the nuScenes dataset show that ForeSight achieves state-of-the-art performance, achieving an EPA of 54.9%, surpassing previous methods by 9.3%, while also attaining the best mAP and minADE among multi-view detection and forecasting models.

cs.CV↗

arXiv2025

OpenNav: Open-World Navigation with Multimodal Large Language Models

Pre-trained large language models (LLMs) have demonstrated strong common-sense reasoning abilities, making them promising for robotic navigation and planning tasks. However, despite recent progress, bridging the gap between language descriptions and actual robot actions in the open-world, beyond merely invoking limited predefined motion primitives, remains an open challenge. In this work, we aim to enable robots to interpret and decompose complex language instructions, ultimately synthesizing a sequence of trajectory points to complete diverse navigation tasks given open-set instructions and open-set objects. We observe that multi-modal large language models (MLLMs) exhibit strong cross-modal understanding when processing free-form language instructions, demonstrating robust scene comprehension. More importantly, leveraging their code-generation capability, MLLMs can interact with vision-language perception models to generate compositional 2D bird-eye-view value maps, effectively integrating semantic knowledge from MLLMs with spatial information from maps to reinforce the robot's spatial understanding. To further validate our approach, we effectively leverage large-scale autonomous vehicle datasets (AVDs) to validate our proposed zero-shot vision-language navigation framework in outdoor navigation tasks, demonstrating its capability to execute a diverse range of free-form natural language navigation instructions while maintaining robustness against object detection errors and linguistic ambiguities. Furthermore, we validate our system on a Husky robot in both indoor and outdoor scenes, demonstrating its real-world robustness and applicability. Supplementary videos are available at https://trailab.github.io/OpenNav-website/

cs.RO↗