arXiv subjects

Letian Wang

Publications and source records attributed to Letian Wang.

At least 37 records · Page 2Linked to original sources

arXiv2025

ForeSight: Multi-View Streaming Joint Object Detection and Trajectory Forecasting

We introduce ForeSight, a novel joint detection and forecasting framework for vision-based 3D perception in autonomous vehicles. Traditional approaches treat detection and forecasting as separate sequential tasks, limiting their ability to leverage temporal cues. ForeSight addresses this limitation with a multi-task streaming and bidirectional learning approach, allowing detection and forecasting to share query memory and propagate information seamlessly. The forecast-aware detection transformer enhances spatial reasoning by integrating trajectory predictions from a multiple hypothesis forecast memory queue, while the streaming forecast transformer improves temporal consistency using past forecasts and refined detections. Unlike tracking-based methods, ForeSight eliminates the need for explicit object association, reducing error propagation with a tracking-free model that efficiently scales across multi-frame sequences. Experiments on the nuScenes dataset show that ForeSight achieves state-of-the-art performance, achieving an EPA of 54.9%, surpassing previous methods by 9.3%, while also attaining the best mAP and minADE among multi-view detection and forecasting models.

cs.CV↗

arXiv2025

OpenNav: Open-World Navigation with Multimodal Large Language Models

Pre-trained large language models (LLMs) have demonstrated strong common-sense reasoning abilities, making them promising for robotic navigation and planning tasks. However, despite recent progress, bridging the gap between language descriptions and actual robot actions in the open-world, beyond merely invoking limited predefined motion primitives, remains an open challenge. In this work, we aim to enable robots to interpret and decompose complex language instructions, ultimately synthesizing a sequence of trajectory points to complete diverse navigation tasks given open-set instructions and open-set objects. We observe that multi-modal large language models (MLLMs) exhibit strong cross-modal understanding when processing free-form language instructions, demonstrating robust scene comprehension. More importantly, leveraging their code-generation capability, MLLMs can interact with vision-language perception models to generate compositional 2D bird-eye-view value maps, effectively integrating semantic knowledge from MLLMs with spatial information from maps to reinforce the robot's spatial understanding. To further validate our approach, we effectively leverage large-scale autonomous vehicle datasets (AVDs) to validate our proposed zero-shot vision-language navigation framework in outdoor navigation tasks, demonstrating its capability to execute a diverse range of free-form natural language navigation instructions while maintaining robustness against object detection errors and linguistic ambiguities. Furthermore, we validate our system on a Husky robot in both indoor and outdoor scenes, demonstrating its real-world robustness and applicability. Supplementary videos are available at https://trailab.github.io/OpenNav-website/

cs.RO↗

arXiv2025

An H$α$ Cloud in the HI Tail: Recent Star Formation in the Outskirts of NGC 4258 Revealed by Nanshan 1-m Telescope

We present first-light deep H$α$ imaging taken with the Nanshan 1-meter wide-field telescope on the local galaxy NGC 4258, alongside archival data from Hubble Space telescope (HST), Westerbork Synthesis Radio Telescope, and The Dark Energy Camera Legacy Survey. The H$α$ image shows ongoing star formation not only inside the galaxy but also in an HI cloud in the eastern HI tail, which is roughly 16 kpc away from the main galaxy. The HST images reveal several ultra-blue compact objects ($\rm F555W - F814W <-0.5 mag,\, FWHM\sim 0.2''$) in the H$α$ bright region, aligned with the HI tail, suggesting the presence of young open cluster candidate in the HI tail. Our results suggest that wide field H$α$ imaging is a valuable tool for investigating recent star formation in the extended regions of NGC 4258. Furthermore, the star formation in diffuse HI tails could highlight an potential aspect of galaxy stellar halo formation, warranting further investigation of the impact of star formation in halos on galaxy evolution.

astro-ph.GA↗

arXiv2025

TEM^3-Learning: Time-Efficient Multimodal Multi-Task Learning for Advanced Assistive Driving

Multi-task learning (MTL) can advance assistive driving by exploring inter-task correlations through shared representations. However, existing methods face two critical limitations: single-modality constraints limiting comprehensive scene understanding and inefficient architectures impeding real-time deployment. This paper proposes TEM^3-Learning (Time-Efficient Multimodal Multi-task Learning), a novel framework that jointly optimizes driver emotion recognition, driver behavior recognition, traffic context recognition, and vehicle behavior recognition through a two-stage architecture. The first component, the mamba-based multi-view temporal-spatial feature extraction subnetwork (MTS-Mamba), introduces a forward-backward temporal scanning mechanism and global-local spatial attention to efficiently extract low-cost temporal-spatial features from multi-view sequential images. The second component, the MTL-based gated multimodal feature integrator (MGMI), employs task-specific multi-gating modules to adaptively highlight the most relevant modality features for each task, effectively alleviating the negative transfer problem in MTL. Evaluation on the AIDE dataset, our proposed model achieves state-of-the-art accuracy across all four tasks, maintaining a lightweight architecture with fewer than 6 million parameters and delivering an impressive 142.32 FPS inference speed. Rigorous ablation studies further validate the effectiveness of the proposed framework and the independent contributions of each module. The code is available on https://github.com/Wenzhuo-Liu/TEM3-Learning.

cs.CV↗

arXiv2025

SN 2021hpr: A Normal Type Ia Supernova Showing Excess Emission in the Early Rising Phase

We present extensive optical observations of a nearby Type Ia supernova (SN Ia), SN 2021hpr, located in the spiral galaxy NGC 3147 at a distance of $\sim$ 45 Mpc. Our observations cover a phase within $\sim 1-2$ days to $\sim 290$ days after the explosion. SN 2021hpr is found to be a spectroscopically normal SN Ia, with an absolute B-band peak magnitude of $M_{max}(B) \approx -19.16 \pm 0.14$ mag and a post-peak decline rate of $Δm_{15}(B)= 1.00 \pm 0.01 $ mag. Early-time light curves showed a $\sim 7.0 \%$ excess emission compared to a homogeneously expanding fireball model, likely due to SN ejecta interacting with a companion or immediate circumstellar matter. The optical spectra of SN 2021hpr are overall similar to those of normal SNe Ia, but characterized by prominent detached high-velocity features (HVFs) of Si {\sc ii} and Ca {\sc ii} in the early phase. After examining a small sample of well-observed normal SNe Ia, we find that the HVFs are likely common for the subgroup with early-excess emission. The association of early bump feature with the HVFs could be attributed to density or abundance enhancement at the outer layer of the exploding star, likely as a result of interactions with companion$/$CSM or experiencing more complete burning. Nevertheless, the redshifted Fe {\sc ii} and Ni {\sc ii} lines in the nebular-phase spectra of SN 2021hpr, contrary to the blueshift trend seen in other SNe Ia showing early bump features, indicate its peculiarity in the explosion that remains to be understood.

astro-ph.HE↗

arXiv2025

SmartPretrain: Model-Agnostic and Dataset-Agnostic Representation Learning for Motion Prediction

Predicting the future motion of surrounding agents is essential for autonomous vehicles (AVs) to operate safely in dynamic, human-robot-mixed environments. However, the scarcity of large-scale driving datasets has hindered the development of robust and generalizable motion prediction models, limiting their ability to capture complex interactions and road geometries. Inspired by recent advances in natural language processing (NLP) and computer vision (CV), self-supervised learning (SSL) has gained significant attention in the motion prediction community for learning rich and transferable scene representations. Nonetheless, existing pre-training methods for motion prediction have largely focused on specific model architectures and single dataset, limiting their scalability and generalizability. To address these challenges, we propose SmartPretrain, a general and scalable SSL framework for motion prediction that is both model-agnostic and dataset-agnostic. Our approach integrates contrastive and reconstructive SSL, leveraging the strengths of both generative and discriminative paradigms to effectively represent spatiotemporal evolution and interactions without imposing architectural constraints. Additionally, SmartPretrain employs a dataset-agnostic scenario sampling strategy that integrates multiple datasets, enhancing data volume, diversity, and robustness. Extensive experiments on multiple datasets demonstrate that SmartPretrain consistently improves the performance of state-of-the-art prediction models across datasets, data splits and main metrics. For instance, SmartPretrain significantly reduces the MissRate of Forecast-MAE by 10.6%. These results highlight SmartPretrain's effectiveness as a unified, scalable solution for motion prediction, breaking free from the limitations of the small-data regime. Codes are available at https://github.com/youngzhou1999/SmartPretrain

cs.CV↗

arXiv2024

Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning

Multi-Modal Large Language Models (MLLMs) have demonstrated impressive performance in various VQA tasks. However, they often lack interpretability and struggle with complex visual inputs, especially when the resolution of the input image is high or when the interested region that could provide key information for answering the question is small. To address these challenges, we collect and introduce the large-scale Visual CoT dataset comprising 438k question-answer pairs, annotated with intermediate bounding boxes highlighting key regions essential for answering the questions. Additionally, about 98k pairs of them are annotated with detailed reasoning steps. Importantly, we propose a multi-turn processing pipeline that dynamically focuses on visual inputs and provides interpretable thoughts. We also introduce the related benchmark to evaluate the MLLMs in scenarios requiring specific local region identification. Extensive experiments demonstrate the effectiveness of our framework and shed light on better inference strategies. The Visual CoT dataset, benchmark, and pre-trained models are available on https://hao-shao.com/projects/viscot.html to support further research in this area.

cs.CV↗

arXiv2024

Optical turbulence in the atmospheric surface layer at the Pamir Plateau Muztagh-ata site

In this paper, we conducted a detailed analysis of optical turbulence in the Atmospheric Surface Layer (ASL) at Muztagh-ata site during on-site testing. We utilized ultrasonic anemometers positioned on a 30-meter tower to collect and process data at five height levels, obtaining data from October 1, 2021 to the present. We investigated the behavior of optical turbulence parameters ($C_n^2$ and seeing $\varepsilon$) in the ASL. Nighttime $C_n^2$ primarily fluctuated in the range of $10^{-16}$ to $10^{-14}$, exhibiting an exponential decrease with height. During the day, it showed a $h^{-0.82}$ dependency, while at night, it displayed a $h^{-0.48}$ dependency. Additionally, we presented the distribution of seeing across different layers within the ASL, showing a gradual decrease with increasing height, with a median seeing of 0.24 arcseconds at nighttime and 0.48 arcseconds at daytime between 6-30m. We investigated the relationship between surface temperature inversion, seeing in the ASL, and wind speed at the site. Our results show that under temperature inversion conditions, seeing significantly improves and is often accompanied by low to moderate wind speeds, while high wind speeds are usually associated with poorer seeing. Preliminary calculations and observational results, combined with the high altitude and unique geographical location, suggest that Muztagh-ata site has the potential to be an outstanding optical astronomical observatory in the western plateau of china.

physics.ao-ph↗

arXiv2024

DistillNeRF: Perceiving 3D Scenes from Single-Glance Images by Distilling Neural Fields and Foundation Model Features

We propose DistillNeRF, a self-supervised learning framework addressing the challenge of understanding 3D environments from limited 2D observations in outdoor autonomous driving scenes. Our method is a generalizable feedforward model that predicts a rich neural scene representation from sparse, single-frame multi-view camera inputs with limited view overlap, and is trained self-supervised with differentiable rendering to reconstruct RGB, depth, or feature images. Our first insight is to exploit per-scene optimized Neural Radiance Fields (NeRFs) by generating dense depth and virtual camera targets from them, which helps our model to learn enhanced 3D geometry from sparse non-overlapping image inputs. Second, to learn a semantically rich 3D representation, we propose distilling features from pre-trained 2D foundation models, such as CLIP or DINOv2, thereby enabling various downstream tasks without the need for costly 3D human annotations. To leverage these two insights, we introduce a novel model architecture with a two-stage lift-splat-shoot encoder and a parameterized sparse hierarchical voxel representation. Experimental results on the NuScenes and Waymo NOTR datasets demonstrate that DistillNeRF significantly outperforms existing comparable state-of-the-art self-supervised methods for scene reconstruction, novel view synthesis, and depth estimation; and it allows for competitive zero-shot 3D semantic occupancy prediction, as well as open-world scene understanding through distilled foundation model features. Demos and code will be available at https://distillnerf.github.io/.

cs.CV↗

arXiv2024

The first low-mass eclipsing binary within the fully convective zone from TMTS

We present a comprehensive photometric and spectroscopic analysis of the short-period ($\sim$5.32 hours) and low-mass eclipsing binary TMTSJ0803 discovered by Tsinghua-Ma Huateng Telescope for Survey (TMTS). By fitting the light curves and radial velocity data with the Wilson--Devinney code, we find that the binary is composed of two late spotted active M dwarfs below the fully convective boundary. This is supported by the discovery of a significant Balmer emission lines in the LAMOST spectrum and prominent coronal X-ray emission. In comparison with the typical luminosity of rapidly rotating fully convective stars, the much brighter X-ray luminosity ($L_{X}/L_{\rm{bol}} = 0.0159 \pm 0.0059$) suggests the stellar magnetic activity of fully convective stars could be enhanced in such a close binary system. Given the metallicity of [M/H] = $-$ 0.35 dex as inferred from the LAMOST spectrum, we measure the masses and radii of both stars to be $M_{1} = 0.169 \pm 0.010~M_{\odot}$, $M_{2} = 0.162 \pm 0.016~M_{\odot}$, $R_{1} = 0.170 \pm 0.006~R_{\odot}$, and $R_{2} = 0.156 \pm 0.006~R_{\odot}$, respectively. Based on the luminosity ratio from the light curve modeling, the effective temperatures of two components are also estimated. In comparison with the stellar evolution models, the radii and effective temperatures of two components are all below the isochrones. The radius deflation might be mainly biased by a small radial velocity (RV) data or (and) a simple correction on RVs, while the discrepancy in effective temperature might be due to the enhanced magnetic activity in this binary.

astro-ph.SR↗

arXiv2024

SmartRefine: A Scenario-Adaptive Refinement Framework for Efficient Motion Prediction

Predicting the future motion of surrounding agents is essential for autonomous vehicles (AVs) to operate safely in dynamic, human-robot-mixed environments. Context information, such as road maps and surrounding agents' states, provides crucial geometric and semantic information for motion behavior prediction. To this end, recent works explore two-stage prediction frameworks where coarse trajectories are first proposed, and then used to select critical context information for trajectory refinement. However, they either incur a large amount of computation or bring limited improvement, if not both. In this paper, we introduce a novel scenario-adaptive refinement strategy, named SmartRefine, to refine prediction with minimal additional computation. Specifically, SmartRefine can comprehensively adapt refinement configurations based on each scenario's properties, and smartly chooses the number of refinement iterations by introducing a quality score to measure the prediction quality and remaining refinement potential of each scenario. SmartRefine is designed as a generic and flexible approach that can be seamlessly integrated into most state-of-the-art motion prediction models. Experiments on Argoverse (1 & 2) show that our method consistently improves the prediction accuracy of multiple state-of-the-art prediction models. Specifically, by adding SmartRefine to QCNet, we outperform all published ensemble-free works on the Argoverse 2 leaderboard (single agent track) at submission. Comprehensive studies are also conducted to ablate design choices and explore the mechanism behind multi-iteration refinement. Codes are available at https://github.com/opendilab/SmartRefine/

cs.CV↗

arXiv2024

Variable white dwarfs in TMTS: Asteroseismological analysis of a ZZ Ceti star, TMTS J17184064+2524314

The Tsinghua University-Ma Huateng Telescope for Survey (TMTS) has been constantly monitoring the northern sky since 2020 in search of rapidly variable stars. To find variable white dwarfs (WDs), the TMTS catalog is cross-matched with the WD catalog of Gaia EDR3, resulting in over 3000 light curves of WD candidates. The WD TMTS J17184064+2524314 (hereafter J1718) is the second ZZ~Ceti star discovered among these common sources. Based on the light curves from TMTS, follow-up photometric observations, and TESS, 10 periods and 3 combination periods are detected. A rotation period of $25.12\pm0.18$ hr is derived, according to the identified rotational splitting. Our spectroscopic observation indicates that this WD belongs to DA type with $T_{\rm eff}=11,670\pm604$ K, log $g=8.16\pm0.36$, $M = 0.70\pm0.23$ M$_{\odot}$, and age=$0.51\pm0.34$ Gyr. Based on core-parameterized asteroseismological model grids ($\geqslant$ 14 million), we derive a best-fit solution of $T_{\rm eff}=11,640\pm20$ K, log $g=8.267\pm0.008$, and $M = 0.750\pm0.005$ M$_{\odot}$ for J1718, consistent with the spectral fitting results. For this WD, the corresponding carbon and oxygen abundances in the core are 0.43 and 0.57, respectively. The distance derived from the intrinsic luminosity given by asteroseismology is $64\pm15$ pc, in accord with the distance of $70.1\pm0.2$ pc from Gaia DR3 within the uncertainties.

astro-ph.SR↗

arXiv2023

LMDrive: Closed-Loop End-to-End Driving with Large Language Models

Despite significant recent progress in the field of autonomous driving, modern methods still struggle and can incur serious accidents when encountering long-tail unforeseen events and challenging urban scenarios. On the one hand, large language models (LLM) have shown impressive reasoning capabilities that approach "Artificial General Intelligence". On the other hand, previous autonomous driving methods tend to rely on limited-format inputs (e.g. sensor data and navigation waypoints), restricting the vehicle's ability to understand language information and interact with humans. To this end, this paper introduces LMDrive, a novel language-guided, end-to-end, closed-loop autonomous driving framework. LMDrive uniquely processes and integrates multi-modal sensor data with natural language instructions, enabling interaction with humans and navigation software in realistic instructional settings. To facilitate further research in language-based closed-loop autonomous driving, we also publicly release the corresponding dataset which includes approximately 64K instruction-following data clips, and the LangAuto benchmark that tests the system's ability to handle complex instructions and challenging driving scenarios. Extensive closed-loop experiments are conducted to demonstrate LMDrive's effectiveness. To the best of our knowledge, we're the very first work to leverage LLMs for closed-loop end-to-end autonomous driving. Codes, models, and datasets can be found at https://github.com/opendilab/LMDrive

cs.CV↗

arXiv2023

Properties and Asteroseismological analysis of a new ZZ ceti discovered by TMTS

Tsinghua university-Ma Huateng Telescope for Survey (TMTS) aims to discover rapidly evolving transients by monitoring the northern sky. The TMTS catalog is cross-matched with the white dwarf (WD) catalog of Gaia EDR3, and light curves of more than a thousand WD candidates are obtained so far. Among them, the WD TMTS J23450729+5813146 (hereafter J2345) is one interesting common source. Based on the light curves from the TMTS and follow-up photometric observations, periods of 967.113 s, 973.734 s, 881.525 s, 843.458 s, 806.916 s and 678.273 s are identified. In addition, the TESS observations suggest a 3.39 h period but this can be attributed to the rotation of a comoving M dwarf located within 3". The spectroscopic observation indicates that this WD is DA type with Teff = 11778+/-617K,log g = 8.38+/-0.31,mass=0.84+/-0.20Msun and age=0.704+/-0.377 Gyrs. Asteroseismological analysis reveals a global best-fit solution of Teff =12110+/-10K and mass=0.760+/-0.005Msun,consistent with the spectral fitting results, and Oxygen and Carbon abundances in the core center are 0.73 and 0.27, respectively. The distance derived from the intrinsic luminosity given by asteroseismology is 93 parsec, which is in agreement with the distance of 98 parsec from Gaia DR3. Additionally, kinematic study shows that this WD is likely a thick disk star. The mass of its zero-age main-sequence mass is estimated to be 3.08 Msun and has a main-sequence plus cooling age of roughly 900 Myrs.

astro-ph.SR↗

arXiv2023

ReasonNet: End-to-End Driving with Temporal and Global Reasoning

The large-scale deployment of autonomous vehicles is yet to come, and one of the major remaining challenges lies in urban dense traffic scenarios. In such cases, it remains challenging to predict the future evolution of the scene and future behaviors of objects, and to deal with rare adverse events such as the sudden appearance of occluded objects. In this paper, we present ReasonNet, a novel end-to-end driving framework that extensively exploits both temporal and global information of the driving scene. By reasoning on the temporal behavior of objects, our method can effectively process the interactions and relationships among features in different frames. Reasoning about the global information of the scene can also improve overall perception performance and benefit the detection of adverse events, especially the anticipation of potential danger from occluded objects. For comprehensive evaluation on occlusion events, we also release publicly a driving simulation benchmark DriveOcclusionSim consisting of diverse occlusion events. We conduct extensive experiments on multiple CARLA benchmarks, where our model outperforms all prior methods, ranking first on the sensor track of the public CARLA Leaderboard.

cs.CV↗

arXiv2023

Efficient Reinforcement Learning for Autonomous Driving with Parameterized Skills and Priors

When autonomous vehicles are deployed on public roads, they will encounter countless and diverse driving situations. Many manually designed driving policies are difficult to scale to the real world. Fortunately, reinforcement learning has shown great success in many tasks by automatic trial and error. However, when it comes to autonomous driving in interactive dense traffic, RL agents either fail to learn reasonable performance or necessitate a large amount of data. Our insight is that when humans learn to drive, they will 1) make decisions over the high-level skill space instead of the low-level control space and 2) leverage expert prior knowledge rather than learning from scratch. Inspired by this, we propose ASAP-RL, an efficient reinforcement learning algorithm for autonomous driving that simultaneously leverages motion skills and expert priors. We first parameterized motion skills, which are diverse enough to cover various complex driving scenarios and situations. A skill parameter inverse recovery method is proposed to convert expert demonstrations from control space to skill space. A simple but effective double initialization technique is proposed to leverage expert priors while bypassing the issue of expert suboptimality and early performance degradation. We validate our proposed method on interactive dense-traffic driving tasks given simple and sparse rewards. Experimental results show that our method can lead to higher learning efficiency and better driving performance relative to previous methods that exploit skills and priors differently. Code is open-sourced to facilitate further research.

cs.RO↗

arXiv2022

Safety-Enhanced Autonomous Driving Using Interpretable Sensor Fusion Transformer

Large-scale deployment of autonomous vehicles has been continually delayed due to safety concerns. On the one hand, comprehensive scene understanding is indispensable, a lack of which would result in vulnerability to rare but complex traffic situations, such as the sudden emergence of unknown objects. However, reasoning from a global context requires access to sensors of multiple types and adequate fusion of multi-modal sensor signals, which is difficult to achieve. On the other hand, the lack of interpretability in learning models also hampers the safety with unverifiable failure causes. In this paper, we propose a safety-enhanced autonomous driving framework, named Interpretable Sensor Fusion Transformer(InterFuser), to fully process and fuse information from multi-modal multi-view sensors for achieving comprehensive scene understanding and adversarial event detection. Besides, intermediate interpretable features are generated from our framework, which provide more semantics and are exploited to better constrain actions to be within the safe sets. We conducted extensive experiments on CARLA benchmarks, where our model outperforms prior methods, ranking the first on the public CARLA Leaderboard. Our code will be made available at https://github.com/opendilab/InterFuser

cs.CV↗

arXiv2022

Social Interactions for Autonomous Driving: A Review and Perspectives

No human drives a car in a vacuum; she/he must negotiate with other road users to achieve their goals in social traffic scenes. A rational human driver can interact with other road users in a socially-compatible way through implicit communications to complete their driving tasks smoothly in interaction-intensive, safety-critical environments. This paper aims to review the existing approaches and theories to help understand and rethink the interactions among human drivers toward social autonomous driving. We take this survey to seek the answers to a series of fundamental questions: 1) What is social interaction in road traffic scenes? 2) How to measure and evaluate social interaction? 3) How to model and reveal the process of social interaction? 4) How do human drivers reach an implicit agreement and negotiate smoothly in social interaction? This paper reviews various approaches to modeling and learning the social interactions between human drivers, ranging from optimization theory and graphical models to social force theory and behavioral & cognitive science. We also highlight some new directions, critical challenges, and opening questions for future research.

cs.RO↗