arXiv subjects

Weidong Sheng

Publications and source records attributed to Weidong Sheng.

13 recordsLinked to original sources

arXiv2026

Understand Before Detect: Vision--Language Learning for Omni-Domain Infrared Small Target Detection

Omni-domain infrared small target (IRST) detection is crucial for infrared surveillance, yet remains challenging due to heterogeneous imaging domains and inconsistent target characteristics. Previous deep learning-based methods have been developed for visual-only paradigms and achieved promising performance on domain-specific tasks. However, existing methods follow the task-specific supervised learning paradigm. This paradigm simplifies the full-scene infrared observations to sparse target supervision, discarding the semantics that remain invariant across heterogeneous domains. Consequently, detection performance suffers substantially under domain shifts. To handle this issue, we introduce \textbf{``understand before detect''}, a paradigm that formulates omni-domain IRST detection as an understanding-driven process, where holistic infrared target understanding precedes precise detection. Building on this paradigm, we propose \textbf{JinSight}, which first develops holistic IRST understanding through language supervision and then transfers the learned cross-domain representations to precise small-target detection. By grounding infrared representations in language semantics, JinSight enables a single model to generalize across heterogeneous infrared domains. We then introduce Latent Semantic Interaction (LSI), which exchanges language-aligned global semantics with fine-grained spatial features in a compact low-rank space. To address the lack of multimodal omni-domain IRST benchmarks, we build \textbf{OmniIRST-VL}, the first large-scale, highly diverse vision--language dataset for omni-domain IRST detection. It comprises over 39k annotations across six complementary instruction tasks covering both scene-level understanding and target-centric reasoning.

cs.CV↗

arXiv2026

Focus on What Really Matters in Low-Altitude Governance: A Management-Centric Multi-Modal Benchmark with Implicitly Coordinated Vision-Language Reasoning Framework

Low-altitude vision systems are becoming a critical infrastructure for smart city governance. However, existing object-centric perception paradigms and loosely coupled vision-language pipelines are still difficult to support management-oriented anomaly understanding required in real-world urban governance. To bridge this gap, we introduce GovLA-10K, the first management-oriented multi-modal benchmark for low-altitude intelligence, along with GovLA-Reasoner, a unified vision-language reasoning framework tailored for governance-aware aerial perception. Unlike existing studies that aim to exhaustively annotate all visible objects, GovLA-10K is deliberately designed around functionally salient targets that directly correspond to practical management needs, and further provides actionable management suggestions grounded in these observations. To effectively coordinate the fine-grained visual grounding with high-level contextual language reasoning, GovLA-Reasoner introduces an efficient Spatially-aware Grounding Adapter (SGA) that implicitly coordinates discriminative representation sharing between the visual detector and the large language model (LLM). Different from existing adapters that primarily focus on global embedding alignment, our SGA is specifically designed to compress and aggregate multi-stream grounding-aware representations, thereby preserving fine-grained spatial cues while enabling their effective integration into the language reasoning process. Extensive experiments indicate that our GovLA-Reasoner effectively improves performance while avoiding the need of fine-tuning for any task-specific individual components. We believe our work offers a new perspective and foundation for future studies on management-aware low-altitude vision-language systems. The code and dataset will be publicly released after further organization.

cs.CV↗

arXiv2024

Heterogeneous Graph Transformer for Multiple Tiny Object Tracking in RGB-T Videos

Tracking multiple tiny objects is highly challenging due to their weak appearance and limited features. Existing multi-object tracking algorithms generally focus on single-modality scenes, and overlook the complementary characteristics of tiny objects captured by multiple remote sensors. To enhance tracking performance by integrating complementary information from multiple sources, we propose a novel framework called {HGT-Track (Heterogeneous Graph Transformer based Multi-Tiny-Object Tracking)}. Specifically, we first employ a Transformer-based encoder to embed images from different modalities. Subsequently, we utilize Heterogeneous Graph Transformer to aggregate spatial and temporal information from multiple modalities to generate detection and tracking features. Additionally, we introduce a target re-detection module (ReDet) to ensure tracklet continuity by maintaining consistency across different modalities. Furthermore, this paper introduces the first benchmark VT-Tiny-MOT (Visible-Thermal Tiny Multi-Object Tracking) for RGB-T fused multiple tiny object tracking. Extensive experiments are conducted on VT-Tiny-MOT, and the results have demonstrated the effectiveness of our method. Compared to other state-of-the-art methods, our method achieves better performance in terms of MOTA (Multiple-Object Tracking Accuracy) and ID-F1 score. The code and dataset will be made available at https://github.com/xuqingyu26/HGTMT.

cs.CV↗

arXiv2024

Visible-Thermal Tiny Object Detection: A Benchmark Dataset and Baselines

Small object detection (SOD) has been a longstanding yet challenging task for decades, with numerous datasets and algorithms being developed. However, they mainly focus on either visible or thermal modality, while visible-thermal (RGBT) bimodality is rarely explored. Although some RGBT datasets have been developed recently, the insufficient quantity, limited category, misaligned images and large target size cannot provide an impartial benchmark to evaluate multi-category visible-thermal small object detection (RGBT SOD) algorithms. In this paper, we build the first large-scale benchmark with high diversity for RGBT SOD (namely RGBT-Tiny), including 115 paired sequences, 93K frames and 1.2M manual annotations. RGBT-Tiny contains abundant targets (7 categories) and high-diversity scenes (8 types that cover different illumination and density variations). Note that, over 81% of targets are smaller than 16x16, and we provide paired bounding box annotations with tracking ID to offer an extremely challenging benchmark with wide-range applications, such as RGBT fusion, detection and tracking. In addition, we propose a scale adaptive fitness (SAFit) measure that exhibits high robustness on both small and large targets. The proposed SAFit can provide reasonable performance evaluation and promote detection performance. Based on the proposed RGBT-Tiny dataset and SAFit measure, extensive evaluations have been conducted, including 23 recent state-of-the-art algorithms that cover four different types (i.e., visible generic detection, visible SOD, thermal SOD and RGBT object detection). Project is available at https://github.com/XinyiYing/RGBT-Tiny.

cs.CV↗

arXiv2023

Local Motion and Contrast Priors Driven Deep Network for Infrared Small Target Super-Resolution

Infrared small target super-resolution (SR) aims to recover reliable and detailed high-resolution image with high-contrast targets from its low-resolution counterparts. Since the infrared small target lacks color and fine structure information, it is significant to exploit the supplementary information among sequence images to enhance the target. In this paper, we propose the first infrared small target SR method named local motion and contrast prior driven deep network (MoCoPnet) to integrate the domain knowledge of infrared small target into deep network, which can mitigate the intrinsic feature scarcity of infrared small targets. Specifically, motivated by the local motion prior in the spatio-temporal dimension, we propose a local spatio-temporal attention module to perform implicit frame alignment and incorporate the local spatio-temporal information to enhance the local features (especially for small targets). Motivated by the local contrast prior in the spatial dimension, we propose a central difference residual group to incorporate the central difference convolution into the feature extraction backbone, which can achieve center-oriented gradient-aware feature extraction to further improve the target contrast. Extensive experiments have demonstrated that our method can recover accurate spatial dependency and improve the target contrast. Comparative results show that MoCoPnet can outperform the state-of-the-art video SR and single image SR methods in terms of both SR performance and target enhancement. Based on the SR results, we further investigate the influence of SR on infrared small target detection and the experimental results demonstrate that MoCoPnet promotes the detection performance. The code is available at https://github.com/XinyiYing/MoCoPnet.

eess.IV↗

arXiv2023

Mapping Degeneration Meets Label Evolution: Learning Infrared Small Target Detection with Single Point Supervision

Training a convolutional neural network (CNN) to detect infrared small targets in a fully supervised manner has gained remarkable research interests in recent years, but is highly labor expensive since a large number of per-pixel annotations are required. To handle this problem, in this paper, we make the first attempt to achieve infrared small target detection with point-level supervision. Interestingly, during the training phase supervised by point labels, we discover that CNNs first learn to segment a cluster of pixels near the targets, and then gradually converge to predict groundtruth point labels. Motivated by this "mapping degeneration" phenomenon, we propose a label evolution framework named label evolution with single point supervision (LESPS) to progressively expand the point label by leveraging the intermediate predictions of CNNs. In this way, the network predictions can finally approximate the updated pseudo labels, and a pixel-level target mask can be obtained to train CNNs in an end-to-end manner. We conduct extensive experiments with insightful visualizations to validate the effectiveness of our method. Experimental results show that CNNs equipped with LESPS can well recover the target masks from corresponding point labels, {and can achieve over 70% and 95% of their fully supervised performance in terms of pixel-level intersection over union (IoU) and object-level probability of detection (Pd), respectively. Code is available at https://github.com/XinyiYing/LESPS.

cs.CV↗

arXiv2020

Deformable 3D Convolution for Video Super-Resolution

The spatio-temporal information among video sequences is significant for video super-resolution (SR). However, the spatio-temporal information cannot be fully used by existing video SR methods since spatial feature extraction and temporal motion compensation are usually performed sequentially. In this paper, we propose a deformable 3D convolution network (D3Dnet) to incorporate spatio-temporal information from both spatial and temporal dimensions for video SR. Specifically, we introduce deformable 3D convolution (D3D) to integrate deformable convolution with 3D convolution, obtaining both superior spatio-temporal modeling capability and motion-aware modeling flexibility. Extensive experiments have demonstrated the effectiveness of D3D in exploiting spatio-temporal information. Comparative results show that our network achieves state-of-the-art SR performance. Code is available at: https://github.com/XinyiYing/D3Dnet.

cs.CV↗

arXiv2015

Observation of coupling between zero- and two-dimensional semiconductor systems based on anomalous diamagnetic effects

We report the direct observation of coupling between a single self-assembled InAs quantum dot and a wetting layer, based on strong diamagnetic shifts of many-body exciton states using magneto-photoluminescence spectroscopy. An extremely large positive diamagnetic coefficient is observed when an electron in the wetting layer combines with a hole in the quantum dot; the coefficient is nearly one order of magnitude larger than that of the exciton states confined in the quantum dots. Recombination of electrons with holes in a quantum dot of the coupled system leads to an unusual negative diamagnetic effect, which is five times stronger than that in a pure quantum dot system. This effect can be attributed to the expansion of the wavefunction of remaining electrons in the wetting layer or the spread of electrons in the excited states of the quantum dot to the wetting layer after recombination. In this case, the wavefunction extent of the final states in the quantum dot plane is much larger than that of the initial states because of the absence of holes in the quantum dot to attract electrons. The properties of emitted photons that depend on the large electron wavefunction extents in the wetting layer indicate that the coupling occurs between systems of different dimensionality, which is also verified from the results obtained by applying a magnetic field in different configurations. This study paves a new way to observe hybrid states with zero- and two-dimensional structures, which could be useful for investigating the Kondo physics and implementing spin-based solid-state quantum information processing.

cond-mat.mes-hall↗

arXiv2015

Longitudinal wave function control in single quantum dots with an applied magnetic field

Controlling single-particle wave functions in single semiconductor quantum dots is in demand to implement solid-state quantum information processing and spintronics. Normally, particle wave functions can be tuned transversely by an perpendicular magnetic field. We report a longitudinal wave function control in single quantum dots with a magnetic field. For a pure InAs quantum dot with a shape of pyramid or truncated pyramid, the hole wave function always occupies the base because of the less confinement at base, which induces a permanent dipole oriented from base to apex. With applying magnetic field along the base-apex direction, the hole wave function shrinks in the base plane. Because of the linear changing of the confinement for hole wave function from base to apex, the center of effective mass moves up during shrinking process. Due to the uniform confine potential for electrons, the center of effective mass of electrons does not move much, which results in a permanent dipole moment change and an inverted electron-hole alignment along the magnetic field direction. Manipulating the wave function longitudinally not only provides an alternative way to control the charge distribution with magnetic field but also a new method to tune electron-hole interaction in single quantum dots.

cond-mat.mes-hall↗

arXiv2005

Atomistic theory of electronic and optical properties of InAs/InP self-assembled quantum dots on patterned substrates

We report on a atomistic theory of electronic structure and optical properties of a single InAs quantum dot grown on InP patterned substrate. The spatial positioning of individual dots using InP nano-templates results in a quantum dot embedded in InP pyramid. The strain distribution of a quantum dot in InP pyramid is calculated using the continuum elasticity theory. The electron and valence hole single-particle states are calculated using atomistic effective-bond-orbital model with second nearest-neighbor interactions, coupled to strain via Bir-Pikus Hamiltonian. The optical properties are determined by solving many-exciton Hamiltonian for interacting electron and hole complexes using the configuration-interaction method. The effect of positioning of quantum dots using nanotemplate on their optical spectra is determined by a comparison with dots on unpatterned substrates, and with experimental results. The possibility of tuning the quantum dot properties with varying the nano-template is explored.

cond-mat.mtrl-sci↗

arXiv2004

Multiband theory of multi-exciton complexes in self-assembled quantum dots

We report on a multiband microscopic theory of many-exciton complexes in self-assembled quantum dots. The single particle states are obtained by three methods: single-band effective-mass approximation, the multiband $k\cdot p$ method, and the tight-binding method. The electronic structure calculations are coupled with strain calculations via Bir-Pikus Hamiltonian. The many-body wave functions of $N$ electrons and $N$ valence holes are expanded in the basis of Slater determinants. The Coulomb matrix elements are evaluated using statically screened interaction for the three different sets of single particle states and the correlated $N$-exciton states are obtained by the configuration interaction method. The theory is applied to the excitonic recombination spectrum in InAs/GaAs self-assembled quantum dots. The results of the single-band effective-mass approximation are successfully compared with those obtained by using the of $k\cdot p$ and tight-binding methods.

cond-mat.mtrl-sci↗

arXiv2002

Absence of correlation between built-in electric dipole moment and quantum Stark effect in InAs/GaAs self-assembled quantum dots

We report significant deviations from the usual quadratic dependence of the ground state interband transition energy on applied electric fields in InAs/GaAs self-assembled quantum dots. In particular, we show that conventional second-order perturbation theory fails to correctly describe the Stark shift for electric field below $F = 10$ kV/cm in high dots. Eight-band ${\bf k}\cdot{\bf p}$ calculations demonstrate this effect is predominantly due to the three-dimensional strain field distribution which for various dot shapes and stoichiometric compositions drastically affects the hole ground state. Our conclusions are supported by two independent experiments.

cond-mat.mes-hall↗

arXiv2001

Anomalous quantum confined Stark effects in stacked InAs/GaAs self-assembled quantum dots

Vertically stacked and coupled InAs/GaAs self-assembled quantum dots (SADs) are predicted to exhibit a strong non-parabolic dependence of the interband transition energy on the electric field, which is not encountered in single SAD structures nor in other types of quantum structures. Our study based on an eight-band strain-dependent ${\bf k}\cdot{\bf p}$ Hamiltonian indicates that this anomalous quantum confined Stark effect is caused by the three-dimensional strain field distribution which influences drastically the hole states in the stacked SAD structures.

cond-mat.mes-hall↗