arXiv subjects

Wenjie Jiang

Publications and source records attributed to Wenjie Jiang.

At least 19 recordsLinked to original sources

arXiv2026

PanoWorld: Real-World Panoramic Generation

In this work, we aim to address the challenge of long-range memory in panoramic world models by exploiting the rotation-equivariant property of omnidirectional representations, where rotation can be treated as an implicit geometric transformation.Building on this insight, we propose PanoWorld, which simplifies camera trajectories into translations via fixed headings for both current-action modeling and long-range memory through Dense Panoramic Ray-Conditioning (DPRC) and Geometry-aware Memory Augmentation (GMA).Then, a three-stage training pipeline is introduced to progressively optimize each component. To better evaluate physical consistency under large-scale spatial variations and diverse illumination conditions, where existing datasets are relatively stable, we construct World360, a large-scale dataset consisting of both real-world video clips collected via panoramic unmanned aerial vehicles and high-quality simulated clips generated by AirSim360.Extensive experiments on World360 demonstrate the effectiveness of PanoWorld, outperforming alternative methods by a large margin.Our models, training code, and dataset will be publicly available. More information can be found on our project page: https://lihaoy-ux.github.io/panoworld-page/.

cs.CV↗

arXiv2026

Accessible Fine-grained Data Representation via Spatial Audio

Pitch-based sonification of quantitative data increases the accessibility of data visualizations that are otherwise inaccessible for blind and low-vision (BLV) individuals. We argue that, although pitch representations can reveal the coarse-grained information of data, such as data trend and value comparison, they cannot effectively convey the fine-grained details like the sign and exact value of individual data points. Informed by existing sound perception research, we propose a spatial audio-based approach by representing data values as the sound direction in the azimuth plane to achieve accessible fine-grained data representation. We conducted a user study with 26 participants (including 10 BLV participants) on four data perception tasks. The results show our approach significantly outperforms pitch representation on fine-grained data perception tasks like recognizing data signs and exact values, and performs similarly on data trend identification, despite its inferior accuracy on data value comparison.

cs.HC↗

arXiv2025

AirSim360: A Panoramic Simulation Platform within Drone View

The field of 360-degree omnidirectional understanding has been receiving increasing attention for advancing spatial intelligence. However, the lack of large-scale and diverse data remains a major limitation. In this work, we propose AirSim360, a simulation platform for omnidirectional data from aerial viewpoints, enabling wide-ranging scene sampling with drones. Specifically, AirSim360 focuses on three key aspects: a render-aligned data and labeling paradigm for pixel-level geometric, semantic, and entity-level understanding; an interactive pedestrian-aware system for modeling human behavior; and an automated trajectory generation paradigm to support navigation tasks. Furthermore, we collect more than 60K panoramic samples and conduct extensive experiments across various tasks to demonstrate the effectiveness of our simulator. Unlike existing simulators, our work is the first to systematically model the 4D real world under an omnidirectional setting. The entire platform, including the toolkit, plugins, and collected datasets, will be made publicly available at https://insta360-research-team.github.io/AirSim360-website.

cs.CV↗

arXiv2025

One Flight Over the Gap: A Survey from Perspective to Panoramic Vision

Driven by the demand for spatial intelligence and holistic scene perception, omnidirectional images (ODIs), which provide a complete 360\textdegree{} field of view, are receiving growing attention across diverse applications such as virtual reality, autonomous driving, and embodied robotics. Despite their unique characteristics, ODIs exhibit remarkable differences from perspective images in geometric projection, spatial distribution, and boundary continuity, making it challenging for direct domain adaption from perspective methods. This survey reviews recent panoramic vision techniques with a particular emphasis on the perspective-to-panorama adaptation. We first revisit the panoramic imaging pipeline and projection methods to build the prior knowledge required for analyzing the structural disparities. Then, we summarize three challenges of domain adaptation: severe geometric distortions near the poles, non-uniform sampling in Equirectangular Projection (ERP), and periodic boundary continuity. Building on this, we cover 20+ representative tasks drawn from more than 300 research papers in two dimensions. On one hand, we present a cross-method analysis of representative strategies for addressing panoramic specific challenges across different tasks. On the other hand, we conduct a cross-task comparison and classify panoramic vision into four major categories: visual quality enhancement and assessment, visual understanding, multimodal understanding, and visual generation. In addition, we discuss open challenges and future directions in data, models, and applications that will drive the advancement of panoramic vision research. We hope that our work can provide new insight and forward looking perspectives to advance the development of panoramic vision technologies. Our project page is https://insta360-research-team.github.io/Survey-of-Panorama

cs.CV↗

arXiv2025

Spatially-Resolved Atmospheric Turbulence Sensing with Two-Dimensional Orbital Angular Momentum Spectroscopy

Atmospheric turbulence characterization is crucial for technologies like free-space optical communications. Existing methods using a spatially-integrated one-dimensional (1D) orbital angular momentum (OAM) spectrum, P(m), obscure the heterogeneous nature of atmospheric distortions. This study introduces a two-dimensional (2D) OAM spectroscopy, P(m, n), which resolves the OAM spectrum (topological charge m) across discrete radial annuli (index n). Integrating this high-dimensional spectral analysis with a Support Vector Machine (SVM) classifier significantly improves the accuracy of atmospheric turbulence parameter inversion. The full potential of complex probe beams, such as multi-ringed Bessel-Gaussian beams, is realized with this radially-resolved 2D analysis. Through a co-design of the probe beam's spatial structure and the OAM spectral analysis dimensionality, a median classification accuracy of 85.47% was achieved across 20 turbulence conditions, a 23% absolute improvement over 1D techniques. The radial index also mitigates insufficient OAM spectral range, and a targeted feature-selection protocol addresses noise from low signal-to-noise ratio outer radial regions. This framework emphasizes co-design of the optical probe field and its OAM spectral analysis for enhanced fidelity in turbulence characterization.

physics.optics↗

arXiv2025

Machine learning assisted speckle and OAM spectrum analysis for enhanced turbulence characterisation

Atmospheric turbulence degrades the performance of free-space optical (FSO) communication and remote sensing systems by introducing phase and intensity distortions. While a majority of research focuses on mitigating these effects to ensure robust signal transmission, an underexplored alternative is to leverage the transformation of structured light to characterize the turbulent medium itself. Here, we introduce a deep learning framework that fuses post-propagation intensity speckle patterns and orbital angular momentum (OAM) spectral data for atmospheric turbulence parameter inference. Our architecture, based on a modified InceptionNet backbone, is optimized to extract and integrate multi-scale features from these distinct optical modalities. This multimodal approach achieves validation accuracies exceeding 80%, substantially outperforming conventional single-modality baselines. The framework demonstrates high inference accuracy and enhanced training stability across a broad range of simulated turbulent conditions, quantified by varying Fried parameters (r0) and Reynolds numbers (Re). This work presents a scalable and data-efficient method for turbulence characterization, offering a pathway toward robust environmental sensing and the optimization of dynamic FSO systems.

physics.optics↗

arXiv2025

A Systematic Investigation on Deep Learning-Based Omnidirectional Image and Video Super-Resolution

Omnidirectional image and video super-resolution is a crucial research topic in low-level vision, playing an essential role in virtual reality and augmented reality applications. Its goal is to reconstruct high-resolution images or video frames from low-resolution inputs, thereby enhancing detail preservation and enabling more accurate scene analysis and interpretation. In recent years, numerous innovative and effective approaches have been proposed, predominantly based on deep learning techniques, involving diverse network architectures, loss functions, projection strategies, and training datasets. This paper presents a systematic review of recent progress in omnidirectional image and video super-resolution, focusing on deep learning-based methods. Given that existing datasets predominantly rely on synthetic degradation and fall short in capturing real-world distortions, we introduce a new dataset, 360Insta, that comprises authentically degraded omnidirectional images and videos collected under diverse conditions, including varying lighting, motion, and exposure settings. This dataset addresses a critical gap in current omnidirectional benchmarks and enables more robust evaluation of the generalization capabilities of omnidirectional super-resolution methods. We conduct comprehensive qualitative and quantitative evaluations of existing methods on both public datasets and our proposed dataset. Furthermore, we provide a systematic overview of the current status of research and discuss promising directions for future exploration. All datasets, methods, and evaluation metrics introduced in this work are publicly available and will be regularly updated. Project page: https://github.com/nqian1/Survey-on-ODISR-and-ODVSR.

cs.CV↗

arXiv2025

DEFOM-Stereo: Depth Foundation Model Based Stereo Matching

Stereo matching is a key technique for metric depth estimation in computer vision and robotics. Real-world challenges like occlusion and non-texture hinder accurate disparity estimation from binocular matching cues. Recently, monocular relative depth estimation has shown remarkable generalization using vision foundation models. Thus, to facilitate robust stereo matching with monocular depth cues, we incorporate a robust monocular relative depth model into the recurrent stereo-matching framework, building a new framework for depth foundation model-based stereo-matching, DEFOM-Stereo. In the feature extraction stage, we construct the combined context and matching feature encoder by integrating features from conventional CNNs and DEFOM. In the update stage, we use the depth predicted by DEFOM to initialize the recurrent disparity and introduce a scale update module to refine the disparity at the correct scale. DEFOM-Stereo is verified to have much stronger zero-shot generalization compared with SOTA methods. Moreover, DEFOM-Stereo achieves top performance on the KITTI 2012, KITTI 2015, Middlebury, and ETH3D benchmarks, ranking $1^{st}$ on many metrics. In the joint evaluation under the robust vision challenge, our model simultaneously outperforms previous models on the individual benchmarks, further demonstrating its outstanding capabilities.

cs.CV↗

arXiv2025

Hierarchical Flow Diffusion for Efficient Frame Interpolation

Most recent diffusion-based methods still show a large gap compared to non-diffusion methods for video frame interpolation, in both accuracy and efficiency. Most of them formulate the problem as a denoising procedure in latent space directly, which is less effective caused by the large latent space. We propose to model bilateral optical flow explicitly by hierarchical diffusion models, which has much smaller search space in the denoising procedure. Based on the flow diffusion model, we then use a flow-guided images synthesizer to produce the final result. We train the flow diffusion model and the image synthesizer end to end. Our method achieves state of the art in accuracy, and 10+ times faster than other diffusion-based methods. The project page is at: https://hfd-interpolation.github.io.

cs.CV↗

arXiv2025

A2Perf: Real-World Autonomous Agents Benchmark

Autonomous agents and systems cover a number of application areas, from robotics and digital assistants to combinatorial optimization, all sharing common, unresolved research challenges. It is not sufficient for agents to merely solve a given task; they must generalize to out-of-distribution tasks, perform reliably, and use hardware resources efficiently during training and inference, among other requirements. Several methods, such as reinforcement learning and imitation learning, are commonly used to tackle these problems, each with different trade-offs. However, there is a lack of benchmarking suites that define the environments, datasets, and metrics which can be used to provide a meaningful way for the community to compare progress on applying these methods to real-world problems. We introduce A2Perf--a benchmark with three environments that closely resemble real-world domains: computer chip floorplanning, web navigation, and quadruped locomotion. A2Perf provides metrics that track task performance, generalization, system resource efficiency, and reliability, which are all critical to real-world applications. Using A2Perf, we demonstrate that web navigation agents can achieve latencies comparable to human reaction times on consumer hardware, reveal reliability trade-offs between algorithms for quadruped locomotion, and quantify the energy costs of different learning approaches for computer chip-design. In addition, we propose a data cost metric to account for the cost incurred acquiring offline data for imitation learning and hybrid algorithms, which allows us to better compare these approaches. A2Perf also contains several standard baselines, enabling apples-to-apples comparisons across methods and facilitating progress in real-world autonomy. As an open-source benchmark, A2Perf is designed to remain accessible, up-to-date, and useful to the research community over the long term.

cs.LG↗

arXiv2024

Quantum continual learning on a programmable superconducting processor

Quantum computers may outperform classical computers on machine learning tasks. In recent years, a variety of quantum algorithms promising unparalleled potential to enhance, speed up, or innovate machine learning have been proposed. Yet, quantum learning systems, similar to their classical counterparts, may likewise suffer from the catastrophic forgetting problem, where training a model with new tasks would result in a dramatic performance drop for the previously learned ones. This problem is widely believed to be a crucial obstacle to achieving continual learning of multiple sequential tasks. Here, we report an experimental demonstration of quantum continual learning on a fully programmable superconducting processor. In particular, we sequentially train a quantum classifier with three tasks, two about identifying real-life images and the other on classifying quantum states, and demonstrate its catastrophic forgetting through experimentally observed rapid performance drops for prior tasks. To overcome this dilemma, we exploit the elastic weight consolidation strategy and show that the quantum classifier can incrementally learn and retain knowledge across the three distinct tasks, with an average prediction accuracy exceeding 92.3%. In addition, for sequential tasks involving quantum-engineered data, we demonstrate that the quantum classifier can achieve a better continual learning performance than a commonly used classical feedforward network with a comparable number of variational parameters. Our results establish a viable strategy for empowering quantum learning systems with desirable adaptability to multiple sequential tasks, marking an important primary experimental step towards the long-term goal of achieving quantum artificial general intelligence.

quant-ph↗

arXiv2024

Evaluation of Pedestrian Safety in a High-Fidelity Simulation Environment Framework

Pedestrians' safety is a crucial factor in assessing autonomous driving scenarios. However, pedestrian safety evaluation is rarely considered by existing autonomous driving simulation platforms. This paper proposes a pedestrian safety evaluation method for autonomous driving, in which not only the collision events but also the conflict events together with the characteristics of pedestrians are fully considered. Moreover, to apply the pedestrian safety evaluation system, we construct a high-fidelity simulation framework embedded with pedestrian safety-critical characteristics. We demonstrate our simulation framework and pedestrian safety evaluation with a comparative experiment with two kinds of autonomous driving perception algorithms -- single-vehicle perception and vehicle-to-infrastructure (V2I) cooperative perception. The results show that our framework can evaluate different autonomous driving algorithms with detailed and quantitative pedestrian safety indexes. To this end, the proposed simulation method and framework can be used to access different autonomous driving algorithms and evaluate pedestrians' safety performance in future autonomous driving simulations, which can inspire more pedestrian-friendly autonomous driving algorithms.

cs.AI↗

arXiv2024

Experimental demonstration of reconstructing quantum states with generative models

Quantum state tomography, a process that reconstructs a quantum state from measurements on an ensemble of identically prepared copies, plays a crucial role in benchmarking quantum devices. However, brute-force approaches to quantum state tomography would become impractical for large systems, as the required resources scale exponentially with the system size. Here, we explore a machine learning approach and report an experimental demonstration of reconstructing quantum states based on neural network generative models with an array of programmable superconducting transmon qubits. In particular, we experimentally prepare the Greenberger-Horne-Zeilinger states and random states up to five qubits and demonstrate that the machine learning approach can efficiently reconstruct these states with the number of required experimental samples scaling linearly with system size. Our results experimentally showcase the intriguing potential for exploiting machine learning techniques in validating and characterizing complex quantum devices, offering a valuable guide for the future development of quantum technologies.

quant-ph↗

arXiv2024

Probing many-body Bell correlation depth with superconducting qubits

Quantum nonlocality describes a stronger form of quantum correlation than that of entanglement. It refutes Einstein's belief of local realism and is among the most distinctive and enigmatic features of quantum mechanics. It is a crucial resource for achieving quantum advantages in a variety of practical applications, ranging from cryptography and certified random number generation via self-testing to machine learning. Nevertheless, the detection of nonlocality, especially in quantum many-body systems, is notoriously challenging. Here, we report an experimental certification of genuine multipartite Bell correlations, which signal nonlocality in quantum many-body systems, up to 24 qubits with a fully programmable superconducting quantum processor. In particular, we employ energy as a Bell correlation witness and variationally decrease the energy of a many-body system across a hierarchy of thresholds, below which an increasing Bell correlation depth can be certified from experimental data. As an illustrating example, we variationally prepare the low-energy state of a two-dimensional honeycomb model with 73 qubits and certify its Bell correlations by measuring an energy that surpasses the corresponding classical bound with up to 48 standard deviations. In addition, we variationally prepare a sequence of low-energy states and certify their genuine multipartite Bell correlations up to 24 qubits via energies measured efficiently by parity oscillation and multiple quantum coherence techniques. Our results establish a viable approach for preparing and certifying multipartite Bell correlations, which provide not only a finer benchmark beyond entanglement for quantum devices, but also a valuable guide towards exploiting multipartite Bell correlation in a wide spectrum of practical applications.

quant-ph↗

arXiv2024

Prethermal Time-Crystalline Corner Modes

We demonstrate the existence of prethermal discrete time crystals whose sub-harmonic response is entirely localized to zero-dimensional corner modes. Within the exponentially long prethermal regime, we show that the robustness of these corner modes arises from two related, yet distinct mechanisms: the presence of a higher-order symmetry-protected topological phase in the effective Hamiltonian, or the emergence of a dynamical constraint that prevents the decay of the corner mode. While the first mechanism ensures the stability of the sub-harmonic response throughout the entirety of the prethermal regime, it is restricted to initial states in the ground state manifold of the effective Hamiltonian. By contrast, the second mechanism enables the observation of the prethermal time-crystalline order for arbitrary initial states, albeit with a time scale that is not only determined by the frequency of the drive, but also the relative energy scale across the system's sublattices. We characterize these two mechanisms by simulating the dynamics of a periodically driven two-dimensional spin model, and discuss natural extensions of our model to all other dimensions.

quant-ph↗

arXiv2024

Non-Abelian braiding of Fibonacci anyons with a superconducting processor

Non-Abelian topological orders offer an intriguing path towards fault-tolerant quantum computation, where information can be encoded and manipulated in a topologically protected manner immune to arbitrary local noises and perturbations. However, realizing non-Abelian topologically ordered states is notoriously challenging in both condensed matter and programmable quantum systems, and it was not until recently that signatures of non-Abelian statistics were observed through digital quantum simulation approaches. Despite these exciting progresses, none of them has demonstrated the appropriate type of topological orders and associated non-Abelian anyons whose braidings alone support universal quantum computation. Here, we report the realization of non-Abelian topologically ordered states of the Fibonacci string-net model and demonstrate braidings of Fibonacci anyons featuring universal computational power, with a superconducting quantum processor. We exploit efficient quantum circuits to prepare the desired states and verify their nontrivial topological nature by measuring the topological entanglement entropy. In addition, we create two pairs of Fibonacci anyons and demonstrate their fusion rule and non-Abelian braiding statistics by applying unitary gates on the underlying physical qubits. Our results establish a versatile digital approach to exploring exotic non-Abelian topological states and their associated braiding statistics with current noisy intermediate-scale quantum processors.

quant-ph↗

arXiv2024

RomniStereo: Recurrent Omnidirectional Stereo Matching

Omnidirectional stereo matching (OSM) is an essential and reliable means for $360^{\circ}$ depth sensing. However, following earlier works on conventional stereo matching, prior state-of-the-art (SOTA) methods rely on a 3D encoder-decoder block to regularize the cost volume, causing the whole system complicated and sub-optimal results. Recently, the Recurrent All-pairs Field Transforms (RAFT) based approach employs the recurrent update in 2D and has efficiently improved image-matching tasks, ie, optical flow, and stereo matching. To bridge the gap between OSM and RAFT, we mainly propose an opposite adaptive weighting scheme to seamlessly transform the outputs of spherical sweeping of OSM into the required inputs for the recurrent update, thus creating a recurrent omnidirectional stereo matching (RomniStereo) algorithm. Furthermore, we introduce two techniques, ie, grid embedding and adaptive context feature generation, which also contribute to RomniStereo's performance. Our best model improves the average MAE metric by 40.7\% over the previous SOTA baseline across five datasets. When visualizing the results, our models demonstrate clear advantages on both synthetic and realistic examples. The code is available at \url{https://github.com/HalleyJiang/RomniStereo}.

cs.CV↗

arXiv2024

Long-lived topological time-crystalline order on a quantum processor

Topologically ordered phases of matter elude Landau's symmetry-breaking theory, featuring a variety of intriguing properties such as long-range entanglement and intrinsic robustness against local perturbations. Their extension to periodically driven systems gives rise to exotic new phenomena that are forbidden in thermal equilibrium. Here, we report the observation of signatures of such a phenomenon -- a prethermal topologically ordered time crystal -- with programmable superconducting qubits arranged on a square lattice. By periodically driving the superconducting qubits with a surface-code Hamiltonian, we observe discrete time-translation symmetry breaking dynamics that is only manifested in the subharmonic temporal response of nonlocal logical operators. We further connect the observed dynamics to the underlying topological order by measuring a nonzero topological entanglement entropy and studying its subsequent dynamics. Our results demonstrate the potential to explore exotic topologically ordered nonequilibrium phases of matter with noisy intermediate-scale quantum processors.

quant-ph↗