arXiv subjects

Shijie Lin

Publications and source records attributed to Shijie Lin.

At least 19 recordsLinked to original sources

arXiv2026

E2Pano: Learning Event-to-Panorama Image Reconstruction

Event cameras offer microsecond-level temporal resolution and high dynamic range, potentially facilitating motion-blur-free panoramic imaging from fast rotational scanning. Nonetheless, existing optimization-based methods remain computationally demanding, while prior learning-based reconstruction methods are largely designed for perspective imagery and lack geometry-aware support for panoramic outputs. We present E2Pano, a geometry-guided event-to-panorama pipeline with an end-to-end learnable photometric reconstruction stage. Our framework preserves real spherical coordinates from geometric mapping throughout the pipeline, employs a lightweight enhancement module with frequency-domain supervision to bridge the event-image domain gap, and leverages a spherical Transformer with 3D positional embeddings for photometric reconstruction. Experiments on synthetic data and captured rotational scans show improved reconstruction quality and lower photometric reconstruction cost than optimization-based baselines, together with encouraging transfer to real captures under our acquisition protocol despite training purely on synthetic data. Additionally, we construct PanoScan, a dataset with 4,370 synthetic and 30 real-world panoramic scenes paired with event streams. Our dataset and code will be released.

cs.CV↗

arXiv2026

See Silhouettes in Motion with Neuromorphic Vision

Quasi-bimodal objects, such as text, road signs, and barcodes, play a basic yet vital role in daily visual communication. By boiling these down to clear silhouettes, binarization uses a minimal language to convey essential vision cues for maximum downstream efficiency, especially for tasks that require simple geometric, topological reasoning rather than heavy appearance modeling. The catch is that frame-based imaging often struggles on mobile platforms like drones, self-driving cars, and underwater vehicles, in which rapid motion causes severe motion blur and harsh lighting washes out scene details. To overcome these physical limits, neuromorphic vision via event cameras, featuring microsecond time resolution and high dynamic range, steps in as a natural solution. Building upon this event-driven paradigm, we propose a simple yet effective dual-modal approach that harnesses the synergy between frames and events for training-free, real-time, high-frame-rate binarization on CPU-only devices. Extensive evaluations show that it earns competitive performance against leading techniques in reducing blur artifacts and delivers impressive improvements under challenging illumination at a lower computational cost. Besides, its asynchronous nature bypasses long-standing event-scarcity issues that break traditional time-binning reconstruction at fixed time slots, maintaining clear target shapes even at extreme kilohertz frame rates. Its binary results further serve as reliable representations to facilitate a range of downstream tasks. This work paves the way towards lightweight perception and interaction in embodied intelligence on resource-constrained edge platforms.

eess.IV↗

arXiv2026

$\mathcal{H}$-EFTCAMB: A Cobaya-Integrated, Python-Wrapped Extension of EFTCAMB for Covariant Horndeski Gravity

We present $\mathcal{H}\mathtt{-EFTCAMB}$, the official successor to $\mathtt{EFTCAMB}$. The original $\mathtt{EFTCAMB}$ is designed as a consistent and numerically stable implementation of the effective field theory (EFT) of dark energy in the Einstein-Boltzmann code $\mathtt{CAMB}$. On top of this, $\mathcal{H}\mathtt{-EFTCAMB}$ introduces a new Horndeski module that supports computing cosmology for an arbitrary input covariant Horndeski Lagragian. $\mathcal{H}\mathtt{-EFTCAMB}$ supports both mapping the Horndeski theory to an EFT lagrangian to solve in the EFT framework as well as directly solving for the scalar field equations of motion derived from the covariant Lagrangian. The latter approach also works for the cases when the Horndeski field experiences turn-overs, e.g. oscillation, where the EFT approach breaks down. The Horndeski module has been validated by comparing internally with existing models in the original $\mathtt{EFTCAMB}$ and externally with $\mathtt{hi\_class}$. $\mathcal{H}\mathtt{-EFTCAMB}$ features a flexible Python wrapper that is seamlessly integrated into the widely utilized cosmological sampler $\mathtt{Cobaya}$. \heft~is publicly available and serves as a comprehensive tool for testing gravity against the precision data from current and next-generation surveys.

gr-qc↗

arXiv2025

Lyapunov Stability Learning with Nonlinear Control via Inductive Biases

Finding a control Lyapunov function (CLF) in a dynamical system with a controller is an effective way to guarantee stability, which is a crucial issue in safety-concerned applications. Recently, deep learning models representing CLFs have been applied into a learner-verifier framework to identify satisfiable candidates. However, the learner treats Lyapunov conditions as complex constraints for optimisation, which is hard to achieve global convergence. It is also too complicated to implement these Lyapunov conditions for verification. To improve this framework, we treat Lyapunov conditions as inductive biases and design a neural CLF and a CLF-based controller guided by this knowledge. This design enables a stable optimisation process with limited constraints, and allows end-to-end learning of both the CLF and the controller. Our approach achieves a higher convergence rate and larger region of attraction (ROA) in learning the CLF compared to existing methods among abundant experiment cases. We also thoroughly reveal why the success rate decreases with previous methods during learning.

cs.LG↗

arXiv2025

Impact of Large-Scale Structure along Line-of-Sight on Time-Delay Cosmography

Time-delay cosmography, by monitoring the multiply imaged gravitational lenses in the time domain, offers a promising and independent method for measuring cosmological distances. However, in addition to the main deflector that produces the multiple images, the large-scale structure along the line-of-sight (LoS) will also deflect the traveling light rays, known as weak lensing (WL). Due to resolution limitations, accurately measuring WL on arcsecond scales is highly challenging. In this work, we evaluate the LoS effects on both lensing images and time-delay measurements using a more straightforward, high-resolution N-body simulation that provides a more realistic matter distribution compared to the traditional, computationally cheaper halo rendering method. We employ the multi-plane ray tracing technique, which is traditionally utilized to compute WL effects at the arcminute scale, extending its application to the strong lensing regime at the arcsecond scale. We focus on the quadruple-image system and present the following findings: 1. In addition to a constant external convergence, large-scale structures within a region approximately 2 arcminutes in angular size act as external perturbers, inducing inhomogeneous fluctuations on the arcsecond scale; 2. These fluctuations cannot be fully accounted for by external shear alone, necessitating the inclusion of external flexion; 3. While incorporating flexion provides a reasonably good fit to the lensing image, the time-delay distance still exhibits a $6.2$\textperthousand~bias and a $2.5\%$ uncertainty. This underscores the limitations of the single-plane approximation, as time-delay errors accumulate along the LoS.

astro-ph.CO↗

arXiv2025

Lattice Boltzmann Model for Learning Real-World Pixel Dynamicity

This work proposes the Lattice Boltzmann Model (LBM) to learn real-world pixel dynamicity for visual tracking. LBM decomposes visual representations into dynamic pixel lattices and solves pixel motion states through collision-streaming processes. Specifically, the high-dimensional distribution of the target pixels is acquired through a multilayer predict-update network to estimate the pixel positions and visibility. The predict stage formulates lattice collisions among the spatial neighborhood of target pixels and develops lattice streaming within the temporal visual context. The update stage rectifies the pixel distributions with online visual representations. Compared with existing methods, LBM demonstrates practical applicability in an online and real-time manner, which can efficiently adapt to real-world visual tracking tasks. Comprehensive evaluations of real-world point tracking benchmarks such as TAP-Vid and RoboTAP validate LBM's efficiency. A general evaluation of large-scale open-world object tracking benchmarks such as TAO, BFT, and OVT-B further demonstrates LBM's real-world practicality.

cs.CV↗

arXiv2025

MDN: Mamba-Driven Dualstream Network For Medical Hyperspectral Image Segmentation

Medical Hyperspectral Imaging (MHSI) offers potential for computational pathology and precision medicine. However, existing CNN and Transformer struggle to balance segmentation accuracy and speed due to high spatial-spectral dimensionality. In this study, we leverage Mamba's global context modeling to propose a dual-stream architecture for joint spatial-spectral feature extraction. To address the limitation of Mamba's unidirectional aggregation, we introduce a recurrent spectral sequence representation to capture low-redundancy global spectral features. Experiments on a public Multi-Dimensional Choledoch dataset and a private Cervical Cancer dataset show that our method outperforms state-of-the-art approaches in segmentation accuracy while minimizing resource usage and achieving the fastest inference speed. Our code will be available at https://github.com/DeepMed-Lab-ECNU/MDN.

eess.IV↗

arXiv2024

A Haptic-Based Proximity Sensing System for Buried Object in Granular Material

The proximity perception of objects in granular materials is significant, especially for applications like minesweeping. However, due to particles' opacity and complex properties, existing proximity sensors suffer from high costs from sophisticated hardware and high user-cost from unintuitive results. In this paper, we propose a simple yet effective proximity sensing system for underground stuff based on the haptic feedback of the sensor-granules interaction. We study and employ the unique characteristic of particles -- failure wedge zone, and combine the machine learning method -- Gaussian process regression, to identify the force signal changes induced by the proximity of objects, so as to achieve near-field perception. Furthermore, we design a novel trajectory to control the probe searching in granules for a wide range of perception. Also, our proximity sensing system can adaptively determine optimal parameters for robustness operation in different particles. Experiments demonstrate our system can perceive underground objects over 0.5 to 7 cm in advance among various materials.

cs.RO↗

arXiv2024

EROAM: Event-based Camera Rotational Odometry and Mapping in Real-time

This paper presents EROAM, a novel event-based rotational odometry and mapping system that achieves real-time, accurate camera rotation estimation. Unlike existing approaches that rely on event generation models or contrast maximization, EROAM employs a spherical event representation by projecting events onto a unit sphere and introduces Event Spherical Iterative Closest Point (ES-ICP), a novel geometric optimization framework designed specifically for event camera data. The spherical representation simplifies rotational motion formulation while operating in a continuous spherical domain, enabling enhanced spatial resolution. Our system features an efficient map management approach using incremental k-d tree structures and intelligent regional density control, ensuring optimal computational performance during long-term operation. Combined with parallel point-to-line optimization, EROAM achieves efficient computation without compromising accuracy. Extensive experiments on both synthetic and real-world datasets show that EROAM significantly outperforms state-of-the-art methods in terms of accuracy, robustness, and computational efficiency. Our method maintains consistent performance under challenging conditions, including high angular velocities and extended sequences, where other methods often fail or show significant drift. Additionally, EROAM produces high-quality panoramic reconstructions with preserved fine structural details.

cs.CV↗

arXiv2024

The role of LRG1 and LRG2's monopole in inferring the DESI 2024 BAO cosmology

The Dark Energy Spectroscopic Instrument (DESI) collaboration recently released its first year of data (DR1) on baryon acoustic oscillations (BAO) in galaxy, quasar, and Lyman-$\alpha$ forest tracers. When combined with CMB and SNIa data, DESI BAO results suggest potential thawing behavior in dark energy. Cosmological analyses utilize comoving distances along ($D_H$) and perpendicular to ($D_M$) the line of sight. Notably, there are $1\sim2\sigma$ deviations in $D_M$ and $D_H$ from Planck cosmology values in the luminous red galaxies (LRG) bins LRG1 and LRG2.This study examines the role of LRG1 and LRG2 in diverging DESI 2024 BAO cosmology from Planck cosmology. We use angle-averaged distance $D_V$ and the ratio $F_{\rm AP}=D_M/D_H$, which are more directly related to the measured monopole and quadrupole components of the galaxy power spectrum or correlation function, instead of the officially adopted $D_M$ and $D_H$. This transformation aims to isolate the influence of monopoles in LRG1 and LRG2 on deviations from $w=-1$. Our findings indicate that removing the $D_V$ data point in LRG2 aligns DESI + CMB + SNIa data compilation with $w=-1$ within a $2\sigma$ contour and reduces the $H_0$ discrepancy from the Planck 2018 results from $0.63\sigma$ to $0.31\sigma$. Similarly, excluding the $D_V$ data point from LRG1 shifts the $w_0/w_a$ contour toward $w=-1$, although no intersection occurs. This highlights the preference of both LRG1 and LRG2 BAO monopole components for the thawing dark energy model, with LRG2 showing a stronger preference. We provide the $D_V$ and $F_{\rm AP}$ data and their covariance alongside this paper.

astro-ph.CO↗

arXiv2024

NetTrack: Tracking Highly Dynamic Objects with a Net

The complex dynamicity of open-world objects presents non-negligible challenges for multi-object tracking (MOT), often manifested as severe deformations, fast motion, and occlusions. Most methods that solely depend on coarse-grained object cues, such as boxes and the overall appearance of the object, are susceptible to degradation due to distorted internal relationships of dynamic objects. To address this problem, this work proposes NetTrack, an efficient, generic, and affordable tracking framework to introduce fine-grained learning that is robust to dynamicity. Specifically, NetTrack constructs a dynamicity-aware association with a fine-grained Net, leveraging point-level visual cues. Correspondingly, a fine-grained sampler and matching method have been incorporated. Furthermore, NetTrack learns object-text correspondence for fine-grained localization. To evaluate MOT in extremely dynamic open-world scenarios, a bird flock tracking (BFT) dataset is constructed, which exhibits high dynamicity with diverse species and open-world scenarios. Comprehensive evaluation on BFT validates the effectiveness of fine-grained learning on object dynamicity, and thorough transfer experiments on challenging open-world benchmarks, i.e., TAO, TAO-OW, AnimalTrack, and GMOT-40, validate the strong generalization ability of NetTrack even without finetuning. Project page: https://george-zhuang.github.io/nettrack/.

cs.CV↗

arXiv2024

NeuPAN: Direct Point Robot Navigation with End-to-End Model-based Learning

Navigating a nonholonomic robot in a cluttered, unknown environment requires accurate perception and precise motion control for real-time collision avoidance. This paper presents NeuPAN: a real-time, highly accurate, map-free, easy-to-deploy, and environment-invariant robot motion planner. Leveraging a tightly coupled perception-to-control framework, NeuPAN has two key innovations compared to existing approaches: 1) it directly maps raw point cloud data to a latent distance feature space for collision-free motion generation, avoiding error propagation from the perception to control pipeline; 2) it is interpretable from an end-to-end model-based learning perspective. The crux of NeuPAN is solving an end-to-end mathematical model with numerous point-level constraints using a plug-and-play (PnP) proximal alternating-minimization network (PAN), incorporating neurons in the loop. This allows NeuPAN to generate real-time, physically interpretable motions. It seamlessly integrates data and knowledge engines, and its network parameters can be fine-tuned via backpropagation. We evaluate NeuPAN on a ground mobile robot, a wheel-legged robot, and an autonomous vehicle, in extensive simulated and real-world environments. Results demonstrate that NeuPAN outperforms existing baselines in terms of accuracy, efficiency, robustness, and generalization capabilities across various environments, including the cluttered sandbox, office, corridor, and parking lot. We show that NeuPAN works well in unknown and unstructured environments with arbitrarily shaped objects, transforming impassable paths into passable ones.

cs.RO↗

arXiv2024

Neuromorphic Synergy for Video Binarization

Bimodal objects, such as the checkerboard pattern used in camera calibration, markers for object tracking, and text on road signs, to name a few, are prevalent in our daily lives and serve as a visual form to embed information that can be easily recognized by vision systems. While binarization from intensity images is crucial for extracting the embedded information in the bimodal objects, few previous works consider the task of binarization of blurry images due to the relative motion between the vision sensor and the environment. The blurry images can result in a loss in the binarization quality and thus degrade the downstream applications where the vision system is in motion. Recently, neuromorphic cameras offer new capabilities for alleviating motion blur, but it is non-trivial to first deblur and then binarize the images in a real-time manner. In this work, we propose an event-based binary reconstruction method that leverages the prior knowledge of the bimodal target's properties to perform inference independently in both event space and image space and merge the results from both domains to generate a sharp binary image. We also develop an efficient integration method to propagate this binary image to high frame rate binary video. Finally, we develop a novel method to naturally fuse events and images for unsupervised threshold identification. The proposed method is evaluated in publicly available and our collected data sequence, and shows the proposed method can outperform the SOTA methods to generate high frame rate binary video in real-time on CPU-only devices.

cs.CV↗

arXiv2023

Simultaneous Synchronization and Calibration for Wide-baseline Stereo Event Cameras

Event-based cameras are increasingly utilized in various applications, owing to their high temporal resolution and low power consumption. However, a fundamental challenge arises when deploying multiple such cameras: they operate on independent time systems, leading to temporal misalignment. This misalignment can significantly degrade performance in downstream applications. Traditional solutions, which often rely on hardware-based synchronization, face limitations in compatibility and are impractical for long-distance setups. To address these challenges, we propose a novel algorithm that exploits the motion of objects in the shared field of view to achieve millisecond-level synchronization among multiple event-based cameras. Our method also concurrently estimates extrinsic parameters. We validate our approach in both simulated and real-world indoor/outdoor scenarios, demonstrating successful synchronization and accurate extrinsic parameters estimation.

cs.RO↗

arXiv2023

GRAINS: Proximity Sensing of Objects in Granular Materials

Proximity sensing detects an object's presence without contact. However, research has rarely explored proximity sensing in granular materials (GM) due to GM's lack of visual and complex properties. In this paper, we propose a granular-material-embedded autonomous proximity sensing system (GRAINS) based on three granular phenomena (fluidization, jamming, and failure wedge zone). GRAINS can automatically sense buried objects beneath GM in real-time manner (at least ~20 hertz) and perceive them 0.5 ~ 7 centimeters ahead in different granules without the use of vision or touch. We introduce a new spiral trajectory for the probe raking in GM, combining linear and circular motions, inspired by a common granular fluidization technique. Based on the observation of force-raising when granular jamming occurs in the failure wedge zone in front of the probe during its raking, we employ Gaussian process regression to constantly learn and predict the force patterns and detect the force anomaly resulting from granular jamming to identify the proximity sensing of buried objects. Finally, we apply GRAINS to a Bayesian-optimization-algorithm-guided exploration strategy to successfully localize underground objects and outline their distribution using proximity sensing without contact or digging. This work offers a simple yet reliable method with potential for safe operation in building habitation infrastructure on an alien planet without human intervention.

cs.RO↗

arXiv2023

Fast Event-based Double Integral for Real-time Robotics

Motion deblurring is a critical ill-posed problem that is important in many vision-based robotics applications. The recently proposed event-based double integral (EDI) provides a theoretical framework for solving the deblurring problem with the event camera and generating clear images at high frame-rate. However, the original EDI is mainly designed for offline computation and does not support real-time requirement in many robotics applications. In this paper, we propose the fast EDI, an efficient implementation of EDI that can achieve real-time online computation on single-core CPU devices, which is common for physical robotic platforms used in practice. In experiments, our method can handle event rates at as high as 13 million event per second in a wide variety of challenging lighting conditions. We demonstrate the benefit on multiple downstream real-time applications, including localization, visual tag detection, and feature matching.

cs.RO↗

arXiv2023

Target-free Extrinsic Calibration of Event-LiDAR Dyad using Edge Correspondences

Calibrating the extrinsic parameters of sensory devices is crucial for fusing multi-modal data. Recently, event cameras have emerged as a promising type of neuromorphic sensors, with many potential applications in fields such as mobile robotics and autonomous driving. When combined with LiDAR, they can provide more comprehensive information about the surrounding environment. Nonetheless, due to the distinctive representation of event cameras compared to traditional frame-based cameras, calibrating them with LiDAR presents a significant challenge. In this paper, we propose a novel method to calibrate the extrinsic parameters between a dyad of an event camera and a LiDAR without the need for a calibration board or other equipment. Our approach takes advantage of the fact that when an event camera is in motion, changes in reflectivity and geometric edges in the environment trigger numerous events, which can also be captured by LiDAR. Our proposed method leverages the edges extracted from events and point clouds and correlates them to estimate extrinsic parameters. Experimental results demonstrate that our proposed method is highly robust and effective in various scenes.

cs.RO↗

arXiv2023

GRB 221009A: An ordinary nearby GRB with extraordinary observational properties

The gamma-ray burst GRB 221009A, known as the ``brightest-of-all-time" (BOAT), is the closest energetic burst detected so far, with an energy of $E_{\gamma,\rm iso} \sim 10^{55}$ ergs. This study aims to assess its compatibility with known GRB energy and luminosity distributions. Our analysis indicates that the energy/luminosity function of GRBs is consistent across various redshift intervals, and that the inclusion of GRB 221009A does not significantly impact the function at low redshifts. Additionally, our evaluation of the best-fitting result of the entire GRB sample suggests that the expected number of GRBs with energy greater than $10^{55}$ ergs at a low redshift is 0.2, so that the emergence of GRB 221009A is consistent with expected energy/luminosity functions within $\sim 2\sigma$ Poisson fluctuation error, still adhering to the principles of small number statistics. Furthermore, we find that GRB 221009A and other energetic bursts, defined as $E_{\gamma,\rm iso} \gtrsim10^{54} {\rm ergs}$, exhibit no significant differences in terms of distributions of $T_{90}$, minimum timescale, Amati relation, $E_{\rm \gamma,iso}$-$E_{\rm X,iso}$ relation, $L_{\gamma,\rm iso}-\Gamma_0$ relation, $E_{\gamma,\rm iso}-\Gamma_0$ relation, $L_{\gamma,\rm iso}-E_{\rm p,i}-\Gamma_0$ relation, and host galaxy properties, compared to normal long GRBs. This suggests that energetic GRBs (including GRB 221009A) and other long GRBs likely have similar progenitor systems and undergo similar energy dissipation and radiation processes. The generation of energetic GRBs may be due to more extreme central engine properties or, more likely, a rarer viewing configuration of a quasi-universal structured jet.

astro-ph.HE↗