arXiv subjects

Jianing Li

Publications and source records attributed to Jianing Li.

At least 73 records · Page 4Linked to original sources

arXiv2023

SODFormer: Streaming Object Detection with Transformer Using Events and Frames

DAVIS camera, streaming two complementary sensing modalities of asynchronous events and frames, has gradually been used to address major object detection challenges (e.g., fast motion blur and low-light). However, how to effectively leverage rich temporal cues and fuse two heterogeneous visual streams remains a challenging endeavor. To address this challenge, we propose a novel streaming object detector with Transformer, namely SODFormer, which first integrates events and frames to continuously detect objects in an asynchronous manner. Technically, we first build a large-scale multimodal neuromorphic object detection dataset (i.e., PKU-DAVIS-SOD) over 1080.1k manual labels. Then, we design a spatiotemporal Transformer architecture to detect objects via an end-to-end sequence prediction problem, where the novel temporal Transformer module leverages rich temporal cues from two visual streams to improve the detection performance. Finally, an asynchronous attention-based fusion module is proposed to integrate two heterogeneous sensing modalities and take complementary advantages from each end, which can be queried at any time to locate objects and break through the limited output frequency from synchronized frame-based fusion strategies. The results show that the proposed SODFormer outperforms four state-of-the-art methods and our eight baselines by a significant margin. We also show that our unifying framework works well even in cases where the conventional frame-based camera fails, e.g., high-speed motion and low-light conditions. Our dataset and code can be available at https://github.com/dianzl/SODFormer.

cs.CV↗

arXiv2023

Deep Directly-Trained Spiking Neural Networks for Object Detection

Spiking neural networks (SNNs) are brain-inspired energy-efficient models that encode information in spatiotemporal dynamics. Recently, deep SNNs trained directly have shown great success in achieving high performance on classification tasks with very few time steps. However, how to design a directly-trained SNN for the regression task of object detection still remains a challenging problem. To address this problem, we propose EMS-YOLO, a novel directly-trained SNN framework for object detection, which is the first trial to train a deep SNN with surrogate gradients for object detection rather than ANN-SNN conversion strategies. Specifically, we design a full-spike residual block, EMS-ResNet, which can effectively extend the depth of the directly-trained SNN with low power consumption. Furthermore, we theoretically analyze and prove the EMS-ResNet could avoid gradient vanishing or exploding. The results demonstrate that our approach outperforms the state-of-the-art ANN-SNN conversion methods (at least 500 time steps) in extremely fewer time steps (only 4 time steps). It is shown that our model could achieve comparable performance to the ANN with the same architecture while consuming 5.83 times less energy on the frame-based COCO Dataset and the event-based Gen1 Dataset.

cs.CV↗

arXiv2023

Full Resolution Repetition Counting

Given an untrimmed video, repetitive actions counting aims to estimate the number of repetitions of class-agnostic actions. To handle the various length of videos and repetitive actions, also optimization challenges in end-to-end video model training, down-sampling is commonly utilized in recent state-of-the-art methods, leading to ignorance of several repetitive samples. In this paper, we attempt to understand repetitive actions from a full temporal resolution view, by combining offline feature extraction and temporal convolution networks. The former step enables us to train repetition counting network without down-sampling while preserving all repetition regardless of the video length and action frequency, and the later network models all frames in a flexible and dynamically expanding temporal receptive field to retrieve all repetitions with a global aspect. We experimentally demonstrate that our method achieves better or comparable performance in three public datasets, i.e., TransRAC, UCFRep and QUVA. We expect this work will encourage our community to think about the importance of full temporal resolution.

cs.CV↗

arXiv2023

Cycle Consistency-based Uncertainty Quantification of Neural Networks in Inverse Imaging Problems

Uncertainty estimation is critical for numerous applications of deep neural networks and draws growing attention from researchers. Here, we demonstrate an uncertainty quantification approach for deep neural networks used in inverse problems based on cycle consistency. We build forward-backward cycles using the physical forward model available and a trained deep neural network solving the inverse problem at hand, and accordingly derive uncertainty estimators through regression analysis on the consistency of these forward-backward cycles. We theoretically analyze cycle consistency metrics and derive their relationship with respect to uncertainty, bias, and robustness of the neural network inference. To demonstrate the effectiveness of these cycle consistency-based uncertainty estimators, we classified corrupted and out-of-distribution input image data using some of the widely used image deblurring and super-resolution neural networks as testbeds. The blind testing of our method outperformed other models in identifying unseen input data corruption and distribution shifts. This work provides a simple-to-implement and rapid uncertainty quantification method that can be universally applied to various neural networks used for solving inverse problems.

cs.CV↗

arXiv2023

Self-consistent thermodynamic potential for magnetized QCD matter

Within the two-flavor Nambu--Jona-Lasinio model, we derive a self-consistent thermodynamic potential $Ω$ for a QCD matter in an external magnetic field $B$. To be consistent with Schwinger's renormalization spirit, counter terms with vacuum quark mass are introduced into $Ω$ and then the explicit $B$-dependent parts can be regularized in a cutoff-free way. Following that, explicit expressions of gap equation and magnetization can be consistently obtained according to the standard thermodynamic relations. The formalism is able to reproduce the paramagnetic feature of a QCD matter without ambiguity. For more realistic study, a running coupling constant is also adopted to account for the inverse magnetic catalysis effect. It turns out that the running coupling would greatly suppress magnetization at large $B$ and is important to reproduce the temperature enhancement effect to magnetization. The case with finite baryon chemical potential is also explored: no sign of first-order transition is found by varying $B$ for the running coupling and the de Haas-van Alphen oscillation shows up in the small $B$ region.

hep-ph↗

arXiv2023

Investigation of the tetraquark states $Qq\bar{Q} \bar{q}$ in the improved chromomagnetic interaction model

In the framework of the improved chromomagnetic interaction model, we complete a systematic study of the $S$-wave tetraquark states $Qq\bar{Q}\bar{q}$ ($Q=c,b$, and $q=u,d,s$) with different quantum numbers, $J^{PC}=0^{+(+)}$, $1^{+(\pm)}$, and $2^{+(+)}$. The mass spectra of tetraquark states are predicted and the possible decay channels are analyzed by considering both the angular momentum and $\mathcal{C}$-parity conservation. The recently observed hidden-charm tetraquark states with strangeness, such as $Z_{cs}(3985)^-$, $X(3960)$, and $Z_{cs}(4220)^+$, can be well explained in our model. Besides, based on the wave function of each tetraquark state, we find that the low-lying states of each $Qq\bar{Q}\bar{q}$ configuration have a large overlap to the $Q\bar Q$ and $q\bar q$ meson basis, instead of $Q\bar q$ and $q\bar Q$ meson basis. This indicates one can search these tetraquark states in future experiments via the channel of $Q\bar Q$ and $q\bar q$ mesons.

hep-ph↗

arXiv2023

Bi-color atomic beam slower and magnetic field compensation for ultracold gases

Transversely loaded bidimensional-magneto-optical-traps (2D-MOT) have been recently developed as high flux sources for cold strontium atoms to realize a new generation of compact experimental setups. Here, we discuss on the implementation of a cross-polarized bi-color slower for a strontium atomic beam improving the 2D-MOT loading, and increasing the number of atoms in a final MOT by eleven times. Our slowing scheme addresses simultaneously two excited Zeeman substates of the 88Sr 1S0->1P1 transition at 461 nm. We also realized a 3-axis active feedback control of the magnetic field down to the microgauss regime. Such a compensation is performed thanks to a network of eight magnetic field probes arranged in a cuboid configuration around the atomic cold sample, and a pair of coils in Helmholtz configuration along each of three Cartesian directions. Our active feedback is capable of efficiently suppressing most of the magnetically-induced position fluctuations of the 689~nm intercombination-line MOT.

physics.atom-ph↗

arXiv2022

Event-based Monocular Dense Depth Estimation with Recurrent Transformers

Event cameras, offering high temporal resolutions and high dynamic ranges, have brought a new perspective to address common challenges (e.g., motion blur and low light) in monocular depth estimation. However, how to effectively exploit the sparse spatial information and rich temporal cues from asynchronous events remains a challenging endeavor. To this end, we propose a novel event-based monocular depth estimator with recurrent transformers, namely EReFormer, which is the first pure transformer with a recursive mechanism to process continuous event streams. Technically, for spatial modeling, a novel transformer-based encoder-decoder with a spatial transformer fusion module is presented, having better global context information modeling capabilities than CNN-based methods. For temporal modeling, we design a gate recurrent vision transformer unit that introduces a recursive mechanism into transformers, improving temporal modeling capabilities while alleviating the expensive GPU memory cost. The experimental results show that our EReFormer outperforms state-of-the-art methods by a margin on both synthetic and real-world datasets. We hope that our work will attract further research to develop stunning transformers in the event-based vision community. Our open-source code can be found in the supplemental material.

cs.CV↗

arXiv2022

BEV-LGKD: A Unified LiDAR-Guided Knowledge Distillation Framework for BEV 3D Object Detection

Recently, Bird's-Eye-View (BEV) representation has gained increasing attention in multi-view 3D object detection, which has demonstrated promising applications in autonomous driving. Although multi-view camera systems can be deployed at low cost, the lack of depth information makes current approaches adopt large models for good performance. Therefore, it is essential to improve the efficiency of BEV 3D object detection. Knowledge Distillation (KD) is one of the most practical techniques to train efficient yet accurate models. However, BEV KD is still under-explored to the best of our knowledge. Different from image classification tasks, BEV 3D object detection approaches are more complicated and consist of several components. In this paper, we propose a unified framework named BEV-LGKD to transfer the knowledge in the teacher-student manner. However, directly applying the teacher-student paradigm to BEV features fails to achieve satisfying results due to heavy background information in RGB cameras. To solve this problem, we propose to leverage the localization advantage of LiDAR points. Specifically, we transform the LiDAR points to BEV space and generate the foreground mask and view-dependent mask for the teacher-student paradigm. It is to be noted that our method only uses LiDAR points to guide the KD between RGB models. As the quality of depth estimation is crucial for BEV perception, we further introduce depth distillation to our framework. Our unified framework is simple yet effective and achieves a significant performance boost. Code will be released.

cs.CV↗

arXiv2022

Formation of a supergiant quantum vortex in a relativistic Bose-Einstein condensate driven by rotation and a parallel magnetic field

Analysis based on the energy spectrum of noninteracting bosons shows that, under the circumstance of parallel rotation and magnetic field, charged bosons form a Bose-Einstein condensate because of the lift of the Landau level degeneracy by rotation [\textcolor{blue}{Y. Liu and I. Zahed, Phys. Rev. Lett. {\bf120}, 032001 (2018)}]. In this work, we study the interaction effect on the ground state of this Bose-Einstein condensate of charged bosons from the viewpoint of spontaneous symmetry breaking. We employ a minimal model for charged bosons with repulsive self-interaction. We find that the ground state of such a Bose-Einstein condensate is a supergiant quantum vortex, i.e., a quantized vortex with a large circulation. The size of the vortex is as large as the system size. The low-energy dispersion of the excitation spectra exhibits quadratic behavior, which is an anisotropic realization of the type-II Goldstone boson. Our study may give some implications to off-central relativistic heavy ion collisions, where large vorticity and magnetic fields can be generated.

nucl-th↗

arXiv2022

Do we need dense matter equation of state in curved spacetime for neutron stars?

Neutron stars are regarded as natural laboratories for the study of dense strong interaction matter. The equation of state (EoS) of dense matter computed in flat spacetime is used to predict the structure of neutron stars by solving the Tolman-Oppenheimer-Volkoff (TOV) equation. Recently, it has been reported that the curved spacetime effect or specifically gravitational time dilation effect on the EoS of dense matter leads to a significant increase of the maximum mass limit of neutron stars [Phys. Rev. D \textbf{104}, 123005 (2021) and J. Cosmol. Astropart. Phys. 02 (2021) 026]. However, in this work, we show that to study the hydrostatic equilibrium of dense matter within the framework of general relativity and relativistic fluid dynamics, the grand canonical EoS of dense matter, $p(T,μ)$, should be the same as that computed in flat spacetime, otherwise it is not consistent with local thermodynamic relations and energy-momentum conservation of the fluid. The gravitation influences the pressure $p$ only through enhancing the temperature $T$ and the chemical potential $μ$, known as Tolman's law and Klein's law. We rewrite the TOV equation as an alternative version so that the grand canonical EoS computed by using field theoretical methods can be used as a direct input. This may provide a tool to study the grand canonical EoS of dense matter via deep learning.

gr-qc↗

arXiv2022

Uncertainty Guided Depth Fusion for Spike Camera

Depth estimation is essential for various important real-world applications such as autonomous driving. However, it suffers from severe performance degradation in high-velocity scenario since traditional cameras can only capture blurred images. To deal with this problem, the spike camera is designed to capture the pixel-wise luminance intensity at high frame rate. However, depth estimation with spike camera remains very challenging using traditional monocular or stereo depth estimation algorithms, which are based on the photometric consistency. In this paper, we propose a novel Uncertainty-Guided Depth Fusion (UGDF) framework to fuse the predictions of monocular and stereo depth estimation networks for spike camera. Our framework is motivated by the fact that stereo spike depth estimation achieves better results at close range while monocular spike depth estimation obtains better results at long range. Therefore, we introduce a dual-task depth estimation architecture with a joint training strategy and estimate the distributed uncertainty to fuse the monocular and stereo results. In order to demonstrate the advantage of spike depth estimation over traditional camera depth estimation, we contribute a spike-depth dataset named CitySpike20K, which contains 20K paired samples, for spike depth estimation. UGDF achieves state-of-the-art results on CitySpike20K, surpassing all monocular or stereo spike depth estimation baselines. We conduct extensive experiments to evaluate the effectiveness and generalization of our method on CitySpike20K. To the best of our knowledge, our framework is the first dual-task fusion framework for spike camera depth estimation. Code and dataset will be released.

cs.CV↗

arXiv2022

Temporal Up-Sampling for Asynchronous Events

The event camera is a novel bio-inspired vision sensor. When the brightness change exceeds the preset threshold, the sensor generates events asynchronously. The number of valid events directly affects the performance of event-based tasks, such as reconstruction, detection, and recognition. However, when in low-brightness or slow-moving scenes, events are often sparse and accompanied by noise, which poses challenges for event-based tasks. To solve these challenges, we propose an event temporal up-sampling algorithm1 to generate more effective and reliable events. The main idea of our algorithm is to generate up-sampling events on the event motion trajectory. First, we estimate the event motion trajectory by contrast maximization algorithm and then up-sampling the events by temporal point processes. Experimental results show that up-sampling events can provide more effective information and improve the performance of downstream tasks, such as improving the quality of reconstructed images and increasing the accuracy of object detection.

cs.CV↗

arXiv2022

Event-based Video Reconstruction via Potential-assisted Spiking Neural Network

Neuromorphic vision sensor is a new bio-inspired imaging paradigm that reports asynchronous, continuously per-pixel brightness changes called `events' with high temporal resolution and high dynamic range. So far, the event-based image reconstruction methods are based on artificial neural networks (ANN) or hand-crafted spatiotemporal smoothing techniques. In this paper, we first implement the image reconstruction work via fully spiking neural network (SNN) architecture. As the bio-inspired neural networks, SNNs operating with asynchronous binary spikes distributed over time, can potentially lead to greater computational efficiency on event-driven hardware. We propose a novel Event-based Video reconstruction framework based on a fully Spiking Neural Network (EVSNN), which utilizes Leaky-Integrate-and-Fire (LIF) neuron and Membrane Potential (MP) neuron. We find that the spiking neurons have the potential to store useful temporal information (memory) to complete such time-dependent tasks. Furthermore, to better utilize the temporal information, we propose a hybrid potential-assisted framework (PA-EVSNN) using the membrane potential of spiking neuron. The proposed neuron is referred as Adaptive Membrane Potential (AMP) neuron, which adaptively updates the membrane potential according to the input spikes. The experimental results demonstrate that our models achieve comparable performance to ANN-based models on IJRR, MVSEC, and HQF datasets. The energy consumptions of EVSNN and PA-EVSNN are 19.36$\times$ and 7.75$\times$ more computationally efficient than their ANN architectures, respectively.

cs.CV↗

arXiv2022

1000x Faster Camera and Machine Vision with Ordinary Devices

In digital cameras, we find a major limitation: the image and video form inherited from a film camera obstructs it from capturing the rapidly changing photonic world. Here, we present vidar, a bit sequence array where each bit represents whether the accumulation of photons has reached a threshold, to record and reconstruct the scene radiance at any moment. By employing only consumer-level CMOS sensors and integrated circuits, we have developed a vidar camera that is 1,000x faster than conventional cameras. By treating vidar as spike trains in biological vision, we have further developed a spiking neural network-based machine vision system that combines the speed of the machine and the mechanism of biological vision, achieving high-speed object detection and tracking 1,000x faster than human vision. We demonstrate the utility of the vidar camera and the super vision system in an assistant referee and target pointing system. Our study is expected to fundamentally revolutionize the image and video concepts and related industries, including photography, movies, and visual media, and to unseal a new spiking neural network-enabled speed-free machine vision era.

cs.CV↗

arXiv2022

On abelian $2$-ramification torsion modules of quadratic fields

For a number field $F$ and a prime number $p$, the $\mathbb{Z}_p$-torsion module of the Galois group of the maximal abelian pro-$p$ extension of $F$ unramified outside $p$ over $F$, denoted as $\mathcal{T}_p(F)$, is an important subject in abelian $p$-ramification theory. In this paper we study the group $\mathcal{T}_2(F)=\mathcal{T}_2(m)$ of the quadratic field $F=\mathbb{Q}(\sqrt{ m})$. Firstly, assuming $m>0$, we prove an explicit $4$-rank formula for $\mathcal{T}_2(-m)$. Furthermore, applying this formula, we obtain the $4$-rank density of $\mathcal{T}_2$-groups of imaginary quadratic fields. Secondly, for $l$ an odd prime, we obtain results about the $2$-divisibility of orders of $\mathcal{T}_2(\pm l)$ and $\mathcal{T}_2(\pm 2l)$. In particular we find that $\#\mathcal{T}_2(l)\equiv 2\# \mathcal{T}_2(2l)\equiv h_2(-2l)\bmod{16}$ if $l\equiv 7\bmod{8}$ where $h_2(-2l)$ is the $2$-class number of $\mathbb{Q}(\sqrt{-2l})$. We then obtain density results for $\mathcal{T}_2(\pm l)$ and $\mathcal{T}_2(\pm 2l)$. Finally, based on our density results and numerical data, we propose distribution conjectures about $\mathcal{T}_p(F)$ when $F$ varies over real or imaginary quadratic fields for any prime $p$, and about $\mathcal{T}_2(\pm l)$ and $\mathcal{T}_2(\pm 2 l)$ when $l$ varies, in the spirit of Cohen-Lenstra heuristics. Our conjecture in the $\mathcal{T}_2(l)$ case is closely connected to Shanks-Sime-Washington's speculation on the distributions of the zeros of $2$-adic $L$-functions and to the distributions of the fundamental units.

math.NT↗

arXiv2022

Mass spectra of doubly heavy tetraquarks in an improved chromomagnetic interaction model

Doubly heavy tetraquark states are the prime candidates for tightly bound exotic states. We present a systematic study of the mass spectra of the $S$-wave doubly heavy tetraquark states $QQ\bar{q}\bar{q}$ ($q=u, d, s$ and $Q=c, b$) with different quantum numbers $J^P=0^+$, $1^+$, and $2^+$ in the framework of the improved chromomagnetic interaction (ICMI) model. The parameters in the ICMI model are obtained by fitting the conventional hadron spectra and are used directly to predict the masses of the tetraquark states. For heavy quarks, the uncertainties of the parameters are obtained by comparing the masses of doubly and triply heavy baryons with those given by lattice QCD, QCD sum rules, and potential models. Several compact and stable bound states are found in both the doubly charmed and doubly bottomed tetraquark systems. The predicted mass of the $cc\bar u\bar d$ state is consistent with the recent measurement from the LHCb collaboration.

hep-ph↗

arXiv2021

Concerted Rolling and Membrane Penetration Revealed by Atomistic Simulations of Antimicrobial Peptides

Short peptides with antimicrobial activity have therapeutic potential for treating bacterial infections. Mechanisms of actions for antimicrobial peptides require binding the biological membrane of their target, which often represents a key mechanistic step. A multitude of data-driven approaches have been developed to predict potential antimicrobial peptide sequences; however, these methods are usually agnostic to the physical interactions between the peptide and the membrane. Towards developing higher throughput screening methodologies, here we use Markov State Modeling and all-atom molecular dynamics simulations to quantify the membrane binding and insertion kinetics of three prototypical and antimicrobial peptides (alpha-helical magainin 2 and PGLa and beta-hairpin tachyplesin 1). By leveraging a set of collective variables that capture the essential physics of the amphiphilic and cationic peptide-membrane interactions we reveal how the slowest kinetic process of membrane insertion is the dynamic rolling of the peptide from a prebound to fully inserted state. These results add critical details to how antimicrobial peptides insert into bacterial membranes.

q-bio.BM↗