arXiv subjects

Hu Zhang

Publications and source records attributed to Hu Zhang.

At least 37 records · Page 2Linked to original sources

arXiv2024

Accurate and Efficient Event-based Semantic Segmentation Using Adaptive Spiking Encoder-Decoder Network

Spiking neural networks (SNNs), known for their low-power, event-driven computation and intrinsic temporal dynamics, are emerging as promising solutions for processing dynamic, asynchronous signals from event-based sensors. Despite their potential, SNNs face challenges in training and architectural design, resulting in limited performance in challenging event-based dense prediction tasks compared to artificial neural networks (ANNs). In this work, we develop an efficient spiking encoder-decoder network (SpikingEDN) for large-scale event-based semantic segmentation tasks. To enhance the learning efficiency from dynamic event streams, we harness the adaptive threshold which improves network accuracy, sparsity and robustness in streaming inference. Moreover, we develop a dual-path Spiking Spatially-Adaptive Modulation module, which is specifically tailored to enhance the representation of sparse events and multi-modal inputs, thereby considerably improving network performance. Our SpikingEDN attains a mean intersection over union (MIoU) of 72.57\% on the DDD17 dataset and 58.32\% on the larger DSEC-Semantic dataset, showing competitive results to the state-of-the-art ANNs while requiring substantially fewer computational resources. Our results shed light on the untapped potential of SNNs in event-based vision applications. The source code will be made publicly available.

cs.CV↗

arXiv2024

Approaching Outside: Scaling Unsupervised 3D Object Detection from 2D Scene

The unsupervised 3D object detection is to accurately detect objects in unstructured environments with no explicit supervisory signals. This task, given sparse LiDAR point clouds, often results in compromised performance for detecting distant or small objects due to the inherent sparsity and limited spatial resolution. In this paper, we are among the early attempts to integrate LiDAR data with 2D images for unsupervised 3D detection and introduce a new method, dubbed LiDAR-2D Self-paced Learning (LiSe). We argue that RGB images serve as a valuable complement to LiDAR data, offering precise 2D localization cues, particularly when scarce LiDAR points are available for certain objects. Considering the unique characteristics of both modalities, our framework devises a self-paced learning pipeline that incorporates adaptive sampling and weak model aggregation strategies. The adaptive sampling strategy dynamically tunes the distribution of pseudo labels during training, countering the tendency of models to overfit easily detected samples, such as nearby and large-sized objects. By doing so, it ensures a balanced learning trajectory across varying object scales and distances. The weak model aggregation component consolidates the strengths of models trained under different pseudo label distributions, culminating in a robust and powerful final model. Experimental evaluations validate the efficacy of our proposed LiSe method, manifesting significant improvements of +7.1% AP$_{BEV}$ and +3.4% AP$_{3D}$ on nuScenes, and +8.3% AP$_{BEV}$ and +7.4% AP$_{3D}$ on Lyft compared to existing techniques.

cs.CV↗

arXiv2024

EMBANet: A Flexible Efffcient Multi-branch Attention Network

This work presents a novel module, namely multi-branch concat (MBC), to process the input tensor and obtain the multi-scale feature map. The proposed MBC module brings new degrees of freedom (DoF) for the design of attention networks by allowing the type of transformation operators and the number of branches to be flexibly adjusted. Two important transformation operators, multiplex and split, are considered in this work, both of which can represent multi-scale features at a more granular level and increase the range of receptive fields. By integrating the MBC and attention module, a multi-branch attention (MBA) module is consequently developed to capture the channel-wise interaction of feature maps for establishing the long-range channel dependency. By substituting the 3x3 convolutions in the bottleneck blocks of the ResNet with the proposed MBA, a novel block namely efficient multi-branch attention (EMBA) is obtained, which can be easily plugged into the state-of-the-art backbone CNN models. Furthermore, a new backbone network called EMBANet is established by stacking the EMBA blocks. The proposed EMBANet is extensively evaluated on representative computer vision tasks including: classification, detection, and segmentation. And it demonstrates consistently superior performance over the popular backbones.

cs.CV↗

arXiv2024

Elemental noncentrosymmetric superconductors

Superconductivity has been discovered in many materials including elements, cuprates and nickelates. Noncentrosymmetric superconductors get attention due to their physical properties results from spatial inversion symmetry breaking. Here we propose the existence of elemental noncentrosymmetric superconductors which needs to break spatial inversion symmetry in single element phases realized by occupations of more than one Wyckoff positions. Elemental noncentrosymmetric superconductors allow us to study the admixture of spin-singlet and spin-triplet pairing states in elements with simple crystal structures and chemical conditions. We find that four-layer La thin films coming from bulk α-La with a double hexagonal close-packed structure are elemental noncentrosymmetric superconductors since the dimension reduction breaks the spatial inversion symmetry. In the normal state, four-layer La is a polar rare-earth metal with a spontaneous polarization having a polar layer group like ferroelectrics. Hence, to be more precise, four-layer La can also be called the elemental polar superconductor in the superconductivity state. Based on this method of breaking inversion symmetry, we discovery polar rare-earth metals, polar transition metals, polar alkali metals and polar alkali earth metals. Other candidate materials are bulk polar materials formed with group 15 elements, Bi thin films with polar stacking, polar double layer plumbene and Al thin films. The question of building the periodic table of elemental noncentrosymmetric superconductors is raised.

cond-mat.supr-con↗

arXiv2024

Automotive Object Detection via Learning Sparse Events by Spiking Neurons

Event-based sensors, distinguished by their high temporal resolution of 1 $\mathrmμ\text{s}$ and a dynamic range of 120 $\text{dB}$, stand out as ideal tools for deployment in fast-paced settings like vehicles and drones. Traditional object detection techniques that utilize Artificial Neural Networks (ANNs) face challenges due to the sparse and asynchronous nature of the events these sensors capture. In contrast, Spiking Neural Networks (SNNs) offer a promising alternative, providing a temporal representation that is inherently aligned with event-based data. This paper explores the unique membrane potential dynamics of SNNs and their ability to modulate sparse events. We introduce an innovative spike-triggered adaptive threshold mechanism designed for stable training. Building on these insights, we present a specialized spiking feature pyramid network (SpikeFPN) optimized for automotive event-based object detection. Comprehensive evaluations demonstrate that SpikeFPN surpasses both traditional SNNs and advanced ANNs enhanced with attention mechanisms. Evidently, SpikeFPN achieves a mean Average Precision (mAP) of 0.477 on the GEN1 Automotive Detection (GAD) benchmark dataset, marking significant increases over the selected SNN baselines. Moreover, the efficient design of SpikeFPN ensures robust performance while optimizing computational resources, attributed to its innate sparse computation capabilities. Source codes are publicly accessible at https://github.com/EMI-Group/spikefpn.

cs.CV↗

arXiv2024

STDEN: Towards Physics-Guided Neural Networks for Traffic Flow Prediction

High-performance traffic flow prediction model designing, a core technology of Intelligent Transportation System, is a long-standing but still challenging task for industrial and academic communities. The lack of integration between physical principles and data-driven models is an important reason for limiting the development of this field. In the literature, physics-based methods can usually provide a clear interpretation of the dynamic process of traffic flow systems but are with limited accuracy, while data-driven methods, especially deep learning with black-box structures, can achieve improved performance but can not be fully trusted due to lack of a reasonable physical basis. To bridge the gap between purely data-driven and physics-driven approaches, we propose a physics-guided deep learning model named Spatio-Temporal Differential Equation Network (STDEN), which casts the physical mechanism of traffic flow dynamics into a deep neural network framework. Specifically, we assume the traffic flow on road networks is driven by a latent potential energy field (like water flows are driven by the gravity field), and model the spatio-temporal dynamic process of the potential energy field as a differential equation network. STDEN absorbs both the performance advantage of data-driven models and the interpretability of physics-based models, so is named a physics-guided prediction model. Experiments on three real-world traffic datasets in Beijing show that our model outperforms state-of-the-art baselines by a significant margin. A case study further verifies that STDEN can capture the mechanism of urban traffic and generate accurate predictions with physical meaning. The proposed framework of differential equation network modeling may also cast light on other similar applications.

cs.LG↗

arXiv2024

Unified understanding to the rich electronic-structure evolutions of 2D black phosphorus under pressure

The electronic structure evolutions of few-layer black phosphorus (BP) under pressure shows a wealth of phenomena, such as the nonmonotonic change of direct gap at the Γ point, the layer-number dependence, and the distinct responses to normal and hydrostatic pressures. A full and unified understanding to these rich phenomena remains lacking. Here, we provide a unified understanding from the competition between interlayer quasi-bonding (QB) interactions and intralayer chemical bonding interactions. The former decreases while the latter increases the band gap under pressure and the origin can be correlated to different combinations of inter- and intra-layer antibonding or bonding interactions at the band edges. More interestingly, the interlayer QB interactions are a coexistence of two categories of interactions, namely, the coexistence of interactions between bands of the same occupancy (occupied-occupied and empty-empty interactions) and of different occupancies (occupied-empty interaction); and, the overall effect is a four-level interaction, which explains the anomalous interlayer-antibonding feature of the conduction band edge of bilayer BP. Our current study lay the foundation for the electronic structure tuning of two-dimensional (2D) BP, and, our analysis method for multi-energy-level interactions can be applied to other 2D semiconductor homo- and hetero-structures that have occupied-empty interlayer interactions.

cond-mat.mtrl-sci↗

arXiv2024

A Boosted-DCA with Power-Sum-DC Decomposition for Linearly Constrained Polynomial Programs

This paper proposes a novel Difference-of-Convex (DC) decomposition for polynomials using a power-sum representation, achieved by solving a sparse linear system. We introduce the Boosted DCA with Exact Line Search (BDCAe) for addressing linearly constrained polynomial programs within the DC framework. Notably, we demonstrate that the exact line search equates to determining the roots of a univariate polynomial in an interval, with coefficients being computed explicitly based on the power-sum DC decompositions. The subsequential convergence of BDCAe to critical points is proven, and its convergence rate under the Kurdyka-Lojasiewicz property is established. To efficiently tackle the convex subproblems, we integrate the Fast Dual Proximal Gradient (FDPG) method by exploiting the separable block structure of the power-sum DC decompositions. We validate our approach through numerical experiments on the Mean-Variance-Skewness-Kurtosis (MVSK) portfolio optimization model and box-constrained polynomial optimization problems. Comparative analysis of BDCAe against DCA, BDCA with Armijo line search, UDCA, and UBDCA with projective DC decomposition, alongside standard nonlinear optimization solvers FMINCON and FILTERSD, substantiates the efficiency of our proposed approach.

math.OC↗

arXiv2023

OpenSight: A Simple Open-Vocabulary Framework for LiDAR-Based Object Detection

Traditional LiDAR-based object detection research primarily focuses on closed-set scenarios, which falls short in complex real-world applications. Directly transferring existing 2D open-vocabulary models with some known LiDAR classes for open-vocabulary ability, however, tends to suffer from over-fitting problems: The obtained model will detect the known objects, even presented with a novel category. In this paper, we propose OpenSight, a more advanced 2D-3D modeling framework for LiDAR-based open-vocabulary detection. OpenSight utilizes 2D-3D geometric priors for the initial discernment and localization of generic objects, followed by a more specific semantic interpretation of the detected objects. The process begins by generating 2D boxes for generic objects from the accompanying camera images of LiDAR. These 2D boxes, together with LiDAR points, are then lifted back into the LiDAR space to estimate corresponding 3D boxes. For better generic object perception, our framework integrates both temporal and spatial-aware constraints. Temporal awareness correlates the predicted 3D boxes across consecutive timestamps, recalibrating the missed or inaccurate boxes. The spatial awareness randomly places some ``precisely'' estimated 3D boxes at varying distances, increasing the visibility of generic objects. To interpret the specific semantics of detected objects, we develop a cross-modal alignment and fusion module to first align 3D features with 2D image embeddings and then fuse the aligned 3D-2D features for semantic decoding. Our experiments indicate that our method establishes state-of-the-art open-vocabulary performance on widely used 3D detection benchmarks and effectively identifies objects for new categories of interest.

cs.CV↗

arXiv2023

Divide and Ensemble: Progressively Learning for the Unknown

In the wheat nutrient deficiencies classification challenge, we present the DividE and EnseMble (DEEM) method for progressive test data predictions. We find that (1) test images are provided in the challenge; (2) samples are equipped with their collection dates; (3) the samples of different dates show notable discrepancies. Based on the findings, we partition the dataset into discrete groups by the dates and train models on each divided group. We then adopt the pseudo-labeling approach to label the test data and incorporate those with high confidence into the training set. In pseudo-labeling, we leverage models ensemble with different architectures to enhance the reliability of predictions. The pseudo-labeling and ensembled model training are iteratively conducted until all test samples are labeled. Finally, the separated models for each group are unified to obtain the model for the whole dataset. Our method achieves an average of 93.6\% Top-1 test accuracy~(94.0\% on WW2020 and 93.2\% on WR2021) and wins the 1$st$ place in the Deep Nutrient Deficiency Challenge~\footnote{https://cvppa2023.github.io/challenges/}.

cs.AI↗

arXiv2023

Elemental topological ferroelectrics and polar metals of few-layer materials

Ferroelectricity can exist in elemental phases as a result of charge transfers between atoms occupying inequivalent Wyckoff positions. We investigate the emergence of ferroelectricity in two-dimensional elemental materials with buckled honeycomb lattices. Various multi-bilayer structures hosting ferroelectricity are designed by stacking-engineering. Ferroelectric materials candidates formed by group IV and V elements are predicted theoretically. Ultrathin Bi films show layer-stacking-dependent physical properties of ferroelectricity, topology, and metallicity. The two-bilayer Bi film with a polar stacking sequence is found to be an elemental topological ferroelectric material. Three and four bilayers Bi films with polar structures are ferroelectric-like elemental polar metals with topological nontrivial edge states. For Ge and Sn, trivial elemental polar metals are predicted. Our work reveals the possibility of design two-dimensional elemental topological ferroelectrics and polar metals by stacking-engineering.

cond-mat.mtrl-sci↗

arXiv2023

BAVS: Bootstrapping Audio-Visual Segmentation by Integrating Foundation Knowledge

Given an audio-visual pair, audio-visual segmentation (AVS) aims to locate sounding sources by predicting pixel-wise maps. Previous methods assume that each sound component in an audio signal always has a visual counterpart in the image. However, this assumption overlooks that off-screen sounds and background noise often contaminate the audio recordings in real-world scenarios. They impose significant challenges on building a consistent semantic mapping between audio and visual signals for AVS models and thus impede precise sound localization. In this work, we propose a two-stage bootstrapping audio-visual segmentation framework by incorporating multi-modal foundation knowledge. In a nutshell, our BAVS is designed to eliminate the interference of background noise or off-screen sounds in segmentation by establishing the audio-visual correspondences in an explicit manner. In the first stage, we employ a segmentation model to localize potential sounding objects from visual data without being affected by contaminated audio signals. Meanwhile, we also utilize a foundation audio classification model to discern audio semantics. Considering the audio tags provided by the audio foundation model are noisy, associating object masks with audio tags is not trivial. Thus, in the second stage, we develop an audio-visual semantic integration strategy (AVIS) to localize the authentic-sounding objects. Here, we construct an audio-visual tree based on the hierarchical correspondence between sounds and object categories. We then examine the label concurrency between the localized objects and classified audio tags by tracing the audio-visual tree. With AVIS, we can effectively segment real-sounding objects. Extensive experiments demonstrate the superiority of our method on AVS datasets, particularly in scenarios involving background noise. Our project website is https://yenanliu.github.io/AVSS.github.io/.

cs.CV↗

arXiv2023

Audio-Visual Segmentation by Exploring Cross-Modal Mutual Semantics

The audio-visual segmentation (AVS) task aims to segment sounding objects from a given video. Existing works mainly focus on fusing audio and visual features of a given video to achieve sounding object masks. However, we observed that prior arts are prone to segment a certain salient object in a video regardless of the audio information. This is because sounding objects are often the most salient ones in the AVS dataset. Thus, current AVS methods might fail to localize genuine sounding objects due to the dataset bias. In this work, we present an audio-visual instance-aware segmentation approach to overcome the dataset bias. In a nutshell, our method first localizes potential sounding objects in a video by an object segmentation network, and then associates the sounding object candidates with the given audio. We notice that an object could be a sounding object in one video but a silent one in another video. This would bring ambiguity in training our object segmentation network as only sounding objects have corresponding segmentation masks. We thus propose a silent object-aware segmentation objective to alleviate the ambiguity. Moreover, since the category information of audio is unknown, especially for multiple sounding sources, we propose to explore the audio-visual semantic correlation and then associate audio with potential objects. Specifically, we attend predicted audio category scores to potential instance masks and these scores will highlight corresponding sounding instances while suppressing inaudible ones. When we enforce the attended instance masks to resemble the ground-truth mask, we are able to establish audio-visual semantics correlation. Experimental results on the AVS benchmarks demonstrate that our method can effectively segment sounding objects without being biased to salient objects.

cs.SD↗

arXiv2023

RVD: A Handheld Device-Based Fundus Video Dataset for Retinal Vessel Segmentation

Retinal vessel segmentation is generally grounded in image-based datasets collected with bench-top devices. The static images naturally lose the dynamic characteristics of retina fluctuation, resulting in diminished dataset richness, and the usage of bench-top devices further restricts dataset scalability due to its limited accessibility. Considering these limitations, we introduce the first video-based retinal dataset by employing handheld devices for data acquisition. The dataset comprises 635 smartphone-based fundus videos collected from four different clinics, involving 415 patients from 50 to 75 years old. It delivers comprehensive and precise annotations of retinal structures in both spatial and temporal dimensions, aiming to advance the landscape of vasculature segmentation. Specifically, the dataset provides three levels of spatial annotations: binary vessel masks for overall retinal structure delineation, general vein-artery masks for distinguishing the vein and artery, and fine-grained vein-artery masks for further characterizing the granularities of each artery and vein. In addition, the dataset offers temporal annotations that capture the vessel pulsation characteristics, assisting in detecting ocular diseases that require fine-grained recognition of hemodynamic fluctuation. In application, our dataset exhibits a significant domain shift with respect to data captured by bench-top devices, thus posing great challenges to existing methods. In the experiments, we provide evaluation metrics and benchmark results on our dataset, reflecting both the potential and challenges it offers for vessel segmentation tasks. We hope this challenging dataset would significantly contribute to the development of eye disease diagnosis and early prevention.

cs.CV↗

arXiv2023

Momentum matching and band-alignment type in van der Waals heterostructures: Interfacial effects and materials screening

Momentum-matched type II van der Waals heterostructures (vdWHs) have been designed by assembling layered two-dimensional semiconductors (2DSs) with special band-structure combinations - that is, the valence band edge at the Gamma point (the Brillouin-zone center) for one 2DS and the conduction band edge at the Gamma point for the other [Ubrig et al., Nat. Mater. 19, 299 (2020)]. However, the band offset sizes, band-alignment types, and whether momentum matched or not, all are affected by the interfacial effects between the component 2DSs, such as the quasichemical-bonding (QB) interaction between layers and the electrical dipole moment formed around the vdW interface. Here, based on density-functional theory calculations, first we probe the interfacial effects (including different QBs for valence and conduction bands, interface dipole, and, the synergistic effects of these two aspects) on band-edge evolution in energy and valley (location in the Brillouin zone) and the resulting changes in band alignment and momentum matching for a typical vdWH of monolayer InSe and bilayer WS2, in which the band edges of subsystems satisfy the special band-structure combination for a momentum-matched type II vdWH. Then, based on the conclusions of the studied interfacial effects, we propose a practical screening method for robust momentum-matched type II vdWHs. This practical screening method can also be applied to other band alignment types. Our current study opens a way for practical screening and designing of vdWHs with robust momentum-matching and band alignment type.

cond-mat.mtrl-sci↗

arXiv2023

Copper-based charge transfer multiferroics with a $d^9$ configuration

Multiferroics are materials with a coexistence of magnetic and ferroelectric order allowing the manipulation of magnetism by applications of an electric field through magnetoelectric coupling effects. Here we propose an idea to design a class of multiferroics with a $d^9$ configuration using the magnetic order in copper-oxygen layers appearing in copper oxide high-temperature superconductors by inducing ferroelectricity. Copper-based charge transfer multiferroics SnCuO2 and PbCuO2 having the inversion symmetry breaking $P4mm$ polar space group are predicted to be such materials. The active inner s electrons in Sn and Pb hybridize with O $2p$ states leading the buckling in copper-oxygen layers and thus induces ferroelectricity, which is known as the lone pair mechanism. As a result of the $d^9$ configuration, SnCuO2 and PbCuO2 are charge transfer insulators with the antiferromagnetic ground state of the moment on Cu retaining some strongly correlated physical properties of parent compounds of copper oxide high-temperature superconductors. Our work reveals the possibility of designing multiferroics based on copper oxide high-temperature superconductors.

cond-mat.str-el↗

arXiv2023

From a superconductor NdNiO$_2$ to a Mott multiferroic BiNiO$_2$

Motivated by the recently discovered superconductivity in Sr-doped nickelate oxides NdNiO$_2$, we predict a material BiNiO$_2$ that provides an opportunity to study the intertwined ferroelectricity, metallicity, and magnetism in a crystal with very simple atomic structures. There is a ferroelectric structural transition from the nonpolar phase with the P4/mmm space group to the polar phase with the P4mm space group, which is driven by the lone pair on Bi. Calculations based on the Heyd-Scuseria-Ernzerhof hybrid density functional reveal that both the nonmagnetic and ferromagnetic states are metallic for nonpolar and polar phases, while the lowest energy ground-state for polar BiNiO$_2$ is a Hubbard Mott insulator with the G-type antiferromagnetic spin configurations. As a ferroelectric material with an electric polarization of 0.49 C/m2, it may be possible to control the magnetic order in BiNiO$_2$ by an applied electric field. The replacement of Nd by Bi serves as a connecting link between a high-temperature superconductor and a Mott multiferroic. Our work supports a route towards strongly correlated ferroelectrics.

cond-mat.str-el↗

arXiv2022

Birefringent Majorana fermions

We propose the concept of birefringent Majorana fermions as collective excitations of quantum interacting many-body states motivated by the already known quasiparticles including Dirac fermions, birefringent Dirac fermions and Majorana fermions. Low energy effective models are constructed to understand the basic mathematical structures of equations of motion and physical properties. We find that the equation of motion, with two different "speed of light", describing birefringent Majorana fermions in condensed matter physics is the birefringence version of the original Majorana equation of real relativistic fermions.

cond-mat.mes-hall↗