arXiv subjects

Bing Yang

Publications and source records attributed to Bing Yang.

At least 37 records · Page 2Linked to original sources

arXiv2024

STNet: Deep Audio-Visual Fusion Network for Robust Speaker Tracking

Audio-visual speaker tracking aims to determine the location of human targets in a scene using signals captured by a multi-sensor platform, whose accuracy and robustness can be improved by multi-modal fusion methods. Recently, several fusion methods have been proposed to model the correlation in multiple modalities. However, for the speaker tracking problem, the cross-modal interaction between audio and visual signals hasn't been well exploited. To this end, we present a novel Speaker Tracking Network (STNet) with a deep audio-visual fusion model in this work. We design a visual-guided acoustic measurement method to fuse heterogeneous cues in a unified localization space, which employs visual observations via a camera model to construct the enhanced acoustic map. For feature fusion, a cross-modal attention module is adopted to jointly model multi-modal contexts and interactions. The correlated information between audio and visual features is further interacted in the fusion model. Moreover, the STNet-based tracker is applied to multi-speaker cases by a quality-aware module, which evaluates the reliability of multi-modal observations to achieve robust tracking in complex scenarios. Experiments on the AV16.3 and CAV3D datasets show that the proposed STNet-based tracker outperforms uni-modal methods and state-of-the-art audio-visual speaker trackers.

cs.CV↗

arXiv2024

RealMAN: A Real-Recorded and Annotated Microphone Array Dataset for Dynamic Speech Enhancement and Localization

The training of deep learning-based multichannel speech enhancement and source localization systems relies heavily on the simulation of room impulse response and multichannel diffuse noise, due to the lack of large-scale real-recorded datasets. However, the acoustic mismatch between simulated and real-world data could degrade the model performance when applying in real-world scenarios. To bridge this simulation-to-real gap, this paper presents a new relatively large-scale Real-recorded and annotated Microphone Array speech&Noise (RealMAN) dataset. The proposed dataset is valuable in two aspects: 1) benchmarking speech enhancement and localization algorithms in real scenarios; 2) offering a substantial amount of real-world training data for potentially improving the performance of real-world applications. Specifically, a 32-channel array with high-fidelity microphones is used for recording. A loudspeaker is used for playing source speech signals (about 35 hours of Mandarin speech). A total of 83.7 hours of speech signals (about 48.3 hours for static speaker and 35.4 hours for moving speaker) are recorded in 32 different scenes, and 144.5 hours of background noise are recorded in 31 different scenes. Both speech and noise recording scenes cover various common indoor, outdoor, semi-outdoor and transportation environments, which enables the training of general-purpose speech enhancement and source localization networks. To obtain the task-specific annotations, speaker location is annotated with an omni-directional fisheye camera by automatically detecting the loudspeaker. The direct-path signal is set as the target clean speech for speech enhancement, which is obtained by filtering the source speech signal with an estimated direct-path propagation filter.

cs.SD↗

arXiv2024

Self-Supervised Learning of Spatial Acoustic Representation with Cross-Channel Signal Reconstruction and Multi-Channel Conformer

Supervised learning methods have shown effectiveness in estimating spatial acoustic parameters such as time difference of arrival, direct-to-reverberant ratio and reverberation time. However, they still suffer from the simulation-to-reality generalization problem due to the mismatch between simulated and real-world acoustic characteristics and the deficiency of annotated real-world data. To this end, this work proposes a self-supervised method that takes full advantage of unlabeled data for spatial acoustic parameter estimation. First, a new pretext task, i.e. cross-channel signal reconstruction (CCSR), is designed to learn a universal spatial acoustic representation from unlabeled multi-channel microphone signals. We mask partial signals of one channel and ask the model to reconstruct them, which makes it possible to learn spatial acoustic information from unmasked signals and extract source information from the other microphone channel. An encoder-decoder structure is used to disentangle the two kinds of information. By fine-tuning the pre-trained spatial encoder with a small annotated dataset, this encoder can be used to estimate spatial acoustic parameters. Second, a novel multi-channel audio Conformer (MC-Conformer) is adopted as the encoder model architecture, which is suitable for both the pretext and downstream tasks. It is carefully designed to be able to capture the local and global characteristics of spatial acoustics exhibited in the time-frequency domain. Experimental results of five acoustic parameter estimation tasks on both simulated and real-world data show the effectiveness of the proposed method. To the best of our knowledge, this is the first self-supervised learning method in the field of spatial acoustic representation learning and multi-channel audio signal processing.

cs.SD↗

arXiv2024

SigDLA: A Deep Learning Accelerator Extension for Signal Processing

Deep learning and signal processing are closely correlated in many IoT scenarios such as anomaly detection to empower intelligence of things. Many IoT processors utilize digital signal processors (DSPs) for signal processing and build deep learning frameworks on this basis. While deep learning is usually much more computing-intensive than signal processing, the computing efficiency of deep learning on DSPs is limited due to the lack of native hardware support. In this case, we present a contrary strategy and propose to enable signal processing on top of a classical deep learning accelerator (DLA). With the observation that irregular data patterns such as butterfly operations in FFT are the major barrier that hinders the deployment of signal processing on DLAs, we propose a programmable data shuffling fabric and have it inserted between the input buffer and computing array of DLAs such that the irregular data is reorganized and the processing is converted to be regular. With the online data shuffling, the proposed architecture, SigDLA, can adapt to various signal processing tasks without affecting the deep learning processing. Moreover, we build a reconfigurable computing array to suit the various data width requirements of both signal processing and deep learning. According to our experiments, SigDLA achieves an average performance speedup of 4.4$\times$, 1.4$\times$, and 1.52$\times$, and average energy reduction of 4.82$\times$, 3.27$\times$, and 2.15$\times$ compared to an embedded ARM processor with customized DSP instructions, a DSP processor, and an independent DSP-DLA architecture respectively with 17% more chip area over the original DLAs.

cs.AR↗

arXiv2024

IPDnet: A Universal Direct-Path IPD Estimation Network for Sound Source Localization

Extracting direct-path spatial feature is crucial for sound source localization in adverse acoustic environments. This paper proposes the IPDnet, a neural network that estimates direct-path inter-channel phase difference (DP-IPD) of sound sources from microphone array signals. The estimated DP-IPD can be easily translated to source location based on the known microphone array geometry. First, a full-band and narrow-band fusion network is proposed for DP-IPD estimation, in which alternating narrow-band and full-band layers are responsible for estimating the rough DP-IPD information in one frequency band and capturing the frequency correlations of DP-IPD, respectively. Second, a new multi-track DP-IPD learning target is proposed for the localization of flexible number of sound sources. Third, the IPDnet is extend to handling variable microphone arrays, once trained which is able to process arbitrary microphone arrays with different number of channels and array topology. Experiments of multiple-moving-speaker localization are conducted on both simulated and real-world data, which show that the proposed full-band and narrow-band fusion network and the proposed multi-track DP-IPD learning target together achieves excellent sound source localization performance. Moreover, the proposed variable-array model generalizes well to unseen microphone arrays.

eess.AS↗

arXiv2024

Soft Error Reliability Analysis of Vision Transformers

Vision Transformers (ViTs) that leverage self-attention mechanism have shown superior performance on many classical vision tasks compared to convolutional neural networks (CNNs) and gain increasing popularity recently. Existing ViTs works mainly optimize performance and accuracy, but ViTs reliability issues induced by soft errors in large-scale VLSI designs have generally been overlooked. In this work, we mainly study the reliability of ViTs and investigate the vulnerability from different architecture granularities ranging from models, layers, modules, and patches for the first time. The investigation reveals that ViTs with the self-attention mechanism are generally more resilient on linear computing including general matrix-matrix multiplication (GEMM) and full connection (FC) and show a relatively even vulnerability distribution across the patches. ViTs involve more fragile non-linear computing such as softmax and GELU compared to typical CNNs. With the above observations, we propose a lightweight block-wise algorithm-based fault tolerance (LB-ABFT) approach to protect the linear computing implemented with distinct sizes of GEMM and apply a range-based protection scheme to mitigate soft errors in non-linear computing. According to our experiments, the proposed fault-tolerant approaches enhance ViTs accuracy significantly with minor computing overhead in presence of various soft errors.

cs.CR↗

arXiv2024

High-rectification near-field radiative thermal diode using Weyl semimetals

Thermal diodes, which allow heat transfer in a preferential direction while being blocked in a reverse direction, have numerous applications in thermal management, information processing, energy harvesting, etc. Typical materials of thermal diodes in previous works include phase-change and magneto-optical materials. However, such thermal diodes highly depend on specific working temperatures or external magnetic fields. In this work, we propose a near-field radiative thermal diode (NFRTD) based on two Weyl semimetals (WSMs) nanoparticles (NPs) mediated by WSMs planar substrate, which works without external magnetic field and with flexible temperatures. Numerical results show that the maximum rectification ratio of NFRTD can be up to 2673 when the emitter is 200 K and receiver is 180 K, which exceeds the maximum value reported in previous works by more than 10 times. The underlying physical mechanism is the strong coupling of the localized plasmon modes in the NPs and nonreciprocal surface plasmon polaritons in the substrate. In addition, we calculate the distribution of the Green function and reflection coefficient to investigate nonreciprocal energy transfer in NFRTD. Finally, we discuss the effects of momentum-separation on the rectification performance of the NFRTD. This work demonstrates the great potential of WSMs in thermal rectification and paves a novel path in designing high-performance NFRTD.

physics.optics↗

arXiv2024

Ergodicity Breaking Under Confinement in Cold-Atom Quantum Simulators

The quantum simulation of gauge theories on synthetic quantum matter devices has gained a lot of traction in the last decade, making possible the observation of a range of exotic quantum many-body phenomena. In this work, we consider the spin-$1/2$ quantum link formulation of $1+1$D quantum electrodynamics with a topological $θ$-angle, which can be used to tune a confinement-deconfinement transition. Exactly mapping this system onto a PXP model with mass and staggered magnetization terms, we show an intriguing interplay between confinement and the ergodicity-breaking paradigms of quantum many-body scarring and Hilbert-space fragmentation. We map out the rich dynamical phase diagram of this model, finding an ergodic phase at small values of the mass $μ$ and confining potential $χ$, an emergent integrable phase for large $μ$, and a fragmented phase for large values of both parameters. We also show that the latter hosts resonances that lead to a vast array of effective models. We propose experimental probes of our findings, which can be directly accessed in current cold-atom setups.

cond-mat.quant-gas↗

arXiv2024

Asymmetric Nucleation Processes in Spontaneous Mode Switch of Active Matter

Flocking and vortical are two typical motion modes in active matter. Although it is known that the two modes can spontaneously switch between each other in a finite-size system, the switching dynamics remain elusive. In this work, by computer simulation of a two-dimensional Vicsek-like system with 1000 particles, we find from the perspective of classical nucleation theory that the forward and backward switching dynamics are asymmetric: from flocking to vortical is a one-step nucleation process, while the opposite is a two-step nucleation process with the system staying in a metastable state before reaching the final flocking state.

cond-mat.soft↗

arXiv2023

Cold-atom quantum simulators of gauge theories

Gauge theories represent a fundamental framework underlying modern physics, constituting the basis of the Standard Model and also providing useful descriptions of various phenomena in condensed matter. Realizing gauge theories on accessible and tunable tabletop quantum devices offers the possibility to study their dynamics from first principles time evolution and to probe their exotic physics, including that generated by deviations from gauge invariance, which is not possible, e.g., in dedicated particle colliders. Not only do cold-atom quantum simulators hold the potential to provide new insights into outstanding high-energy and nuclear-physics questions, they also provide a versatile tool for the exploration of topological phases and ergodicity-breaking mechanisms relevant to low-energy many-body physics. In recent years, cold-atom quantum simulators have demonstrated impressive progress in the large-scale implementation of $1+1$D Abelian gauge theories. In this Review, we chronicle the progress of cold-atom quantum simulators of gauge theories, highlighting the crucial advancements achieved along the way in order to reliably stabilize gauge invariance and go from building blocks to large-scale realizations where \textit{bona fide} gauge-theory phenomena can be probed. We also provide a brief outlook on where this field is heading, and what is required experimentally and theoretically to bring the technology to the next level by surveying various concrete proposals for advancing these setups to higher spatial dimensions, higher-spin representations of the gauge field, and non-Abelian gauge groups.

cond-mat.quant-gas↗

arXiv2023

Noise Measurement of a Wind Turbine using Thick Blades with Blunt Trailing Edge

The noise generated by wind turbines can potentially cause significant harm to the ecological environment and the living conditions of residents. Therefore, a proper assessment of wind turbine noise is crucial. The IEC 61400-11 standard provides standardized guidelines for measuring turbine noise, facilitating the comparison of noise characteristics among different wind turbine models. This work aims to conduct a comprehensive noise measurement of a 100kW wind turbine using thick blades with blunt trailing edge, which differs from the typical turbines studied previously. The work takes into account the unique design and dynamic characteristics of small-scale wind turbines and adjusts the measurement accordingly, with deviations from the IEC standards will be explicitly addressed.

physics.flu-dyn↗

arXiv2023

DeepBurning-MixQ: An Open Source Mixed-Precision Neural Network Accelerator Design Framework for FPGAs

Mixed-precision neural networks (MPNNs) that enable the use of just enough data width for a deep learning task promise significant advantages of both inference accuracy and computing overhead. FPGAs with fine-grained reconfiguration capability can adapt the processing with distinct data width and models, and hence, can theoretically unleash the potential of MPNNs. Nevertheless, commodity DPUs on FPGAs mostly emphasize generality and have limited support for MPNNs especially the ones with lower data width. In addition, primitive DSPs in FPGAs usually have much larger data width than that is required by MPNNs and haven't been sufficiently co-explored with MPNNs yet. To this end, we propose an open source MPNN accelerator design framework specifically tailored for FPGAs. In this framework, we have a systematic DSP-packing algorithm to pack multiple lower data width MACs in a single primitive DSP and enable efficient implementation of MPNNs. Meanwhile, we take DSP packing efficiency into consideration with MPNN quantization within a unified neural network architecture search (NAS) framework such that it can be aware of the DSP overhead during quantization and optimize the MPNN performance and accuracy concurrently. Finally, we have the optimized MPNN fine-tuned to a fully pipelined neural network accelerator template based on HLS and make best use of available resources for higher performance. Our experiments reveal the resulting accelerators produced by the proposed framework can achieve overwhelming advantages in terms of performance, resource utilization, and inference accuracy for MPNNs when compared with both handcrafted counterparts and prior hardware-aware neural network accelerators on FPGAs.

cs.AR↗

arXiv2023

Aeroacoustic Source Localization

The deconvolutional DAMAS algorithm can effectively eliminate the misconceptions in the usually-used beamforming localization algorithm, allowing for more accurate calculation of the source location as well as the intensity. When solving a linear system of equations, the DAMAS algorithm takes into account the mutual influence of different locations, reducing or even eliminating sidelobes and producing more accurate results. This work first introduces the principles of the DAMAS algorithm. Then it applies both the beamforming algorithm and the DAMAS algorithm to simulate the localization of a single-frequency source from a 1.5 MW wind turbine, a complex line source with the text "UCAS" and a line source downstream of an airfoil trailing edge. Finally, the work presents experimental localization results of the source of a 1.5 MW wind turbine using both the beamforming algorithm and the DAMAS algorithm.

cs.SD↗

arXiv2023

Observation of microscopic confinement dynamics by a tunable topological $θ$-angle

The topological $θ$-angle is central to the understanding of a plethora of phenomena in condensed matter and high-energy physics such as the strong CP problem, dynamical quantum topological phase transitions, and the confinement--deconfinement transition. Difficulties arise when probing the effects of the topological $θ$-angle using classical methods, in particular through the appearance of a sign problem in numerical simulations. Quantum simulators offer a powerful alternate venue for realizing the $θ$-angle, which has hitherto remained an outstanding challenge due to the difficulty of introducing a dynamical electric field in the experiment. Here, we report on the experimental realization of a tunable topological $θ$-angle in a Bose--Hubbard gauge-theory quantum simulator, implemented through a tilted superlattice potential that induces an effective background electric field. We demonstrate the rich physics due to this angle by the direct observation of the confinement--deconfinement transition of $(1+1)$-dimensional quantum electrodynamics. Using an atomic-precision quantum gas microscope, we distinguish between the confined and deconfined phases by monitoring the real-time evolution of particle--antiparticle pairs, which exhibit constrained (ballistic) propagation for a finite (vanishing) deviation of the $θ$-angle from $π$. Our work provides a major step forward in the realization of topological terms on modern quantum simulators, and the exploration of rich physics they have been theorized to entail.

cond-mat.quant-gas↗

arXiv2023

FN-SSL: Full-Band and Narrow-Band Fusion for Sound Source Localization

Extracting direct-path spatial features is critical for sound source localization in adverse acoustic environments. This paper proposes a full-band and narrow-band fusion network for estimating direct-path inter-channel phase difference (DP-IPD) from microphone signals. The alternating full-band and narrow-band layers are responsible for learning the full-band correlation and narrow-band extraction of DP-IPD, respectively. Experiments show that the proposed network noticeably outperforms other advanced methods on both simulated and real-world data.

eess.AS↗

arXiv2023

Attention Mechanisms in Medical Image Segmentation: A Survey

Medical image segmentation plays an important role in computer-aided diagnosis. Attention mechanisms that distinguish important parts from irrelevant parts have been widely used in medical image segmentation tasks. This paper systematically reviews the basic principles of attention mechanisms and their applications in medical image segmentation. First, we review the basic concepts of attention mechanism and formulation. Second, we surveyed over 300 articles related to medical image segmentation, and divided them into two groups based on their attention mechanisms, non-Transformer attention and Transformer attention. In each group, we deeply analyze the attention mechanisms from three aspects based on the current literature work, i.e., the principle of the mechanism (what to use), implementation methods (how to use), and application tasks (where to use). We also thoroughly analyzed the advantages and limitations of their applications to different tasks. Finally, we summarize the current state of research and shortcomings in the field, and discuss the potential challenges in the future, including task specificity, robustness, standard evaluation, etc. We hope that this review can showcase the overall research context of traditional and Transformer attention methods, provide a clear reference for subsequent research, and inspire more advanced attention research, not only in medical image segmentation, but also in other image analysis scenarios.

eess.IV↗

arXiv2023

Observation of many-body scarring in a Bose--Hubbard quantum simulator

The ongoing quest for understanding nonequilibrium dynamics of complex quantum systems underpins the foundation of statistical physics as well as the development of quantum technology. Quantum many-body scarring has recently opened a window into novel mechanisms for delaying the onset of thermalization by preparing the system in special initial states, such as the $\mathbb{Z}_2$ state in a Rydberg atom system. Here we realize many-body scarring in a Bose-Hubbard quantum simulator from previously unknown initial conditions such as the unit-filling state. We develop a quantum-interference protocol for measuring the entanglement entropy and demonstrate that scarring traps the many-body system in a low-entropy subspace. Our work makes the resource of scarring accessible to a broad class of ultracold-atom experiments, and it allows one to explore the relation of scarring to constrained dynamics in lattice gauge theories, Hilbert space fragmentation, and disorder-free localization.

cond-mat.quant-gas↗

arXiv2023

Spin-$S$ $\mathrm{U}(1)$ Quantum Link Models with Dynamical Matter on a Quantum Simulator

Quantum link models (QLMs) offer the realistic prospect for the practical implementation of lattice quantum electrodynamics (QED) on modern quantum simulators, and they provide a venue for exploring various nonergodic phenomena relevant to quantum many-body physics. In these models, gauge and electric fields are represented by spin-$S$ operators. So far, large-scale realizations of QLMs have been restricted to $S=1/2$ representations, whereas the lattice-QED limit is approached at $S\to\infty$. Here, we present a bosonic mapping for the representation of gauge and electric fields with effective spin-$S$ operators for arbitrarily large values of $S$. Based on this mapping, we then propose an experimental scheme for the realization of a large-scale spin-$1$ $\mathrm{U}(1)$ QLM using spinless bosons in an optical superlattice. Using perturbation theory and infinite matrix product state calculations, which work directly in the thermodynamic limit, we demonstrate the faithfulness of the mapping and stability of gauge invariance throughout all accessible evolution times. We further demonstrate the potential of our proposed quantum simulator to address relevant high-energy physics by probing the (de)confinement of an electron--positron pair by tuning the gauge coupling. Our work provides an essential step towards gauge-theory quantum simulators in the quantum-field-theory limit.

cond-mat.quant-gas↗