arXiv subjects

Shilei Zhang

Publications and source records attributed to Shilei Zhang.

At least 37 records · Page 2Linked to original sources

arXiv2024

Exploring Energy-Based Models for Out-of-Distribution Detection in Dialect Identification

The diverse nature of dialects presents challenges for models trained on specific linguistic patterns, rendering them susceptible to errors when confronted with unseen or out-of-distribution (OOD) data. This study introduces a novel margin-enhanced joint energy model (MEJEM) tailored specifically for OOD detection in dialects. By integrating a generative model and the energy margin loss, our approach aims to enhance the robustness of dialect identification systems. Furthermore, we explore two OOD scores for OOD dialect detection, and our findings conclusively demonstrate that the energy score outperforms the softmax score. Leveraging Sharpness-Aware Minimization to optimize the training process of the joint model, we enhance model generalization by minimizing both loss and sharpness. Experiments conducted on dialect identification tasks validate the efficacy of Energy-Based Models and provide valuable insights into their performance.

cs.CL↗

arXiv2024

GenDistiller: Distilling Pre-trained Language Models based on an Autoregressive Generative Model

Pre-trained speech language models such as HuBERT and WavLM leverage unlabeled speech data for self-supervised learning and offer powerful representations for numerous downstream tasks. Despite the success of these models, their high requirements for memory and computing resource hinder their application on resource restricted devices. Therefore, this paper introduces GenDistiller, a novel knowledge distillation framework which generates the hidden representations of the pre-trained teacher model directly by a much smaller student network. The proposed method takes the previous hidden layer as history and implements a layer-by-layer prediction of the teacher model autoregressively. Experiments on SUPERB reveal the advantage of GenDistiller over the baseline distilling method without an autoregressive framework, with 33% fewer parameters, similar time consumption and better performance on most of the SUPERB tasks. Ultimately, the proposed GenDistiller reduces the size of WavLM by 82%.

eess.AS↗

arXiv2024

CEC: A Noisy Label Detection Method for Speaker Recognition

Noisy labels are inevitable, even in well-annotated datasets. The detection of noisy labels is of significant importance to enhance the robustness of speaker recognition models. In this paper, we propose a novel noisy label detection approach based on two new statistical metrics: Continuous Inconsistent Counting (CIC) and Total Inconsistent Counting (TIC). These metrics are calculated through Cross-Epoch Counting (CEC) and correspond to the early and late stages of training, respectively. Additionally, we categorize samples based on their prediction results into three categories: inconsistent samples, hard samples, and easy samples. During training, we gradually increase the difficulty of hard samples to update model parameters, preventing noisy labels from being overfitted. Compared to contrastive schemes, our approach not only achieves the best performance in speaker verification but also excels in noisy label detection.

eess.AS↗

arXiv2024

PolySpeech: Exploring Unified Multitask Speech Models for Competitiveness with Single-task Models

Recently, there have been attempts to integrate various speech processing tasks into a unified model. However, few previous works directly demonstrated that joint optimization of diverse tasks in multitask speech models has positive influence on the performance of individual tasks. In this paper we present a multitask speech model -- PolySpeech, which supports speech recognition, speech synthesis, and two speech classification tasks. PolySpeech takes multi-modal language model as its core structure and uses semantic representations as speech inputs. We introduce semantic speech embedding tokenization and speech reconstruction methods to PolySpeech, enabling efficient generation of high-quality speech for any given speaker. PolySpeech shows competitiveness across various tasks compared to single-task models. In our experiments, multitask optimization achieves performance comparable to single-task optimization and is especially beneficial for specific tasks.

cs.CL↗

arXiv2024

Plugin Speech Enhancement: A Universal Speech Enhancement Framework Inspired by Dynamic Neural Network

The expectation to deploy a universal neural network for speech enhancement, with the aim of improving noise robustness across diverse speech processing tasks, faces challenges due to the existing lack of awareness within static speech enhancement frameworks regarding the expected speech in downstream modules. These limitations impede the effectiveness of static speech enhancement approaches in achieving optimal performance for a range of speech processing tasks, thereby challenging the notion of universal applicability. The fundamental issue in achieving universal speech enhancement lies in effectively informing the speech enhancement module about the features of downstream modules. In this study, we present a novel weighting prediction approach, which explicitly learns the task relationships from downstream training information to address the core challenge of universal speech enhancement. We found the role of deciding whether to employ data augmentation techniques as crucial downstream training information. This decision significantly impacts the expected speech and the performance of the speech enhancement module. Moreover, we introduce a novel speech enhancement network, the Plugin Speech Enhancement (Plugin-SE). The Plugin-SE is a dynamic neural network that includes the speech enhancement module, gate module, and weight prediction module. Experimental results demonstrate that the proposed Plugin-SE approach is competitive or superior to other joint training methods across various downstream tasks.

eess.AS↗

arXiv2023

Cascaded Multi-task Adaptive Learning Based on Neural Architecture Search

Cascading multiple pre-trained models is an effective way to compose an end-to-end system. However, fine-tuning the full cascaded model is parameter and memory inefficient and our observations reveal that only applying adapter modules on cascaded model can not achieve considerable performance as fine-tuning. We propose an automatic and effective adaptive learning method to optimize end-to-end cascaded multi-task models based on Neural Architecture Search (NAS) framework. The candidate adaptive operations on each specific module consist of frozen, inserting an adapter and fine-tuning. We further add a penalty item on the loss to limit the learned structure which takes the amount of trainable parameters into account. The penalty item successfully restrict the searched architecture and the proposed approach is able to search similar tuning scheme with hand-craft, compressing the optimizing parameters to 8.7% corresponding to full fine-tuning on SLURP with an even better performance.

cs.LG↗

arXiv2023

GenDistiller: Distilling Pre-trained Language Models based on Generative Models

Self-supervised pre-trained models such as HuBERT and WavLM leverage unlabeled speech data for representation learning and offer significantly improve for numerous downstream tasks. Despite the success of these methods, their large memory and strong computational requirements hinder their application on resource restricted devices. Therefore, this paper introduces GenDistiller, a novel knowledge distillation framework to distill hidden representations from teacher network based on generative language model. The generative structure enables the proposed model to generate the target teacher hidden layers autoregressively, considering the interactions between hidden layers without instroducing additional inputs. A two-dimensional attention mechanism is implemented to ensure the causality of hidden layers, while preserving bidirectional attention in the time dimension. Experiments reveal the advantage of the generative distiller over the baseline system that predicts the hidden layers of teacher network directly without a generatvie model.

eess.AS↗

arXiv2023

VE-KWS: Visual Modality Enhanced End-to-End Keyword Spotting

The performance of the keyword spotting (KWS) system based on audio modality, commonly measured in false alarms and false rejects, degrades significantly under the far field and noisy conditions. Therefore, audio-visual keyword spotting, which leverages complementary relationships over multiple modalities, has recently gained much attention. However, current studies mainly focus on combining the exclusively learned representations of different modalities, instead of exploring the modal relationships during each respective modeling. In this paper, we propose a novel visual modality enhanced end-to-end KWS framework (VE-KWS), which fuses audio and visual modalities from two aspects. The first one is utilizing the speaker location information obtained from the lip region in videos to assist the training of multi-channel audio beamformer. By involving the beamformer as an audio enhancement module, the acoustic distortions, caused by the far field or noisy environments, could be significantly suppressed. The other one is conducting cross-attention between different modalities to capture the inter-modal relationships and help the representation learning of each modality. Experiments on the MSIP challenge corpus show that our proposed model achieves 2.79% false rejection rate and 2.95% false alarm rate on the Eval set, resulting in a new SOTA performance compared with the top-ranking systems in the ICASSP2022 MISP challenge.

cs.SD↗

arXiv2022

Time-resolved measurement of spin excitations in Cu$_2$OSeO$_3$

Magnetic diffraction in combination with x-ray detected ferromagnetic resonance (DFMR) is a powerful technique for performing time-resolved measurements on individual spin textures. Here, we study the ferromagnetic resonance (FMR) modes of both the conical and field-polarized phases in the chiral magnet Cu$_2$OSeO$_3$. Following the identification of the FMR modes at different temperatures using broadband vector network analyzer FMR, we use DFMR on the crystalline (001) Bragg peak to reveal the time-dependent spin configurations of the selected FMR modes. By being able to measure both the amplitude and phase response of the spin system across the resonance, a continuous phase advance (of 180$^\circ$) in the conical mode, and a phase lag (of 180$^\circ$) in the field-polarized mode is found. By performing dynamic measurements in the conical phase as a function of the linear polarization angle of the x-rays, i.e., successively probing the dynamics of the moments, we find an inversion of the dynamics along the conical axis upon inverting the applied field direction. By allowing for time-resolved measurements of the phase and amplitude of individual magnetic phases, DFMR opens up new opportunities for obtaining a deeper understanding of the complex dynamics of chiral magnets.

cond-mat.mtrl-sci↗

arXiv2022

Persistent exchange splitting in a chiral helimagnet Cr1/3NbS2

Using high-resolution angle-resolved photoemission spectroscopy (ARPES) and ab-initio calculation, we systematically investigate the electronic structure of the chiral helimagnet Cr1/3NbS2 and its temperature evolution. The comparison with NbS2 suggests that the electronic structure of Cr1/3NbS2 is strongly modified by the intercalation of Cr atoms. Our ab-initio calculation, consistent with experimental result, suggests strong hybridization between Nb- and Cr-derived states near the Fermi level. In the chiral helimagnetic state (below the Curie temperature Tc), we observe exchange splitting of the energy bands crossing EF, which follows the temperature evolution of the magnetic moment, suggesting an important role of the conduction electrons in the long-range magnetic ordering. Interestingly, the exchange splitting persists far above Tc with negligible temperature dependence, in drastic contrast to the itinerant ferromagnetism described by the Stoner model, indicating the existence of short-range magnetic order. Our results provide important insights into the microscopic mechanism of the chiral helimagnetic ordering in Cr1/3NbS2.

cond-mat.mtrl-sci↗

arXiv2022

Meta Auxiliary Learning for Low-resource Spoken Language Understanding

Spoken language understanding (SLU) treats automatic speech recognition (ASR) and natural language understanding (NLU) as a unified task and usually suffers from data scarcity. We exploit an ASR and NLU joint training method based on meta auxiliary learning to improve the performance of low-resource SLU task by only taking advantage of abundant manual transcriptions of speech data. One obvious advantage of such method is that it provides a flexible framework to implement a low-resource SLU training task without requiring access to any further semantic annotations. In particular, a NLU model is taken as label generation network to predict intent and slot tags from texts; a multi-task network trains ASR task and SLU task synchronously from speech; and the predictions of label generation network are delivered to the multi-task network as semantic targets. The efficiency of the proposed algorithm is demonstrated with experiments on the public CATSLU dataset, which produces more suitable ASR hypotheses for the downstream NLU task.

eess.AS↗

arXiv2022

A CTC Triggered Siamese Network with Spatial-Temporal Dropout for Speech Recognition

Siamese networks have shown effective results in unsupervised visual representation learning. These models are designed to learn an invariant representation of two augmentations for one input by maximizing their similarity. In this paper, we propose an effective Siamese network to improve the robustness of End-to-End automatic speech recognition (ASR). We introduce spatial-temporal dropout to support a more violent disturbance for Siamese-ASR framework. Besides, we also relax the similarity regularization to maximize the similarities of distributions on the frames that connectionist temporal classification (CTC) spikes occur rather than on all of them. The efficiency of the proposed architecture is evaluated on two benchmarks, AISHELL-1 and Librispeech, resulting in 7.13% and 6.59% relative character error rate (CER) and word error rate (WER) reductions respectively. Analysis shows that our proposed approach brings a better uniformity for the trained model and enlarges the CTC spikes obviously.

eess.AS↗

arXiv2022

Multiple Confidence Gates For Joint Training Of SE And ASR

Joint training of speech enhancement model (SE) and speech recognition model (ASR) is a common solution for robust ASR in noisy environments. SE focuses on improving the auditory quality of speech, but the enhanced feature distribution is changed, which is uncertain and detrimental to the ASR. To tackle this challenge, an approach with multiple confidence gates for jointly training of SE and ASR is proposed. A speech confidence gates prediction module is designed to replace the former SE module in joint training. The noisy speech is filtered by gates to obtain features that are easier to be fitting by the ASR network. The experimental results show that the proposed method has better performance than the traditional robust speech recognition system on test sets of clean speech, synthesized noisy speech, and real noisy speech.

eess.AS↗

arXiv2022

HGCN: Harmonic gated compensation network for speech enhancement

Mask processing in the time-frequency (T-F) domain through the neural network has been one of the mainstreams for single-channel speech enhancement. However, it is hard for most models to handle the situation when harmonics are partially masked by noise. To tackle this challenge, we propose a harmonic gated compensation network (HGCN). We design a high-resolution harmonic integral spectrum to improve the accuracy of harmonic locations prediction. Then we add voice activity detection (VAD) and voiced region detection (VRD) to the convolutional recurrent network (CRN) to filter harmonic locations. Finally, the harmonic gating mechanism is used to guide the compensation model to adjust the coarse results from CRN to obtain the refinedly enhanced results. Our experiments show HGCN achieves substantial gain over a number of advanced approaches in the community.

eess.AS↗

arXiv2022

Harmonic gated compensation network plus for ICASSP 2022 DNS CHALLENGE

The harmonic structure of speech is resistant to noise, but the harmonics may still be partially masked by noise. Therefore, we previously proposed a harmonic gated compensation network (HGCN) to predict the full harmonic locations based on the unmasked harmonics and process the result of a coarse enhancement module to recover the masked harmonics. In addition, the auditory loudness loss function is used to train the network. For the DNS Challenge, we update HGCN with the following aspects, resulting in HGCN+. First, a high-band module is employed to help the model handle full-band signals. Second, cosine is used to model the harmonic structure more accurately. Then, the dual-path encoder and dual-path rnn (DPRNN) are introduced to take full advantage of the features. Finally, a gated residual linear structure replaces the gated convolution in the compensation module to increase the receptive field of frequency. The experimental results show that each updated module brings performance improvement to the model. HGCN+ also outperforms the referenced models on both wide-band and full-band test sets.

eess.AS↗

arXiv2021

Boundary and Context Aware Training for CIF-based Non-Autoregressive End-to-end ASR

Continuous integrate-and-fire (CIF) based models, which use a soft and monotonic alignment mechanism, have been well applied in non-autoregressive (NAR) speech recognition with competitive performance compared with other NAR methods. However, such an alignment learning strategy may suffer from an erroneous acoustic boundary estimation, severely hindering the convergence speed as well as the system performance. In this paper, we propose a boundary and context aware training approach for CIF based NAR models. Firstly, the connectionist temporal classification (CTC) spike information is utilized to guide the learning of acoustic boundaries in the CIF. Besides, an additional contextual decoder is introduced behind the CIF decoder, aiming to capture the linguistic dependencies within a sentence. Finally, we adopt a recently proposed Conformer architecture to improve the capacity of acoustic modeling. Experiments on the open-source Mandarin AISHELL-1 corpus show that the proposed method achieves a comparable character error rates (CERs) of 4.9% with only 1/24 latency compared with a state-of-the-art autoregressive (AR) Conformer model. Futhermore, when evaluating on an internal 7500 hours Mandarin corpus, our model still outperforms other NAR methods and even reaches the AR Conformer model on a challenging real-world noisy test set.

cs.SD↗

arXiv2021

Electrical manipulation of skyrmions in a chiral magnet

Writing, erasing and computing are three fundamental operations required by any working electronic devices. Magnetic skyrmions could be basic bits in promising in emerging topological spintronic devices. In particular, skyrmions in chiral magnets have outstanding properties like compact texture, uniform size and high mobility. However, creating, deleting and driving isolated skyrmions, as prototypes of aforementioned basic operations, have been grand challenge in chiral magnets ever since the discovery of skyrmions, and achieving all these three operations in a single device is highly desirable. Here, by engineering chiral magnet Co$_8$Zn$_{10}$Mn$_2$ into the customized micro-devices for in-situ Lorentz transmission electron microscopy observations, we implement these three operations of skyrmions using nanosecond current pulses with a low a current density about $10^{10}$ A/m$^2$ at room temperature. A notched structure can create or delete magnetic skyrmions depending on the direction and magnitude of current pulses. We further show that the magnetic skyrmions can be deterministically shifted step-by-step by current pulses, allowing the establishment of the universal current-velocity relationship. These experimental results have immediate significance towards the skyrmion-based memory or logic devices.

cond-mat.mes-hall↗

arXiv2020

Creation of a Chiral Bobber Lattice in Helimagnet-Multilayer Heterostructures

A chiral bobber is a localized three-dimensional magnetization configuration, terminated by a singularity. Chiral bobbers coexist with magnetic skyrmions in chiral magnets, lending themselves to new types of skyrmion-complementary bits of information. However, the on-demand creation of bobbers, as well as their direct observation remained elusive. Here, we introduce a new mechanism for creating a stable chiral bobber lattice state via the proximity of two skyrmion species with comparable size. This effect is experimentally demonstrated in a Cu$_2$OSeO$_3$/[Ta/CoFeB/MgO]$_4$ heterostructure in which an exotic bobber lattice state emerges in the phase diagram of Cu$_2$OSeO$_3$. To unambiguously reveal the existence of the chiral bobber lattice state, we have developed a novel characterization technique, magnetic truncation rod analysis, which is based on resonant elastic x-ray scattering.

cond-mat.mtrl-sci↗