arXiv subjects

Xin Lei

Publications and source records attributed to Xin Lei.

At least 19 recordsLinked to original sources

arXiv2026

T-Mimi: A Transformer-based Mimi Decoder for Real-Time On-Phone TTS

Neural audio codecs provide promising acoustic features for speech synthesis, with representative streaming codecs like Mimi providing high-quality acoustic features for real-time Text-to-Speech (TTS) applications. However, Mimi's decoder, which employs a hybrid transformer and convolution architecture, introduces significant latency bottlenecks on edge devices due to the the compute intensive nature of deconvolution layers which are not friendly for mobile-CPUs, such as the most representative framework XNNPACK. This paper introduces T-Mimi, a novel modification of the Mimi codec decoder that replaces its convolutional components with a purely transformer-based decoder, inspired by the TS3-Codec architecture. This change dramatically reduces on-device TTS latency from 42.1ms to just 4.4ms. Furthermore, we conduct quantization aware training and derive a crucial finding: the final two transformer layers and the concluding linear layers of the decoder, which are close to the waveform, are highly sensitive to quantization and must be preserved at full precision to maintain audio quality.

eess.AS↗

arXiv2025

Directional Source Separation for Robust Speech Recognition on Smart Glasses

Modern smart glasses leverage advanced audio sensing and machine learning technologies to offer real-time transcribing and captioning services, considerably enriching human experiences in daily communications. However, such systems frequently encounter challenges related to environmental noises, resulting in degradation to speech recognition and speaker change detection. To improve voice quality, this work investigates directional source separation using the multi-microphone array. We first explore multiple beamformers to assist source separation modeling by strengthening the directional properties of speech signals. In addition to relying on predetermined beamformers, we investigate neural beamforming in multi-channel source separation, demonstrating that automatic learning directional characteristics effectively improves separation quality. We further compare the ASR performance leveraging separated outputs to noisy inputs. Our results show that directional source separation benefits ASR for the wearer but not for the conversation partner. Lastly, we perform the joint training of the directional source separation and ASR model, achieving the best overall ASR performance.

cs.SD↗

arXiv2025

A Domain Adaptation Framework for Speech Recognition Systems with Only Synthetic data

We introduce DAS (Domain Adaptation with Synthetic data), a novel domain adaptation framework for pre-trained ASR model, designed to efficiently adapt to various language-defined domains without requiring any real data. In particular, DAS first prompts large language models (LLMs) to generate domain-specific texts before converting these texts to speech via text-to-speech technology. The synthetic data is used to fine-tune Whisper with Low-Rank Adapters (LoRAs) for targeted domains such as music, weather, and sports. We introduce a novel one-pass decoding strategy that merges predictions from multiple LoRA adapters efficiently during the auto-regressive text generation process. Experimental results show significant improvements, reducing the Word Error Rate (WER) by 10% to 17% across all target domains compared to the original model, with minimal performance regression in out-of-domain settings (e.g., -1% on Librispeech test sets). We also demonstrate that DAS operates efficiently during inference, introducing an additional 9% increase in Real Time Factor (RTF) compared to the original model when inferring with three LoRA adapters.

eess.AS↗

arXiv2024

Query-by-Example Keyword Spotting Using Spectral-Temporal Graph Attentive Pooling and Multi-Task Learning

Existing keyword spotting (KWS) systems primarily rely on predefined keyword phrases. However, the ability to recognize customized keywords is crucial for tailoring interactions with intelligent devices. In this paper, we present a novel Query-by-Example (QbyE) KWS system that employs spectral-temporal graph attentive pooling and multi-task learning. This framework aims to effectively learn speaker-invariant and linguistic-informative embeddings for QbyE KWS tasks. Within this framework, we investigate three distinct network architectures for encoder modeling: LiCoNet, Conformer and ECAPA_TDNN. The experimental results on a substantial internal dataset of $629$ speakers have demonstrated the effectiveness of the proposed QbyE framework in maximizing the potential of simpler models such as LiCoNet. Particularly, LiCoNet, which is 13x more efficient, achieves comparable performance to the computationally intensive Conformer model (1.98% vs. 1.63\% FRR at 0.3 FAs/Hr).

cs.CL↗

arXiv2024

Medium recoil mode of $Δ$ production in single isobaric charge-exchange reactions

The dynamic mechanisms underlying single charge-exchange reactions have been investigated using a theoretical framework that combines the Isospin-dependent Quantum Molecular Dynamics (IQMD) model with the statistical decay model GEMINI++. Two distinct channels contribute to the single isobaric charge-exchange reaction: quasi-elastic channel, where neutron-proton scattering drives the charge-exchange, and inelastic channel, where the $Δ$ particle is produced during the process. In a referenced study [Phys.RevC 106.014618(2022)], experimental data have revealed that the inelastic channel accounts for approximately 50 percent of the single isobaric charge-exchange reaction. However, our current model fails in reproducing the significant contribution of inelastic channel unless the novel medium recoil mode associated with $Δ$ production is considered in the calculations. Notably, this in-medium effect arising from inelastic nucleon-nucleon collisions is not yet incorporated into mainstream microscopic transport models. The dynamical properties of protons and pions emitting in the single isobaric charge-exchange reactions are predicted. This exploration of in-medium effects adds a valuable dimension to our understanding of the intricate dynamics involved in single charge-exchange reactions.

nucl-th↗

arXiv2024

Disentangled Training with Adversarial Examples For Robust Small-footprint Keyword Spotting

A keyword spotting (KWS) engine that is continuously running on device is exposed to various speech signals that are usually unseen before. It is a challenging problem to build a small-footprint and high-performing KWS model with robustness under different acoustic environments. In this paper, we explore how to effectively apply adversarial examples to improve KWS robustness. We propose datasource-aware disentangled learning with adversarial examples to reduce the mismatch between the original and adversarial data as well as the mismatch across original training datasources. The KWS model architecture is based on depth-wise separable convolution and a simple attention module. Experimental results demonstrate that the proposed learning strategy improves false reject rate by $40.31%$ at $1%$ false accept rate on the internal dataset, compared to the strongest baseline without using adversarial examples. Our best-performing system achieves $98.06%$ accuracy on the Google Speech Commands V1 dataset.

cs.SD↗

arXiv2024

LLaMA based Punctuation Restoration With Forward Pass Only Decoding

This paper introduces two advancements in the field of Large Language Model Annotation with a focus on punctuation restoration tasks. Our first contribution is the application of LLaMA for punctuation restoration, which demonstrates superior performance compared to the established benchmark. Despite its impressive quality, LLaMA faces challenges regarding inference speed and hallucinations. To address this, our second contribution presents Forward Pass Only Decoding (FPOD), a novel decoding approach for annotation tasks. This innovative method results in a substantial 19.8x improvement in inference speed, effectively addressing a critical bottleneck and enhancing the practical utility of LLaMA for large-scale data annotation tasks without hallucinations. The combination of these contributions not only solidifies LLaMA as a powerful tool for punctuation restoration but also highlights FPOD as a crucial strategy for overcoming speed constraints.

cs.CL↗

arXiv2024

Microscopic study of deformation and orientation effects in heavy-ion reactions above Coulomb barrier using the Boltzmann-Uehling-Uhlenbeck model

Background: The understanding of the impact of initial deformation and collision orientation on quasi-fission and fusion-fission reactions remains incomplete. Purpose: This article aims to explore how the orientation of deformed nuclei influences quasi-fission and fusion-fission around 1.2 VB, employing a micro dynamical method in systems with diverse shapes, namely 24Mg + 178Hf, 34S + 168Er, and 48Ti + 154Sm. Method: Utilizing the Boltzmann-Uehling-Uhlenbeck model, this study investigates quasi-fission and fusion fission reactions. The model elucidates micro-dynamic processes and microscopic observables through the definition of the window and event-by-event simulations. Results: The findings reveal that the orientation of deformed nuclei significantly influences the nucleus-nucleus interaction potential, thereby impacting the competition between quasi-fission and fusion-fission. Particularly, the orientation of the deformed target nucleus emerges as the primary factor affecting this competition. Notably, a higher proportion of fusion-fission events is observed when the target nucleus is in the belly orientation compared to the tip. The study also observes that the configuration of the dinuclear system contributes to fluctuations and dissipation. Collisions with different orientations result in distinct dinuclear system configurations, with belly-oriented collisions leading to larger fluctuations between events, while tip-oriented collisions exhibit smaller fluctuations. Conclusions: Considering diverse orientations of nuclei with distinct initial deformations, this study concludes that the orientation of the target nucleus is the key factor influencing quasi-fission and fusion-fission reactions around 1.2 VB.

nucl-th↗

arXiv2024

FADI-AEC: Fast Score Based Diffusion Model Guided by Far-end Signal for Acoustic Echo Cancellation

Despite the potential of diffusion models in speech enhancement, their deployment in Acoustic Echo Cancellation (AEC) has been restricted. In this paper, we propose DI-AEC, pioneering a diffusion-based stochastic regeneration approach dedicated to AEC. Further, we propose FADI-AEC, fast score-based diffusion AEC framework to save computational demands, making it favorable for edge devices. It stands out by running the score model once per frame, achieving a significant surge in processing efficiency. Apart from that, we introduce a novel noise generation technique where far-end signals are utilized, incorporating both far-end and near-end signals to refine the score model's accuracy. We test our proposed method on the ICASSP2023 Microsoft deep echo cancellation challenge evaluation dataset, where our method outperforms some of the end-to-end methods and other diffusion based echo cancellation methods.

eess.AS↗

arXiv2023

TODM: Train Once Deploy Many Efficient Supernet-Based RNN-T Compression For On-device ASR Models

Automatic Speech Recognition (ASR) models need to be optimized for specific hardware before they can be deployed on devices. This can be done by tuning the model's hyperparameters or exploring variations in its architecture. Re-training and re-validating models after making these changes can be a resource-intensive task. This paper presents TODM (Train Once Deploy Many), a new approach to efficiently train many sizes of hardware-friendly on-device ASR models with comparable GPU-hours to that of a single training job. TODM leverages insights from prior work on Supernet, where Recurrent Neural Network Transducer (RNN-T) models share weights within a Supernet. It reduces layer sizes and widths of the Supernet to obtain subnetworks, making them smaller models suitable for all hardware types. We introduce a novel combination of three techniques to improve the outcomes of the TODM Supernet: adaptive dropouts, an in-place Alpha-divergence knowledge distillation, and the use of ScaledAdam optimizer. We validate our approach by comparing Supernet-trained versus individually tuned Multi-Head State Space Model (MH-SSM) RNN-T using LibriSpeech. Results demonstrate that our TODM Supernet either matches or surpasses the performance of manually tuned models by up to a relative of 3% better in word error rate (WER), while efficiently keeping the cost of training many models at a small constant.

cs.CL↗

arXiv2022

LiCo-Net: Linearized Convolution Network for Hardware-efficient Keyword Spotting

This paper proposes a hardware-efficient architecture, Linearized Convolution Network (LiCo-Net) for keyword spotting. It is optimized specifically for low-power processor units like microcontrollers. ML operators exhibit heterogeneous efficiency profiles on power-efficient hardware. Given the exact theoretical computation cost, int8 operators are more computation-effective than float operators, and linear layers are often more efficient than other layers. The proposed LiCo-Net is a dual-phase system that uses the efficient int8 linear operators at the inference phase and applies streaming convolutions at the training phase to maintain a high model capacity. The experimental results show that LiCo-Net outperforms single-value decomposition filter (SVDF) on hardware efficiency with on-par detection performance. Compared to SVDF, LiCo-Net reduces cycles by 40% on HiFi4 DSP.

cs.LG↗

arXiv2022

SCA: Streaming Cross-attention Alignment for Echo Cancellation

End-to-End deep learning has shown promising results for speech enhancement tasks, such as noise suppression, dereverberation, and speech separation. However, most state-of-the-art methods for echo cancellation are either classical DSP-based or hybrid DSP-ML algorithms. Components such as the delay estimator and adaptive linear filter are based on traditional signal processing concepts, and deep learning algorithms typically only serve to replace the non-linear residual echo suppressor. This paper introduces an end-to-end echo cancellation network with a streaming cross-attention alignment (SCA). Our proposed method can handle unaligned inputs without requiring external alignment and generate high-quality speech without echoes. At the same time, the end-to-end algorithm simplifies the current echo cancellation pipeline for time-variant echo path cases. We test our proposed method on the ICASSP2022 and Interspeech2021 Microsoft deep echo cancellation challenge evaluation dataset, where our method outperforms some of the other hybrid and end-to-end methods.

eess.AS↗

arXiv2021

U2++: Unified Two-pass Bidirectional End-to-end Model for Speech Recognition

The unified streaming and non-streaming two-pass (U2) end-to-end model for speech recognition has shown great performance in terms of streaming capability, accuracy, real-time factor (RTF), and latency. In this paper, we present U2++, an enhanced version of U2 to further improve the accuracy. The core idea of U2++ is to use the forward and the backward information of the labeling sequences at the same time at training to learn richer information, and combine the forward and backward prediction at decoding to give more accurate recognition results. We also proposed a new data augmentation method called SpecSub to help the U2++ model to be more accurate and robust. Our experiments show that, compared with U2, U2++ shows faster convergence at training, better robustness to the decoding method, as well as consistent 5\% - 8\% word error rate reduction gain over U2. On the experiment of AISHELL-1, we achieve a 4.63\% character error rate (CER) with a non-streaming setup and 5.05\% with a streaming setup with 320ms latency by U2++. To the best of our knowledge, 5.05\% is the best-published streaming result on the AISHELL-1 test set.

cs.SD↗

arXiv2021

Unified Streaming and Non-streaming Two-pass End-to-end Model for Speech Recognition

In this paper, we present a novel two-pass approach to unify streaming and non-streaming end-to-end (E2E) speech recognition in a single model. Our model adopts the hybrid CTC/attention architecture, in which the conformer layers in the encoder are modified. We propose a dynamic chunk-based attention strategy to allow arbitrary right context length. At inference time, the CTC decoder generates n-best hypotheses in a streaming way. The inference latency could be easily controlled by only changing the chunk size. The CTC hypotheses are then rescored by the attention decoder to get the final result. This efficient rescoring process causes very little sentence-level latency. Our experiments on the open 170-hour AISHELL-1 dataset show that, the proposed method can unify the streaming and non-streaming model simply and efficiently. On the AISHELL-1 test set, our unified model achieves 5.60% relative character error rate (CER) reduction in non-streaming ASR compared to a standard non-streaming transformer. The same model achieves 5.42% CER with 640ms latency in a streaming ASR system.

cs.SD↗

arXiv2021

WeNet: Production oriented Streaming and Non-streaming End-to-End Speech Recognition Toolkit

In this paper, we propose an open source, production first, and production ready speech recognition toolkit called WeNet in which a new two-pass approach is implemented to unify streaming and non-streaming end-to-end (E2E) speech recognition in a single model. The main motivation of WeNet is to close the gap between the research and the production of E2E speechrecognition models. WeNet provides an efficient way to ship ASR applications in several real-world scenarios, which is the main difference and advantage to other open source E2E speech recognition toolkits. In our toolkit, a new two-pass method is implemented. Our method propose a dynamic chunk-based attention strategy of the the transformer layers to allow arbitrary right context length modifies in hybrid CTC/attention architecture. The inference latency could be easily controlled by only changing the chunk size. The CTC hypotheses are then rescored by the attention decoder to get the final result. Our experiments on the AISHELL-1 dataset using WeNet show that, our model achieves 5.03\% relative character error rate (CER) reduction in non-streaming ASR compared to a standard non-streaming transformer. After model quantification, our model perform reasonable RTF and latency.

cs.SD↗

arXiv2020

An energy-splitting high order numerical method for multi-material flows

This chapter deals with multi-material flow problems by a kind of effective numerical methods, based on a series of reduced forms of the Baer-Nunziato (BN) model. Numerical simulations often face a host of difficult challenges, typically including the volume fraction positivity and stability of multi-material shocks. To cope with these challenges, we propose a new non-oscillatory {\em energy-splitting} Godunov-type scheme for computing multi-fluid flows in the Eulerian framework. A novel reduced version of the BN model is introduced as the basis for the energy-splitting scheme. In comparison with existing two-material compressible flow models obtained by reducing the BN model in the literature, it is shown that our new reduced model can simulate the kinetic energy exchange around material interfaces very effectively. Then a second-order accurate extension of the energy-splitting Godunov-type scheme is made using the generalized Riemann problem (GRP) solver. Numerical experiments are carried out for the shock-interface interaction, shock-bubble interaction and the Richtmyer-Meshkov instability problems, which demonstrate the excellent performance of this type of schemes.

physics.comp-ph↗

arXiv2020

A staggered-projection Godunov-type method for the Baer-Nunziato two-phase model

When describing the deflagration-to-detonation transition in solid granular explosives mixed with gaseous products of combustion, a well-developed two-phase mixture model is the compressible Baer-Nunziato (BN) model, containing solid and gas phases. If this model is numerically simulated by a conservative Godunov-type scheme, spurious oscillations are likely to generate from porosity interfaces, which may result from the average process of conservative variables that violates the continuity of Riemann invariants across porosity interfaces. In order to suppress the oscillations, this paper proposes a staggered-projection Godunov-type scheme over a fixed gas-solid staggered grid, by enforcing that solid contacts with porosity jumps are always inside gaseous grid cells and other discontinuities appear at gaseous cell interfaces. This scheme is based on a standard Godunov scheme for the Baer-Nunziato model on gaseous cells and guarantees the continuity of the Riemann invariants associated with the solid contact discontinuities across porosity jumps. While porosity interfaces are moving, a projection process fully takes into account the continuity of associated Riemann invariants and ensure that porosity jumps remain inside gaseous cells. This staggered-projection Godunov-type scheme is well-balanced with good numerical performance not only on suppressing spurious oscillations near porosity interfaces but also capturing strong discontinuities such as shocks.

physics.comp-ph↗

arXiv2019

Knowledge Distillation For Recurrent Neural Network Language Modeling With Trust Regularization

Recurrent Neural Networks (RNNs) have dominated language modeling because of their superior performance over traditional N-gram based models. In many applications, a large Recurrent Neural Network language model (RNNLM) or an ensemble of several RNNLMs is used. These models have large memory footprints and require heavy computation. In this paper, we examine the effect of applying knowledge distillation in reducing the model size for RNNLMs. In addition, we propose a trust regularization method to improve the knowledge distillation training for RNNLMs. Using knowledge distillation with trust regularization, we reduce the parameter size to a third of that of the previously published best model while maintaining the state-of-the-art perplexity result on Penn Treebank data. In a speech recognition N-bestrescoring task, we reduce the RNNLM model size to 18.5% of the baseline system, with no degradation in word error rate(WER) performance on Wall Street Journal data set.

cs.CL↗