arXiv subjects

Xie Chen

Publications and source records attributed to Xie Chen.

At least 199 records · Page 11Linked to original sources

arXiv2023

Improving Code-Switching and Named Entity Recognition in ASR with Speech Editing based Data Augmentation

Recently, end-to-end (E2E) automatic speech recognition (ASR) models have made great strides and exhibit excellent performance in general speech recognition. However, there remain several challenging scenarios that E2E models are not competent in, such as code-switching and named entity recognition (NER). Data augmentation is a common and effective practice for these two scenarios. However, the current data augmentation methods mainly rely on audio splicing and text-to-speech (TTS) models, which might result in discontinuous, unrealistic, and less diversified speech. To mitigate these potential issues, we propose a novel data augmentation method by applying the text-based speech editing model. The augmented speech from speech editing systems is more coherent and diversified, also more akin to real speech. The experimental results on code-switching and NER tasks show that our proposed method can significantly outperform the audio splicing and neural TTS based data augmentation systems.

cs.CL↗

arXiv2023

MT4SSL: Boosting Self-Supervised Speech Representation Learning by Integrating Multiple Targets

In this paper, we provide a new perspective on self-supervised speech models from how the training targets are obtained. We generalize the targets extractor into Offline Targets Extractor (Off-TE) and Online Targets Extractor (On-TE). Based on this, we propose a new multi-tasking learning framework for self-supervised learning, MT4SSL, which stands for Boosting Self-Supervised Speech Representation Learning by Integrating Multiple Targets. MT4SSL uses the K-means algorithm as an Off-TE and a teacher network without gradients as an On-TE, respectively. Our model outperforms previous SSL methods by nontrivial margins on the LibriSpeech benchmark, and is comparable to or even better than the best-performing models with fewer data. Furthermore, we find that using both Off-TE and On-TE results in better convergence in the pre-training phase. With both effectiveness and efficiency, we think doing multi-task learning on self-supervised speech models from our perspective is a promising trend.

cs.CL↗

arXiv2023

Blank-regularized CTC for Frame Skipping in Neural Transducer

Neural Transducer and connectionist temporal classification (CTC) are popular end-to-end automatic speech recognition systems. Due to their frame-synchronous design, blank symbols are introduced to address the length mismatch between acoustic frames and output tokens, which might bring redundant computation. Previous studies managed to accelerate the training and inference of neural Transducers by discarding frames based on the blank symbols predicted by a co-trained CTC. However, there is no guarantee that the co-trained CTC can maximize the ratio of blank symbols. This paper proposes two novel regularization methods to explicitly encourage more blanks by constraining the self-loop of non-blank symbols in the CTC. It is interesting to find that the frame reduction ratio of the neural Transducer can approach the theoretical boundary. Experiments on LibriSpeech corpus show that our proposed method accelerates the inference of neural Transducer by 4 times without sacrificing performance. Our work is open-sourced and publicly available https://github.com/k2-fsa/icefall.

eess.AS↗

arXiv2023

Improving Few-Shot Learning for Talking Face System with TTS Data Augmentation

Audio-driven talking face has attracted broad interest from academia and industry recently. However, data acquisition and labeling in audio-driven talking face are labor-intensive and costly. The lack of data resource results in poor synthesis effect. To alleviate this issue, we propose to use TTS (Text-To-Speech) for data augmentation to improve few-shot ability of the talking face system. The misalignment problem brought by the TTS audio is solved with the introduction of soft-DTW, which is first adopted in the talking face task. Moreover, features extracted by HuBERT are explored to utilize underlying information of audio, and found to be superior over other features. The proposed method achieves 17%, 14%, 38% dominance on MSE score, DTW score and user study preference repectively over the baseline model, which shows the effectiveness of improving few-shot learning for talking face system with TTS augmentation.

cs.SD↗

arXiv2023

Front-End Adapter: Adapting Front-End Input of Speech based Self-Supervised Learning for Speech Recognition

Recent years have witnessed a boom in self-supervised learning (SSL) in various areas including speech processing. Speech based SSL models present promising performance in a range of speech related tasks. However, the training of SSL models is computationally expensive and a common practice is to fine-tune a released SSL model on the specific task. It is essential to use consistent front-end input during pre-training and fine-tuning. This consistency may introduce potential issues when the optimal front-end is not the same as that used in pre-training. In this paper, we propose a simple but effective front-end adapter to address this front-end discrepancy. By minimizing the distance between the outputs of different front-ends, the filterbank feature (Fbank) can be compatible with SSL models which are pre-trained with waveform. The experiment results demonstrate the effectiveness of our proposed front-end adapter on several popular SSL models for the speech recognition task.

eess.AS↗

arXiv2023

EmoDiff: Intensity Controllable Emotional Text-to-Speech with Soft-Label Guidance

Although current neural text-to-speech (TTS) models are able to generate high-quality speech, intensity controllable emotional TTS is still a challenging task. Most existing methods need external optimizations for intensity calculation, leading to suboptimal results or degraded quality. In this paper, we propose EmoDiff, a diffusion-based TTS model where emotion intensity can be manipulated by a proposed soft-label guidance technique derived from classifier guidance. Specifically, instead of being guided with a one-hot vector for the specified emotion, EmoDiff is guided with a soft label where the value of the specified emotion and \textit{Neutral} is set to $α$ and $1-α$ respectively. The $α$ here represents the emotion intensity and can be chosen from 0 to 1. Our experiments show that EmoDiff can precisely control the emotion intensity while maintaining high voice quality. Moreover, diverse speech with specified emotion intensity can be generated by sampling in the reverse denoising process.

eess.AS↗

arXiv2023

Ground state degeneracy of the Ising cage-net model

The Ising cage-net model, first proposed in Phys. Rev. X 9, 021010 (2019), is a representative type I fracton model with nontrivial non-abelian features. In this paper, we calculate the ground state degeneracy of this model and find that, even though it follows a similar coupled layer structure as the X-cube model, the Ising cage-net model cannot be "foliated" in the same sense as X-cube as defined in Phys. Rev. X 8, 031051 (2018). A more generalized notion of "foliation'' is hence needed to understand the renormalization group transformation of the Ising cage-net model. The calculation is done using an operator algebra approach that we develop in this paper, and we demonstrate its validity through a series of examples.

cond-mat.str-el↗

arXiv2022

Exploring Effective Fusion Algorithms for Speech Based Self-Supervised Learning Models

Self-supervised learning (SSL) has achieved great success in various areas including speech processing. Recently, it is proven that speech based SSL models are able to extract superior universal representations on a range of downstream tasks compared to traditional hand-craft feature (e.g. FBank, MFCC) in the SUPERB benchmark. However, different types of SSL models might exhibit distinct strengths on different downstream tasks. In order to better utilize the potential power of SSL models, in this work, we explore the effective fusion on multiple SSL models. A series of model fusion algorithms are investigated and compared by combining two types of SSL models, Hubert and Data2vec, on two representative tasks from SUPERB benchmark, which are speaker identification (SID) and automatic speech recognition (ASR) tasks. The experimental results demonstrate that our proposed fusion algorithms can further boost the individual model significantly.

cs.SD↗

arXiv2022

Gapless Infinite-component Chern-Simons-Maxwell Theories

The infinite-component Chern-Simons-Maxwell (iCSM) theory is a 3+1D generalization of the 2+1D Chern-Simons-Maxwell theory by including an infinite number of coupled gauge fields. It can be used to describe interesting 3+1D systems. In Phys. Rev. B 105, 195124 (2022), it was used to construct gapped fracton models both within and beyond the foliation framework. In this paper, we study the nontrivial features of gapless iCSM theories. In particular, we find that while gapless 2+1D Maxwell theories are confined and not robust due to monopole effect, gapless iCSM theories are deconfined and robust against all local perturbation and hence represent a robust 3+1D deconfined gapless order. The gaplessness of the gapless iCSM theory can be understood as a consequence of the spontaneous breaking of an exotic one-form symmetry. Moreover, for a subclass of the gapless iCSM theories, we find interesting topological features in the correlation and response of the system. Finally, for this subclass of theories, we propose a fully continuous field theory description of the model that captures all these features.

cond-mat.str-el↗

arXiv2022

LongFNT: Long-form Speech Recognition with Factorized Neural Transducer

Traditional automatic speech recognition~(ASR) systems usually focus on individual utterances, without considering long-form speech with useful historical information, which is more practical in real scenarios. Simply attending longer transcription history for a vanilla neural transducer model shows no much gain in our preliminary experiments, since the prediction network is not a pure language model. This motivates us to leverage the factorized neural transducer structure, containing a real language model, the vocabulary predictor. We propose the {LongFNT-Text} architecture, which fuses the sentence-level long-form features directly with the output of the vocabulary predictor and then embeds token-level long-form features inside the vocabulary predictor, with a pre-trained contextual encoder RoBERTa to further boost the performance. Moreover, we propose the {LongFNT} architecture by extending the long-form speech to the original speech input and achieve the best performance. The effectiveness of our LongFNT approach is validated on LibriSpeech and GigaSpeech corpora with 19% and 12% relative word error rate~(WER) reduction, respectively.

cs.SD↗

arXiv2022

An Adapter based Multi-label Pre-training for Speech Separation and Enhancement

In recent years, self-supervised learning (SSL) has achieved tremendous success in various speech tasks due to its power to extract representations from massive unlabeled data. However, compared with tasks such as speech recognition (ASR), the improvements from SSL representation in speech separation (SS) and enhancement (SE) are considerably smaller. Based on HuBERT, this work investigates improving the SSL model for SS and SE. We first update HuBERT's masked speech prediction (MSP) objective by integrating the separation and denoising terms, resulting in a multiple pseudo label pre-training scheme, which significantly improves HuBERT's performance on SS and SE but degrades the performance on ASR. To maintain its performance gain on ASR, we further propose an adapter-based architecture for HuBERT's Transformer encoder, where only a few parameters of each layer are adjusted to the multiple pseudo label MSP while other parameters remain frozen as default HuBERT. Experimental results show that our proposed adapter-based multiple pseudo label HuBERT yield consistent and significant performance improvements on SE, SS, and ASR tasks, with a faster pre-training speed, at only marginal parameters increase.

eess.AS↗

arXiv2022

Loops in 4+1d Topological Phases

2+1d topological phases are well characterized by the fusion rules and braiding/exchange statistics of fractional point excitations. In 4+1d, some topological phases contain only fractional loop excitations. What kind of loop statistics exist? We study the 4+1d gauge theory with 2-form $\mathbb{Z}_2$ gauge field (the loop only toric code) and find that while braiding statistics between two different types of loops can be nontrivial, the self `exchange' statistics are all trivial. In particular, we show that the electric, magnetic, and dyonic loop excitations in the 4+1d toric code are not distinguished by their self-statistics. They tunnel into each other across 3+1d invertible domain walls which in turn give explicit unitary circuits that map the loop excitations into each other. The $SL(2,\mathbb{Z}_2)$ symmetry that permutes the loops, however, cannot be consistently gauged and we discuss the associated obstruction in the process. Moreover, we discuss a gapless boundary condition dubbed the `fractional Maxwell theory' and show how it can be Higgsed into gapped boundary conditions. We also discuss the generalization of these results from the $\mathbb{Z}_2$ gauge group to $\mathbb{Z}_N$.

cond-mat.str-el↗

arXiv2022

Internal Language Model Adaptation with Text-Only Data for End-to-End Speech Recognition

Text-only adaptation of an end-to-end (E2E) model remains a challenging task for automatic speech recognition (ASR). Language model (LM) fusion-based approaches require an additional external LM during inference, significantly increasing the computation cost. To overcome this, we propose an internal LM adaptation (ILMA) of the E2E model using text-only data. Trained with audio-transcript pairs, an E2E model implicitly learns an internal LM that characterizes the token sequence probability which is approximated by the E2E model output after zeroing out the encoder contribution. During ILMA, we fine-tune the internal LM, i.e., the E2E components excluding the encoder, to minimize a cross-entropy loss. To make ILMA effective, it is essential to train the E2E model with an internal LM loss besides the standard E2E loss. Furthermore, we propose to regularize ILMA by minimizing the Kullback-Leibler divergence between the output distributions of the adapted and unadapted internal LMs. ILMA is the most effective when we update only the last linear layer of the joint network. ILMA enables a fast text-only adaptation of the E2E model without increasing the run-time computational cost. Experimented with 30K-hour trained transformer transducer models, ILMA achieves up to 34.9% relative word error rate reduction from the unadapted baseline.

cs.CL↗

arXiv2022

Quantum spin liquids bootstrapped from Ising criticality in Rydberg arrays

Arrays of Rydberg atoms constitute a highly tunable, strongly interacting venue for the pursuit of exotic states of matter. We develop a new strategy for accessing a family of fractionalized phases known as quantum spin liquids in two-dimensional Rydberg arrays. We specifically use effective field theory methods to study arrays assembled from Rydberg chains tuned to an Ising phase transition that famously hosts emergent fermions propagating within each chain. This highly entangled starting point allows us to naturally access spin liquids familiar from Kitaev's honeycomb model, albeit from an entirely different framework. In particular, we argue that finite-range repulsive Rydberg interactions, which frustrate nearby symmetry-breaking orders, can enable coherent propagation of emergent fermions between the chains in which they were born. Delocalization of emergent fermions across the full two-dimensional Rydberg array yields a gapless Z2 spin liquid with a single massless Dirac cone. Here, the Rydberg occupation numbers exhibit universal power-law correlations that provide a straightforward experimental diagnostic of this phase. We further show that explicitly breaking symmetries perturbs the gapless spin liquid into gapped, topologically ordered descendants: Breaking lattice symmetries generates toric-code topological order, whereas introducing chirality generates non-Abelian Ising topological order. In the toric-code phase, we analytically construct microscopic incarnations of non-Abelian defects, which can be created and transported by dynamically controlling the atom positions in the array. Our work suggests that appropriately tuned Rydberg arrays provide a cold-atoms counterpart of solid-state 'Kitaev materials' and, more generally, spotlights a new angle for pursuing experimental platforms for Abelian and non-Abelian fractionalization.

cond-mat.str-el↗

arXiv2021

Low-bit Quantization of Recurrent Neural Network Language Models Using Alternating Direction Methods of Multipliers

The high memory consumption and computational costs of Recurrent neural network language models (RNNLMs) limit their wider application on resource constrained devices. In recent years, neural network quantization techniques that are capable of producing extremely low-bit compression, for example, binarized RNNLMs, are gaining increasing research interests. Directly training of quantized neural networks is difficult. By formulating quantized RNNLMs training as an optimization problem, this paper presents a novel method to train quantized RNNLMs from scratch using alternating direction methods of multipliers (ADMM). This method can also flexibly adjust the trade-off between the compression rate and model performance using tied low-bit quantization tables. Experiments on two tasks: Penn Treebank (PTB), and Switchboard (SWBD) suggest the proposed ADMM quantization achieved a model size compression factor of up to 31 times over the full precision baseline RNNLMs. Faster convergence of 5 times in model training over the baseline binarized RNNLM quantization was also obtained. Index Terms: Language models, Recurrent neural networks, Quantization, Alternating direction methods of multipliers.

cs.LG↗

arXiv2021

Microscopic characterization of Ising conformal field theory in Rydberg chains

Rydberg chains provide an appealing platform for probing conformal field theories (CFTs) that capture universal behavior in a myriad of physical settings. Focusing on a Rydberg chain at the Ising transition separating charge density wave and disordered phases, we establish a detailed link between microscopics and low-energy physics emerging at criticality. We first construct lattice incarnations of primary fields in the underlying Ising CFT including chiral fermions -- a nontrivial task given that the Rydberg chain Hamiltonian does not admit an exact fermionization. With this dictionary in hand, we compute correlations of microscopic Rydberg operators, paying special attention to finite, open chains of immediate experimental relevance. We further develop a method to quantify how second-neighbor Rydberg interactions tune the sign and strength of four-fermion couplings in the Ising CFT. Finally, we determine how the Ising fields evolve when four-fermion couplings drive an instability to Ising tricriticality. Our results pave the way to a thorough experimental characterization of Ising criticality in Rydberg arrays, and can inform the design of novel higher-dimensional phases based on coupled critical chains.

cond-mat.str-el↗

arXiv2021

Factorized Neural Transducer for Efficient Language Model Adaptation

In recent years, end-to-end (E2E) based automatic speech recognition (ASR) systems have achieved great success due to their simplicity and promising performance. Neural Transducer based models are increasingly popular in streaming E2E based ASR systems and have been reported to outperform the traditional hybrid system in some scenarios. However, the joint optimization of acoustic model, lexicon and language model in neural Transducer also brings about challenges to utilize pure text for language model adaptation. This drawback might prevent their potential applications in practice. In order to address this issue, in this paper, we propose a novel model, factorized neural Transducer, by factorizing the blank and vocabulary prediction, and adopting a standalone language model for the vocabulary prediction. It is expected that this factorization can transfer the improvement of the standalone language model to the Transducer for speech recognition, which allows various language model adaptation techniques to be applied. We demonstrate that the proposed factorized neural Transducer yields 15% to 20% WER improvements when out-of-domain text data is used for language model adaptation, at the cost of a minor degradation in WER on a general test set.

cs.CL↗

arXiv2021

Memory-Efficient Pipeline-Parallel DNN Training

Many state-of-the-art ML results have been obtained by scaling up the number of parameters in existing models. However, parameters and activations for such large models often do not fit in the memory of a single accelerator device; this means that it is necessary to distribute training of large models over multiple accelerators. In this work, we propose PipeDream-2BW, a system that supports memory-efficient pipeline parallelism. PipeDream-2BW uses a novel pipelining and weight gradient coalescing strategy, combined with the double buffering of weights, to ensure high throughput, low memory footprint, and weight update semantics similar to data parallelism. In addition, PipeDream-2BW automatically partitions the model over the available hardware resources, while respecting hardware constraints such as memory capacities of accelerators and interconnect topologies. PipeDream-2BW can accelerate the training of large GPT and BERT language models by up to 20$\times$ with similar final model accuracy.

cs.LG↗