arXiv subjects

Zhuo Chen

Publications and source records attributed to Zhuo Chen.

At least 325 records · Page 18Linked to original sources

arXiv2022

Simulating realistic speech overlaps improves multi-talker ASR

Multi-talker automatic speech recognition (ASR) has been studied to generate transcriptions of natural conversation including overlapping speech of multiple speakers. Due to the difficulty in acquiring real conversation data with high-quality human transcriptions, a naïve simulation of multi-talker speech by randomly mixing multiple utterances was conventionally used for model training. In this work, we propose an improved technique to simulate multi-talker overlapping speech with realistic speech overlaps, where an arbitrary pattern of speech overlaps is represented by a sequence of discrete tokens. With this representation, speech overlapping patterns can be learned from real conversations based on a statistical language model, such as N-gram, which can be then used to generate multi-talker speech for training. In our experiments, multi-talker ASR models trained with the proposed method show consistent improvement on the word error rates across multiple datasets.

eess.AS↗

arXiv2022

An Adapter based Multi-label Pre-training for Speech Separation and Enhancement

In recent years, self-supervised learning (SSL) has achieved tremendous success in various speech tasks due to its power to extract representations from massive unlabeled data. However, compared with tasks such as speech recognition (ASR), the improvements from SSL representation in speech separation (SS) and enhancement (SE) are considerably smaller. Based on HuBERT, this work investigates improving the SSL model for SS and SE. We first update HuBERT's masked speech prediction (MSP) objective by integrating the separation and denoising terms, resulting in a multiple pseudo label pre-training scheme, which significantly improves HuBERT's performance on SS and SE but degrades the performance on ASR. To maintain its performance gain on ASR, we further propose an adapter-based architecture for HuBERT's Transformer encoder, where only a few parameters of each layer are adjusted to the multiple pseudo label MSP while other parameters remain frozen as default HuBERT. Experimental results show that our proposed adapter-based multiple pseudo label HuBERT yield consistent and significant performance improvements on SE, SS, and ASR tasks, with a faster pre-training speed, at only marginal parameters increase.

eess.AS↗

arXiv2022

Self-supervised learning with bi-label masked speech prediction for streaming multi-talker speech recognition

Self-supervised learning (SSL), which utilizes the input data itself for representation learning, has achieved state-of-the-art results for various downstream speech tasks. However, most of the previous studies focused on offline single-talker applications, with limited investigations in multi-talker cases, especially for streaming scenarios. In this paper, we investigate SSL for streaming multi-talker speech recognition, which generates transcriptions of overlapping speakers in a streaming fashion. We first observe that conventional SSL techniques do not work well on this task due to the poor representation of overlapping speech. We then propose a novel SSL training objective, referred to as bi-label masked speech prediction, which explicitly preserves representations of all speakers in overlapping speech. We investigate various aspects of the proposed system including data configuration and quantizer selection. The proposed SSL setup achieves substantially better word error rates on the LibriSpeechMix dataset.

eess.AS↗

arXiv2022

The standard cohomology of regular Courant algebroids

For any regular Courant algebroid $E$ over a smooth manifold $M$ with characteristic distribution $F$ and ample Lie algebroid $A_E$, we prove that there exists a canonical homological vector field on the graded manifold $A_E[1] \oplus (TM/F)^\ast[2]$ such that the resulting dg manifold $\mathcal{M}_E$, which we call the minimal model of the Courant algebroid $E$, encodes all cohomological information of $E$. Indeed, the standard cohomology of $E$ can be identified with the cohomology of the function space on $\mathcal{M}_E$, which can be computed by a Hodge-to-de Rham type spectral sequence. We apply this result to generalized exact Courant algebroids and those arising from regular Lie algebroids.

math.DG↗

arXiv2022

Benchmarking Knowledge-driven Zero-shot Learning

External knowledge (a.k.a. side information) plays a critical role in zero-shot learning (ZSL) which aims to predict with unseen classes that have never appeared in training data. Several kinds of external knowledge, such as text and attribute, have been widely investigated, but they alone are limited with incomplete semantics. Some very recent studies thus propose to use Knowledge Graph (KG) due to its high expressivity and compatibility for representing kinds of knowledge. However, the ZSL community is still in short of standard benchmarks for studying and comparing different external knowledge settings and different KG-based ZSL methods. In this paper, we proposed six resources covering three tasks, i.e., zero-shot image classification (ZS-IMGC), zero-shot relation extraction (ZS-RE), and zero-shot KG completion (ZS-KGC). Each resource has a normal ZSL benchmark and a KG containing semantics ranging from text to attribute, from relational knowledge to logical expressions. We have clearly presented these resources including their construction, statistics, data formats and usage cases w.r.t. different ZSL methods. More importantly, we have conducted a comprehensive benchmarking study, with two general and state-of-the-art methods, two setting-specific methods and one interpretable method. We discussed and compared different ZSL paradigms w.r.t. different external knowledge settings, and found that our resources have great potential for developing more advanced ZSL methods and more solutions for applying KGs for augmenting machine learning. All the resources are available at https://github.com/China-UK-ZSL/Resources_for_KZSL.

cs.AI↗

arXiv2022

Real-time Speech Interruption Analysis: From Cloud to Client Deployment

Meetings are an essential form of communication for all types of organizations, and remote collaboration systems have been much more widely used since the COVID-19 pandemic. One major issue with remote meetings is that it is challenging for remote participants to interrupt and speak. We have recently developed the first speech interruption analysis model, which detects failed speech interruptions, shows very promising performance, and is being deployed in the cloud. To deliver this feature in a more cost-efficient and environment-friendly way, we reduced the model complexity and size to ship the WavLM_SI model in client devices. In this paper, we first describe how we successfully improved the True Positive Rate (TPR) at a 1% False Positive Rate (FPR) from 50.9% to 68.3% for the failed speech interruption detection model by training on a larger dataset and fine-tuning. We then shrank the model size from 222.7 MB to 9.3 MB with an acceptable loss in accuracy and reduced the complexity from 31.2 GMACS (Giga Multiply-Accumulate Operations per Second) to 4.3 GMACS. We also estimated the environmental impact of the complexity reduction, which can be used as a general guideline for large Transformer-based models, and thus make those models more accessible with less computation overhead.

cs.CL↗

arXiv2022

Locally finite infinity-modules and weak Loday-Pirashvili modules over differential graded Lie algebras

Motivated by recent developments of $\infty$-categorical theories related to differential graded (dg for short) Lie algebras, we develop a general framework for locally finite $\infty$-$\mathfrak{g}$-modules over a dg Lie algebra $\mathfrak{g}$. We show that the category of such locally finite $\infty$-$\mathfrak{g}$-modules is almost a model category in the sense of Vallette. As a homotopy theoretical generalization of Loday and Pirashvili's Lie algebra objects in the tensor category of linear maps, we further study weak Loday-Pirashvili modules consisting of $\infty$-morphisms from locally finite $\infty$-$\mathfrak{g}$-modules to the adjoint module $\mathfrak{g}$. From the category of such weak Loday-Pirashvili modules over $\mathfrak{g}$, we find a functor that maps to the category of Leibniz$_\infty$ algebras enriched over the Chevalley-Eilenberg dg algebra of $\mathfrak{g}$. This functor can be regarded as the homotopy lifting of Loday and Pirashvili's original method to realize Leibniz algebras from Lie algebra objects in the category of linear maps.

math.RT↗

arXiv2022

VarArray Meets t-SOT: Advancing the State of the Art of Streaming Distant Conversational Speech Recognition

This paper presents a novel streaming automatic speech recognition (ASR) framework for multi-talker overlapping speech captured by a distant microphone array with an arbitrary geometry. Our framework, named t-SOT-VA, capitalizes on independently developed two recent technologies; array-geometry-agnostic continuous speech separation, or VarArray, and streaming multi-talker ASR based on token-level serialized output training (t-SOT). To combine the best of both technologies, we newly design a t-SOT-based ASR model that generates a serialized multi-talker transcription based on two separated speech signals from VarArray. We also propose a pre-training scheme for such an ASR model where we simulate VarArray's output signals based on monaural single-talker ASR training data. Conversation transcription experiments using the AMI meeting corpus show that the system based on the proposed framework significantly outperforms conventional ones. Our system achieves the state-of-the-art word error rates of 13.7% and 15.5% for the AMI development and evaluation sets, respectively, in the multiple-distant-microphone setting while retaining the streaming inference capability.

eess.AS↗

arXiv2022

Floquet Engineering of Nonequilibrium Valley-Polarized Quantum Anomalous Hall Effect with Tunable Chern Number

Numerous attempts have been made so far to explore the quantum anomalous Hall effect (QAHE), but the ultralow observed temperature strongly hinders its practical applications. Hence, it is of great interest to go beyond the existing paradigm of QAHE. Here, we propose that Floquet engineering offers a strategy to realize the QAHE via hybridization of Floquet-Bloch bands. Based on first-principles calculations and Floquet theorem, we unveil that nonequilibrium valley-polarized QAHE (VP-QAHE), independent of magnetic orders, is widely present in ferromagnetic and nonmagnetic members of two-dimensional family materials $M$Si$_2$$Z_4$ ($M$ = Mo, W, V; $Z$ = N, P, As) by irradiating circularly polarized light (CPL). Remarkably, by tuning the frequency, intensity, and handedness of incident CPL, the Chern number of VP-QAHE is highly tunable and up to $\mathcal{C}=\pm 4$. We reveal that such Chern number tunable VP-QAHE attributes to light-induced trigonal warping and multiple band inversion at different valleys. The valley-resolved chiral edge states and quantized plateau of Hall conductance, which facilitates the experimental measurement, are visible inside the global band gap. Our work not only establishes Floquet Engineering of nonequilibrium VP-QAHE with tunable Chern number in realistic materials, but also provides a promising avenue to explore emergent topological phases under light irradiation.

cond-mat.mes-hall↗

arXiv2022

The Microsoft System for VoxCeleb Speaker Recognition Challenge 2022

In this report, we describe our submitted system for track 2 of the VoxCeleb Speaker Recognition Challenge 2022 (VoxSRC-22). We fuse a variety of good-performing models ranging from supervised models to self-supervised learning(SSL) pre-trained models. The supervised models, trained using VoxCeleb-2 dev data, consist of ECAPA-TDNN and Res2Net in a very deep structure. The SSL pre-trained models, wav2vec and wavLM, are trained using large scale unlabeled speech data up to million hours. These models are cascaded with ECAPA-TDNN and further fine-tuned in a supervised fashion to extract the speaker representations. All 13 models are applied with score normalization and calibration and then fused into the the submitted system. We also explore the audio quality measures in the calibration stage such as duration, SNR, T60, and MOS. The best submitted system achieves 0.073 in minDCF and 1.436% in EER on the VoxSRC-22 evaluation set.

cs.SD↗

arXiv2022

Hochschild cohomology of dg manifolds associated to integrable distributions

For the field $\mathbb{K} = \mathbb{R}$ or $\mathbb{C}$, and an integrable distribution $F \subseteq T_M \otimes_{\mathbb{R}} \mathbb{K}$ on a smooth manifold $M$, we study the Hochschild cohomology of the dg manifold $(F[1],d_F)$ and establish a canonical isomorphism with the Hochschild cohomology of the algebra of functions on leaf space in terms of transversal polydifferential operators of $F$. In particular, for the dg manifold $(T_X^{0,1}[1],\bar{\partial})$ associated with a complex manifold $X$, we prove that its Hochschild cohomology is canonically isomorphic to the Hochschild cohomology $HH^{\bullet}(X)$ of the complex manifold $X$. As an application, we show that the Duflo-Kontsevich type theorem for the dg manifold $(T_X^{0,1}[1],\bar{\partial})$ implies the Duflo-Kontsevich theorem for complex manifolds.

math.DG↗

arXiv2022

Collaboration of Experts: Achieving 80% Top-1 Accuracy on ImageNet with 100M FLOPs

In this paper, we propose a Collaboration of Experts (CoE) framework to pool together the expertise of multiple networks towards a common aim. Each expert is an individual network with expertise on a unique portion of the dataset, which enhances the collective capacity. Given a sample, an expert is selected by the delegator, which simultaneously outputs a rough prediction to support early termination. To fulfill this framework, we propose three modules to impel each model to play its role, namely weight generation module (WGM), label generation module (LGM) and variance calculation module (VCM). Our method achieves the state-of-the-art performance on ImageNet, 80.7% top-1 accuracy with 194M FLOPs. Combined with PWLU activation function and CondConv, CoE further achieves the accuracy of 80.0% with only 100M FLOPs for the first time. More importantly, our method is hardware friendly and achieves a 3-6x speedup compared with some existing conditional computation approaches.

cs.CV↗

arXiv2022

Streaming Multi-Talker ASR with Token-Level Serialized Output Training

This paper proposes a token-level serialized output training (t-SOT), a novel framework for streaming multi-talker automatic speech recognition (ASR). Unlike existing streaming multi-talker ASR models using multiple output branches, the t-SOT model has only a single output branch that generates recognition tokens (e.g., words, subwords) of multiple speakers in chronological order based on their emission times. A special token that indicates the change of ``virtual'' output channels is introduced to keep track of the overlapping utterances. Compared to the prior streaming multi-talker ASR models, the t-SOT model has the advantages of less inference cost and a simpler model architecture. Moreover, in our experiments with LibriSpeechMix and LibriCSS datasets, the t-SOT-based transformer transducer model achieves the state-of-the-art word error rates by a significant margin to the prior results. For non-overlapping speech, the t-SOT model is on par with a single-talker ASR model in terms of both accuracy and computational cost, opening the door for deploying one model for both single- and multi-talker scenarios.

eess.AS↗

arXiv2022

Streaming Speaker-Attributed ASR with Token-Level Speaker Embeddings

This paper presents a streaming speaker-attributed automatic speech recognition (SA-ASR) model that can recognize ``who spoke what'' with low latency even when multiple people are speaking simultaneously. Our model is based on token-level serialized output training (t-SOT) which was recently proposed to transcribe multi-talker speech in a streaming fashion. To further recognize speaker identities, we propose an encoder-decoder based speaker embedding extractor that can estimate a speaker representation for each recognized token not only from non-overlapping speech but also from overlapping speech. The proposed speaker embedding, named t-vector, is extracted synchronously with the t-SOT ASR model, enabling joint execution of speaker identification (SID) or speaker diarization (SD) with the multi-talker transcription with low latency. We evaluate the proposed model for a joint task of ASR and SID/SD by using LibriSpeechMix and LibriCSS corpora. The proposed model achieves substantially better accuracy than a prior streaming model and shows comparable or sometimes even superior results to the state-of-the-art offline SA-ASR model.

eess.AS↗

arXiv2022

Internal symmetry of the $L_{\leqslant 3}$ algebra arising from a Lie pair

A Lie pair is an inclusion $A$ to $L$ of Lie algebroids over the same base manifold. In an earlier work, the third author with Bandiera, Stiénon, and Xu introduced a canonical $L_{\leqslant 3}$ algebra $Γ(\wedge^\bullet A^\vee \otimes L/A)$ whose unary bracket is the Chevalley-Eilenberg differential arising from every Lie pair $(L,A)$. In this note, we prove that to such a Lie pair there is an associated Lie algebra action by $\mathrm{Der}(L)$ on the $L_{\leqslant 3}$ algebra $Γ(\wedge^\bullet A^\vee \otimes L/A)$. Here $\mathrm{Der}(L)$ is the space of derivations on the Lie algebroid $L$, or infinitesimal automorphisms of $L$. The said action gives rise to a larger scope of gauge equivalences of Maurer-Cartan elements in $Γ(\wedge^\bullet A^\vee \otimes L/A)$, and for this reason we elect to call the $\mathrm{Der}(L)$-action internal symmetry of $Γ(\wedge^\bullet A^\vee \otimes L/A)$.

math.DG↗

arXiv2022

Why does Self-Supervised Learning for Speech Recognition Benefit Speaker Recognition?

Recently, self-supervised learning (SSL) has demonstrated strong performance in speaker recognition, even if the pre-training objective is designed for speech recognition. In this paper, we study which factor leads to the success of self-supervised learning on speaker-related tasks, e.g. speaker verification (SV), through a series of carefully designed experiments. Our empirical results on the Voxceleb-1 dataset suggest that the benefit of SSL to SV task is from a combination of mask speech prediction loss, data scale, and model size, while the SSL quantizer has a minor impact. We further employ the integrated gradients attribution method and loss landscape visualization to understand the effectiveness of self-supervised learning for speaker recognition performance.

cs.CL↗

arXiv2022

Dirac generating operators of split Courant algebroids

Given a vector bundle $A$ over a smooth manifold $M$ such that the square root $\mathcal{L}$ of the line bundle $\wedge^{\mathrm{top}}A^\ast \otimes \wedge^{\mathrm{top}}T^\ast M$ exists, the Clifford bundle associated to the split pseudo-Euclidean vector bundle $(E = A \oplus A^\ast, \langle \cdot, \cdot \rangle)$, admits a spinor bundle $\wedge^\bullet A \otimes \mathcal{L}$, whose section space can be thought of as that of Berezinian half-densities of the graded manifold $A^\ast[1]$. We give an explicit construction of Dirac generating operators of split Courant algebroid (or proto-bialgebroid) structures on $A \oplus A^\ast$ introduced by Alekseev and Xu. We also prove that the square of the Dirac generating operator gives rise to an invariant of the split Courant algebroid.

math.DG↗

arXiv2022

WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing

Self-supervised learning (SSL) achieves great success in speech recognition, while limited exploration has been attempted for other speech processing tasks. As speech signal contains multi-faceted information including speaker identity, paralinguistics, spoken content, etc., learning universal representations for all speech tasks is challenging. To tackle the problem, we propose a new pre-trained model, WavLM, to solve full-stack downstream speech tasks. WavLM jointly learns masked speech prediction and denoising in pre-training. By this means, WavLM does not only keep the speech content modeling capability by the masked speech prediction, but also improves the potential to non-ASR tasks by the speech denoising. In addition, WavLM employs gated relative position bias for the Transformer structure to better capture the sequence ordering of input speech. We also scale up the training dataset from 60k hours to 94k hours. WavLM Large achieves state-of-the-art performance on the SUPERB benchmark, and brings significant improvements for various speech processing tasks on their representative benchmarks. The code and pre-trained models are available at https://aka.ms/wavlm.

cs.CL↗