arXiv subjects

Zhuo Chen

Publications and source records attributed to Zhuo Chen.

At least 361 records · Page 20Linked to original sources

arXiv2021

A Comparative Study of Modular and Joint Approaches for Speaker-Attributed ASR on Monaural Long-Form Audio

Speaker-attributed automatic speech recognition (SA-ASR) is a task to recognize "who spoke what" from multi-talker recordings. An SA-ASR system usually consists of multiple modules such as speech separation, speaker diarization and ASR. On the other hand, considering the joint optimization, an end-to-end (E2E) SA-ASR model has recently been proposed with promising results on simulation data. In this paper, we present our recent study on the comparison of such modular and joint approaches towards SA-ASR on real monaural recordings. We develop state-of-the-art SA-ASR systems for both modular and joint approaches by leveraging large-scale training data, including 75 thousand hours of ASR training data and the VoxCeleb corpus for speaker representation learning. We also propose a new pipeline that performs the E2E SA-ASR model after speaker clustering. Our evaluation on the AMI meeting corpus reveals that after fine-tuning with a small real data, the joint system performs 8.9--29.9% better in accuracy compared to the best modular system while the modular system performs better before such fine-tuning. We also conduct various error analyses to show the remaining issues for the monaural SA-ASR.

eess.AS↗

arXiv2021

AISHELL-4: An Open Source Dataset for Speech Enhancement, Separation, Recognition and Speaker Diarization in Conference Scenario

In this paper, we present AISHELL-4, a sizable real-recorded Mandarin speech dataset collected by 8-channel circular microphone array for speech processing in conference scenario. The dataset consists of 211 recorded meeting sessions, each containing 4 to 8 speakers, with a total length of 120 hours. This dataset aims to bridge the advanced research on multi-speaker processing and the practical application scenario in three aspects. With real recorded meetings, AISHELL-4 provides realistic acoustics and rich natural speech characteristics in conversation such as short pause, speech overlap, quick speaker turn, noise, etc. Meanwhile, accurate transcription and speaker voice activity are provided for each meeting in AISHELL-4. This allows the researchers to explore different aspects in meeting processing, ranging from individual tasks such as speech front-end processing, speech recognition and speaker diarization, to multi-modality modeling and joint optimization of relevant tasks. Given most open source dataset for multi-speaker tasks are in English, AISHELL-4 is the only Mandarin dataset for conversation speech, providing additional value for data diversity in speech community. We also release a PyTorch-based training and evaluation framework as baseline system to promote reproducible research in this field.

cs.SD↗

arXiv2021

Target-speaker Voice Activity Detection with Improved I-Vector Estimation for Unknown Number of Speaker

Target-speaker voice activity detection (TS-VAD) has recently shown promising results for speaker diarization on highly overlapped speech. However, the original model requires a fixed (and known) number of speakers, which limits its application to real conversations. In this paper, we extend TS-VAD to speaker diarization with unknown numbers of speakers. This is achieved by two steps: first, an initial diarization system is applied for speaker number estimation, followed by TS-VAD network output masking according to this estimate. We further investigate different diarization methods, including clustering-based and region proposal networks, for estimating the initial i-vectors. Since these systems have complementary strengths, we propose a fusion-based method to combine frame-level decisions from the systems for an improved initialization. We demonstrate through experiments on variants of the LibriCSS meeting corpus that our proposed approach can improve the DER by up to 50\% relative across varying numbers of speakers. This improvement also results in better downstream ASR performance approaching that using oracle segments.

eess.AS↗

arXiv2021

Spacetime Neural Network for High Dimensional Quantum Dynamics

We develop a spacetime neural network method with second order optimization for solving quantum dynamics from the high dimensional Schrödinger equation. In contrast to the standard iterative first order optimization and the time-dependent variational principle, our approach utilizes the implicit mid-point method and generates the solution for all spatial and temporal values simultaneously after optimization. We demonstrate the method in the Schrödinger equation with a self-normalized autoregressive spacetime neural network construction. Future explorations for solving different high dimensional differential equations are discussed.

cond-mat.dis-nn↗

arXiv2021

Investigation of Practical Aspects of Single Channel Speech Separation for ASR

Speech separation has been successfully applied as a frontend processing module of conversation transcription systems thanks to its ability to handle overlapped speech and its flexibility to combine with downstream tasks such as automatic speech recognition (ASR). However, a speech separation model often introduces target speech distortion, resulting in a sub-optimum word error rate (WER). In this paper, we describe our efforts to improve the performance of a single channel speech separation system. Specifically, we investigate a two-stage training scheme that firstly applies a feature level optimization criterion for pretraining, followed by an ASR-oriented optimization criterion using an end-to-end (E2E) speech recognition model. Meanwhile, to keep the model light-weight, we introduce a modified teacher-student learning technique for model compression. By combining those approaches, we achieve a absolute average WER improvement of 2.70% and 0.77% using models with less than 10M parameters compared with the previous state-of-the-art results on the LibriCSS dataset for utterance-wise evaluation and continuous evaluation, respectively

eess.AS↗

arXiv2021

Modeling and Reasoning in Event Calculus using Goal-Directed Constraint Answer Set Programming

Automated commonsense reasoning is essential for building human-like AI systems featuring, for example, explainable AI. Event Calculus (EC) is a family of formalisms that model commonsense reasoning with a sound, logical basis. Previous attempts to mechanize reasoning using EC faced difficulties in the treatment of the continuous change in dense domains (e.g., time and other physical quantities), constraints among variables, default negation, and the uniform application of different inference methods, among others. We propose the use of s(CASP), a query-driven, top-down execution model for Predicate Answer Set Programming with Constraints, to model and reason using EC. We show how EC scenarios can be naturally and directly encoded in s(CASP) and how it enables deductive and abductive reasoning tasks in domains featuring constraints involving both dense time and dense fluents.

cs.AI↗

arXiv2021

Lifting The Grey Curtain: A First Look at the Ecosystem of CULPRITWARE

Mobile apps are extensively involved in cyber-crimes. Some apps are malware which compromise users' devices, while some others may lead to privacy leakage. Apart from them, there also exist apps which directly make profit from victims through deceiving, threatening or other criminal actions. We name these apps as CULPRITWARE. They have become emerging threats in recent years. However, the characteristics and the ecosystem of CULPRITWARE remain mysterious. This paper takes the first step towards systematically studying CULPRITWARE and its ecosystem. Specifically, we first characterize CULPRITWARE by categorizing and comparing them with benign apps and malware. The result shows that CULPRITWARE have unique features, e.g., the usage of app generators (25.27%) deviates from that of benign apps (5.08%) and malware (0.43%). Such a discrepancy can be used to distinguish CULPRITWARE from benign apps and malware. Then we understand the structure of the ecosystem by revealing the four participating entities (i.e., developer, agent, operator and reaper) and the workflow. After that, we further reveal the characteristics of the ecosystem by studying the participating entities. Our investigation shows that the majority of CULPRITWARE (at least 52.08%) are propagated through social media rather than the official app markets, and most CULPRITWARE (96%) indirectly rely on the covert fourth-party payment services to transfer the profits. Our findings shed light on the ecosystem, and can facilitate the community and law enforcement authorities to mitigate the threats. We will release the source code of our tools to engage the community.

cs.CR↗

arXiv2021

Human Listening and Live Captioning: Multi-Task Training for Speech Enhancement

With the surge of online meetings, it has become more critical than ever to provide high-quality speech audio and live captioning under various noise conditions. However, most monaural speech enhancement (SE) models introduce processing artifacts and thus degrade the performance of downstream tasks, including automatic speech recognition (ASR). This paper proposes a multi-task training framework to make the SE models unharmful to ASR. Because most ASR training samples do not have corresponding clean signal references, we alternately perform two model update steps called SE-step and ASR-step. The SE-step uses clean and noisy signal pairs and a signal-based loss function. The ASR-step applies a pre-trained ASR model to training signals enhanced with the SE model. A cross-entropy loss between the ASR output and reference transcriptions is calculated to update the SE model parameters. Experimental results with realistic large-scale settings using ASR models trained on 75,000-hour data show that the proposed framework improves the word error rate for the SE output by 11.82% with little compromise in the SE quality. Performance analysis is also carried out by changing the ASR model, the data used for the ASR-step, and the schedule of the two update steps.

eess.AS↗

arXiv2021

Hopf algebras arising from dg manifolds

Let $(\mathcal{M}, Q)$ be a dg manifold. The space of vector fields with shifted degrees $(\mathcal{X}(\mathcal{M})[-1], L_Q)$ is a Lie algebra object in the homology category $\mathrm{H}((C^{\infty}_{\mathcal{M}},Q)\mathrm{-}\mathbf{mod})$ of dg modules over $(\mathcal{M},Q)$, the Atiyah class $α_{\mathcal{M}}$ being its Lie bracket. The triple $(\mathcal{X}(\mathcal{M})[-1], L_Q; α_{\mathcal{M}})$ is also a Lie algebra object in the Gabriel-Zisman homotopy category $Π((C^{\infty}_{\mathcal{M}},Q)\mathrm{-}\mathbf{mod})$. In this paper, we describe the universal enveloping algebra of $(\mathcal{X}(\mathcal{M})[-1], L_Q; α_{\mathcal{M}})$ and prove that it is a Hopf algebra object in $Π((C^{\infty}_{\mathcal{M}},Q)\mathrm{-}\mathbf{mod})$. As an application, we study Fedosov dg Lie algebroids and recover a result of Stiénon, Xu, and the second author on the Hopf algebra arising from a Lie pair.

math.DG↗

arXiv2021

Knowledge-aware Zero-Shot Learning: Survey and Perspective

Zero-shot learning (ZSL) which aims at predicting classes that have never appeared during the training using external knowledge (a.k.a. side information) has been widely investigated. In this paper we present a literature review towards ZSL in the perspective of external knowledge, where we categorize the external knowledge, review their methods and compare different external knowledge. With the literature review, we further discuss and outlook the role of symbolic knowledge in addressing ZSL and other machine learning sample shortage issues.

cs.AI↗

arXiv2021

Observation of eccentric binary black hole mergers with second and third generation gravitational wave detector networks

[Abridged] We introduce an improved version of the Eccentric, Non-spinning, Inspiral-Gaussian-process Merger Approximant (ENIGMA) waveform model. We find that this ready-to-use model can: (i) produce physically consistent signals when sampling over 1M samples chosen over the $m_{\{1,\,2\}}\in[5M_{\odot},\,50M_{\odot}]$ parameter space, and the entire range of binary inclination angles; (ii) produce waveforms within 0.04 seconds from an initial gravitational wave frequency $f_{\textrm{GW}} =15\,\textrm{Hz}$ and at a sample rate of 8192 Hz; and (iii) reproduce the physics of quasi-circular mergers. We utilize ENIGMA to compute the expected signal-to-noise ratio (SNR) distributions of eccentric binary black hole mergers assuming the existence of second and third generation gravitational wave detector networks that include the twin LIGO detectors, Virgo, KAGRA, LIGO-India, a LIGO-type detector in Australia, Cosmic Explorer, and the Einstein Telescope. In the context of advanced LIGO-type detectors, we find that the SNR of eccentric mergers is always larger than quasi-circular mergers for systems with $e_0\leq0.4$ at $f_{\textrm{GW}} =10\,\textrm{Hz}$, even if the timespan of eccentric signals is just a third of quasi-circular systems with identical total mass and mass-ratio. For Cosmic Explorer-type detector networks, we find that eccentric mergers have similar SNRs than quasi-circular systems for $e_0\leq0.3$ at $f_{\textrm{GW}} =10\,\textrm{Hz}$. Systems with $e_0\sim0.5$ at $f_{\textrm{GW}} =10\,\textrm{Hz}$ have SNRs that range between 50%-90% of the SNR produced by quasi-circular mergers, even if these eccentric signals are just between a third to a tenth the length of quasi-circular systems. For Einstein Telescope-type detectors, we find that eccentric mergers have similar SNRs than quasi-circular systems for $e_0\leq0.4$ at $f_{\textrm{GW}} =5\,\textrm{Hz}$.

gr-qc↗

arXiv2021

Large-Scale Pre-Training of End-to-End Multi-Talker ASR for Meeting Transcription with Single Distant Microphone

Transcribing meetings containing overlapped speech with only a single distant microphone (SDM) has been one of the most challenging problems for automatic speech recognition (ASR). While various approaches have been proposed, all previous studies on the monaural overlapped speech recognition problem were based on either simulation data or small-scale real data. In this paper, we extensively investigate a two-step approach where we first pre-train a serialized output training (SOT)-based multi-talker ASR by using large-scale simulation data and then fine-tune the model with a small amount of real meeting data. Experiments are conducted by utilizing 75 thousand (K) hours of our internal single-talker recording to simulate a total of 900K hours of multi-talker audio segments for supervised pre-training. With fine-tuning on the 70 hours of the AMI-SDM training data, our SOT ASR model achieves a word error rate (WER) of 21.2% for the AMI-SDM evaluation set while automatically counting speakers in each test segment. This result is not only significantly better than the previous state-of-the-art WER of 36.4% with oracle utterance boundary information but also better than a result by a similarly fine-tuned single-talker ASR model applied to beamformed audio.

eess.AS↗

arXiv2021

End-to-End Speaker-Attributed ASR with Transformer

This paper presents our recent effort on end-to-end speaker-attributed automatic speech recognition, which jointly performs speaker counting, speech recognition and speaker identification for monaural multi-talker audio. Firstly, we thoroughly update the model architecture that was previously designed based on a long short-term memory (LSTM)-based attention encoder decoder by applying transformer architectures. Secondly, we propose a speaker deduplication mechanism to reduce speaker identification errors in highly overlapped regions. Experimental results on the LibriSpeechMix dataset shows that the transformer-based architecture is especially good at counting the speakers and that the proposed model reduces the speaker-attributed word error rate by 47% over the LSTM-based baseline. Furthermore, for the LibriCSS dataset, which consists of real recordings of overlapped speech, the proposed model achieves concatenated minimum-permutation word error rates of 11.9% and 16.3% with and without target speaker profiles, respectively, both of which are the state-of-the-art results for LibriCSS with the monaural setting.

eess.AS↗

arXiv2021

Continuous Speech Separation with Ad Hoc Microphone Arrays

Speech separation has been shown effective for multi-talker speech recognition. Under the ad hoc microphone array setup where the array consists of spatially distributed asynchronous microphones, additional challenges must be overcome as the geometry and number of microphones are unknown beforehand. Prior studies show, with a spatial-temporalinterleaving structure, neural networks can efficiently utilize the multi-channel signals of the ad hoc array. In this paper, we further extend this approach to continuous speech separation. Several techniques are introduced to enable speech separation for real continuous recordings. First, we apply a transformer-based network for spatio-temporal modeling of the ad hoc array signals. In addition, two methods are proposed to mitigate a speech duplication problem during single talker segments, which seems more severe in the ad hoc array scenarios. One method is device distortion simulation for reducing the acoustic mismatch between simulated training data and real recordings. The other is speaker counting to detect the single speaker segments and merge the output signal channels. Experimental results for AdHoc-LibiCSS, a new dataset consisting of continuous recordings of concatenated LibriSpeech utterances obtained by multiple different devices, show the proposed separation method can significantly improve the ASR accuracy for overlapped speech with little performance degradation for single talker segments.

cs.SD↗

arXiv2021

Dual-Path Modeling for Long Recording Speech Separation in Meetings

The continuous speech separation (CSS) is a task to separate the speech sources from a long, partially overlapped recording, which involves a varying number of speakers. A straightforward extension of conventional utterance-level speech separation to the CSS task is to segment the long recording with a size-fixed window and process each window separately. Though effective, this extension fails to model the long dependency in speech and thus leads to sub-optimum performance. The recent proposed dual-path modeling could be a remedy to this problem, thanks to its capability in jointly modeling the cross-window dependency and the local-window processing. In this work, we further extend the dual-path modeling framework for CSS task. A transformer-based dual-path system is proposed, which integrates transform layers for global modeling. The proposed models are applied to LibriCSS, a real recorded multi-talk dataset, and consistent WER reduction can be observed in the ASR evaluation for separated speech. Also, a dual-path transformer equipped with convolutional layers is proposed. It significantly reduces the computation amount by 30% with better WER evaluation. Furthermore, the online processing dual-path models are investigated, which shows 10% relative WER reduction compared to the baseline.

eess.AS↗

arXiv2021

OntoZSL: Ontology-enhanced Zero-shot Learning

Zero-shot Learning (ZSL), which aims to predict for those classes that have never appeared in the training data, has arisen hot research interests. The key of implementing ZSL is to leverage the prior knowledge of classes which builds the semantic relationship between classes and enables the transfer of the learned models (e.g., features) from training classes (i.e., seen classes) to unseen classes. However, the priors adopted by the existing methods are relatively limited with incomplete semantics. In this paper, we explore richer and more competitive prior knowledge to model the inter-class relationship for ZSL via ontology-based knowledge representation and semantic embedding. Meanwhile, to address the data imbalance between seen classes and unseen classes, we developed a generative ZSL framework with Generative Adversarial Networks (GANs). Our main findings include: (i) an ontology-enhanced ZSL framework that can be applied to different domains, such as image classification (IMGC) and knowledge graph completion (KGC); (ii) a comprehensive evaluation with multiple zero-shot datasets from different domains, where our method often achieves better performance than the state-of-the-art models. In particular, on four representative ZSL baselines of IMGC, the ontology-based class semantics outperform the previous priors e.g., the word embeddings of classes by an average of 12.4 accuracy points in the standard ZSL across two example datasets (see Figure 4).

cs.AI↗

arXiv2020

Continuous Speech Separation Using Speaker Inventory for Long Multi-talker Recording

Leveraging additional speaker information to facilitate speech separation has received increasing attention in recent years. Recent research includes extracting target speech by using the target speaker's voice snippet and jointly separating all participating speakers by using a pool of additional speaker signals, which is known as speech separation using speaker inventory (SSUSI). However, all these systems ideally assume that the pre-enrolled speaker signals are available and are only evaluated on simple data configurations. In realistic multi-talker conversations, the speech signal contains a large proportion of non-overlapped regions, where we can derive robust speaker embedding of individual talkers. In this work, we adopt the SSUSI model in long recordings and propose a self-informed, clustering-based inventory forming scheme for long recording, where the speaker inventory is fully built from the input signal without the need for external speaker signals. Experiment results on simulated noisy reverberant long recording datasets show that the proposed method can significantly improve the separation performance across various conditions.

eess.AS↗

arXiv2020

Task Offloading for Large-Scale Asynchronous Mobile Edge Computing: An Index Policy Approach

Mobile-edge computing (MEC) offloads computational tasks from wireless devices to network edge, and enables real-time information transmission and computing. Most existing work concerns a small-scale synchronous MEC system. In this paper, we focus on a large-scale asynchronous MEC system with random task arrivals, distinct workloads, and diverse deadlines. We formulate the offloading policy design as a restless multi-armed bandit (RMAB) to maximize the total discounted reward over the time horizon. However, the formulated RMAB is related to a PSPACE-hard sequential decision-making problem, which is intractable. To address this issue, by exploiting the Whittle index (WI) theory, we rigorously establish the WI indexability and derive a scalable closed-form solution. Consequently, in our WI policy, each user only needs to calculate its WI and report it to the BS, and the users with the highest indices are selected for task offloading. Furthermore, when the task completion ratio becomes the focus, the shorter slack time less remaining workload (STLW) priority rule is introduced into the WI policy for performance improvement. When the knowledge of user offloading energy consumption is not available prior to the offloading, we develop Bayesian learning-enabled WI policies, including maximum likelihood estimation, Bayesian learning with conjugate prior, and prior-swapping techniques. Simulation results show that the proposed policies significantly outperform the other existing policies.

cs.IT↗