arXiv subjects

Guoqing Zhao

Publications and source records attributed to Guoqing Zhao.

At least 19 recordsLinked to original sources

arXiv2024

Boosting Multi-Speaker Expressive Speech Synthesis with Semi-supervised Contrastive Learning

This paper aims to build a multi-speaker expressive TTS system, synthesizing a target speaker's speech with multiple styles and emotions. To this end, we propose a novel contrastive learning-based TTS approach to transfer style and emotion across speakers. Specifically, contrastive learning from different levels, i.e. utterance and category level, is leveraged to extract the disentangled style, emotion, and speaker representations from speech for style and emotion transfer. Furthermore, a semi-supervised training strategy is introduced to improve the data utilization efficiency by involving multi-domain data, including style-labeled data, emotion-labeled data, and abundant unlabeled data. To achieve expressive speech with diverse styles and emotions for a target speaker, the learned disentangled representations are integrated into an improved VITS model. Experiments on multi-domain data demonstrate the effectiveness of the proposed method.

eess.AS↗

arXiv2024

Factorized Learning Assisted with Large Language Model for Gloss-free Sign Language Translation

Previous Sign Language Translation (SLT) methods achieve superior performance by relying on gloss annotations. However, labeling high-quality glosses is a labor-intensive task, which limits the further development of SLT. Although some approaches work towards gloss-free SLT through jointly training the visual encoder and translation network, these efforts still suffer from poor performance and inefficient use of the powerful Large Language Model (LLM). Most seriously, we find that directly introducing LLM into SLT will lead to insufficient learning of visual representations as LLM dominates the learning curve. To address these problems, we propose Factorized Learning assisted with Large Language Model (FLa-LLM) for gloss-free SLT. Concretely, we factorize the training process into two stages. In the visual initialing stage, we employ a lightweight translation model after the visual encoder to pre-train the visual encoder. In the LLM fine-tuning stage, we freeze the acquired knowledge in the visual encoder and integrate it with a pre-trained LLM to inspire the LLM's translation potential. This factorized training strategy proves to be highly effective as evidenced by significant improvements achieved across three SLT datasets which are all conducted under the gloss-free setting.

cs.CL↗

arXiv2024

SELM: Speech Enhancement Using Discrete Tokens and Language Models

Language models (LMs) have shown superior performances in various speech generation tasks recently, demonstrating their powerful ability for semantic context modeling. Given the intrinsic similarity between speech generation and speech enhancement, harnessing semantic information holds potential advantages for speech enhancement tasks. In light of this, we propose SELM, a novel paradigm for speech enhancement, which integrates discrete tokens and leverages language models. SELM comprises three stages: encoding, modeling, and decoding. We transform continuous waveform signals into discrete tokens using pre-trained self-supervised learning (SSL) models and a k-means tokenizer. Language models then capture comprehensive contextual information within these tokens. Finally, a detokenizer and HiFi-GAN restore them into enhanced speech. Experimental results demonstrate that SELM achieves comparable performance in objective metrics alongside superior results in subjective perception. Our demos are available https://honee-w.github.io/SELM/.

eess.AS↗

arXiv2023

VoxBlink: A Large Scale Speaker Verification Dataset on Camera

In this paper, we introduce a large-scale and high-quality audio-visual speaker verification dataset, named VoxBlink. We propose an innovative and robust automatic audio-visual data mining pipeline to curate this dataset, which contains 1.45M utterances from 38K speakers. Due to the inherent nature of automated data collection, introducing noisy data is inevitable. Therefore, we also utilize a multi-modal purification step to generate a cleaner version of the VoxBlink, named VoxBlink-clean, comprising 18K identities and 1.02M utterances. In contrast to the VoxCeleb, the VoxBlink sources from short videos of ordinary users, and the covered scenarios can better align with real-life situations. To our best knowledge, the VoxBlink dataset is one of the largest publicly available speaker verification datasets. Leveraging the VoxCeleb and VoxBlink-clean datasets together, we employ diverse speaker verification models with multiple architectural backbones to conduct comprehensive evaluations on the VoxCeleb test sets. Experimental results indicate a substantial enhancement in performance,ranging from 12% to 30% relatively, across various backbone architectures upon incorporating the VoxBlink-clean into the training process. The details of the dataset can be found on http://voxblink.github.io

eess.AS↗

arXiv2023

Haha-Pod: An Attempt for Laughter-based Non-Verbal Speaker Verification

It is widely acknowledged that discriminative representation for speaker verification can be extracted from verbal speech. However, how much speaker information that non-verbal vocalization carries is still a puzzle. This paper explores speaker verification based on the most ubiquitous form of non-verbal voice, laughter. First, we use a semi-automatic pipeline to collect a new Haha-Pod dataset from open-source podcast media. The dataset contains over 240 speakers' laughter clips with corresponding high-quality verbal speech. Second, we propose a Two-Stage Teacher-Student (2S-TS) framework to minimize the within-speaker embedding distance between verbal and non-verbal (laughter) signals. Considering Haha-Pod as a test set, two trials (S2L-Eval) are designed to verify the speaker's identity through laugh sounds. Experimental results demonstrate that our method can significantly improve the performance of the S2L-Eval test set with only a minor degradation on the VoxCeleb1 test set. The resources for the Haha-Pod dataset can be found at https://github.com/nevermoreLin/HahaPod.

eess.AS↗

arXiv2023

Multi-objective Progressive Clustering for Semi-supervised Domain Adaptation in Speaker Verification

Utilizing the pseudo-labeling algorithm with large-scale unlabeled data becomes crucial for semi-supervised domain adaptation in speaker verification tasks. In this paper, we propose a novel pseudo-labeling method named Multi-objective Progressive Clustering (MoPC), specifically designed for semi-supervised domain adaptation. Firstly, we utilize limited labeled data from the target domain to derive domain-specific descriptors based on multiple distinct objectives, namely within-graph denoising, intra-class denoising and inter-class denoising. Then, the Infomap algorithm is adopted for embedding clustering, and the descriptors are leveraged to further refine the target domain's pseudo-labels. Moreover, to further improve the quality of pseudo labels, we introduce the subcenter-purification and progressive-merging strategy for label denoising. Our proposed MoPC method achieves 4.95% EER and ranked the 1$^{st}$ place on the evaluation set of VoxSRC 2023 track 3. We also conduct additional experiments on the FFSVC dataset and yield promising results.

eess.AS↗

arXiv2023

The DKU-MSXF Diarization System for the VoxCeleb Speaker Recognition Challenge 2023

This paper describes the DKU-MSXF submission to track 4 of the VoxCeleb Speaker Recognition Challenge 2023 (VoxSRC-23). Our system pipeline contains voice activity detection, clustering-based diarization, overlapped speech detection, and target-speaker voice activity detection, where each procedure has a fused output from 3 sub-models. Finally, we fuse different clustering-based and TSVAD-based diarization systems using DOVER-Lap and achieve the 4.30% diarization error rate (DER), which ranks first place on track 4 of the challenge leaderboard.

eess.AS↗

arXiv2023

The DKU-MSXF Speaker Verification System for the VoxCeleb Speaker Recognition Challenge 2023

This paper is the system description of the DKU-MSXF System for the track1, track2 and track3 of the VoxCeleb Speaker Recognition Challenge 2023 (VoxSRC-23). For Track 1, we utilize a network structure based on ResNet for training. By constructing a cross-age QMF training set, we achieve a substantial improvement in system performance. For Track 2, we inherite the pre-trained model from Track 1 and conducte mixed training by incorporating the VoxBlink-clean dataset. In comparison to Track 1, the models incorporating VoxBlink-clean data exhibit a performance improvement by more than 10% relatively. For Track3, the semi-supervised domain adaptation task, a novel pseudo-labeling method based on triple thresholds and sub-center purification is adopted to make domain adaptation. The final submission achieves mDCF of 0.1243 in task1, mDCF of 0.1165 in Track 2 and EER of 4.952% in Track 3.

eess.AS↗

arXiv2023

The NPU-MSXF Speech-to-Speech Translation System for IWSLT 2023 Speech-to-Speech Translation Task

This paper describes the NPU-MSXF system for the IWSLT 2023 speech-to-speech translation (S2ST) task which aims to translate from English speech of multi-source to Chinese speech. The system is built in a cascaded manner consisting of automatic speech recognition (ASR), machine translation (MT), and text-to-speech (TTS). We make tremendous efforts to handle the challenging multi-source input. Specifically, to improve the robustness to multi-source speech input, we adopt various data augmentation strategies and a ROVER-based score fusion on multiple ASR model outputs. To better handle the noisy ASR transcripts, we introduce a three-stage fine-tuning strategy to improve translation accuracy. Finally, we build a TTS model with high naturalness and sound quality, which leverages a two-stage framework, using network bottleneck features as a robust intermediate representation for speaker timbre and linguistic content disentanglement. Based on the two-stage framework, pre-trained speaker embedding is leveraged as a condition to transfer the speaker timbre in the source English speech to the translated Chinese speech. Experimental results show that our system has high translation accuracy, speech naturalness, sound quality, and speaker similarity. Moreover, it shows good robustness to multi-source data.

cs.SD↗

arXiv2023

TreeMAN: Tree-enhanced Multimodal Attention Network for ICD Coding

ICD coding is designed to assign the disease codes to electronic health records (EHRs) upon discharge, which is crucial for billing and clinical statistics. In an attempt to improve the effectiveness and efficiency of manual coding, many methods have been proposed to automatically predict ICD codes from clinical notes. However, most previous works ignore the decisive information contained in structured medical data in EHRs, which is hard to be captured from the noisy clinical notes. In this paper, we propose a Tree-enhanced Multimodal Attention Network (TreeMAN) to fuse tabular features and textual features into multimodal representations by enhancing the text representations with tree-based features via the attention mechanism. Tree-based features are constructed according to decision trees learned from structured multimodal medical data, which capture the decisive information about ICD coding. We can apply the same multi-label classifier from previous text models to the multimodal representations to predict ICD codes. Experiments on two MIMIC datasets show that our method outperforms prior state-of-the-art ICD coding approaches. The code is available at https://github.com/liu-zichen/TreeMAN.

cs.CL↗

arXiv2023

MADI: Inter-domain Matching and Intra-domain Discrimination for Cross-domain Speech Recognition

End-to-end automatic speech recognition (ASR) usually suffers from performance degradation when applied to a new domain due to domain shift. Unsupervised domain adaptation (UDA) aims to improve the performance on the unlabeled target domain by transferring knowledge from the source to the target domain. To improve transferability, existing UDA approaches mainly focus on matching the distributions of the source and target domains globally and/or locally, while ignoring the model discriminability. In this paper, we propose a novel UDA approach for ASR via inter-domain MAtching and intra-domain DIscrimination (MADI), which improves the model transferability by fine-grained inter-domain matching and discriminability by intra-domain contrastive discrimination simultaneously. Evaluations on the Libri-Adapt dataset demonstrate the effectiveness of our approach. MADI reduces the relative word error rate (WER) on cross-device and cross-environment ASR by 17.7% and 22.8%, respectively.

cs.CL↗

arXiv2022

MoSE: Modality Split and Ensemble for Multimodal Knowledge Graph Completion

Multimodal knowledge graph completion (MKGC) aims to predict missing entities in MKGs. Previous works usually share relation representation across modalities. This results in mutual interference between modalities during training, since for a pair of entities, the relation from one modality probably contradicts that from another modality. Furthermore, making a unified prediction based on the shared relation representation treats the input in different modalities equally, while their importance to the MKGC task should be different. In this paper, we propose MoSE, a Modality Split representation learning and Ensemble inference framework for MKGC. Specifically, in the training phase, we learn modality-split relation embeddings for each modality instead of a single modality-shared one, which alleviates the modality interference. Based on these embeddings, in the inference phase, we first make modality-split predictions and then exploit various ensemble methods to combine the predictions with different weights, which models the modality importance dynamically. Experimental results on three KG datasets show that MoSE outperforms state-of-the-art MKGC methods. Codes are available at https://github.com/OreOZhao/MoSE4MKGC.

cs.CL↗

arXiv2022

Overcoming Language Priors in Visual Question Answering via Distinguishing Superficially Similar Instances

Despite the great progress of Visual Question Answering (VQA), current VQA models heavily rely on the superficial correlation between the question type and its corresponding frequent answers (i.e., language priors) to make predictions, without really understanding the input. In this work, we define the training instances with the same question type but different answers as \textit{superficially similar instances}, and attribute the language priors to the confusion of VQA model on such instances. To solve this problem, we propose a novel training framework that explicitly encourages the VQA model to distinguish between the superficially similar instances. Specifically, for each training instance, we first construct a set that contains its superficially similar counterparts. Then we exploit the proposed distinguishing module to increase the distance between the instance and its counterparts in the answer space. In this way, the VQA model is forced to further focus on the other parts of the input beyond the question type, which helps to overcome the language priors. Experimental results show that our method achieves the state-of-the-art performance on VQA-CP v2. Codes are available at \href{https://github.com/wyk-nku/Distinguishing-VQA.git}{Distinguishing-VQA}.

cs.CL↗

arXiv2022

TPSNet: Reverse Thinking of Thin Plate Splines for Arbitrary Shape Scene Text Representation

The research focus of scene text detection and recognition has shifted to arbitrary shape text in recent years, where the text shape representation is a fundamental problem. An ideal representation should be compact, complete, efficient, and reusable for subsequent recognition in our opinion. However, previous representations have flaws in one or more aspects. Thin-Plate-Spline (TPS) transformation has achieved great success in scene text recognition. Inspired by this, we reversely think of its usage and sophisticatedly take TPS as an exquisite representation for arbitrary shape text representation. The TPS representation is compact, complete, and efficient. With the predicted TPS parameters, the detected text region can be directly rectified to a near-horizontal one to assist the subsequent recognition. To further exploit the potential of the TPS representation, the Border Alignment Loss is proposed. Based on these designs, we implement the text detector TPSNet, which can be extended to a text spotter conveniently. Extensive evaluation and ablation of several public benchmarks demonstrate the effectiveness and superiority of the proposed method for text representation and spotting. Particularly, TPSNet achieves the detection F-Measure improvement of 4.4\% (78.4\% vs. 74.0\%) on Art dataset and the end-to-end spotting F-Measure improvement of 5.0\% (78.5\% vs. 73.5\%) on Total-Text, which are large margins with no bells and whistles.

cs.CV↗

arXiv2021

Growth of Outward Propagating Fast-Magnetosonic/Whistler Waves in the Inner Heliosphere Observed by Parker Solar Probe

The solar wind in the inner heliosphere has been observed by Parker Solar Probe (PSP) to exhibit abundant wave activities. The cyclotron wave modes in the sense of ions or electrons are among the most crucial wave components. However, their origin and evolution in the inner heliosphere close to the Sun remain mysteries. Specifically, it remains unknown whether it is an emitted signal from the solar atmosphere or an eigenmode growing locally in the heliosphere due to plasma instability. To address and resolve this controversy, we must investigate the key quantity of the energy change rate of the wave mode. We develop a new technique to measure the energy change rate of plasma waves, and apply this technique to the wave electromagnetic fields measured by PSP. We provide the wave Poynting flux in the solar wind frame, identify the wave nature to be the outward propagating fast-magnetosonic/whistler wave mode instead of the sunward propagating waves. We provide the first evidence for growth of the fast-magnetosonic/whistler wave mode in the inner heliosphere based on the derived spectra of the real and imaginary parts of the wave frequencies. The energy change rate rises and stays at a positive level in the same wavenumber range as the bumps of the electromagnetic field power spectral densities, clearly manifesting that the observed fast-magnetosonic/whistler waves are locally growing to a large amplitude.

astro-ph.SR↗

arXiv2019

Comparison of counterstreaming suprathermal electron signatures of ICMEs with and without magnetic cloud: are all ICMEs flux ropes?

Magnetic clouds (MCs), as large-scale interplanetary magnetic flux ropes, are usually still connected to the sun at both ends near 1 AU. Many researchers believe that all non-MC interplanetary coronal mass ejections (ICMEs) also have magnetic flux rope structures, which are inconspicuous because the observing spacecraft crosses the flanks of the rope structures. If so, the field lines of non-MC ICMEs should also be usually connected to the Sun on both ends. Then we want to know whether the field lines of most non-MC ICMEs are still connected to the sun at both ends or not. This study examined the counterstreaming suprathermal electron (CSE) signatures of 266 ICMEs observed by the \emph{Advanced Composition Explorer} (\emph{ACE}) spacecraft from 1998 to 2008 and compared the CSE signatures of MCs and non-MC ICMEs. Results show that only 10 of the 101 MC events ($9.9\%$ ) and 75 of the 171 non-MC events ($43.9\%$) have no CSEs. Moreover, 21 of the non-MC ICMEs have high CSE percentages (more than $70\%$) and show relatively stable magnetic field components with slight rotations, which are in line with the expectations that spacecraft passes through the flank of magnetic flux ropes. So the 21 events may be magnetic flux ropes but the \emph{ACE} spacecraft passes through their flanks of magnetic flux ropes. Considering that most other non-MC events have disordered magnetic fields, we suggest that some non-MC ICMEs inherently have disordered magnetic fields, namely have no magnetic flux rope structures.

physics.space-ph↗

arXiv2018

Observations on a series of merging magnetic flux ropes within an interplanetary coronal mass ejection

Coronal mass ejections (CMEs) are intense solar explosive eruptions. CMEs are highly important players in solar-terrestrial relationships, and they have important consequences for major geomagnetic storms and energetic particle events. It has been unclear how CMEs evolve when they propagate in the heliosphere. Here we report an interplanetary coronal mass ejection (ICME) consisting of multiple magnetic flux ropes measured by WIND on March 25-26, 1998. These magnetic flux ropes were merging with each other. The observations indicate that internal interactions (reconnections) within multi-flux-rope CME can coalesce into large-scale ropes, which may improve our understanding of the interplanetary evolution of CMEs. In addition, we speculated that the reported rope-rope interactions may also exist between successive rope-like CMEs and are important for the space weather forecasting.

physics.space-ph↗

arXiv2018

Cold prominence materials detected within magnetic clouds during 1998-2007

Coronal mass ejections (CMEs) are intense solar explosive eruptions, and they are frequently correlated with prominence eruptions. Previous observations show that about $70\%$ of CMEs are associated with prominence eruptions. However, there are only a handful of reported observations of prominence plasma materials within interplanetary CMEs (ICMEs), which are the interplanetary manifestations of CMEs. Moreover, approximately $4\%$ of ICMEs exhibit the presence of prominence materials, and approximately $12\%$ of magnetic clouds (MCs) contain prominence materials. We aim to comprehensively search for cold prominence materials in MCs observed by the Advanced Composition Explorer (ACE) spacecraft during 1998-2007. Using the criteria of unusual $O^{5+}$ and (or) $Fe^{6+}$ abundances, we examined 76 MCs observed by ACE during 1998-2007 to search for cold prominence materials. Our results revealed that out of the 76 MCs, 27 ($36\%$) events contained prominence material regions with low-charge-state signatures. Although the fraction is still lower than the approximately $70\%$ of CMEs associated with prominence eruptions, it is much higher than $12\%$. The unusual $O^{5+}$ and (or) $Fe^{6+}$ abundances may be simple and reliable criteria to investigate prominence materials in the interplanetary medium.

physics.space-ph↗