arXiv subjects

Cheng Yu

Publications and source records attributed to Cheng Yu.

At least 73 records · Page 4Linked to original sources

arXiv2021

MetricGAN-U: Unsupervised speech enhancement/ dereverberation based only on noisy/ reverberated speech

Most of the deep learning-based speech enhancement models are learned in a supervised manner, which implies that pairs of noisy and clean speech are required during training. Consequently, several noisy speeches recorded in daily life cannot be used to train the model. Although certain unsupervised learning frameworks have also been proposed to solve the pair constraint, they still require clean speech or noise for training. Therefore, in this paper, we propose MetricGAN-U, which stands for MetricGAN-unsupervised, to further release the constraint from conventional unsupervised learning. In MetricGAN-U, only noisy speech is required to train the model by optimizing non-intrusive speech quality metrics. The experimental results verified that MetricGAN-U outperforms baselines in both objective and subjective metrics.

cs.SD↗

arXiv2021

MetricGAN+: An Improved Version of MetricGAN for Speech Enhancement

The discrepancy between the cost function used for training a speech enhancement model and human auditory perception usually makes the quality of enhanced speech unsatisfactory. Objective evaluation metrics which consider human perception can hence serve as a bridge to reduce the gap. Our previously proposed MetricGAN was designed to optimize objective metrics by connecting the metric with a discriminator. Because only the scores of the target evaluation functions are needed during training, the metrics can even be non-differentiable. In this study, we propose a MetricGAN+ in which three training techniques incorporating domain-knowledge of speech processing are proposed. With these techniques, experimental results on the VoiceBank-DEMAND dataset show that MetricGAN+ can increase PESQ score by 0.3 compared to the previous MetricGAN and achieve state-of-the-art results (PESQ score = 3.15).

cs.SD↗

arXiv2021

Improving Perceptual Quality by Phone-Fortified Perceptual Loss using Wasserstein Distance for Speech Enhancement

Speech enhancement (SE) aims to improve speech quality and intelligibility, which are both related to a smooth transition in speech segments that may carry linguistic information, e.g. phones and syllables. In this study, we propose a novel phone-fortified perceptual loss (PFPL) that takes phonetic information into account for training SE models. To effectively incorporate the phonetic information, the PFPL is computed based on latent representations of the wav2vec model, a powerful self-supervised encoder that renders rich phonetic information. To more accurately measure the distribution distances of the latent representations, the PFPL adopts the Wasserstein distance as the distance measure. Our experimental results first reveal that the PFPL is more correlated with the perceptual evaluation metrics, as compared to signal-level losses. Moreover, the results showed that the PFPL can enable a deep complex U-Net SE model to achieve highly competitive performance in terms of standardized quality and intelligibility evaluations on the Voice Bank-DEMAND dataset.

cs.SD↗

arXiv2021

Global ill-posedness for a dense set of initial data to the Isentropic system of gas dynamics

In dimension $n=2$ and $3$, we show that for any initial datum belonging to a dense subset of the energy space, there exist infinitely many global-in-time admissible weak solutions to the isentropic Euler system whenever $1<γ\leq 1+\frac2n$. This result can be regarded as a compressible counterpart of the one obtained by Szekelyhidi--Wiedemann (ARMA, 2012) for incompressible flows. Similarly to the incompressible result, the admissibility condition is defined in its integral form. Our result is based on a generalization of a key step of the convex integration procedure. This generalization allows, even in the compressible case, to convex integrate any smooth positive Reynolds stress. A large family of subsolutions can then be considered. These subsolutions can be generated, for instance, via regularization of any weak inviscid limit of an associated compressible Navier--Stokes system with degenerate viscosities.

math.AP↗

arXiv2021

Boosting Objective Scores of a Speech Enhancement Model by MetricGAN Post-processing

The Transformer architecture has demonstrated a superior ability compared to recurrent neural networks in many different natural language processing applications. Therefore, our study applies a modified Transformer in a speech enhancement task. Specifically, positional encoding in the Transformer may not be necessary for speech enhancement, and hence, it is replaced by convolutional layers. To further improve the perceptual evaluation of the speech quality (PESQ) scores of enhanced speech, the L_1 pre-trained Transformer is fine-tuned using a MetricGAN framework. The proposed MetricGAN can be treated as a general post-processing module to further boost the objective scores of interest. The experiments were conducted using the data sets provided by the organizer of the Deep Noise Suppression (DNS) challenge. Experimental results demonstrated that the proposed system outperformed the challenge baseline, in both subjective and objective evaluations, with a large margin.

eess.AS↗

arXiv2021

Attention-based multi-task learning for speech-enhancement and speaker-identification in multi-speaker dialogue scenario

Multi-task learning (MTL) and attention mechanism have been proven to effectively extract robust acoustic features for various speech-related tasks in noisy environments. In this study, we propose an attention-based MTL (ATM) approach that integrates MTL and the attention-weighting mechanism to simultaneously realize a multi-model learning structure that performs speech enhancement (SE) and speaker identification (SI). The proposed ATM system consists of three parts: SE, SI, and attention-Net (AttNet). The SE part is composed of a long-short-term memory (LSTM) model, and a deep neural network (DNN) model is used to develop the SI and AttNet parts. The overall ATM system first extracts the representative features and then enhances the speech signals in LSTM-SE and specifies speaker identity in DNN-SI. The AttNet computes weights based on DNN-SI to prepare better representative features for LSTM-SE. We tested the proposed ATM system on Taiwan Mandarin hearing in noise test sentences. The evaluation results confirmed that the proposed system can effectively enhance speech quality and intelligibility of a given noisy input. Moreover, the accuracy of the SI can also be notably improved by using the proposed ATM system.

eess.AS↗

arXiv2021

Dissipative solutions to the compressible isentropic Navier-Stokes equations

The existence of dissipative solutions to the compressible isentropic Navier-Stokes equations was established in this paper. This notion was inspired by the concept of dissipative solutions to the incompressible Euler equations of Lions (\cite{Lions-1996}, Section 4.4). Our method is to recover such solutions by passing to the limits from approximated solutions, thanks to compactness argument.

math.AP↗

arXiv2021

Inviscid limit of the inhomogeneous incompressible Navier-Stokes equations under the weak Kolmogorov hypothesis in $\mathbb{R}^3$

In this paper, we consider the inviscid limit of inhomogeneous incompressible Navier-Stokes equations under the weak Kolmogorov hypothesis in $\mathbb{R}^3$. In particular, we first deduce the Kolmogorov-type hypothesis in $\mathbb{R}^3$, which yields the uniform bounds of $α^{th}$-order fractional derivatives of $\sqrt{ρ^μ}{\bf u}^μ$ in $L^2_x$ for some $α>0$, independent of the viscosity. The uniform bounds can provide strong convergence of $\sqrt{ρ^μ}\bf u^μ$ in $L^2$ space. This shows that the inviscid limit is a weak solution to the corresponding Euler equations.

math.AP↗

arXiv2020

Speech Enhancement based on Denoising Autoencoder with Multi-branched Encoders

Deep learning-based models have greatly advanced the performance of speech enhancement (SE) systems. However, two problems remain unsolved, which are closely related to model generalizability to noisy conditions: (1) mismatched noisy condition during testing, i.e., the performance is generally sub-optimal when models are tested with unseen noise types that are not involved in the training data; (2) local focus on specific noisy conditions, i.e., models trained using multiple types of noises cannot optimally remove a specific noise type even though the noise type has been involved in the training data. These problems are common in real applications. In this paper, we propose a novel denoising autoencoder with a multi-branched encoder (termed DAEME) model to deal with these two problems. In the DAEME model, two stages are involved: training and testing. In the training stage, we build multiple component models to form a multi-branched encoder based on a decision tree (DSDT). The DSDT is built based on prior knowledge of speech and noisy conditions (the speaker, environment, and signal factors are considered in this paper), where each component of the multi-branched encoder performs a particular mapping from noisy to clean speech along the branch in the DSDT. Finally, a decoder is trained on top of the multi-branched encoder. In the testing stage, noisy speech is first processed by each component model. The multiple outputs from these models are then integrated into the decoder to determine the final enhanced speech. Experimental results show that DAEME is superior to several baseline models in terms of objective evaluation metrics, automatic speech recognition results, and quality in subjective human listening tests.

eess.AS↗

arXiv2020

Shaping Deep Feature Space towards Gaussian Mixture for Visual Classification

The softmax cross-entropy loss function has been widely used to train deep models for various tasks. In this work, we propose a Gaussian mixture (GM) loss function for deep neural networks for visual classification. Unlike the softmax cross-entropy loss, our method explicitly shapes the deep feature space towards a Gaussian Mixture distribution. With a classification margin and a likelihood regularization, the GM loss facilitates both high classification performance and accurate modeling of the feature distribution. The GM loss can be readily used to distinguish abnormal inputs, such as the adversarial examples, based on the discrepancy between feature distributions of the inputs and the training set. Furthermore, theoretical analysis shows that a symmetric feature space can be achieved by using the GM loss, which enables the models to perform robustly against adversarial attacks. The proposed model can be implemented easily and efficiently without using extra trainable parameters. Extensive evaluations demonstrate that the proposed method performs favorably not only on image classification but also on robust detection of adversarial examples generated by strong attacks under different threat models.

cs.CV↗

arXiv2020

Multi-Scale Networks for 3D Human Pose Estimation with Inference Stage Optimization

Estimating 3D human poses from a monocular video is still a challenging task. Many existing methods' performance drops when the target person is occluded by other objects, or the motion is too fast/slow relative to the scale and speed of the training data. Moreover, many of these methods are not designed or trained under severe occlusion explicitly, making their performance on handling occlusion compromised. Addressing these problems, we introduce a spatio-temporal network for robust 3D human pose estimation. As humans in videos may appear in different scales and have various motion speeds, we apply multi-scale spatial features for 2D joints or keypoints prediction in each individual frame, and multi-stride temporal convolutional networks (TCNs) to estimate 3D joints or keypoints. Furthermore, we design a spatio-temporal discriminator based on body structures as well as limb motions to assess whether the predicted pose forms a valid pose and a valid movement. During training, we explicitly mask out some keypoints to simulate various occlusion cases, from minor to severe occlusion, so that our network can learn better and becomes robust to various degrees of occlusion. As there are limited 3D ground-truth data, we further utilize 2D video data to inject a semi-supervised learning capability to our network. Moreover, we observe that there is a discrepancy between 3D pose prediction and 2D pose estimation due to different pose variations between video and image training datasets. We, therefore propose a confidence-based inference stage optimization to adaptively enforce 3D pose projection to match 2D pose estimation to further improve final pose prediction accuracy. Experiments on public datasets validate the effectiveness of our method, and our ablation studies show the strengths of our network's individual submodules.

cs.CV↗

arXiv2020

Global existence of weak solutions to the Navier-Stokes equations with temperature-depending viscosity coefficient

In this paper, the initial-boundary value problem to the three-dimensional inhomogeneous, incompressible and heat-conducting Navier-Stokes equations with temperature-depending viscosity coefficient is considered in a bounded domain. The viscosity coefficient is degenerate and may vanish in the region of absolutely zero temperature. Global existence of weak solutions to such a system is established for the large initial data. The proof is based on a three-level approximate scheme, the De Giorgi's method and compactness arguments.

math.AP↗

arXiv2020

HLT-NUS Submission for NIST 2019 Multimedia Speaker Recognition Evaluation

This work describes the speaker verification system developed by Human Language Technology Laboratory, National University of Singapore (HLT-NUS) for 2019 NIST Multimedia Speaker Recognition Evaluation (SRE). The multimedia research has gained attention to a wide range of applications and speaker recognition is no exception to it. In contrast to the previous NIST SREs, the latest edition focuses on a multimedia track to recognize speakers with both audio and visual information. We developed separate systems for audio and visual inputs followed by a score level fusion of the systems from the two modalities to collectively use their information. The audio systems are based on x-vector based speaker embedding, whereas the face recognition systems are based on ResNet and InsightFace based face embeddings. With post evaluation studies and refinements, we obtain an equal error rate (EER) of 0.88% and an actual detection cost function (actDCF) of 0.026 on the evaluation set of 2019 NIST multimedia SRE corpus.

eess.AS↗

arXiv2020

Waveform-based Voice Activity Detection Exploiting Fully Convolutional networks with Multi-Branched Encoders

In this study, we propose an encoder-decoder structured system with fully convolutional networks to implement voice activity detection (VAD) directly on the time-domain waveform. The proposed system processes the input waveform to identify its segments to be either speech or non-speech. This novel waveform-based VAD algorithm, with a short-hand notation "WVAD", has two main particularities. First, as compared to most conventional VAD systems that use spectral features, raw-waveforms employed in WVAD contain more comprehensive information and thus are supposed to facilitate more accurate speech/non-speech predictions. Second, based on the multi-branched architecture, WVAD can be extended by using an ensemble of encoders, referred to as WEVAD, that incorporate multiple attribute information in utterances, and thus can yield better VAD performance for specified acoustic conditions. We evaluated the presented WVAD and WEVAD for the VAD task in two datasets: First, the experiments conducted on AURORA2 reveal that WVAD outperforms many state-of-the-art VAD algorithms. Next, the TMHINT task confirms that through combining multiple attributes in utterances, WEVAD behaves even better than WVAD.

eess.AS↗

arXiv2020

Time-Domain Multi-modal Bone/air Conducted Speech Enhancement

Previous studies have proven that integrating video signals, as a complementary modality, can facilitate improved performance for speech enhancement (SE). However, video clips usually contain large amounts of data and pose a high cost in terms of computational resources and thus may complicate the SE system. As an alternative source, a bone-conducted speech signal has a moderate data size while manifesting speech-phoneme structures, and thus complements its air-conducted counterpart. In this study, we propose a novel multi-modal SE structure in the time domain that leverages bone- and air-conducted signals. In addition, we examine two ensemble-learning-based strategies, early fusion (EF) and late fusion (LF), to integrate the two types of speech signals, and adopt a deep learning-based fully convolutional network to conduct the enhancement. The experiment results on the Mandarin corpus indicate that this newly presented multi-modal (integrating bone- and air-conducted signals) SE structure significantly outperforms the single-source SE counterparts (with a bone- or air-conducted signal only) in various speech evaluation metrics. In addition, the adoption of an LF strategy other than an EF in this novel SE multi-modal structure achieves better results.

eess.AS↗

arXiv2019

Increasing Compactness Of Deep Learning Based Speech Enhancement Models With Parameter Pruning And Quantization Techniques

Most recent studies on deep learning based speech enhancement (SE) focused on improving denoising performance. However, successful SE applications require striking a desirable balance between denoising performance and computational cost in real scenarios. In this study, we propose a novel parameter pruning (PP) technique, which removes redundant channels in a neural network. In addition, a parameter quantization (PQ) technique was applied to reduce the size of a neural network by representing weights with fewer cluster centroids. Because the techniques are derived based on different concepts, the PP and PQ can be integrated to provide even more compact SE models. The experimental results show that the PP and PQ techniques produce a compacted SE model with a size of only 10.03% compared to that of the original model, resulting in minor performance losses of 1.43% (from 0.70 to 0.69) for STOI and 3.24% (from 1.85 to 1.79) for PESQ. The promising results suggest that the PP and PQ techniques can be used in a SE system in devices with limited storage and computation resources.

eess.AS↗

arXiv2019

Global Existence of Entropy-Weak Solutions to the Compressible Navier-Stokes Equations with Non-Linear Density Dependent Viscosities

In this paper, we extend considerably the global existence results of entropy-weak solutions related to compressible Navier-Stokes system with density dependent viscosities obtained, independently (using different strategies), by Vasseur-Yu [Inventiones mathematicae (2016) and arXiv:1501.06803 (2015)] and by Li-Xin [arXiv:1504.06826 (2015)].More precisely we are able to consider a physical symmetric viscous stress tensor $σ=2μ(ρ)\,{\mathbb{D}}(u)+\bigl(λ(ρ){\rm div}u -P(ρ)\bigr)\, {\rm Id}$ where ${\mathbb D}(u) = [\nabla u + \nabla^T u]/2$ with a shear and bulk viscosities (respectively $μ(ρ)$ and $λ(ρ)$) satisfying the BD relation $λ(ρ)=2(μ'(ρ)ρ- μ(ρ))$ and a pressure law $P(ρ)=aρ^γ$ (with $a>0$ a given constant) for any adiabatic constant $γ>1$. The nonlinear shear viscosity $μ(ρ)$ satisfies some lower and upper bounds for low and high densities (our mathematical result includes the case $μ(ρ)= μρ^α$ with $2/3 < α< 4$ and $μ>0$ constant). This provides an answer to a longstanding mathematical question on compressible Navier-Stokes equations with density dependent viscosities as mentioned for instance by F. Rousset in the Bourbaki 69ème année, 2016--2017, no 1135.

math.AP↗

arXiv2019

Modeling of fracture geometry alteration and fracture flow evolution under geostress and water-rock interaction

A coupled mech-hydro-chemical model for rock geometry alteration of fractures under water-rock interaction (WRI) and geostress is developed. Processes including WRI, asperity deformation, mineral chemical dissolution and pressure dissolution etc., are taken into account. A feature of this model lies in its approximate linearization to the non-linear pressure dissolution process, which makes this model compatible with existing numerical solute transport models. Case study shows that although usually only a thin layer of rock surface is invaded by WRI, the mechanical weakening of this thin layer tend to induce significant increase in rock surface deformation. Thus, the distributed flow field, mineral dissolution rates, and surface alteration increments etc., are all affected. This indicates that when fracture flow related issues are concerned, we should focus on the top thin layer of rock surface rather than the matrix. Results also show that fluid flow enhances the dissolution of rocks and development of flowing channels; however, at places where stress induces high pressure-dissolution but fluid flows slowly, minerals precipitate and fill up fracture voids. This suggests the hydraulic condition plays a key role in the development of fracture flow channels and the evolution of fracture geometry.

physics.geo-ph↗