arXiv subjects

Fan Yu

Publications and source records attributed to Fan Yu.

At least 73 records · Page 4Linked to original sources

arXiv2021

WeNet: Production oriented Streaming and Non-streaming End-to-End Speech Recognition Toolkit

In this paper, we propose an open source, production first, and production ready speech recognition toolkit called WeNet in which a new two-pass approach is implemented to unify streaming and non-streaming end-to-end (E2E) speech recognition in a single model. The main motivation of WeNet is to close the gap between the research and the production of E2E speechrecognition models. WeNet provides an efficient way to ship ASR applications in several real-world scenarios, which is the main difference and advantage to other open source E2E speech recognition toolkits. In our toolkit, a new two-pass method is implemented. Our method propose a dynamic chunk-based attention strategy of the the transformer layers to allow arbitrary right context length modifies in hybrid CTC/attention architecture. The inference latency could be easily controlled by only changing the chunk size. The CTC hypotheses are then rescored by the attention decoder to get the final result. Our experiments on the AISHELL-1 dataset using WeNet show that, our model achieves 5.03\% relative character error rate (CER) reduction in non-streaming ASR compared to a standard non-streaming transformer. After model quantification, our model perform reasonable RTF and latency.

cs.SD↗

arXiv2021

Solving Partial Differential Equations with Point Source Based on Physics-Informed Neural Networks

In recent years, deep learning technology has been used to solve partial differential equations (PDEs), among which the physics-informed neural networks (PINNs) emerges to be a promising method for solving both forward and inverse PDE problems. PDEs with a point source that is expressed as a Dirac delta function in the governing equations are mathematical models of many physical processes. However, they cannot be solved directly by conventional PINNs method due to the singularity brought by the Dirac delta function. We propose a universal solution to tackle this problem with three novel techniques. Firstly the Dirac delta function is modeled as a continuous probability density function to eliminate the singularity; secondly a lower bound constrained uncertainty weighting algorithm is proposed to balance the PINNs losses between point source area and other areas; and thirdly a multi-scale deep neural network with periodic activation function is used to improve the accuracy and convergence speed of the PINNs method. We evaluate the proposed method with three representative PDEs, and the experimental results show that our method outperforms existing deep learning-based methods with respect to the accuracy, the efficiency and the versatility.

cs.LG↗

arXiv2021

Towards More Effective and Economic Sparsely-Activated Model

The sparsely-activated models have achieved great success in natural language processing through large-scale parameters and relatively low computational cost, and gradually become a feasible technique for training and implementing extremely large models. Due to the limit of communication cost, activating multiple experts is hardly affordable during training and inference. Therefore, previous work usually activate just one expert at a time to alleviate additional communication cost. Such routing mechanism limits the upper bound of model performance. In this paper, we first investigate a phenomenon that increasing the number of activated experts can boost the model performance with higher sparse ratio. To increase the number of activated experts without an increase in computational cost, we propose SAM (Switch and Mixture) routing, an efficient hierarchical routing mechanism that activates multiple experts in a same device (GPU). Our methods shed light on the training of extremely large sparse models and experiments prove that our models can achieve significant performance gain with great efficiency improvement.

cs.CL↗

arXiv2021

Boundary and Context Aware Training for CIF-based Non-Autoregressive End-to-end ASR

Continuous integrate-and-fire (CIF) based models, which use a soft and monotonic alignment mechanism, have been well applied in non-autoregressive (NAR) speech recognition with competitive performance compared with other NAR methods. However, such an alignment learning strategy may suffer from an erroneous acoustic boundary estimation, severely hindering the convergence speed as well as the system performance. In this paper, we propose a boundary and context aware training approach for CIF based NAR models. Firstly, the connectionist temporal classification (CTC) spike information is utilized to guide the learning of acoustic boundaries in the CIF. Besides, an additional contextual decoder is introduced behind the CIF decoder, aiming to capture the linguistic dependencies within a sentence. Finally, we adopt a recently proposed Conformer architecture to improve the capacity of acoustic modeling. Experiments on the open-source Mandarin AISHELL-1 corpus show that the proposed method achieves a comparable character error rates (CERs) of 4.9% with only 1/24 latency compared with a state-of-the-art autoregressive (AR) Conformer model. Futhermore, when evaluating on an internal 7500 hours Mandarin corpus, our model still outperforms other NAR methods and even reaches the AR Conformer model on a challenging real-world noisy test set.

cs.SD↗

arXiv2021

Weighted and shifted BDF2 methods on variable grids

Variable steps implicit-explicit multistep methods for PDEs have been presented in [17], where the zero-stability is studied for ODEs; however, the stability analysis still remains an open question for PDEs. Based on the idea of linear multistep methods, we present a simple weighted and shifted BDF2 methods with variable steps for the parabolic problems, which serve as a bridge between BDF2 and Crank-Nicolson scheme. The contributions of this paper are as follows: we first prove that the optimal adjacent time-step ratios for the weighted and shifted BDF2, which greatly improve the maximum time-step ratios for BDF2 in [11,15]. Moreover, the unconditional stability and optimal convergence are rigorous proved, which make up for the vacancy of the theory for PDEs in [17]. Finally, numerical experiments are given to illustrate theoretical results.

math.NA↗

arXiv2021

BDF$6$ SAV schemes for time-fractional Allen-Cahn dissipative systems

Recently, the error analysis of BDF$k$ $(1\leqslant k\leqslant5)$ SAV (scalar auxiliary variable) schemes are given in \cite{Huangg:20} for the classical Allen-Cahn equation. However, it remains unavailable for BDF$6$ SAV schemes. In this paper, we construct and analyze BDF$6$ SAV schemes for the time-fractional dissipative systems. We carry out a rigorous error analysis for the time-fractional Allen-Cahn equation, which also fills up a gap for the classical case. Finally, numerical experiment is shown to illustrate the effectiveness of the presented methods. As far as we know, this is the first SAV schemes for the time-fractional dissipative systems.

math.NA↗

arXiv2021

The Accented English Speech Recognition Challenge 2020: Open Datasets, Tracks, Baselines, Results and Methods

The variety of accents has posed a big challenge to speech recognition. The Accented English Speech Recognition Challenge (AESRC2020) is designed for providing a common testbed and promoting accent-related research. Two tracks are set in the challenge -- English accent recognition (track 1) and accented English speech recognition (track 2). A set of 160 hours of accented English speech collected from 8 countries is released with labels as the training set. Another 20 hours of speech without labels is later released as the test set, including two unseen accents from another two countries used to test the model generalization ability in track 2. We also provide baseline systems for the participants. This paper first reviews the released dataset, track setups, baselines and then summarizes the challenge results and major techniques used in the submissions.

cs.SD↗

arXiv2021

AsymptoticNG: A regularized natural gradient optimization algorithm with look-ahead strategy

Optimizers that further adjust the scale of gradient, such as Adam, Natural Gradient (NG), etc., despite widely concerned and used by the community, are often found poor generalization performance, compared with Stochastic Gradient Descent (SGD). They tend to converge excellently at the beginning of training but are weak at the end. An immediate idea is to complement the strengths of these algorithms with SGD. However, a truncated replacement of optimizer often leads to a crash of the update pattern, and new algorithms often require many iterations to stabilize their search direction. Driven by this idea and to address this problem, we design and present a regularized natural gradient optimization algorithm with look-ahead strategy, named asymptotic natural gradient (ANG). According to the total iteration step, ANG dynamic assembles NG and Euclidean gradient, and updates parameters along the new direction using the intensity of NG. Validation experiments on CIFAR10 and CIFAR100 data sets show that ANG can update smoothly and stably at the second-order speed, and achieve better generalization performance.

cs.LG↗

arXiv2020

Eigenvalue-corrected Natural Gradient Based on a New Approximation

Using second-order optimization methods for training deep neural networks (DNNs) has attracted many researchers. A recently proposed method, Eigenvalue-corrected Kronecker Factorization (EKFAC) (George et al., 2018), proposes an interpretation of viewing natural gradient update as a diagonal method, and corrects the inaccurate re-scaling factor in the Kronecker-factored eigenbasis. Gao et al. (2020) considers a new approximation to the natural gradient, which approximates the Fisher information matrix (FIM) to a constant multiplied by the Kronecker product of two matrices and keeps the trace equal before and after the approximation. In this work, we combine the ideas of these two methods and propose Trace-restricted Eigenvalue-corrected Kronecker Factorization (TEKFAC). The proposed method not only corrects the inexact re-scaling factor under the Kronecker-factored eigenbasis, but also considers the new approximation method and the effective damping technique proposed in Gao et al. (2020). We also discuss the differences and relationships among the Kronecker-factored approximations. Empirically, our method outperforms SGD with momentum, Adam, EKFAC and TKFAC on several DNNs.

cs.LG↗

arXiv2020

A Trace-restricted Kronecker-Factored Approximation to Natural Gradient

Second-order optimization methods have the ability to accelerate convergence by modifying the gradient through the curvature matrix. There have been many attempts to use second-order optimization methods for training deep neural networks. Inspired by diagonal approximations and factored approximations such as Kronecker-Factored Approximate Curvature (KFAC), we propose a new approximation to the Fisher information matrix (FIM) called Trace-restricted Kronecker-factored Approximate Curvature (TKFAC) in this work, which can hold the certain trace relationship between the exact and the approximate FIM. In TKFAC, we decompose each block of the approximate FIM as a Kronecker product of two smaller matrices and scaled by a coefficient related to trace. We theoretically analyze TKFAC's approximation error and give an upper bound of it. We also propose a new damping technique for TKFAC on convolutional neural networks to maintain the superiority of second-order optimization methods during training. Experiments show that our method has better performance compared with several state-of-the-art algorithms on some deep network architectures.

cs.LG↗

arXiv2020

The SLT 2021 children speech recognition challenge: Open datasets, rules and baselines

Automatic speech recognition (ASR) has been significantly advanced with the use of deep learning and big data. However improving robustness, including achieving equally good performance on diverse speakers and accents, is still a challenging problem. In particular, the performance of children speech recognition (CSR) still lags behind due to 1) the speech and language characteristics of children's voice are substantially different from those of adults and 2) sizable open dataset for children speech is still not available in the research community. To address these problems, we launch the Children Speech Recognition Challenge (CSRC), as a flagship satellite event of IEEE SLT 2021 workshop. The challenge will release about 400 hours of Mandarin speech data for registered teams and set up two challenge tracks and provide a common testbed to benchmark the CSR performance. In this paper, we introduce the datasets, rules, evaluation method as well as baselines.

cs.SD↗

arXiv2020

Backward difference formula: The energy technique for subdiffusion equation

Based on the equivalence of A-stability and G-stability, the energy technique of the six-step BDF method for the heat equation has been discussed in [Akrivis, Chen, Yu, Zhou, Math. Comp., Revised]. Unfortunately, this theory is hard to extend the time-fractional PDEs. In this work, we consider three types of subdiffusion models, namely single-term, multi-term and distributed order fractional diffusion equations. We present a novel and concise stability analysis of time stepping schemes generated by $k$-step backward difference formula (BDF$k$), for approximately solving the subdiffusion equation. The analysis mainly relies on the energy technique by applying Grenander-Szegö theorem. This kind of argument has been widely used to confirm the stability of various $A$-stable schemes (e.g., $k=1,2$). However, it is not an easy task for the higher-order BDF methods, due to the loss the $A$-stability. The core object of this paper is to fill in this gap.

math.NA↗

arXiv2020

Finite difference/spectral approximations for the two-dimensional time Caputo-Fabrizio fractional diffusion equation

The main contribution of this work is to construct and analyze stable and high order schemes to efficiently solve the two-dimensional time Caputo-Fabrizio fractional diffusion equation. Based on a third-order finite difference method in time and spectral methods in space, the proposed scheme is unconditionally stable and has the global truncation error $\mathcal{O}(τ^3+N^{-m})$, where $τ$, $N$ and $m$ are the time step size, polynomial degree and regularity in the space variable of the exact solution, respectively. It should be noted that the global truncation error $\mathcal{O}(τ^2+N^{-m})$ is well established in [ Li, Lv and Xu, {\em Numer. Methods Partial Differ. Equ}. (2019)]. Finally, some numerical experiments are carried out to verify the theoretical analysis. To the best of our knowledge, this is the first proof for the stability of the third-order scheme for the Caputo-Fabrizio fractional operator.

math.NA↗

arXiv2020

The energy technique for the six-step BDF method

In combination with the Grenander--Szegö theorem, we observe that a relaxed positivity condition on multipliers, milder than the basic %fundamental requirement of the Nevanlinna--Odeh multipliers that the sum of the absolute values of their components is strictly less than $1$, makes the energy technique applicable to the stability analysis of BDF methods for parabolic equations with selfadjoint elliptic part. This is particularly useful for the six-step BDF method for which no Nevanlinna--Odeh multiplier exists. We introduce multipliers satisfying the positivity property for the six-step BDF method and establish stability of the method for parabolic equations.

math.NA↗

arXiv2020

TensorOpt: Exploring the Tradeoffs in Distributed DNN Training with Auto-Parallelism

A good parallelization strategy can significantly improve the efficiency or reduce the cost for the distributed training of deep neural networks (DNNs). Recently, several methods have been proposed to find efficient parallelization strategies but they all optimize a single objective (e.g., execution time, memory consumption) and produce only one strategy. We propose FT, an efficient algorithm that searches for an optimal set of parallelization strategies to allow the trade-off among different objectives. FT can adapt to different scenarios by minimizing the memory consumption when the number of devices is limited and fully utilize additional resources to reduce the execution time. For popular DNN models (e.g., vision, language), an in-depth analysis is conducted to understand the trade-offs among different objectives and their influence on the parallelization strategies. We also develop a user-friendly system, called TensorOpt, which allows users to run their distributed DNN training jobs without caring the details of parallelization strategies. Experimental results show that FT runs efficiently and provides accurate estimation of runtime costs, and TensorOpt is more flexible in adapting to resource availability compared with existing frameworks.

cs.DC↗

arXiv2019

Elastic deep learning in multi-tenant GPU cluster

We study how to support elasticity, i.e., the ability to dynamically adjust the parallelism (number of GPUs), for deep neural network (DNN) training. Elasticity can benefit multi-tenant GPU cluster management in many ways, e.g., achieving various scheduling objectives (e.g., job throughput, job completion time, GPU efficiency) according to cluster load variations, maximizing the use of transient idle resources, performance profiling, job migration, and straggler mitigation. However, existing parallelism adjustment strategies incur high overheads, which hinder many applications from making effective use of elasticity. We propose EDL to enable low-overhead elastic deep learning with a simple API. We present techniques that are necessary to reduce the overhead of parallelism adjustments, such as stop-free scaling and dynamic data pipeline. We also demonstrate that EDL can indeed bring significant benefits to the above-listed applications in GPU cluster management.

cs.DC↗

arXiv2019

Growth and Thermo-driven Crystalline Phase Transition of Metastable Monolayer 1T'-WSe2 Thin Film

Two-dimensional (2D) transition metal dichalcogenides MX2 (M = Mo, W, X = S, Se, Te) attracts enormous research interests in recent years. Its 2H phase possesses an indirect to direct bandgap transition in 2D limit, and thus shows great application potentials in optoelectronic devices [1]. The 1T' crystalline phase transition can drive the monolayer MX2 to be a 2D topological insulator. Here we realized the molecular beam epitaxial (MBE) growth of both the 1T' and 2H phase monolayer WSe2 on bilayer graphene (BLG) substrate. The crystalline structures of these two phases were characterized using scanning tunneling microscopy. The monolayer 1T'-WSe2 was found to be metastable, and can transform into 2H phase under post-annealing procedure. The phase transition temperature of 1T'-WSe2 grown on BLG is lower than that of 1T' phase grown on 2H-WSe2 layers. This thermo-driven crystalline phase transition makes the monolayer WSe2 to be an ideal platform for the controlling of topological phase transitions in 2D materials family.

cond-mat.mtrl-sci↗

arXiv2017

Quantifying the relative effects of environmental and direct transmission of norovirus

Background: Norovirus is a common cause of outbreaks of acute gastroenteritis in health- and child-care settings, with serial outbreaks also frequently observed aboard cruise ships. The relative contributions of environmental and direct person-to-person transmission of norovirus has hitherto not been quantified. Objective: We employ a novel mathematical model of norovirus transmission, and fit the model to daily incidence data from a major norovirus outbreak on a cruise ship, and examine the relative efficacy of potential control strategies aimed at reducing environmental and/or direct transmission. Results: The reproduction number for environmental and direct transmission combined is Rtot = 11.0 [9.4,15.6], and of environmental transmission alone is Renviron = 0.85 [0.18,2.04]. Direct transmission is overwhelmingly due to 0 passenger-to-passenger contacts, but crew can act as a reservoir of infection from cruise-to-cruise. Implications: This is the first quantification of the relative roles of environmental and direct transmission of norovirus. While environmental transmission has the potential to maintain a sustained series of outbreaks aboard a cruise ship in the absence of strict sanitation practices, direct transmission dominates. Quarantine of ill passengers and cleaning are likely to have little impact on final outbreak size, but intensive promotion of good hand washing practices can prevent outbreaks.

q-bio.QM↗