Searcharxiv⌕ Search

arXiv subjects

William Chan

Publications and source records attributed to William Chan.

At least 37 records · Page 2Linked to original sources
arXiv2021

SpeechStew: Simply Mix All Available Speech Recognition Data to Train One Large Neural Network

We present SpeechStew, a speech recognition model that is trained on a combination of various publicly available speech recognition datasets: AMI, Broadcast News, Common Voice, LibriSpeech, Switchboard/Fisher, Tedlium, and Wall Street Journal. SpeechStew simply mixes all of these datasets together, without any special re-weighting or re-balancing of the datasets. SpeechStew achieves SoTA or near SoTA results across a variety of tasks, without the use of an external language model. Our results include 9.0\% WER on AMI-IHM, 4.7\% WER on Switchboard, 8.3\% WER on CallHome, and 1.3\% on WSJ, which significantly outperforms prior work with strong external language models. We also demonstrate that SpeechStew learns powerful transfer learning representations. We fine-tune SpeechStew on a noisy low resource speech dataset, CHiME-6. We achieve 38.9\% WER without a language model, which compares to 38.6\% WER to a strong HMM baseline with a language model.

cs.CL↗
arXiv2020

Non-Autoregressive Machine Translation with Latent Alignments

This paper presents two strong methods, CTC and Imputer, for non-autoregressive machine translation that model latent alignments with dynamic programming. We revisit CTC for machine translation and demonstrate that a simple CTC model can achieve state-of-the-art for single-step non-autoregressive machine translation, contrary to what prior work indicates. In addition, we adapt the Imputer model for non-autoregressive machine translation and demonstrate that Imputer with just 4 generation steps can match the performance of an autoregressive Transformer baseline. Our latent alignment models are simpler than many existing non-autoregressive translation baselines; for example, we do not require target length prediction or re-scoring with an autoregressive model. On the competitive WMT'14 En$\rightarrow$De task, our CTC model achieves 25.7 BLEU with a single generation step, while Imputer achieves 27.5 BLEU with 2 generation steps, and 28.0 BLEU with 4 generation steps. This compares favourably to the autoregressive Transformer baseline at 27.8 BLEU.

cs.CL↗
arXiv2020

WaveGrad: Estimating Gradients for Waveform Generation

This paper introduces WaveGrad, a conditional model for waveform generation which estimates gradients of the data density. The model is built on prior work on score matching and diffusion probabilistic models. It starts from a Gaussian white noise signal and iteratively refines the signal via a gradient-based sampler conditioned on the mel-spectrogram. WaveGrad offers a natural way to trade inference speed for sample quality by adjusting the number of refinement steps, and bridges the gap between non-autoregressive and autoregressive models in terms of audio quality. We find that it can generate high fidelity audio samples using as few as six iterations. Experiments reveal WaveGrad to generate high fidelity audio, outperforming adversarial non-autoregressive baselines and matching a strong likelihood-based autoregressive baseline using fewer sequential operations. Audio samples are available at https://wavegrad.github.io/.

eess.AS↗
arXiv2020

Multichannel Generative Language Model: Learning All Possible Factorizations Within and Across Channels

A channel corresponds to a viewpoint or transformation of an underlying meaning. A pair of parallel sentences in English and French express the same underlying meaning, but through two separate channels corresponding to their languages. In this work, we present the Multichannel Generative Language Model (MGLM). MGLM is a generative joint distribution model over channels. MGLM marginalizes over all possible factorizations within and across all channels. MGLM endows flexible inference, including unconditional generation, conditional generation (where 1 channel is observed and other channels are generated), and partially observed generation (where incomplete observations are spread across all the channels). We experiment with the Multi30K dataset containing English, French, Czech, and German. We demonstrate experiments with unconditional, conditional, and partially conditional generation. We provide qualitative samples sampled unconditionally from the generative joint distribution. We also quantitatively analyze the quality-diversity trade-offs and find MGLM outperforms traditional bilingual discriminative models.

cs.CL↗
arXiv2020

Imputer: Sequence Modelling via Imputation and Dynamic Programming

This paper presents the Imputer, a neural sequence model that generates output sequences iteratively via imputations. The Imputer is an iterative generative model, requiring only a constant number of generation steps independent of the number of input or output tokens. The Imputer can be trained to approximately marginalize over all possible alignments between the input and output sequences, and all possible generation orders. We present a tractable dynamic programming training algorithm, which yields a lower bound on the log marginal likelihood. When applied to end-to-end speech recognition, the Imputer outperforms prior non-autoregressive models and achieves competitive results to autoregressive models. On LibriSpeech test-other, the Imputer achieves 11.1 WER, outperforming CTC at 13.0 WER and seq2seq at 12.5 WER.

eess.AS↗
arXiv2020

Insertion-Deletion Transformer

We propose the Insertion-Deletion Transformer, a novel transformer-based neural architecture and training method for sequence generation. The model consists of two phases that are executed iteratively, 1) an insertion phase and 2) a deletion phase. The insertion phase parameterizes a distribution of insertions on the current output hypothesis, while the deletion phase parameterizes a distribution of deletions over the current output hypothesis. The training method is a principled and simple algorithm, where the deletion model obtains its signal directly on-policy from the insertion model output. We demonstrate the effectiveness of our Insertion-Deletion Transformer on synthetic translation tasks, obtaining significant BLEU score improvement over an insertion-only model.

cs.LG↗
arXiv2019

SpecAugment on Large Scale Datasets

Recently, SpecAugment, an augmentation scheme for automatic speech recognition that acts directly on the spectrogram of input utterances, has shown to be highly effective in enhancing the performance of end-to-end networks on public datasets. In this paper, we demonstrate its effectiveness on tasks with large scale datasets by investigating its application to the Google Multidomain Dataset (Narayanan et al., 2018). We achieve improvement across all test domains by mixing raw training data augmented with SpecAugment and noise-perturbed training data when training the acoustic model. We also introduce a modification of SpecAugment that adapts the time mask size and/or multiplicity depending on the length of the utterance, which can potentially benefit large scale tasks. By using adaptive masking, we are able to further improve the performance of the Listen, Attend and Spell model on LibriSpeech to 2.2% WER on test-clean and 5.2% WER on test-other.

eess.AS↗
arXiv2019

SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition

We present SpecAugment, a simple data augmentation method for speech recognition. SpecAugment is applied directly to the feature inputs of a neural network (i.e., filter bank coefficients). The augmentation policy consists of warping the features, masking blocks of frequency channels, and masking blocks of time steps. We apply SpecAugment on Listen, Attend and Spell networks for end-to-end speech recognition tasks. We achieve state-of-the-art performance on the LibriSpeech 960h and Swichboard 300h tasks, outperforming all prior work. On LibriSpeech, we achieve 6.8% WER on test-other without the use of a language model, and 5.8% WER with shallow fusion with a language model. This compares to the previous state-of-the-art hybrid system of 7.5% WER. For Switchboard, we achieve 7.2%/14.6% on the Switchboard/CallHome portion of the Hub5'00 test set without the use of a language model, and 6.8%/14.1% with shallow fusion, which compares to the previous state-of-the-art hybrid system at 8.3%/17.3% WER.

eess.AS↗
arXiv2019

Big Bidirectional Insertion Representations for Documents

The Insertion Transformer is well suited for long form text generation due to its parallel generation capabilities, requiring $O(\log_2 n)$ generation steps to generate $n$ tokens. However, modeling long sequences is difficult, as there is more ambiguity captured in the attention mechanism. This work proposes the Big Bidirectional Insertion Representations for Documents (Big BIRD), an insertion-based model for document-level translation tasks. We scale up the insertion-based models to long form documents. Our key contribution is introducing sentence alignment via sentence-positional embeddings between the source and target document. We show an improvement of +4.3 BLEU on the WMT'19 English$\rightarrow$German document-level translation task compared with the Insertion Transformer baseline.

cs.CL↗
arXiv2019

An Empirical Study of Generation Order for Machine Translation

In this work, we present an empirical study of generation order for machine translation. Building on recent advances in insertion-based modeling, we first introduce a soft order-reward framework that enables us to train models to follow arbitrary oracle generation policies. We then make use of this framework to explore a large variety of generation orders, including uninformed orders, location-based orders, frequency-based orders, content-based orders, and model-based orders. Curiously, we find that for the WMT'14 English $\to$ German translation task, order does not have a substantial impact on output quality, with unintuitive orderings such as alphabetical and shortest-first matching the performance of a standard Transformer. This demonstrates that traditional left-to-right generation is not strictly necessary to achieve high performance. On the other hand, results on the WMT'18 English $\to$ Chinese task tend to vary more widely, suggesting that translation for less well-aligned language pairs may be more sensitive to generation order.

cs.CL↗
arXiv2019

Bounds on Continuous Scott Rank

An analog of Nadel's effective bound for the continuous Scott rank of metric structures, developed by Ben Yaacov, Doucha, Nies, and Tsankov, will be established: Let $\mathscr{L}$ be a language of continuous logic with code $\hat{\mathscr{L}}$. Let $Ω$ be a weak modulus of uniform continuity with code $\hatΩ$. Let $\mathcal{D}$ be a countable $\mathscr{L}$-pre-structure. Let $\bar{\mathcal{D}}$ denote the completion structure of $\mathcal{D}$. Then $\mathrm{SR}_Ω(\bar{D}) \leq ω_1^{\hat{\mathscr{L}}\oplus\hatΩ\oplus\mathcal{D}}$, the Church-Kleene ordinal relative to $\hat{\mathscr{L}}\oplus\hatΩ\oplus\mathcal{D}$.

math.LO↗
arXiv2019

Speaker-Targeted Audio-Visual Models for Speech Recognition in Cocktail-Party Environments

Speech recognition in cocktail-party environments remains a significant challenge for state-of-the-art speech recognition systems, as it is extremely difficult to extract an acoustic signal of an individual speaker from a background of overlapping speech with similar frequency and temporal characteristics. We propose the use of speaker-targeted acoustic and audio-visual models for this task. We complement the acoustic features in a hybrid DNN-HMM model with information of the target speaker's identity as well as visual features from the mouth region of the target speaker. Experimentation was performed using simulated cocktail-party data generated from the GRID audio-visual corpus by overlapping two speakers's speech on a single acoustic channel. Our audio-only baseline achieved a WER of 26.3%. The audio-visual model improved the WER to 4.4%. Introducing speaker identity information had an even more pronounced effect, improving the WER to 3.6%. Combining both approaches, however, did not significantly improve performance further. Our work demonstrates that speaker-targeted models can significantly improve the speech recognition in cocktail party environments.

eess.AS↗
arXiv2019

Bounds on Scott Ranks of Some Polish Metric Spaces

If $\mathcal{N}$ is a proper Polish metric space and $\mathcal{M}$ is any countable dense submetric space of $\mathcal{N}$, then the Scott rank of $\mathcal{N}$ in the natural first order language of metric spaces is countable and in fact at most $ω_1^{\mathcal{M}} + 1$, where $ω_1^{\mathcal{M}}$ is the Church-Kleene ordinal of $\mathcal{M}$ (construed as a subset of $ω$) which is the least ordinal with no presentation on $ω$ computable from $\mathcal{M}$. If $\mathcal{N}$ is a rigid Polish metric space and $\mathcal{M}$ is any countable dense submetric space, then the Scott rank of $\mathcal{N}$ is countable and in fact less than $ω_1^{\mathcal{M}}$.

math.LO↗
arXiv2019

KERMIT: Generative Insertion-Based Modeling for Sequences

We present KERMIT, a simple insertion-based approach to generative modeling for sequences and sequence pairs. KERMIT models the joint distribution and its decompositions (i.e., marginals and conditionals) using a single neural network and, unlike much prior work, does not rely on a prespecified factorization of the data distribution. During training, one can feed KERMIT paired data $(x, y)$ to learn the joint distribution $p(x, y)$, and optionally mix in unpaired data $x$ or $y$ to refine the marginals $p(x)$ or $p(y)$. During inference, we have access to the conditionals $p(x \mid y)$ and $p(y \mid x)$ in both directions. We can also sample from the joint distribution or the marginals. The model supports both serial fully autoregressive decoding and parallel partially autoregressive decoding, with the latter exhibiting an empirically logarithmic runtime. We demonstrate through experiments in machine translation, representation learning, and zero-shot cloze question answering that our unified approach is capable of matching or exceeding the performance of dedicated state-of-the-art systems across a wide range of tasks without the need for problem-specific architectural adaptation.

cs.CL↗
arXiv2019

The Destruction of the Axiom of Determinacy by Forcings on $\mathbb{R}$ when $Θ$ is Regular

$\mathsf{ZF + AD}$ proves that for all nontrivial forcings $\mathbb{P}$ on a wellorderable set of cardinality less than $Θ$, $1_{\mathbb{P}} \Vdash_{\mathbb{P}} \neg\mathsf{AD}$. $\mathsf{ZF + AD} + Θ$ is regular proves that for all nontrivial forcing $\mathbb{P}$ which is a surjective image of $\mathbb{R}$, $1_{\mathbb{P}} \Vdash_{\mathbb{P}} \neg\mathsf{AD}$. In particular, $\mathsf{ZF + AD + V = L(\mathbb{R})}$ proves that for every nontrivial forcing $\mathbb{P} \in L_Θ(\mathbb{R})$, $1_{\mathbb{P}} \Vdash_{\mathbb{P}} \neg\mathsf{AD}$.

math.LO↗
arXiv2019

Cardinality of Wellordered Disjoint Unions of Quotients of Smooth Equivalence Relations

Assume $\mathsf{ZF + AD^+ + V = L(\mathscr{P}(\mathbb{R}))}$. Let $\approx$ denote the relation of being in bijection. Let $κ\in \mathrm{ON}$ and $\langle E_α: α< κ\rangle$ be a sequence of equivalence relations on $\mathbb{R}$ with all classes countable and for all $α< κ$, $\mathbb{R} / E_α\approx \mathbb{R}$. Then the disjoint union $\bigsqcup_{α< κ} \mathbb{R} / E_α$ is in bijection with $\mathbb{R} \times κ$ and $\bigsqcup_{α< κ} \mathbb{R} / E_α$ has the Jónsson property. Assume $\mathsf{ZF + AD^+ + V = L(\mathscr{P}(\mathbb{R}))}$. A set $X \subseteq [ω_1]^{<ω_1}$ has a sequence $\langle E_α: α< ω_1\rangle$ of equivalence relations on $\mathbb{R}$ such that $\mathbb{R} / E_α\approx \mathbb{R}$ and $X \approx \bigsqcup_{α< ω_1} \mathbb{R} / E_α$ if and only if $\mathbb{R} \sqcup ω_1$ injects into $X$. Assume $\mathsf{AD}$. Suppose $R \subseteq [ω_1]^ω\times \mathbb{R}$ is a relation such that for all $f \in [ω_1]^ω$, $R_f = \{x \in \mathbb{R} : R(f,x)\}$ is nonempty and countable. Then there is an uncountable $X \subseteq ω_1$ and function $Φ: [X]^ω\rightarrow \mathbb{R}$ which uniformizes $R$ on $[X]^ω$: that is, for all $f \in [X]^ω$, $R(f,Φ(f))$. Under $\mathsf{AD}$, if $κ$ is an ordinal and $\langle E_α: α< κ\rangle$ is a sequence of equivalence relations on $\mathbb{R}$ with all classes countable, then $[ω_1]^ω$ does not inject into $\bigsqcup_{α< κ} \mathbb{R} / E_α$.

math.LO↗
arXiv2019

Lingvo: a Modular and Scalable Framework for Sequence-to-Sequence Modeling

Lingvo is a Tensorflow framework offering a complete solution for collaborative deep learning research, with a particular focus towards sequence-to-sequence models. Lingvo models are composed of modular building blocks that are flexible and easily extensible, and experiment configurations are centralized and highly customizable. Distributed training and quantized inference are supported directly within the framework, and it contains existing implementations of a large number of utilities, helper functions, and the newest research ideas. Lingvo has been used in collaboration by dozens of researchers in more than 20 papers over the last two years. This document outlines the underlying design of Lingvo and serves as an introduction to the various pieces of the framework, while also offering examples of advanced features that showcase the capabilities of the framework.

cs.LG↗