arXiv subjects

William Chan

Publications and source records attributed to William Chan.

54 records · Page 3Linked to original sources

arXiv2019

Insertion Transformer: Flexible Sequence Generation via Insertion Operations

We present the Insertion Transformer, an iterative, partially autoregressive model for sequence generation based on insertion operations. Unlike typical autoregressive models which rely on a fixed, often left-to-right ordering of the output, our approach accommodates arbitrary orderings by allowing for tokens to be inserted anywhere in the sequence during decoding. This flexibility confers a number of advantages: for instance, not only can our model be trained to follow specific orderings such as left-to-right generation or a binary tree traversal, but it can also be trained to maximize entropy over all valid insertions for robustness. In addition, our model seamlessly accommodates both fully autoregressive generation (one insertion at a time) and partially autoregressive generation (simultaneous insertions at multiple locations). We validate our approach by analyzing its performance on the WMT 2014 English-German machine translation task under various settings for training and decoding. We find that the Insertion Transformer outperforms many prior non-autoregressive approaches to translation at comparable or better levels of parallelism, and successfully recovers the performance of the original Transformer while requiring only logarithmically many iterations during decoding.

cs.CL↗

arXiv2019

Optimal Completion Distillation for Sequence Learning

We present Optimal Completion Distillation (OCD), a training procedure for optimizing sequence to sequence models based on edit distance. OCD is efficient, has no hyper-parameters of its own, and does not require pretraining or joint optimization with conditional log-likelihood. Given a partial sequence generated by the model, we first identify the set of optimal suffixes that minimize the total edit distance, using an efficient dynamic programming algorithm. Then, for each position of the generated sequence, we use a target distribution that puts equal probability on the first token of all the optimal suffixes. OCD achieves the state-of-the-art performance on end-to-end speech recognition, on both Wall Street Journal and Librispeech datasets, achieving $9.3\%$ WER and $4.5\%$ WER respectively.

cs.LG↗

arXiv2018

Privacy Partitioning: Protecting User Data During the Deep Learning Inference Phase

We present a practical method for protecting data during the inference phase of deep learning based on bipartite topology threat modeling and an interactive adversarial deep network construction. We term this approach \emph{Privacy Partitioning}. In the proposed framework, we split the machine learning models and deploy a few layers into users' local devices, and the rest of the layers into a remote server. We propose an approach to protect user's data during the inference phase, while still achieve good classification accuracy. We conduct an experimental evaluation of this approach on benchmark datasets of three computer vision tasks. The experimental results indicate that this approach can be used to significantly attenuate the capacity for an adversary with access to the state-of-the-art deep network's intermediate states to learn privacy-sensitive inputs to the network. For example, we demonstrate that our approach can prevent attackers from inferring the private attributes such as gender from the Face image dataset without sacrificing the classification accuracy of the original machine learning task such as Face Identification.

cs.CR↗

arXiv2018

Bytes are All You Need: End-to-End Multilingual Speech Recognition and Synthesis with Bytes

We present two end-to-end models: Audio-to-Byte (A2B) and Byte-to-Audio (B2A), for multilingual speech recognition and synthesis. Prior work has predominantly used characters, sub-words or words as the unit of choice to model text. These units are difficult to scale to languages with large vocabularies, particularly in the case of multilingual processing. In this work, we model text via a sequence of Unicode bytes, specifically, the UTF-8 variable length byte sequence for each character. Bytes allow us to avoid large softmaxes in languages with large vocabularies, and share representations in multilingual models. We show that bytes are superior to grapheme characters over a wide variety of languages in monolingual end-to-end speech recognition. Additionally, our multilingual byte model outperform each respective single language baseline on average by 4.4% relatively. In Japanese-English code-switching speech, our multilingual byte model outperform our monolingual baseline by 38.6% relatively. Finally, we present an end-to-end multilingual speech synthesis model using byte representations which matches the performance of our monolingual baselines.

eess.AS↗

arXiv2018

$L(\mathbb{R})$ with Determinacy Satisfies the Suslin Hypothesis

The Suslin hypothesis states that there are no nonseparable complete dense linear orderings without endpoints which have the countable chain condition. $\mathsf{ZF + AD^+ + V = L(\mathscr{P}(\mathbb{R}))}$ proves the Suslin hypothesis. In particular, if $L(\mathbb{R}) \models \mathsf{AD}$, then $L(\mathbb{R})$ satisfies the Suslin hypothesis, which answers a question of Foreman.

math.LO↗

arXiv2017

Scott Ranks of Classifications of the Admissibility Equivalence Relation

Let $\mathscr{L}$ be a recursive language. Let $S(\mathscr{L})$ be the set of $\mathscr{L}$-structures with domain $ω$. Let $Φ: {}^ω2 \rightarrow S(\mathscr{L})$ be a $Δ_1^1$ function with the property that for all $x,y \in {}^ω2$, $ω_1^x = ω_1^y$ if and only if $Φ(x) \approx_{\mathscr{L}} Φ(y)$. Then there is some $x \in {}^ω2$ so that $\mathrm{SR}(Φ(x)) = ω_1^x + 1$.

math.LO↗

arXiv2017

Ordinal Definability and Combinatorics of Equivalence Relations

Assume $\mathsf{ZF + AD^+ + V = L(\mathscr{P}(\mathbb{R}))}$. Let $E$ be a $\mathbfΣ^1_1$ equivalence relation coded in $\mathrm{HOD}$. $E$ has an ordinal definable equivalence class without any ordinal definable elements if and only if $\mathrm{HOD} \models E$ is unpinned. $\mathsf{ZF + AD^+ + V = L(\mathscr{P}(\mathbb{R}))}$ proves $E$-class section uniformization when $E$ is a $\mathbfΣ^1_1$ equivalence relation on $\mathbb{R}$ which is pinned in every transitive model of $\mathsf{ZFC}$ containing the real which codes $E$: Suppose $R$ is a relation on $\mathbb{R}$ such that each section $R_x = \{y : (x,y) \in R\}$ is an $E$-class, then there is a function $f : \mathbb{R} \rightarrow \mathbb{R}$ such that for all $x \in \mathbb{R}$, $R(x,f(x))$. $\mathsf{ZF + AD}$ proves that $\mathbb{R} \times κ$ is Jónsson whenever $κ$ is an ordinal: For every function $f : [\mathbb{R} \times κ]^{<ω}_= \rightarrow \mathbb{R} \times κ$, there is an $A \subseteq \mathbb{R} \times κ$ with $A$ in bijection with $\mathbb{R} \times κ$ and $f[[A]^{<ω}_=] \neq \mathbb{R} \times κ$.

math.LO↗

arXiv2017

Definable Combinatorics of Some Borel Equivalence Relations

If $X$ is a set, $E$ is an equivalence relation on $X$, and $n \in ω$, then define $$[X]^n_E = \{(x_0, ..., x_{n - 1}) \in {}^nX : (\forall i,j)(i \neq j \Rightarrow \neg(x_i \ E \ x_j))\}.$$ For $n \in ω$, a set $X$ has the $n$-Jónsson property if and only if for every function $f : [X]^n_= \rightarrow X$, there exists some $Y \subseteq X$ with $X$ and $Y$ in bijection so that $f[[Y]^n_=] \neq X$. A set $X$ has the Jónsson property if and only for every function $f : (\bigcup_{n \in ω}[X]^n_=) \rightarrow X$, there exists some $Y \subseteq X$ with $X$ and $Y$ in bijection so that $f[\bigcup_{n \in ω} [Y]^n_=] \neq X$. Let $n \in ω$, $X$ be a Polish space, and $E$ be an equivalence relation on $X$. $E$ has the $n$-Mycielski property if and only if for all comeager $C \subseteq {}^nX$, there is some $\mathbf{Δ_1^1}$ $A \subseteq X$ so that $E \leq_{\mathbf{Δ_1^1}} E \upharpoonright A$ and $[A]^n_E \subseteq C$. The following equivalence relations will be considered: $E_0$ is defined on ${}^\omega2$ by $x \ E_0 \ y$ if and only if $(\exists n)(\forall k > n)(x(k) = y(k))$. $E_1$ is defined on ${}^ω({}^\omega2)$ by $x \ E_1 \ y$ if and only if $(\exists n)(\forall k > n)(x(k) = y(k))$. $E_2$ is defined on ${}^\omega2$ by $x \ E_2 \ y$ if and only if $\sum\{\frac{1}{n + 1} : n \in x \ \triangle \ y\} < \infty$, where $\triangle$ denotes the symmetric difference. $E_3$ is defined on ${}^ω({}^\omega2)$ by $x \ E_3 \ y$ if and only if $(\forall n)(x(n) \ E_0 \ y(n))$. Holshouser and Jackson have shown that $\mathbb{R}$ is Jónsson under $\mathsf{AD}$. It will be shown that $E_0$ does not have the $3$-Mycielski property and that $E_1$, $E_2$, and $E_3$ do not have the $2$-Mycielski property. Under $\mathsf{ZF + AD}$, ${}^ω2 / E_0$ does not have the $3$-Jónsson property.

math.LO↗

arXiv2017

Advances in Joint CTC-Attention based End-to-End Speech Recognition with a Deep CNN Encoder and RNN-LM

We present a state-of-the-art end-to-end Automatic Speech Recognition (ASR) model. We learn to listen and write characters with a joint Connectionist Temporal Classification (CTC) and attention-based encoder-decoder network. The encoder is a deep Convolutional Neural Network (CNN) based on the VGG network. The CTC network sits on top of the encoder and is jointly trained with the attention-based decoder. During the beam search process, we combine the CTC predictions, the attention-based decoder predictions and a separately trained LSTM language model. We achieve a 5-10\% error reduction compared to prior systems on spontaneous Japanese and Chinese speech, and our end-to-end model beats out traditional hybrid ASR systems.

cs.CL↗

arXiv2017

Latent Sequence Decompositions

We present the Latent Sequence Decompositions (LSD) framework. LSD decomposes sequences with variable lengthed output units as a function of both the input sequence and the output sequence. We present a training algorithm which samples valid extensions and an approximate decoding algorithm. We experiment with the Wall Street Journal speech recognition task. Our LSD model achieves 12.9% WER compared to a character baseline of 14.8% WER. When combined with a convolutional network on the encoder, we achieve 9.6% WER.

stat.ML↗

arXiv2016

Very Deep Convolutional Networks for End-to-End Speech Recognition

Sequence-to-sequence models have shown success in end-to-end speech recognition. However these models have only used shallow acoustic encoder networks. In our work, we successively train very deep convolutional networks to add more expressive power and better generalization for end-to-end ASR models. We apply network-in-network principles, batch normalization, residual connections and convolutional LSTMs to build very deep recurrent and convolutional structures. Our models exploit the spectral structure in the feature space and add computational depth without overfitting issues. We experiment with the WSJ ASR task and achieve 10.5\% word error rate without any dictionary or language using a 15 layer deep network.

cs.CL↗

arXiv2016

When an Equivalence Relation with All Borel Classes will be Borel Somewhere?

In $\mathsf{ZFC}$, if there is a measurable cardinal with infinitely many Woodin cardinals below it, then for every equivalence relation $E \in L(\mathbb{R})$ on $\mathbb{R}$ with all $\mathbfΔ_1^1$ classes and every $σ$-ideal $I$ on $\mathbb{R}$ so that the associated forcing $\mathbb{P}_I$ of $I^+$ $\mathbfΔ_1^1$ subsets is proper, there exists some $I^+$ $\mathbfΔ_1^1$ set $C$ so that $E \upharpoonright C$ is a $\mathbfΔ_1^1$ equivalence relation. In $\mathsf{ZF} + \mathsf{DC} + \mathsf{AD}_\mathbb{R} + V = L(\mathscr{P}(\mathbb{R}))$, for every equivalence relation $E$ on $\mathbb{R}$ with all $\mathbfΔ_1^1$ classes and every $σ$-ideal $I$ on $\mathbb{R}$ so that the associated forcing $\mathbb{P}_I$ is proper, there is some $I^+$ $\mathbfΔ_1^1$ set $C$ so that $E \upharpoonright C$ is a $\mathbfΔ_1^1$ equivalence relation.

math.LO↗

arXiv2016

The Countable Admissible Ordinal Equivalence Relation

Let $F_{ω_1}$ be the countable admissible ordinal equivalence relation defined on ${}^ω2$ by $x \ F_{ω_1} \ y$ if and only if $ω_1^x = ω_1^y$. It will be shown that $F_{ω_1}$ is classifiable by countable structures and must be classified by structures of high Scott rank. If $E$ and $F$ are equivalence relations, then $E$ is almost Borel reducible to $F$ if and only if there is a Borel reduction of $E$ to $F$, except possibly on countably many $E$-classes. Let $E_{ω_1}$ denote the equivalence of order types of reals coding well-orderings. It will be shown that in the constructible universe $L$ and set generic extensions of $L$, $E_{ω_1}$ is not almost Borel reducible to $F_{ω_1}$, although a result of Zapletal implies such an almost Borel reduction exists if there is a measurable cardinal. Lastly, it will be shown that the isomorphism relation induced by a counterexample to Vaught's conjecture cannot be Borel reducible to $F_{ω_1}$ in $L$ and set generic extensions of $L$. This shows the consistency of a negative answer to a question of Sy-David Friedman.

math.LO↗

arXiv2015

Equivalence Relations Which Are Borel Somewhere

The following will be shown: Let $I$ be a $σ$-ideal on a Polish space $X$ with the property that the associated forcing of $I^+$ Borel subsets ordered by $\subseteq$ is a proper forcing. Let E be an analytic or coanalytic equivalence relation on this Polish space with all equivalence classes Borel. If sharps of certain sets exist, then there is an $I^+$ Borel subset $C$ of $X$ such that $E \upharpoonright C$ is a Borel equivalence relation.

math.LO↗

arXiv2015

Listen, Attend and Spell

We present Listen, Attend and Spell (LAS), a neural network that learns to transcribe speech utterances to characters. Unlike traditional DNN-HMM models, this model learns all the components of a speech recognizer jointly. Our system has two components: a listener and a speller. The listener is a pyramidal recurrent network encoder that accepts filter bank spectra as inputs. The speller is an attention-based recurrent network decoder that emits characters as outputs. The network produces character sequences without making any independence assumptions between the characters. This is the key improvement of LAS over previous end-to-end CTC models. On a subset of the Google voice search task, LAS achieves a word error rate (WER) of 14.1% without a dictionary or a language model, and 10.3% with language model rescoring over the top 32 beams. By comparison, the state-of-the-art CLDNN-HMM model achieves a WER of 8.0%.

cs.CL↗

arXiv2015

Deep Recurrent Neural Networks for Acoustic Modelling

We present a novel deep Recurrent Neural Network (RNN) model for acoustic modelling in Automatic Speech Recognition (ASR). We term our contribution as a TC-DNN-BLSTM-DNN model, the model combines a Deep Neural Network (DNN) with Time Convolution (TC), followed by a Bidirectional Long Short-Term Memory (BLSTM), and a final DNN. The first DNN acts as a feature processor to our model, the BLSTM then generates a context from the sequence acoustic signal, and the final DNN takes the context and models the posterior probabilities of the acoustic states. We achieve a 3.47 WER on the Wall Street Journal (WSJ) eval92 task or more than 8% relative improvement over the baseline DNN models.

cs.LG↗

arXiv2015

Transferring Knowledge from a RNN to a DNN

Deep Neural Network (DNN) acoustic models have yielded many state-of-the-art results in Automatic Speech Recognition (ASR) tasks. More recently, Recurrent Neural Network (RNN) models have been shown to outperform DNNs counterparts. However, state-of-the-art DNN and RNN models tend to be impractical to deploy on embedded systems with limited computational capacity. Traditionally, the approach for embedded platforms is to either train a small DNN directly, or to train a small DNN that learns the output distribution of a large DNN. In this paper, we utilize a state-of-the-art RNN to transfer knowledge to small DNN. We use the RNN model to generate soft alignments and minimize the Kullback-Leibler divergence against the small DNN. The small DNN trained on the soft RNN alignments achieved a 3.93 WER on the Wall Street Journal (WSJ) eval92 task compared to a baseline 4.54 WER or more than 13% relative improvement.

cs.LG↗

arXiv2012

Leveraging Subjective Human Annotation for Clustering Historic Newspaper Articles

The New York Public Library is participating in the Chronicling America initiative to develop an online searchable database of historically significant newspaper articles. Microfilm copies of the newspapers are scanned and high resolution Optical Character Recognition (OCR) software is run on them. The text from the OCR provides a wealth of data and opinion for researchers and historians. However, categorization of articles provided by the OCR engine is rudimentary and a large number of the articles are labeled editorial without further grouping. Manually sorting articles into fine-grained categories is time consuming if not impossible given the size of the corpus. This paper studies techniques for automatic categorization of newspaper articles so as to enhance search and retrieval on the archive. We explore unsupervised (e.g. KMeans) and semi-supervised (e.g. constrained clustering) learning algorithms to develop article categorization schemes geared towards the needs of end-users. A pilot study was designed to understand whether there was unanimous agreement amongst patrons regarding how articles can be categorized. It was found that the task was very subjective and consequently automated algorithms that could deal with subjective labels were used. While the small scale pilot study was extremely helpful in designing machine learning algorithms, a much larger system needs to be developed to collect annotations from users of the archive. The "BODHI" system currently being developed is a step in that direction, allowing users to correct wrongly scanned OCR and providing keywords and tags for newspaper articles used frequently. On successful implementation of the beta version of this system, we hope that it can be integrated with existing software being developed for the Chronicling America project.

cs.IR↗