arXiv subjects
Publications and source records attributed to Ben Krause.
We prove that the discrete fractional integration operators along the primes \[ T^λ_{\mathbb{P}}f(x) := \sum_{p} \frac{f(x-p)}{p^λ} \cdot \log p \] are bounded $\ell^p\to \ell^{p'}$ whenever $ \frac{1}{p'} < \frac{1}{p} - (1-λ), \ p > 1.$ Here, the sum runs only over prime $p$.
This research note combines two methods that have recently improved the state of the art in language modeling: Transformers and dynamic evaluation. Transformers use stacked layers of self-attention that allow them to capture long range dependencies in sequential data. Dynamic evaluation fits models to the recent sequence history, allowing them to assign higher probabilities to re-occurring sequential patterns. By applying dynamic evaluation to Transformer-XL models, we improve the state of the art on enwik8 from 0.99 to 0.94 bits/char, text8 from 1.08 to 1.04 bits/char, and WikiText-103 from 18.3 to 16.4 perplexity points.
Suppose that $A \subset \mathbb{R}$ has positive upper density, \[ \limsup_{|I| \to \infty} \frac{|A \cap I|}{|I|} = δ> 0,\] and $P(t) \in \mathbb{R}[t]$ is a polynomial with no constant or linear term, or more generally a non-flat curve: a locally differentiable curve which doesn't "resemble a line" near $0$ or $\infty$. Then for any $R_0 \leq R$ sufficiently large, there exists some $x_R \in A$ so that \[ \inf_{R_0 \leq T \leq R} \frac{|\{ 0 \leq t < T : x_R - t \in A, \ x_R - P(t) \in A \}|}{T} \geq c_P \cdot δ^2 \] for some absolute constant $c_P > 0$, that depends only on $P$.
Conversational agents are gaining popularity with the increasing ubiquity of smart devices. However, training agents in a data driven manner is challenging due to a lack of suitable corpora. This paper presents a novel method for gathering topical, unstructured conversational data in an efficient way: self-dialogues through crowd-sourcing. Alongside this paper, we include a corpus of 3.6 million words across 23 topics. We argue the utility of the corpus by comparing self-dialogues with standard two-party conversations as well as data from other corpora.
For polynomial $ P (x,y)$, and any Calderón-Zygmund kernel, $K$, the operator below satisfies a $ (1,r)$ sparse bound, for $ 1< r \leq 2$. $$ \sup _{ε>0} \Bigl\lvert \int_{|y| > ε} f (x-y) e ^{2 πi P (x,y) } K(y) \; dy \Bigr\rvert $$ The implied bound depends upon $ P (x,y)$ only through the degree of $ P$. We derive from this a range of weighted inequalities, including weak type inequalities on $ L ^{1} (w)$, which are new, even in the unweighted case. The unweighted weak-type estimate, without maximal truncations, is due to Chanillo and Christ (1987).
Motivated by Bourgain's work on pointwise ergodic theorems, and the work of Stein and Stein-Wainger on maximally modulated singular integrals without linear terms, we prove that the maximally monomially modulated discrete Hilbert transform, \[ \mathcal{C}_df(x) := \sup_λ\left| \sum_{m \neq 0} f(x-m) \frac{e^{2πi λm^d}}{m} \right| \] is bounded on all $\ell^p, \ 2 - \frac{1}{d^2 + 1} < p < \infty$, for any $d \geq 2$. We also establish almost everywhere pointwise convergence of the modulated ergodic Hilbert transforms (as $λ\to 0$) \[ \sum_{m \neq 0} T^m f(x) \cdot \frac{e^{2πi λm^d}}{m} \] for any measure-preserving system $(X,μ,T)$, and any $f \in L^p(X), \ 2 - \frac{1}{d^2 +1} < p < \infty$.
We present methodology for using dynamic evaluation to improve neural sequence models. Models are adapted to recent history via a gradient descent based mechanism, causing them to assign higher probabilities to re-occurring sequential patterns. Dynamic evaluation outperforms existing adaptation approaches in our comparisons. Dynamic evaluation improves the state-of-the-art word-level perplexities on the Penn Treebank and WikiText-2 datasets to 51.1 and 44.3 respectively, and the state-of-the-art character-level cross-entropies on the text8 and Hutter Prize datasets to 1.19 bits/char and 1.08 bits/char respectively.
We introduce multiplicative LSTM (mLSTM), a recurrent neural network architecture for sequence modelling that combines the long short-term memory (LSTM) and multiplicative recurrent neural network architectures. mLSTM is characterised by its ability to have different recurrent transition functions for each possible input, which we argue makes it more expressive for autoregressive density estimation. We demonstrate empirically that mLSTM outperforms standard LSTM and its deep variants for a range of character level language modelling tasks. In this version of the paper, we regularise mLSTM to achieve 1.27 bits/char on text8 and 1.24 bits/char on Hutter Prize. We also apply a purely byte-level mLSTM on the WikiText-2 dataset to achieve a character level entropy of 1.26 bits/char, corresponding to a word level perplexity of 88.8, which is comparable to word level LSTMs regularised in similar ways on the same task.
We present Edina, the University of Edinburgh's social bot for the Amazon Alexa Prize competition. Edina is a conversational agent whose responses utilize data harvested from Amazon Mechanical Turk (AMT) through an innovative new technique we call self-dialogues. These are conversations in which a single AMT Worker plays both participants in a dialogue. Such dialogues are surprisingly natural, efficient to collect and reflective of relevant and/or trending topics. These self-dialogues provide training data for a generative neural network as well as a basis for soft rules used by a matching score component. Each match of a soft rule against a user utterance is associated with a confidence score which we show is strongly indicative of reply quality, allowing this component to self-censor and be effectively integrated with other components. Edina's full architecture features a rule-based system backing off to a matching score, backing off to a generative neural network. Our hybrid data-driven methodology thus addresses both coverage limitations of a strictly rule-based approach and the lack of guarantees of a strictly machine-learning approach.
We extend the main result of Harrow, Kolla, and Schulman -- the existence of dimension-free $L^2$-bounds for the spherical maximal function in the hypercube -- to all $L^p, p > 1$. Our approach is motivated by the spectral technique developed by Stein and Nevo and Stein in the context of pointwise ergodic theorems on general groups. We provide an example which demonstrates that no dimension-free weak-type (1-1) bound exists at the endpoint.
Let $a_n$ be the random increasing sequence of natural numbers which takes each value independently with decreasing probability of order $n^{-α}$, $0 < α< 1/2$. We prove that, almost surely, for every measure-preserving system $(X,T)$ and every $f \in L^1(X)$ orthogonal to the invariant factor, the modulated, random averages \[ \sup_{b} \Big| \frac{1}{N} \sum_{n = 1}^N b(n) T^{a_{n}} f \Big| \] converge to $0$ pointwise almost everywhere, where the supremum is taken over a set of bounded functions with certain uniform approximation properties.
For each $ d \geq 2$, the Hilbert transform with a polynomial oscillation as below satisfies a $ (1, r )$ sparse bound, for all $ r>1$ $$ H _{ \ast } f (x) = \sup _{ε} \Bigl\lvert \int_{|y| > ε} f (x-y) \frac { e ^{2 πi y ^d }} y\; dy \Bigr\rvert. $$ This quickly implies weak-type inequalities for the maximal truncations, which hold for $A_1$ weights, but are new even in the case of Lebesgue measure. The unweighted weak-type estimate \emph{without maximal truncations} but with arbitrary polynomials, is due to Chanillo and Christ (1987).
We introduce sufficient conditions on discrete singular integral operators for their maximal truncations to satisfy a sparse bound. The latter imply a range of quantitative weighted inequalities, which are new. As an application, we prove the following ergodic theorem: let $p(t)$ be a Hardy field function which grows "super-linearly" and stays "sufficiently far" from polynomials. We show that for each measure-preserving system, $(X,Σ,μ,τ)$, with $τ$ a measure-preserving $\mathbb{Z}$-action, the modulated one-sided ergodic Hilbert transform \[ \sum_{n=1}^\infty \frac{e^{2πi p(n)}}{n} τ^n f(x) \] converges $μ$-a.e. for each $f \in L^r(X), \ 1 \leq r < \infty$. This affirmatively answers a question of J. Rosenblatt. In the second part of the paper, we establish almost sure sparse bounds for random one-sided ergodic Hilbert transforms, \[ \sum_{n=1}^\infty \frac{X_n}{n} τ^n f(x), \] where $\{ X_n \}$ are uniformly bounded, independent, and mean-zero random variables.
Let $σ$ be arc-length measure on $S^1\subset \mathbb R^2$ and $Θ$ denote rotation by an angle $θ\in (0, π]$. Define a model bilinear generalized Radon transform, $$B_θ(f,g)(x)=\int_{S^1} f(x-y)g(x-Θy)\, dσ(y),$$ an analogue of the linear generalized Radon transforms of Guillemin and Sternberg \cite{GS} and Phong and Stein (e.g., \cite{PhSt91,St93}). Operators such as $B_θ$ are motivated by problems in geometric measure theory and combinatorics. For $θ<π$, we show that $B_θ: L^p({\Bbb R}^2) \times L^q({\Bbb R}^2) \to L^r({\Bbb R}^2)$ if $\left(\frac{1}{p},\frac{1}{q},\frac{1}{r}\right)\in Q$, the polyhedron with the vertices $(0,0,0)$, $(\frac{2}{3}, \frac{2}{3}, 1)$, $(0, \frac{2}{3}, \frac{1}{3})$, $(\frac{2}{3},0,\frac{1}{3})$, $(1,0,1)$, $(0,1,1)$ and $(\frac{1}{2},\frac{1}{2},\frac{1}{2})$, except for $\left( \frac{1}{2},\frac{1}{2},\frac{1}{2} \right)$, where we obtain a restricted strong type estimate. For the degenerate case $θ=π$, a more restrictive set of exponents holds. In the scale of normed spaces, $p,q,r \ge 1$, the type set $Q$ is sharp. Estimates for the same exponents are also proved for a class of bilinear generalized Radon transforms in $\mathbb R^2$ of the form $$ B(f,g)(x)=\int \int δ(ϕ_1(x,y)-t_1)δ(ϕ_2(x,z)-t_2) δ(ϕ_3(y,z)-t_3) f(y)g(z) ψ(y,z) \, dy\, dz, $$ where $δ$ denotes the Dirac distribution, $t_1,t_2,t_3\in\mathbb R$, $ψ$ is a smooth cut-off and the defining functions $ϕ_j$ satisfy some natural geometric assumptions.
We prove new endpoint bounds for the lacunary spherical maximal operator and as a consequence obtain almost everywhere pointwise convergence of lacunary spherical means for functions locally in $L\log\log\log L(\log\log\log\log L)^{1+ε}$ for any $ε>0$.
We study discrete random variants of the Carleson maximal operator. Intriguingly, these questions remain subtle and difficult, even in this setting. Let $\{X_m\}$ be an independent sequence of $\{0,1\}$ random variables with expectations \[ \mathbb E X_m = σ_m = m^{-α}, \ 0 < α< 1/2, \] and $ S_m = \sum_{k=1} ^{m} X_k$. Then the maximal operator below almost surely is bounded from $ \ell ^{p}$ to $ \ell ^{p}$, provided the Minkowski dimension of $ Λ\subset [-1/2, 1/2]$ is strictly less than $ 1- α$. \[ \sup_{λ\in Λ} \Bigl| \sum_{m\neq 0} X_{\lvert m\rvert } \frac{e( λm )}{ {\rm sgn} (m)S_{ |m| }} f(x- m) \Bigr|. \] This operator also satisfies a sparse type bound. The form of the sparse bound immediately implies weighted estimates in all $ \ell ^{2}$, which are novel in this setting. Variants and extensions are also considered.
Consider the discrete maximal function acting on $\ell^2(\mathbb Z)$ functions \[ \mathcal{C}_Λ f( n ) := \sup_{ λ\in Λ} \left| \sum_{m \neq 0} f(n-m) \frac{e^{2 πiλm^2}} {m} \right| \] where $Λ\subset [0,1]$. We give sufficient conditions on $Λ$, met by certain kinds of Cantor sets, for this to be a bounded sublinear operator. This result is a discrete analogue of E. M. Stein's integral result, that the maximal operator below is bounded on $L^2(\mathbb R)$. \[ \mathcal{C}_2 f(x):= \sup_{λ\in \mathbb R} \left| \int f(x-y) \frac{e^{2πi λy^2}}{y} \ dy \right|.\] The proof of our result relies heavily on Bourgain's work on arithmetic ergodic theorems, with novel complexity arising from the oscillatory nature of the question at hand, and difficulties arising from the the parameter $λ$ above.
Consider the discrete maximal function acting on finitely supported functions on the integers, \[ \mathcal{C}_Λf(n) := \sup_{λ\in Λ} | \sum_{p \in \pm \mathbb{P}} f(n-p) \log |p| \frac{e^{2πi λp}}{p} |,\] where $\pm \mathbb{P} := \{ \pm p : p \text{ is a prime} \}$, and $Λ\subset [0,1]$. We give sufficient conditions on $Λ$, met by (finite unions of) lacunary sets, for this to be a bounded sublinear operator on $\ell^p(\mathbb{Z})$ for $\frac{3}{2} < p < 4$.