arXiv subjects

Henry WJ Reeve

Publications and source records attributed to Henry WJ Reeve.

9 recordsLinked to original sources

arXiv2020

Optimistic bounds for multi-output prediction

We investigate the challenge of multi-output learning, where the goal is to learn a vector-valued function based on a supervised data set. This includes a range of important problems in Machine Learning including multi-target regression, multi-class classification and multi-label classification. We begin our analysis by introducing the self-bounding Lipschitz condition for multi-output loss functions, which interpolates continuously between a classical Lipschitz condition and a multi-dimensional analogue of a smoothness condition. We then show that the self-bounding Lipschitz condition gives rise to optimistic bounds for multi-output learning, which are minimax optimal up to logarithmic factors. The proof exploits local Rademacher complexity combined with a powerful minoration inequality due to Srebro, Sridharan and Tewari. As an application we derive a state-of-the-art generalization bound for multi-class gradient boosting.

stat.ML↗

arXiv2018

Minimax rates for cost-sensitive learning on manifolds with approximate nearest neighbours

We study the approximate nearest neighbour method for cost-sensitive classification on low-dimensional manifolds embedded within a high-dimensional feature space. We determine the minimax learning rates for distributions on a smooth manifold, in a cost-sensitive setting. This generalises a classic result of Audibert and Tsybakov. Building upon recent work of Chaudhuri and Dasgupta we prove that these minimax rates are attained by the approximate nearest neighbour algorithm, where neighbours are computed in a randomly projected low-dimensional space. In addition, we give a bound on the number of dimensions required for the projection which depends solely upon the reach and dimension of the manifold, combined with the regularity of the marginal.

cs.LG↗

arXiv2018

Diversity and degrees of freedom in regression ensembles

Ensemble methods are a cornerstone of modern machine learning. The performance of an ensemble depends crucially upon the level of diversity between its constituent learners. This paper establishes a connection between diversity and degrees of freedom (i.e. the capacity of the model), showing that diversity may be viewed as a form of inverse regularisation. This is achieved by focusing on a previously published algorithm Negative Correlation Learning (NCL), in which model diversity is explicitly encouraged through a diversity penalty term in the loss function. We provide an exact formula for the effective degrees of freedom in an NCL ensemble with fixed basis functions, showing that it is a continuous, convex and monotonically increasing function of the diversity parameter. We demonstrate a connection to Tikhonov regularisation and show that, with an appropriately chosen diversity parameter, an NCL ensemble can always outperform the unregularised ensemble in the presence of noise. We demonstrate the practical utility of our approach by deriving a method to efficiently tune the diversity parameter. Finally, we use a Monte-Carlo estimator to extend the connection between diversity and degrees of freedom to ensembles of deep neural networks.

cs.LG↗

arXiv2018

The K-Nearest Neighbour UCB algorithm for multi-armed bandits with covariates

In this paper we propose and explore the k-Nearest Neighbour UCB algorithm for multi-armed bandits with covariates. We focus on a setting where the covariates are supported on a metric space of low intrinsic dimension, such as a manifold embedded within a high dimensional ambient feature space. The algorithm is conceptually simple and straightforward to implement. The k-Nearest Neighbour UCB algorithm does not require prior knowledge of the either the intrinsic dimension of the marginal distribution or the time horizon. We prove a regret bound for the k-Nearest Neighbour UCB algorithm which is minimax optimal up to logarithmic factors. In particular, the algorithm automatically takes advantage of both low intrinsic dimensionality of the marginal distribution over the covariates and low noise in the data, expressed as a margin condition. In addition, focusing on the case of bounded rewards, we give corresponding regret bounds for the k-Nearest Neighbour KL-UCB algorithm, which is an analogue of the KL-UCB algorithm adapted to the setting of multi-armed bandits with covariates. Finally, we present empirical results which demonstrate the ability of both the k-Nearest Neighbour UCB and k-Nearest Neighbour KL-UCB to take advantage of situations where the data is supported on an unknown sub-manifold of a high-dimensional feature space.

cs.LG↗

arXiv2012

Multifractal analysis of Birkhoff averages for typical infinitely generated self-affine sets

We develop a thermodynamic formalism for quasi-multiplicative potentials on a countable symbolic space and apply these results to the dimension theory of infinitely generated self-affine sets. The first application is a generalisation of Falconer's dimension formula to include typical infinitely generated self-affine sets and show the existence of an ergodic invariant measure of full dimension whenever the pressure function has a root. Considering the multifractal analysis of Birkhoff averages of general potentials $Φ$ taking values in $\R^{\N}$, we give a formula for the Hausdorff dimension of $J_Φ(α)$, the $α$-level set of the Birkhoff average, on a typical infinitely generated self-affine set. We also show that for bounded potentials $Φ$, the Hausdorff dimension of $J_Φ(α)$ is given by the maximum of the critical value for the pressure and the supremum of Lyapunov dimensions of invariant measures $μ$ for which $\intΦ\,dμ=α$. Our multifractal results are new in both the finitely generated and the infinitely generated setting.

math.DS↗

arXiv2011

Shrinking Targets for Countable Markov Maps

Let $T$ be an expanding Markov map with a countable number of inverse branches and a repeller $Λ$ contained within the unit interval. Given $α\in \R_+$ we consider the set of points $x \in Λ$ for which $T^n(x)$ hits a shrinking ball of radius $e^{-nα}$ around $y$ for infinitely many iterates $n$. Let $s(α)$ denote the infimal value of $s$ for which the pressure of the potential $-s\log|T'|$ is below $s α$. Building on previous work of Hill, Velani and Urbański we show that for all points $y$ contained within the limit set of the associated iterated function system the Hausdorff dimension of the shrinking target set is given by $s(α)$. Moreover, when $\barΛ=[0,1]$ the same holds true for all $y \in [0,1]$. However, given $β\in (0,1)$ we provide an example of an expanding Markov map $T$ with a repeller $Λ$ of Hausdorff dimension $β$ with a point $y\in \barΛ$ such that for all $α\in \R_+$ the dimension of the shrinking target set is zero.

math.DS↗

arXiv2011

Infinite Non-Conformal Iterated Function Systems

We consider a generalisation of the self-affine iterated function systems of Lalley and Gatzouras by allowing for a countable infinity of non-conformal contractions. It is shown that the Hausdorff dimension of the limit set is equal to the supremum of the dimensions of compactly supported ergodic measures. In addition we consider the multifractal analysis for countable families of potentials. We obtain a conditional variational principle for the level sets.

math.DS↗

arXiv2011

The packing spectrum for Birkhoff averages on a self-affine repeller

We consider the multifractal analysis for Birkhoff averages of continuous potentials on a self-affine Sierpiński sponge. In particular, we give a variational principal for the packing dimension of the level sets. Furthermore, we prove that the packing spectrum is concave and continuous. We give a sufficient condition for the packing spectrum to be real analytic, but show that for general Hölder continuous potentials, this need not be the case. We also give a precise criterion for when the packing spectrum attains the full packing dimension of the repeller. Again, we present an example showing that this is not always the case.

math.DS↗

arXiv2011

Multifractal analysis for Birkhoff averages on Lalley-Gatzouras repellers

We consider the multifractal analysis for Birkhoff averages of continuous potentials on a class of non-conformal repellers corresponding to the self-affine limit sets studied by Lalley and Gatzouras. A conditional variational principle is given for the Hausdorff dimension of the set of points for which the Birkhoff averages converge to a given value. This extends a result of Barral and Mensi to certain non-conformal maps with a measure dependent Lyapunov exponent.

math.DS↗