arXiv subjects

Jonathan H. Manton

Publications and source records attributed to Jonathan H. Manton.

At least 37 records · Page 2Linked to original sources

arXiv2021

Online Transfer Learning: Negative Transfer and Effect of Prior Knowledge

Transfer learning is a machine learning paradigm where the knowledge from one task is utilized to resolve the problem in a related task. On the one hand, it is conceivable that knowledge from one task could be useful for solving a related problem. On the other hand, it is also recognized that if not executed properly, transfer learning algorithms could in fact impair the learning performance instead of improving it - commonly known as "negative transfer". In this paper, we study the online transfer learning problems where the source samples are given in an offline way while the target samples arrive sequentially. We define the expected regret of the online transfer learning problem and provide upper bounds on the regret using information-theoretic quantities. We also obtain exact expressions for the bounds when the sample size becomes large. Examples show that the derived bounds are accurate even for small sample sizes. Furthermore, the obtained bounds give valuable insight on the effect of prior knowledge for transfer learning in our formulation. In particular, we formally characterize the conditions under which negative transfer occurs.

cs.LG↗

arXiv2021

Hidden Markov chains and fields with observations in Riemannian manifolds

Hidden Markov chain, or Markov field, models, with observations in a Euclidean space, play a major role across signal and image processing. The present work provides a statistical framework which can be used to extend these models, along with related, popular algorithms (such as the Baum-Welch algorithm), to the case where the observations lie in a Riemannian manifold. It is motivated by the potential use of hidden Markov chains and fields, with observations in Riemannian manifolds, as models for complex signals and images.

math.ST↗

arXiv2020

New Insights on Learning Rules for Hopfield Networks: Memory and Objective Function Minimisation

Hopfield neural networks are a possible basis for modelling associative memory in living organisms. After summarising previous studies in the field, we take a new look at learning rules, exhibiting them as descent-type algorithms for various cost functions. We also propose several new cost functions suitable for learning. We discuss the role of biases (the external inputs) in the learning process in Hopfield networks. Furthermore, we apply Newtons method for learning memories, and experimentally compare the performances of various learning rules. Finally, to add to the debate whether allowing connections of a neuron to itself enhances memory capacity, we numerically investigate the effects of self coupling. Keywords: Hopfield Networks, associative memory, content addressable memory, learning rules, gradient descent, attractor networks

cs.NE↗

arXiv2020

VAR estimators using binary measurements

In this paper, two novel algorithms to estimate a Gaussian Vector Autoregressive (VAR) model from 1-bit measurements are introduced. They are based on the Yule-Walker scheme modified to account for quantisation. The scalar case has been studied before. The main difficulty when going from the scalar to the vector case is how to estimate the ratios of the variances of pairwise components of the VAR model. The first method overcomes this difficulty by requiring the quantisation to be non-symmetric: each component of the VAR model output is replaced by a binary "zero" or a binary "one" depending on whether its value is greater than a strictly positive threshold. Different components of the VAR model can have different thresholds. As the choice of these thresholds has a strong influence on the performance, this first method is best suited for applications where the variance of each time series is approximately known prior to choosing the corresponding threshold. The second method relies instead on symmetric quantisations of not only each component of the VAR model but also on the pairwise differences of the components. These additional measurements are equivalent to a ranking of the instantaneous VAR model output, from the smallest component to the largest component. This avoids the need for choosing thresholds but requires additional hardware for quantising the components in pairs. Numerical simulations show the efficiency of both schemes.

stat.ME↗

arXiv2020

Information-theoretic analysis for transfer learning

Transfer learning, or domain adaptation, is concerned with machine learning problems in which training and testing data come from possibly different distributions (denoted as $μ$ and $μ'$, respectively). In this work, we give an information-theoretic analysis on the generalization error and the excess risk of transfer learning algorithms, following a line of work initiated by Russo and Zhou. Our results suggest, perhaps as expected, that the Kullback-Leibler (KL) divergence $D(mu||mu')$ plays an important role in characterizing the generalization error in the settings of domain adaptation. Specifically, we provide generalization error upper bounds for general transfer learning algorithms and extend the results to a specific empirical risk minimization (ERM) algorithm where data from both distributions are available in the training phase. We further apply the method to iterative, noisy gradient descent algorithms, and obtain upper bounds which can be easily calculated, only using parameters from the learning algorithms. A few illustrative examples are provided to demonstrate the usefulness of the results. In particular, our bound is tighter in specific classification problems than the bound derived using Rademacher complexity.

cs.LG↗

arXiv2020

Asymptotic regime for impropriety tests of complex random vectors

Impropriety testing for complex-valued vector has been considered lately due to potential applications ranging from digital communications to complex media imaging. This paper provides new results for such tests in the asymptotic regime, i.e. when the vector dimension and sample size grow commensurately to infinity. The studied tests are based on invariant statistics named impropriety coefficients. Limiting distributions for these statistics are derived, together with those of the Generalized Likelihood Ratio Test (GLRT) and Roy's test, in the Gaussian case. This characterization in the asymptotic regime allows also to identify a phase transition in Roy's test with potential application in detection of complex-valued low-rank subspace corrupted by proper noise in large datasets. Simulations illustrate the accuracy of the proposed asymptotic approximations.

eess.SP↗

arXiv2019

The Riemannian barycentre as a proxy for global optimisation

Let $M$ be a simply-connected compact Riemannian symmetric space, and $U$ a twice-differentiable function on $M$, with unique global minimum at $x^* \in M$. The idea of the present work is to replace the problem of searching for the global minimum of $U$, by the problem of finding the Riemannian barycentre of the Gibbs distribution $P_{\scriptscriptstyle{T}} \propto \exp(-U/T)$. In other words, instead of minimising the function $U$ itself, to minimise $\mathcal{E}_{\scriptscriptstyle{T}}(x) = \frac{1}{2}\int d^{\scriptscriptstyle 2}(x,z)P_{\scriptscriptstyle{T}}(dz)$, where $d(\cdot,\cdot)$ denotes Riemannian distance. The following original result is proved : if $U$ is invariant by geodesic symmetry about $x^*$, then for each $δ< \frac{1}{2} r_{\scriptscriptstyle cx}$ ($r_{\scriptscriptstyle cx}$ the convexity radius of $M$), there exists $T_{\scriptscriptstyle δ}$ such that $T \leq T_{\scriptscriptstyle δ}$ implies $\mathcal{E}_{\scriptscriptstyle{T}}$ is strongly convex on the geodesic ball $B(x^*,δ)\,$, and $x^*$ is the unique global minimum of $\mathcal{E}_{\scriptscriptstyle{T\,}}$. Moreover, this $T_{\scriptscriptstyle δ}$ can be computed explicitly. This result gives rise to a general algorithm for black-box optimisation, which is briefly described, and will be further explored in future work.

math.ST↗

arXiv2018

Modeling the respiratory Central Pattern Generator with resonate-and-fire Izhikevich-Neurons

Computational models of the respiratory central pattern generator (rCPG) are usually based on biologically-plausible Hodgkin Huxley neuron models. Such models require numerous parameters and thus are prone to overfitting. The HH approach is motivated by the assumption that the biophysical properties of neurons determine the network dynamics. Here, we implement the rCPG using simpler Izhikevich resonate-and-fire neurons. Our rCPG model generates a 3-phase respiratory motor pattern based on established connectivities and can reproduce previous experimental and theoretical observations. Further, we demonstrate the flexibility of the model by testing whether intrinsic bursting properties are necessary for rhythmogenesis. Our simulations demonstrate that replacing predicted mandatory bursting properties of pre-inspiratory neurons with spike adapting properties yields a model that generates comparable respiratory activity patterns. The latter supports our view that the importance of the exact modeling parameters of specific respiratory neurons is overestimated.

q-bio.NC↗

arXiv2018

Source Localisation Using Binary Measurements

This paper considers the problem of localising a stationary signal source using a team of mobile agents which only take binary measurements. Background false detection rates and missed detection probabilities are incorporated into the framework. A Bayesian estimation algorithm that discretises the search environment is employed, and analytical convergence and consistency results for this are derived. Fisher Information is then used as a metric for the design of optimal agent geometries. Knowledge of the probability of detection as a function of the source and agent locations is assumed in the analysis, with special attention given to range-dependent functions. The behaviour of the algorithm under inexact knowledge of the probability of detection is also analysed. Finally, simulation results are presented to demonstrate the effectiveness of the algorithm.

eess.SP↗

arXiv2017

Riemannian Gaussian distributions on the space of positive-definite quaternion matrices

Recently, Riemannian Gaussian distributions were defined on spaces of positive-definite real and complex matrices. The present paper extends this definition to the space of positive-definite quaternion matrices. In order to do so, it develops the Riemannian geometry of the space of positive-definite quaternion matrices, which is shown to be a Riemannian symmetric space of non-positive curvature. The paper gives original formulae for the Riemannian metric of this space, its geodesics, and distance function. Then, it develops the theory of Riemannian Gaussian distributions, including the exact expression of their probability density, their sampling algorithm and statistical inference.

math.ST↗

arXiv2016

Expansion-maximization-compression algorithm with spherical harmonics for single particle imaging with X-ray lasers

In 3D single particle imaging with X-ray free-electron lasers, particle orientation is not recorded during measurement but is instead recovered as a necessary step in the reconstruction of a 3D image from the diffraction data. Here we use harmonic analysis on the sphere to cleanly separate the angu- lar and radial degrees of freedom of this problem, providing new opportunities to efficiently use data and computational resources. We develop the Expansion-Maximization-Compression algorithm into a shell-by-shell approach and implement an angular bandwidth limit that can be gradually raised during the reconstruction. We study the minimum number of patterns and minimum rotation sampling required for a desired angular and radial resolution. These extensions provide new av- enues to improve computational efficiency and speed of convergence, which are critically important considering the very large datasets expected from experiment.

physics.data-an↗

arXiv2015

Density estimation on the rotation group using diffusive wavelets

This paper considers the problem of estimating probability density functions on the rotation group $SO(3)$. Two distinct approaches are proposed, one based on characteristic functions and the other on wavelets using the heat kernel. Expressions are derived for their Mean Integrated Squared Errors. The performance of the estimators is studied numerically and compared with the performance of an existing technique using the De La Vallée Poussin kernel estimator. The heat-kernel wavelet approach appears to offer the best convergence, with faster convergence to the optimal bound and guaranteed positivity of the estimated probability density function.

math.ST↗

arXiv2015

Isotropic Multiple Scattering Processes on Hyperspheres

This paper presents several results about isotropic random walks and multiple scattering processes on hyperspheres ${\mathbb S}^{p-1}$. It allows one to derive the Fourier expansions on ${\mathbb S}^{p-1}$ of these processes. A result of unimodality for the multiconvolution of symmetrical probability density functions (pdf) on ${\mathbb S}^{p-1}$ is also introduced. Such processes are then studied in the case where the scattering distribution is von Mises Fisher (vMF). Asymptotic distributions for the multiconvolution of vMFs on ${\mathbb S}^{p-1}$ are obtained. Both Fourier expansion and asymptotic approximation allows us to compute estimation bounds for the parameters of Compound Cox Processes (CCP) on ${\mathbb S}^{p-1}$.

cs.IT↗

arXiv2015

A Primer on Reproducing Kernel Hilbert Spaces

Reproducing kernel Hilbert spaces are elucidated without assuming prior familiarity with Hilbert spaces. Compared with extant pedagogic material, greater care is placed on motivating the definition of reproducing kernel Hilbert spaces and explaining when and why these spaces are efficacious. The novel viewpoint is that reproducing kernel Hilbert space theory studies extrinsic geometry, associating with each geometric configuration a canonical overdetermined coordinate system. This coordinate system varies continuously with changing geometric configurations, making it well-suited for studying problems whose solutions also vary continuously with changing geometry. This primer can also serve as an introduction to infinite-dimensional linear algebra because reproducing kernel Hilbert spaces have more properties in common with Euclidean spaces than do more general Hilbert spaces.

math.HO↗

arXiv2014

A Framework for Generalising the Newton Method and Other Iterative Methods from Euclidean Space to Manifolds

The Newton iteration is a popular method for minimising a cost function on Euclidean space. Various generalisations to cost functions defined on manifolds appear in the literature. In each case, the convergence rate of the generalised Newton iteration needed establishing from first principles. The present paper presents a framework for generalising iterative methods from Euclidean space to manifolds that ensures local convergence rates are preserved. It applies to any (memoryless) iterative method computing a coordinate independent property of a function (such as a zero or a local minimum). All possible Newton methods on manifolds are believed to come under this framework. Changes of coordinates, and not any Riemannian structure, are shown to play a natural role in lifting the Newton method to a manifold. The framework also gives new insight into the design of Newton methods in general.

math.OC↗

arXiv2013

Differential Calculus, Tensor Products and the Importance of Notation

An efficient coordinate-free notation is elucidated for differentiating matrix expressions and other functions between higher-dimensional vector spaces. This method of differentiation is known, but not explained well, in the literature. Teaching it early in the curriculum would avoid the tedium of element-wise differentiation and provide a better footing for understanding more advanced applications of calculus. Additionally, it is shown to lead naturally to tensor products, a topic previously considered too difficult to motivate quickly in elementary ways.

math.HO↗

arXiv2013

A Primer on Stochastic Differential Geometry for Signal Processing

This primer explains how continuous-time stochastic processes (precisely, Brownian motion and other Ito diffusions) can be defined and studied on manifolds. No knowledge is assumed of either differential geometry or continuous-time processes. The arguably dry approach is avoided of first introducing differential geometry and only then introducing stochastic processes; both areas are motivated and developed jointly.

math.HO↗

arXiv2012

Optimisation Geometry

This article demonstrates how an understanding of the geometry of a family of cost functions can be used to develop efficient numerical algorithms for real-time optimisation. Crucially, it is not the geometry of the individual functions which is studied, but the geometry of the family as a whole. In some respects, this challenges the conventional divide between convex and non-convex optimisation problems because none of the cost functions in a family need be convex in order for efficient numerical algorithms to exist for optimising in real-time any function belonging to the family. The title "Optimisation Geometry" comes by analogy from the study of the geometry of a family of probability distributions being called information geometry.

math.OC↗