arXiv subjects

Terry Lyons

Publications and source records attributed to Terry Lyons.

At least 55 records · Page 3Linked to original sources

arXiv2023

Rectifiable paths with polynomial log-signature are straight lines

The signature of a rectifiable path is a tensor series in the tensor algebra whose coefficients are definite iterated integrals of the path. The signature characterises the path up to a generalised form of reparametrisation. It is a classical result of K. T. Chen that the log-signature (the logarithm of the signature) is a Lie series. A Lie series is polynomial if it has finite degree. We show that the log-signature is polynomial if and only if the path is a straight line up to reparametrisation. Consequently, the log-signature of a rectifiable path either has degree one or infinite support. Though our result pertains to rectifiable paths, the proof uses results from rough path theory, in particular that the signature characterises a rough path up to reparametrisation.

math.RA↗

arXiv2023

Dual Bayesian ResNet: A Deep Learning Approach to Heart Murmur Detection

This study presents our team PathToMyHeart's contribution to the George B. Moody PhysioNet Challenge 2022. Two models are implemented. The first model is a Dual Bayesian ResNet (DBRes), where each patient's recording is segmented into overlapping log mel spectrograms. These undergo two binary classifications: present versus unknown or absent, and unknown versus present or absent. The classifications are aggregated to give a patient's final classification. The second model is the output of DBRes integrated with demographic data and signal features using XGBoost.DBRes achieved our best weighted accuracy of $0.771$ on the hidden test set for murmur classification, which placed us fourth for the murmur task. (On the clinical outcome task, which we neglected, we scored 17th with costs of $12637$.) On our held-out subset of the training set, integrating the demographic data and signal features improved DBRes's accuracy from $0.762$ to $0.820$. However, this decreased DBRes's weighted accuracy from $0.780$ to $0.749$. Our results demonstrate that log mel spectrograms are an effective representation of heart sound recordings, Bayesian networks provide strong supervised classification performance, and treating the ternary classification as two binary classifications increases performance on the weighted accuracy.

cs.LG↗

arXiv2023

Sampling-based Nyström Approximation and Kernel Quadrature

We analyze the Nyström approximation of a positive definite kernel associated with a probability measure. We first prove an improved error bound for the conventional Nyström approximation with i.i.d. sampling and singular-value decomposition in the continuous regime; the proof techniques are borrowed from statistical learning theory. We further introduce a refined selection of subspaces in Nyström approximation with theoretical guarantees that is applicable to non-i.i.d. landmark points. Finally, we discuss their application to convex kernel quadrature and give novel theoretical guarantees as well as numerical observations.

math.NA↗

arXiv2023

Benchmarking optimality of time series classification methods in distinguishing diffusions

Statistical optimality benchmarking is crucial for analyzing and designing time series classification (TSC) algorithms. This study proposes to benchmark the optimality of TSC algorithms in distinguishing diffusion processes by the likelihood ratio test (LRT). The LRT is an optimal classifier by the Neyman-Pearson lemma. The LRT benchmarks are computationally efficient because the LRT does not need training, and the diffusion processes can be efficiently simulated and are flexible to reflect the specific features of real-world applications. We demonstrate the benchmarking with three widely-used TSC algorithms: random forest, ResNet, and ROCKET. These algorithms can achieve the LRT optimality for univariate time series and multivariate Gaussian processes. However, these model-agnostic algorithms are suboptimal in classifying high-dimensional nonlinear multivariate time series. Additionally, the LRT benchmark provides tools to analyze the dependence of classification accuracy on the time length, dimension, temporal sampling frequency, and randomness of the time series.

stat.ML↗

arXiv2023

Optimal Stopping via Distribution Regression: a Higher Rank Signature Approach

Distribution Regression on path-space refers to the task of learning functions mapping the law of a stochastic process to a scalar target. The learning procedure based on the notion of path-signature, i.e. a classical transform from rough path theory, was widely used to approximate weakly continuous functionals, such as the pricing functionals of path--dependent options' payoffs. However, this approach fails for Optimal Stopping Problems arising from mathematical finance, such as the pricing of American options, because the corresponding value functions are in general discontinuous with respect to the weak topology. In this paper we develop a rigorous mathematical framework to resolve this issue by recasting an Optimal Stopping Problem as a higher order kernel mean embedding regression based on the notions of higher rank signatures of measure--valued paths and adapted topologies. The core computational component of our algorithm consists in solving a family of two--dimensional hyperbolic PDEs.

math.PR↗

arXiv2023

New directions in the applications of rough path theory

This article provides a concise overview of some of the recent advances in the application of rough path theory to machine learning. Controlled differential equations (CDEs) are discussed as the key mathematical model to describe the interaction of a stream with a physical control system. A collection of iterated integrals known as the signature naturally arises in the description of the response produced by such interactions. The signature comes equipped with a variety of powerful properties rendering it an ideal feature map for streamed data. We summarise recent advances in the symbiosis between deep learning and CDEs, studying the link with RNNs and culminating with the Neural CDE model. We concluded with a discussion on signature kernel methods.

cs.LG↗

arXiv2022

Positively Weighted Kernel Quadrature via Subsampling

We study kernel quadrature rules with convex weights. Our approach combines the spectral properties of the kernel with recombination results about point measures. This results in effective algorithms that construct convex quadrature rules using only access to i.i.d. samples from the underlying measure and evaluation of the kernel and that result in a small worst-case error. In addition to our theoretical results and the benefits resulting from convex weights, our experiments indicate that this construction can compete with the optimal bounds in well-known examples.

math.NA↗

arXiv2022

Hypercontractivity Meets Random Convex Hulls: Analysis of Randomized Multivariate Cubatures

Given a probability measure $μ$ on a set $\mathcal{X}$ and a vector-valued function $φ$, a common problem is to construct a discrete probability measure on $\mathcal{X}$ such that the push-forward of these two probability measures under $φ$ is the same. This construction is at the heart of numerical integration methods that run under various names such as quadrature, cubature, or recombination. A natural approach is to sample points from $μ$ until their convex hull of their image under $φ$ includes the mean of $φ$. Here we analyze the computational complexity of this approach when $φ$ exhibits a graded structure by using so-called hypercontractivity. The resulting theorem not only covers the classical cubature case of multivariate polynomials, but also integration on pathspace, as well as kernel quadrature for product measures.

math.PR↗

arXiv2022

An asymptotic radius of convergence for the Loewner equation and simulation of $SLE_k$ traces via splitting

In this paper, we shall study the convergence of Taylor approximations for the backward Loewner differential equation (driven by Brownian motion) near the origin. More concretely, whenever the initial condition of the backward Loewner equation (which lies in the upper half plane) is small and has the form $Z_{0} = \varepsilon i$, we show these approximations exhibit an $O(\varepsilon)$ error provided the time horizon is $\varepsilon^{2+δ}$ for $δ> 0$. Statements of this theorem will be given using both rough path and $L^{2}(\mathbb{P})$ estimates. Furthermore, over the time horizon of $\varepsilon^{2-δ}$, we shall see that "higher degree" terms within the Taylor expansion become larger than "lower degree" terms for small $\varepsilon$. In this sense, the time horizon on which approximations are accurate scales like $\varepsilon^{2}$. This scaling comes naturally from the Loewner equation when growing vector field derivatives are balanced against decaying iterated integrals of the Brownian motion. As well as being of theoretical interest, this scaling may be used as a guiding principle for developing adaptive step size strategies which perform efficiently near the origin. In addition, this result highlights the limitations of using stochastic Taylor methods (such as the Euler-Maruyama and Milstein methods) for approximating $SLE_κ$ traces. Due to the analytically tractable vector fields of the Loewner equation, we will show Ninomiya-Victoir (or Strang) splitting is particularly well suited for SLE simulation. As the singularity at the origin can lead to large numerical errors, we shall employ the adaptive step size proposed by Tom Kennedy to discretize $SLE_κ$ traces using this splitting. We believe that the Ninomiya-Victoir scheme is the first high order numerical method that has been successfully applied to $SLE_κ$ traces.

math.PR↗

arXiv2022

ImageSig: A signature transform for ultra-lightweight image recognition

This paper introduces a new lightweight method for image recognition. ImageSig is based on computing signatures and does not require a convolutional structure or an attention-based encoder. It is striking to the authors that it achieves: a) an accuracy for 64 X 64 RGB images that exceeds many of the state-of-the-art methods and simultaneously b) requires orders of magnitude less FLOPS, power and memory footprint. The pretrained model can be as small as 44.2 KB in size. ImageSig shows unprecedented performance on hardware such as Raspberry Pi and Jetson-nano. ImageSig treats images as streams with multiple channels. These streams are parameterized by spatial directions. We contribute to the functionality of signature and rough path theory to stream-like data and vision tasks on static images beyond temporal streams. With very few parameters and small size models, the key advantage is that one could have many of these "detectors" assembled on the same chip; moreover, the feature acquisition can be performed once and shared between different models of different tasks - further accelerating the process. This contributes to energy efficiency and the advancements of embedded AI at the edge.

cs.CV↗

arXiv2021

Higher Order Kernel Mean Embeddings to Capture Filtrations of Stochastic Processes

Stochastic processes are random variables with values in some space of paths. However, reducing a stochastic process to a path-valued random variable ignores its filtration, i.e. the flow of information carried by the process through time. By conditioning the process on its filtration, we introduce a family of higher order kernel mean embeddings (KMEs) that generalizes the notion of KME and captures additional information related to the filtration. We derive empirical estimators for the associated higher order maximum mean discrepancies (MMDs) and prove consistency. We then construct a filtration-sensitive kernel two-sample test able to pick up information that gets missed by the standard MMD test. In addition, leveraging our higher order MMDs we construct a family of universal kernels on stochastic processes that allows to solve real-world calibration and optimal stopping problems in quantitative finance (such as the pricing of American options) via classical kernel-based regression methods. Finally, adapting existing tests for conditional independence to the case of stochastic processes, we design a causal-discovery algorithm to recover the causal graph of structural dependencies among interacting bodies solely from observations of their multidimensional trajectories.

stat.ML↗

arXiv2021

Logsig-RNN: a novel network for robust and efficient skeleton-based action recognition

This paper contributes to the challenge of skeleton-based human action recognition in videos. The key step is to develop a generic network architecture to extract discriminative features for the spatio-temporal skeleton data. In this paper, we propose a novel module, namely Logsig-RNN, which is the combination of the log-signature layer and recurrent type neural networks (RNNs). The former one comes from the mathematically principled technology of signatures and log-signatures as representations for streamed data, which can manage high sample rate streams, non-uniform sampling and time series of variable length. It serves as an enhancement of the recurrent layer, which can be conveniently plugged into neural networks. Besides we propose two path transformation layers to significantly reduce path dimension while retaining the essential information fed into the Logsig-RNN module. Finally, numerical results demonstrate that replacing the RNN module by the Logsig-RNN module in SOTA networks consistently improves the performance on both Chalearn gesture data and NTU RGB+D 120 action data in terms of accuracy and robustness. In particular, we achieve the state-of-the-art accuracy on Chalearn2013 gesture data by combining simple path transformation layers with the Logsig-RNN. Codes are available at https://github.com/steveliao93/GCN_LogsigRNN.

cs.CV↗

arXiv2021

Efficient and Accurate Gradients for Neural SDEs

Neural SDEs combine many of the best qualities of both RNNs and SDEs: memory efficient training, high-capacity function approximation, and strong priors on model space. This makes them a natural choice for modelling many types of temporal dynamics. Training a Neural SDE (either as a VAE or as a GAN) requires backpropagating through an SDE solve. This may be done by solving a backwards-in-time SDE whose solution is the desired parameter gradients. However, this has previously suffered from severe speed and accuracy issues, due to high computational cost and numerical truncation errors. Here, we overcome these issues through several technical innovations. First, we introduce the \textit{reversible Heun method}. This is a new SDE solver that is \textit{algebraically reversible}: eliminating numerical gradient errors, and the first such solver of which we are aware. Moreover it requires half as many function evaluations as comparable solvers, giving up to a $1.98\times$ speedup. Second, we introduce the \textit{Brownian Interval}: a new, fast, memory efficient, and exact way of sampling \textit{and reconstructing} Brownian motion. With this we obtain up to a $10.6\times$ speed improvement over previous techniques, which in contrast are both approximate and relatively slow. Third, when specifically training Neural SDEs as GANs (Kidger et al. 2021), we demonstrate how SDE-GANs may be trained through careful weight clipping and choice of activation function. This reduces computational cost (giving up to a $1.87\times$ speedup) and removes the numerical truncation errors associated with gradient penalty. Altogether, we outperform the state-of-the-art by substantial margins, with respect to training speed, and with respect to classification, prediction, and MMD test metrics. We have contributed implementations of all of our techniques to the torchsde library to help facilitate their adoption.

cs.LG↗

arXiv2021

SigGPDE: Scaling Sparse Gaussian Processes on Sequential Data

Making predictions and quantifying their uncertainty when the input data is sequential is a fundamental learning challenge, recently attracting increasing attention. We develop SigGPDE, a new scalable sparse variational inference framework for Gaussian Processes (GPs) on sequential data. Our contribution is twofold. First, we construct inducing variables underpinning the sparse approximation so that the resulting evidence lower bound (ELBO) does not require any matrix inversion. Second, we show that the gradients of the GP signature kernel are solutions of a hyperbolic partial differential equation (PDE). This theoretical insight allows us to build an efficient back-propagation algorithm to optimize the ELBO. We showcase the significant computational gains of SigGPDE compared to existing methods, while achieving state-of-the-art performance for classification tasks on large datasets of up to 1 million multivariate time series.

stat.ML↗

arXiv2021

Distribution Regression for Sequential Data

Distribution regression refers to the supervised learning problem where labels are only available for groups of inputs instead of individual inputs. In this paper, we develop a rigorous mathematical framework for distribution regression where inputs are complex data streams. Leveraging properties of the expected signature and a recent signature kernel trick for sequential data from stochastic analysis, we introduce two new learning techniques, one feature-based and the other kernel-based. Each is suited to a different data regime in terms of the number of data streams and the dimensionality of the individual streams. We provide theoretical results on the universality of both approaches and demonstrate empirically their robustness to irregularly sampled multivariate time-series, achieving state-of-the-art performance on both synthetic and real-world examples from thermodynamics, mathematical finance and agricultural science.

cs.LG↗

arXiv2021

SK-Tree: a systematic malware detection algorithm on streaming trees via the signature kernel

The development of machine learning algorithms in the cyber security domain has been impeded by the complex, hierarchical, sequential and multimodal nature of the data involved. In this paper we introduce the notion of a streaming tree as a generic data structure encompassing a large portion of real-world cyber security data. Starting from host-based event logs we represent computer processes as streaming trees that evolve in continuous time. Leveraging the properties of the signature kernel, a machine learning tool that recently emerged as a leading technology for learning with complex sequences of data, we develop the SK-Tree algorithm. SK-Tree is a supervised learning method for systematic malware detection on streaming trees that is robust to irregular sampling and high dimensionality of the underlying streams. We demonstrate the effectiveness of SK-Tree to detect malicious events on a portion of the publicly available DARPA OpTC dataset, achieving an AUROC score of 98%.

cs.CR↗

arXiv2021

Areas of areas generate the shuffle algebra

We consider the anti-symmetrization of the half-shuffle on words, which we call the 'area' operator, since it corresponds to taking the signed area of elements of the iterated-integral signature. The tensor algebra is a so-called Tortkara algebra under this operator. We show that the iterated application of the area operator is sufficient to recover the iterated-integral signature of a path. Just as the "information" the second level adds to the first one is known to be equivalent to the area between components of the path, this means that all the information added by subsequent levels is equivalent to iterated areas. On the way to this main result, we characterize (homogeneous) generating sets of the shuffle algebra. We finally discuss compatibility between the area operator and discrete integration and stochastic integration and conclude with some results on the linear span of the areas of areas.

math.RA↗

arXiv2021

General Signature Kernels

Suppose that $γ$ and $σ$ are two continuous bounded variation paths which take values in a finite-dimensional inner product space $V$. Recent papers have introduced the truncated and the untruncated signature kernel of $γ$ and $σ$, and showed how these concepts can be used in classification and prediction tasks involving multivariate time series. In this paper, we introduce general signature kernels and show how they can be interpreted, in many examples, as an average of PDE solutions, and hence how they can be computed using suitable quadrature rules. We extend this analysis to derive closed-form formulae for expressions involving the expected (Stratonovich) signature of Brownian motion. In doing so, we articulate a novel connection between signature kernels and the hyperbolic development map, the latter of which has been a broadly useful tool in the analysis of the signature. As an application we evaluate the use of different general signature kernels as the basis for non-parametric goodness-of-fit tests to Wiener measure on path space.

math.PR↗