Searcharxiv⌕ Search

arXiv subjects

Sotirios Sabanis

Publications and source records attributed to Sotirios Sabanis.

At least 37 records · Page 2Linked to original sources
arXiv2023

Non-asymptotic estimates for TUSLA algorithm for non-convex learning with applications to neural networks with ReLU activation function

We consider non-convex stochastic optimization problems where the objective functions have super-linearly growing and discontinuous stochastic gradients. In such a setting, we provide a non-asymptotic analysis for the tamed unadjusted stochastic Langevin algorithm (TUSLA) introduced in Lovas et al. (2020). In particular, we establish non-asymptotic error bounds for the TUSLA algorithm in Wasserstein-1 and Wasserstein-2 distances. The latter result enables us to further derive non-asymptotic estimates for the expected excess risk. To illustrate the applicability of the main results, we consider an example from transfer learning with ReLU neural networks, which represents a key paradigm in machine learning. Numerical experiments are presented for the aforementioned example which support our theoretical findings. Hence, in this setting, we demonstrate both theoretically and numerically that the TUSLA algorithm can solve the optimization problem involving neural networks with ReLU activation function. Besides, we provide simulation results for synthetic examples where popular algorithms, e.g. ADAM, AMSGrad, RMSProp, and (vanilla) stochastic gradient descent (SGD) algorithm, may fail to find the minimizer of the objective functions due to the super-linear growth and the discontinuity of the corresponding stochastic gradient, while the TUSLA algorithm converges rapidly to the optimal solution. Moreover, we provide an empirical comparison of the performance of TUSLA with popular stochastic optimizers on real-world datasets, as well as investigate the effect of the key hyperparameters of TUSLA on its performance.

math.OC↗
arXiv2023

Kinetic Langevin MCMC Sampling Without Gradient Lipschitz Continuity -- the Strongly Convex Case

In this article we consider sampling from log concave distributions in Hamiltonian setting, without assuming that the objective gradient is globally Lipschitz. We propose two algorithms based on monotone polygonal (tamed) Euler schemes, to sample from a target measure, and provide non-asymptotic 2-Wasserstein distance bounds between the law of the process of each algorithm and the target measure. Finally, we apply these results to bound the excess risk optimization error of the associated optimization problem.

math.PR↗
arXiv2023

Taming neural networks with TUSLA: Non-convex learning via adaptive stochastic gradient Langevin algorithms

Artificial neural networks (ANNs) are typically highly nonlinear systems which are finely tuned via the optimization of their associated, non-convex loss functions. In many cases, the gradient of any such loss function has superlinear growth, making the use of the widely-accepted (stochastic) gradient descent methods, which are based on Euler numerical schemes, problematic. We offer a new learning algorithm based on an appropriately constructed variant of the popular stochastic gradient Langevin dynamics (SGLD), which is called tamed unadjusted stochastic Langevin algorithm (TUSLA). We also provide a nonasymptotic analysis of the new algorithm's convergence properties in the context of non-convex learning problems with the use of ANNs. Thus, we provide finite-time guarantees for TUSLA to find approximate minimizers of both empirical and population risks. The roots of the TUSLA algorithm are based on the taming technology for diffusion processes with superlinear coefficients as developed in \citet{tamed-euler, SabanisAoAP} and for MCMC algorithms in \citet{tula}. Numerical experiments are presented which confirm the theoretical findings and illustrate the need for the use of the new algorithm in comparison to vanilla SGLD within the framework of ANNs.

cs.LG↗
arXiv2022

A Strongly Monotonic Polygonal Euler Scheme

In recent years tamed schemes have become an important technique for simulating SDEs and SPDEs whose continuous coefficients display superlinear growth. The taming method, which involves curbing the growth of the coefficients as a function of stepsize, has so far however not been adapted to preserve the monotonicity of the coefficients. This has arisen as an issue particularly in \cite{articletam}, where the lack of a strongly monotonic tamed scheme forces strong conditions on the setting. In the present work we give a novel and explicit method for truncating monotonic functions in separable Hilbert spaces, and show how this can be used to define a polygonal (tamed) Euler scheme on finite dimensional space, preserving the monotonicity of the drift coefficient. This new method of truncation is well-defined with almost no assumptions and, unlike the well-known Moreau-Yosida regularisation, does not require an optimisation problem to be solved at each evaluation. Our construction is the first infinite dimensional method for truncating monotone functions that we are aware of, as well as the first explicit method in any number of dimensions.

math.PR↗
arXiv2022

Nonasymptotic estimates for Stochastic Gradient Langevin Dynamics under local conditions in nonconvex optimization

In this paper, we are concerned with a non-asymptotic analysis of sampling algorithms used in nonconvex optimization. In particular, we obtain non-asymptotic estimates in Wasserstein-1 and Wasserstein-2 distances for a popular class of algorithms called Stochastic Gradient Langevin Dynamics (SGLD). In addition, the aforementioned Wasserstein-2 convergence result can be applied to establish a non-asymptotic error bound for the expected excess risk. Crucially, these results are obtained under a local Lipschitz condition and a local dissipativity condition where we remove the uniform dependence in the data stream. We illustrate the importance of this relaxation by presenting examples from variational inference and from index tracking optimization.

math.ST↗
arXiv2022

Existence, uniqueness and approximation of solutions of SDEs with superlinear coefficients in the presence of discontinuities of the drift coefficient

Existence, uniqueness, and $L_p$-approximation results are presented for scalar stochastic differential equations (SDEs) by considering the case where, the drift coefficient has finitely many spatial discontinuities while both coefficients can grow superlinearly (in the space variable). These discontinuities are described by a piecewise local Lipschitz continuity and a piecewise monotone-type condition while the diffusion coefficient is assumed to be locally Lipschitz continuous and non-degenerate at the discontinuity points of the drift coefficient. Moreover, the superlinear nature of the coefficients is dictated by a suitable coercivity condition and a polynomial growth of the (local) Lipschitz constants of the coefficients. Existence and uniqueness of strong solutions of such SDEs are obtained. Furthermore, the classical $L_p$-error rate $1/2$, for a suitable range of values of $p$, is recovered for a tamed Euler scheme which is used for approximating these solutions. To the best of the authors' knowledge, these are the first existence, uniqueness and approximation results for this class of SDEs.

math.PR↗
arXiv2021

Statistical Finite Elements via Langevin Dynamics

The recent statistical finite element method (statFEM) provides a coherent statistical framework to synthesise finite element models with observed data. Through embedding uncertainty inside of the governing equations, finite element solutions are updated to give a posterior distribution which quantifies all sources of uncertainty associated with the model. However to incorporate all sources of uncertainty, one must integrate over the uncertainty associated with the model parameters, the known forward problem of uncertainty quantification. In this paper, we make use of Langevin dynamics to solve the statFEM forward problem, studying the utility of the unadjusted Langevin algorithm (ULA), a Metropolis-free Markov chain Monte Carlo sampler, to build a sample-based characterisation of this otherwise intractable measure. Due to the structure of the statFEM problem, these methods are able to solve the forward problem without explicit full PDE solves, requiring only sparse matrix-vector products. ULA is also gradient-based, and hence provides a scalable approach up to high degrees-of-freedom. Leveraging the theory behind Langevin-based samplers, we provide theoretical guarantees on sampler performance, demonstrating convergence, for both the prior and posterior, in the Kullback-Leibler divergence, and, in Wasserstein-2, with further results on the effect of preconditioning. Numerical experiments are also provided, for both the prior and posterior, to demonstrate the efficacy of the sampler, with a Python package also included.

stat.CO↗
arXiv2021

On stochastic gradient Langevin dynamics with dependent data streams: the fully non-convex case

We consider the problem of sampling from a target distribution, which is \emph {not necessarily logconcave}, in the context of empirical risk minimization and stochastic optimization as presented in Raginsky et al. (2017). Non-asymptotic analysis results are established in the $L^1$-Wasserstein distance for the behaviour of Stochastic Gradient Langevin Dynamics (SGLD) algorithms. We allow the estimation of gradients to be performed even in the presence of \emph{dependent} data streams. Our convergence estimates are sharper and \emph{uniform} in the number of iterations, in contrast to those in previous studies.

math.ST↗
arXiv2020

Model-Independent Price Bounds for Catastrophic Mortality Bonds

In this paper, we are concerned with the valuation of Catastrophic Mortality Bonds and, in particular, we examine the case of the Swiss Re Mortality Bond 2003 as a primary example of this class of assets. This bond was the first Catastrophic Mortality Bond to be launched in the market and encapsulates the behaviour of a well-defined mortality index to generate payoffs for bondholders. Pricing these type of bonds is a challenging task and no closed form solution exists in the literature. In our approach, we express the payoff of such a bond in terms of the payoff of an Asian put option and present a new approach to derive model-independent bounds exploiting comonotonic theory as illustrated in \cite{prime1}, \cite{2} and \cite{Simon} for the pricing of Asian options. We carry out Monte Carlo simulations to estimate the bond price and illustrate the quality of the bounds.

q-fin.PR↗
arXiv2020

A fully data-driven approach to minimizing CVaR for portfolio of assets via SGLD with discontinuous updating

A new approach in stochastic optimization via the use of stochastic gradient Langevin dynamics (SGLD) algorithms, which is a variant of stochastic gradient decent (SGD) methods, allows us to efficiently approximate global minimizers of possibly complicated, high-dimensional landscapes. With this in mind, we extend here the non-asymptotic analysis of SGLD to the case of discontinuous stochastic gradients. We are thus able to provide theoretical guarantees for the algorithm's convergence in (standard) Wasserstein distances for both convex and non-convex objective functions. We also provide explicit upper estimates of the expected excess risk associated with the approximation of global minimizers of these objective functions. All these findings allow us to devise and present a fully data-driven approach for the optimal allocation of weights for the minimization of CVaR of portfolio of assets with complete theoretical guarantees for its performance. Numerical results illustrate our main findings.

q-fin.PM↗
arXiv2019

Higher Order Langevin Monte Carlo Algorithm

A new (unadjusted) Langevin Monte Carlo (LMC) algorithm with improved rates in total variation and in Wasserstein distance is presented. All these are obtained in the context of sampling from a target distribution $π$ that has a density $\hatπ$ on $\mathbb{R}^d$ known up to a normalizing constant. Moreover, $-\log \hatπ$ is assumed to have a locally Lipschitz gradient and its third derivative is locally Hölder continuous with exponent $β\in (0,1]$. Non-asymptotic bounds are obtained for the convergence to stationarity of the new sampling method with convergence rate $1+ β/2$ in Wasserstein distance, while it is shown that the rate is 1 in total variation even in the absence of convexity. Finally, in the case where $-\log \hatπ$ is strongly convex and its gradient is Lipschitz continuous, explicit constants are provided.

math.ST↗
arXiv2019

Optimising portfolio diversification and dimensionality

A new framework for portfolio diversification is introduced which goes beyond the classical mean-variance approach and portfolio allocation strategies such as risk parity. It is based on a novel concept called portfolio dimensionality that connects diversification to the non-Gaussianity of portfolio returns and can typically be defined in terms of the ratio of risk measures which are homogenous functions of equal degree. The latter arises naturally due to our requirement that diversification measures should be leverage invariant. We introduce this new framework and argue the benefits relative to existing measures of diversification in the literature, before addressing the question of optimizing diversification or, equivalently, dimensionality. Maximising portfolio dimensionality leads to highly non-trivial optimization problems with objective functions which are typically non-convex and potentially have multiple local optima. Two complementary global optimization algorithms are thus presented. For problems of moderate size and more akin to asset allocation problems, a deterministic Branch and Bound algorithm is developed, whereas for problems of larger size a stochastic global optimization algorithm based on Gradient Langevin Dynamics is given. We demonstrate analytically and through numerical experiments that the framework reflects the desired properties often discussed in the literature.

q-fin.PM↗
arXiv2018

The Tamed Unadjusted Langevin Algorithm

In this article, we consider the problem of sampling from a probability measure $π$ having a density on $\mathbb{R}^d$ known up to a normalizing constant, $x\mapsto \mathrm{e}^{-U(x)} / \int_{\mathbb{R}^d} \mathrm{e}^{-U(y)} \mathrm{d} y$. The Euler discretization of the Langevin stochastic differential equation (SDE) is known to be unstable in a precise sense, when the potential $U$ is superlinear, i.e. $\liminf_{\Vert x \Vert\to+\infty} \Vert \nabla U(x) \Vert / \Vert x \Vert = +\infty$. Based on previous works on the taming of superlinear drift coefficients for SDEs, we introduce the Tamed Unadjusted Langevin Algorithm (TULA) and obtain non-asymptotic bounds in $V$-total variation norm and Wasserstein distance of order $2$ between the iterates of TULA and $π$, as well as weak error bounds. Numerical experiments are presented which support our findings.

stat.ME↗
arXiv2018

On explicit order 1.5 approximations with varying coefficients: the case of super-linear diffusion coefficients

A conjecture appears in \cite{milsteinscheme}, in the form of a remark, where it is stated that it is possible to construct, in a specified way, any high order explicit numerical schemes to approximate the solutions of SDEs with superlinear coefficients. We answer this conjecture affirmatively for the case of order 1.5 approximations and show that the suggested methodology works. Moreover, we explore the case of having Hölder continuous derivatives for the diffusion coefficients.

math.PR↗
arXiv2017

General Price Bounds for Guaranteed Annuity Options

In this paper, we are concerned with the valuation of Guaranteed Annuity Options (GAOs) under the most generalised modelling framework where both interest and mortality rates are stochastic and correlated. Pricing these type of options in the correlated environment is a challenging task and no closed form solution exists in the literature. We employ the use of doubly stochastic stopping times to incorporate the randomness about the time of death and employ a suitable change of measure to facilitate the valuation of survival benefit, there by adapting the payoff of the GAO in terms of the payoff of a basket call option. We derive general price bounds for GAOs by utilizing a conditioning approach for the lower bound and arithmetic-geometric mean inequality for the upper bound. The theory is then applied to affine models to present some very interesting formulae for the bounds under the affine set up. Numerical examples are furnished and benchmarked against Monte Carlo simulations to estimate the price of a GAO for a variety of affine processes governing the evolution of mortality and the interest rate.

q-fin.PR↗
arXiv2016

On Explicit Approximations for Lévy Driven SDEs with Super-linear Diffusion Coefficients

Motivated by the results of \cite{sabanis2015}, we propose explicit Euler-type schemes for SDEs with random coefficients driven by Lévy noise when the drift and diffusion coefficients can grow super-linearly. As an application of our results, one can construct explicit Euler-type schemes for SDEs with delays (SDDEs) which are driven by Lévy noise and have super-linear coefficients. Strong convergence results are established and their rate of convergence is shown to be equal to that of the classical Euler scheme. It is proved that the optimal rate of convergence is achieved for $\mathcal{L}^2$-convergence which is consistent with the corresponding results available in the literature.

math.PR↗
arXiv2016

Euler approximations with varying coefficients: The case of superlinearly growing diffusion coefficients

A new class of explicit Euler schemes, which approximate stochastic differential equations (SDEs) with superlinearly growing drift and diffusion coefficients, is proposed in this article. It is shown, under very mild conditions, that these explicit schemes converge in probability and in $\mathcal{L}^p$ to the solution of the corresponding SDEs. Moreover, rate of convergence estimates are provided for $\mathcal{L}^p$ and almost sure convergence. In particular, the strong order $1/2$ is recovered in the case of uniform $\mathcal{L}^p$-convergence.

math.PR↗