arXiv subjects

Benedict Leimkuhler

Publications and source records attributed to Benedict Leimkuhler.

36 records · Page 2Linked to original sources

arXiv2021

Constraint-Based Regularization of Neural Networks

We propose a method for efficiently incorporating constraints into a stochastic gradient Langevin framework for the training of deep neural networks. Constraints allow direct control of the parameter space of the model. Appropriately designed, they reduce the vanishing/exploding gradient problem, control weight magnitudes and stabilize deep neural networks and thus improve the robustness of training algorithms and the generalization capabilities of the trained neural network. We present examples of constrained training methods motivated by orthogonality preservation for weight matrices and explicit weight normalizations. We describe the methods in the overdamped formulation of Langevin dynamics and the underdamped form, in which momenta help to improve sampling efficiency. The methods are explored in test examples in image classification and natural language processing.

cs.LG↗

arXiv2021

Better Training using Weight-Constrained Stochastic Dynamics

We employ constraints to control the parameter space of deep neural networks throughout training. The use of customized, appropriately designed constraints can reduce the vanishing/exploding gradients problem, improve smoothness of classification boundaries, control weight magnitudes and stabilize deep neural networks, and thus enhance the robustness of training algorithms and the generalization capabilities of neural networks. We provide a general approach to efficiently incorporate constraints into a stochastic gradient Langevin framework, allowing enhanced exploration of the loss landscape. We also present specific examples of constrained training methods motivated by orthogonality preservation for weight matrices and explicit weight normalizations. Discretization schemes are provided both for the overdamped formulation of Langevin dynamics and the underdamped form, in which momenta further improve sampling efficiency. These optimization schemes can be used directly, without needing to adapt neural network architecture design choices or to modify the objective with regularization terms, and see performance improvements in classification tasks.

cs.LG↗

arXiv2020

Efficient Numerical Algorithms for the Generalized Langevin Equation

We study the design and implementation of numerical methods to solve the generalized Langevin equation (GLE) focusing on canonical sampling properties of numerical integrators. For this purpose, we cast the GLE in an extended phase space formulation and derive a family of splitting methods which generalize existing Langevin dynamics integration methods. We show exponential convergence in law and the validity of a central limit theorem for the Markov chains obtained via these integration methods, and we show that the dynamics of a suggested integration scheme is consistent with asymptotic limits of the exact dynamics and can reproduce (in the short memory limit) a superconvergence property for the analogous splitting of underdamped Langevin dynamics. We then apply our proposed integration method to several model systems, including a Bayesian inference problem. We demonstrate in numerical experiments that our method outperforms other proposed GLE integration schemes in terms of the accuracy of sampling. Moreover, using a parameterization of the memory kernel in the GLE as proposed by Ceriotti et al [9], our experiments indicate that the obtained GLE-based sampling scheme outperforms state-of-the-art sampling schemes based on underdamped Langevin dynamics in terms of robustness and efficiency.

math.NA↗

arXiv2020

TATi-Thermodynamic Analytics ToolkIt: TensorFlow-based software for posterior sampling in machine learning applications

With the advent of GPU-assisted hardware and maturing high-efficiency software platforms such as TensorFlow and PyTorch, Bayesian posterior sampling for neural networks becomes plausible. In this article we discuss Bayesian parametrization in machine learning based on Markov Chain Monte Carlo methods, specifically discretized stochastic differential equations such as Langevin dynamics and extended system methods in which an ensemble of walkers is employed to enhance sampling. We provide a glimpse of the potential of the sampling-intensive approach by studying (and visualizing) the loss landscape of a neural network applied to the MNIST data set. Moreover, we investigate how the sampling efficiency itself can be significantly enhanced through an ensemble quasi-Newton preconditioning method. This article accompanies the release of a new TensorFlow software package, the Thermodynamic Analytics ToolkIt, which is used in the computational experiments.

math.ST↗

arXiv2020

Covariance-Controlled Adaptive Langevin Thermostat for Large-Scale Bayesian Sampling

Monte Carlo sampling for Bayesian posterior inference is a common approach used in machine learning. The Markov Chain Monte Carlo procedures that are used are often discrete-time analogues of associated stochastic differential equations (SDEs). These SDEs are guaranteed to leave invariant the required posterior distribution. An area of current research addresses the computational benefits of stochastic gradient methods in this setting. Existing techniques rely on estimating the variance or covariance of the subsampling error, and typically assume constant variance. In this article, we propose a covariance-controlled adaptive Langevin thermostat that can effectively dissipate parameter-dependent noise while maintaining a desired target distribution. The proposed method achieves a substantial speedup over popular alternative schemes for large-scale machine learning applications.

stat.ML↗

arXiv2020

Pairwise adaptive thermostats for improved accuracy and stability in dissipative particle dynamics

We examine the formulation and numerical treatment of dissipative particle dynamics (DPD) and momentum-conserving molecular dynamics. We show that it is possible to improve both the accuracy and the stability of DPD by employing a pairwise adaptive Langevin thermostat that precisely matches the dynamical characteristics of DPD simulations (e.g., autocorrelation functions) while automatically correcting thermodynamic averages using a negative feedback loop. In the low friction regime, it is possible to replace DPD by a simpler momentum-conserving variant of the Nosé--Hoover--Langevin method based on thermostatting only pairwise interactions; we show that this method has an extra order of accuracy for an important class of observables (a superconvergence result), while also allowing larger timesteps than alternatives. All the methods mentioned in the article are easily implemented. Numerical experiments are performed in both equilibrium and nonequilibrium settings; using Lees--Edwards boundary conditions to induce shear flow.

physics.comp-ph↗

arXiv2020

Assessing numerical methods for molecular and particle simulation

We discuss the design of state-of-the-art numerical methods for molecular dynamics, focusing on the demands of soft matter simulation, where the purposes include sampling and dynamics calculations both in and out of equilibrium. We discuss the characteristics of different algorithms, including their essential conservation properties, the convergence of averages, and the accuracy of numerical discretizations. Formulations of the equations of motion which are suited to both equilibrium and nonequilibrium simulation include Langevin dynamics, dissipative particle dynamics (DPD), and the more recently proposed "pairwise adaptive Langevin" (PAdL) method, which, like DPD but unlike Langevin dynamics, conserves momentum and better matches the relaxation rate of orientational degrees of freedom. PAdL is easy to code and suitable for a variety of problems in nonequilibrium soft matter modeling, our simulations of polymer melts indicate that this method can also provide dramatic improvements in computational efficiency. Moreover we show that PAdL gives excellent control of the relaxation rate to equilibrium. In the nonequilibrium setting, we further demonstrate that while PAdL allows the recovery of accurate shear viscosities at higher shear rates than are possible using the DPD method at identical timestep, it also outperforms Langevin dynamics in terms of stability and accuracy at higher shear rates.

physics.comp-ph↗

arXiv2020

Partitioned integrators for thermodynamic parameterization of neural networks

Traditionally, neural networks are parameterized using optimization procedures such as stochastic gradient descent, RMSProp and ADAM. These procedures tend to drive the parameters of the network toward a local minimum. In this article, we employ alternative "sampling" algorithms (referred to here as "thermodynamic parameterization methods") which rely on discretized stochastic differential equations for a defined target distribution on parameter space. We show that the thermodynamic perspective already improves neural network training. Moreover, by partitioning the parameters based on natural layer structure we obtain schemes with very rapid convergence for data sets with complicated loss landscapes. We describe easy-to-implement hybrid partitioned numerical algorithms, based on discretized stochastic differential equations, which are adapted to feed-forward neural networks, including a multi-layer Langevin algorithm, AdLaLa (combining the adaptive Langevin and Langevin algorithms) and LOL (combining Langevin and Overdamped Langevin); we examine the convergence of these methods using numerical studies and compare their performance among themselves and in relation to standard alternatives such as stochastic gradient descent and ADAM. We present evidence that thermodynamic parameterization methods can be (i) faster, (ii) more accurate, and (iii) more robust than standard algorithms used within machine learning frameworks.

cs.LG↗

arXiv2019

Simulated Tempering Method in the Infinite Switch Limit with Adaptive Weight Learning

We investigate the theoretical foundations of the simulated tempering method and use our findings to design efficient algorithms. Employing a large deviation argument first used for replica exchange molecular dynamics [Plattner et al., J. Chem. Phys. 135:134111 (2011)], we demonstrate that the most efficient approach to simulated tempering is to vary the temperature infinitely rapidly. In this limit, we can replace the equations of motion for the temperature and physical variables by averaged equations for the latter alone, with the forces rescaled according to a position-dependent function defined in terms of temperature weights. The averaged equations are similar to those used in Gao's integrated-over-temperature method, except that we show that it is better to use a continuous rather than a discrete set of temperatures. We give a theoretical argument for the choice of the temperature weights as the reciprocal partition function, thereby relating simulated tempering to Wang-Landau sampling. Finally, we describe a self-consistent algorithm for simultaneously sampling the canonical ensemble and learning the weights during simulation. This algorithm is tested on a system of harmonic oscillators as well as a continuous variant of the Curie-Weiss model, where it is shown to perform well and to accurately capture the second-order phase transition observed in this model.

physics.chem-ph↗

arXiv2018

Ergodic properties of quasi-Markovian generalized Langevin equations with configuration dependent noise and non-conservative force

We discuss the ergodic properties of quasi-Markovian stochastic differential equations, providing general conditions that ensure existence and uniqueness of a smooth invariant distribution and exponential convergence of the evolution operator in suitably weighted $L^{\infty}$ spaces, which implies the validity of central limit theorem for the respective solution processes. The main new result is an ergodicity condition for the generalized Langevin equation with configuration-dependent noise and (non-)conservative force.

math.PR↗

arXiv2016

Ensemble preconditioning for Markov chain Monte Carlo simulation

We describe parallel Markov chain Monte Carlo methods that propagate a collective ensemble of paths, with local covariance information calculated from neighboring replicas. The use of collective dynamics eliminates multiplicative noise and stabilizes the dynamics thus providing a practical approach to difficult anisotropic sampling problems in high dimensions. Numerical experiments with model problems demonstrate that dramatic potential speedups, compared to various alternative schemes, are attainable.

stat.ME↗

arXiv2016

Adaptive Thermostats for Noisy Gradient Systems

We study numerical methods for sampling probability measures in high dimension where the underlying model is only approximately identified with a gradient system. Extended stochastic dynamical methods are discussed which have application to multiscale models, nonequilibrium molecular dynamics, and Bayesian sampling techniques arising in emerging machine learning applications. In addition to providing a more comprehensive discussion of the foundations of these methods, we propose a new numerical method for the adaptive Langevin/stochastic gradient Nosé--Hoover thermostat that achieves a dramatic improvement in numerical efficiency over the most popular stochastic gradient methods reported in the literature. We also demonstrate that the newly established method inherits a superconvergence property (fourth order convergence to the invariant measure for configurational quantities) recently demonstrated in the setting of Langevin dynamics. Our findings are verified by numerical experiments.

math.NA↗

arXiv2015

The computation of averages from equilibrium and nonequilibrium Langevin molecular dynamics

We consider numerical methods for thermodynamic sampling, i.e. computing sequences of points distributed according to the Gibbs-Boltzmann distribution, using Langevin dynamics and overdamped Langevin dynamics (Brownian dynamics). A wide variety of numerical methods for Langevin dynamics may be constructed based on splitting the stochastic differential equations into various component parts, each of which may be propagated exactly in the sense of distributions. Each such method may be viewed as generating samples according to an associated invariant measure that differs from the exact canonical invariant measure by a stepsize-dependent perturbation. We provide error estimates a la Talay-Tubaro on the invariant distribution for small stepsize, and compare the sampling bias obtained for various choices of splitting method. We further investigate the overdamped limit and apply the methods in the context of driven systems where the goal is sampling with respect to a nonequilibrium steady state. Our analyses are illustrated by numerical experiments.

cond-mat.stat-mech↗

arXiv2014

Direct control of the small-scale energy balance in 2D fluid dynamics

We explore the direct modification of the pseudo-spectral truncation of 2D, incompressible fluid dynamics to maintain a prescribed kinetic energy spectrum. The method provides a means of simulating fluid states with defined spectral properties, for the purpose of matching simulation statistics to given information, arising from observations, theoretical prediction or high fidelity simulation. In the scheme outlined here, Nosé-Hoover thermostats, commonly used in molecular dynamics, are introduced as feedback controls applied to energy shells of the Fourier-discretized Navier-Stokes equations. As we demonstrate in numerical experiments, the dynamical properties (quantified using autocorrelation functions) are only modestly perturbed by our device, while ensemble dispersion is significantly enhanced in comparison with simulations of a corresponding truncation incorporating hyperviscosity.

physics.flu-dyn↗

arXiv2014

Least-biased correction of extended dynamical systems using observational data

We consider dynamical systems evolving near an equilibrium statistical state where the interest is in modelling long term behavior that is consistent with thermodynamic constraints. We adjust the distribution using an entropy-optimizing formulation that can be computed on-the- fly, making possible partial corrections using incomplete information, for example measured data or data computed from a different model (or the same model at a different scale). We employ a thermostatting technique to sample the target distribution with the aim of capturing relavant statistical features while introducing mild dynamical perturbation (thermostats). The method is tested for a point vortex fluid model on the sphere, and we demonstrate both convergence of equilibrium quantities and the ability of the formulation to balance stationary and transient- regime errors.

physics.flu-dyn↗

arXiv2014

On the numerical treatment of dissipative particle dynamics and related systems

We review and compare numerical methods that simultaneously control temperature while preserving the momentum, a family of particle simulation methods commonly used for the modelling of complex fluids and polymers. The class of methods considered includes dissipative particle dynamics (DPD) as well as extended stochastic-dynamics models incorporating a generalized pairwise thermostat scheme in which stochastic forces are eliminated and the coefficient of dissipation is treated as an additional auxiliary variable subject to a feedback (kinetic energy) control mechanism. In the latter case, we consider the addition of a coupling of the auxiliary variable, as in the Nosé-Hoover-Langevin (NHL) method, with stochastic dynamics to ensure ergodicity, and find that the convergence of ensemble averages is substantially improved. To this end, splitting methods are developed and studied in terms of their thermodynamic accuracy, two-point correlation functions, and convergence. In terms of computational efficiency as measured by the ratio of thermodynamic accuracy to CPU time, we report significant advantages in simulation for the pairwise NHL method compared to popular alternative schemes (up to an 80\% improvement), without degradation of convergence rate. The momentum-conserving thermostat technique described here provides a consistent hydrodynamic model in the low-friction regime, but it will also be of use in both equilibrium and nonequilibrium molecular simulation applications owing to its efficiency and simple numerical implementation.

physics.comp-ph↗

arXiv2013

Robust and efficient configurational molecular sampling via Langevin Dynamics

A wide variety of numerical methods are evaluated and compared for solving the stochastic differential equations encountered in molecular dynamics. The methods are based on the application of deterministic impulses, drifts, and Brownian motions in some combination. The Baker-Campbell-Hausdorff expansion is used to study sampling accuracy following recent work by the authors, which allows determination of the stepsize-dependent bias in configurational averaging. For harmonic oscillators, configurational averaging is exact for certain schemes, which may result in improved performance in the modelling of biomolecules where bond stretches play a prominent role. For general systems, an optimal method can be identified that has very low bias compared to alternatives. In simulations of the alanine dipeptide reported here (both solvated and unsolvated), higher accuracy is obtained without loss of computational efficiency, while allowing large timestep, and with no impairment of the conformational exploration rate (the effective diffusion rate observed in simulation). The optimal scheme is a uniformly better performing algorithm for molecular sampling, with overall efficiency improvements of 25% or more in practical timestep size achievable in vacuum, and with reductions in the error of configurational averages of a factor of ten or more attainable in solvated simulations at large timestep.

physics.comp-ph↗

arXiv2012

Rational Construction of Stochastic Numerical Methods for Molecular Sampling

In this article, we focus on the sampling of the configurational Gibbs-Boltzmann distribution, that is, the calculation of averages of functions of the position coordinates of a molecular $N$-body system modelled at constant temperature. We show how a formal series expansion of the invariant measure of a Langevin dynamics numerical method can be obtained in a straightforward way using the Baker-Campbell-Hausdorff lemma. We then compare Langevin dynamics integrators in terms of their invariant distributions and demonstrate a superconvergence property (4th order accuracy where only 2nd order would be expected) of one method in the high friction limit; this method, moreover, can be reduced to a simple modification of the Euler-Maruyama method for Brownian dynamics involving a non-Markovian (coloured noise) random process. In the Brownian dynamics case, 2nd order accuracy of the invariant density is achieved. All methods considered are efficient for molecular applications (requiring one force evaluation per timestep) and of a simple form. In fully resolved (long run) molecular dynamics simulations, for our favoured method, we observe up to two orders of magnitude improvement in configurational sampling accuracy for given stepsize with no evident reduction in the size of the largest usable timestep compared to common alternative methods.

math.NA↗