arXiv subjects

Eric Vanden-Eijnden

Publications and source records attributed to Eric Vanden-Eijnden.

At least 73 records · Page 4Linked to original sources

arXiv2021

Sharp Asymptotic Estimates for Expectations, Probabilities, and Mean First Passage Times in Stochastic Systems with Small Noise

Freidlin-Wentzell theory of large deviations can be used to compute the likelihood of extreme or rare events in stochastic dynamical systems via the solution of an optimization problem. The approach gives exponential estimates that often need to be refined via calculation of a prefactor. Here it is shown how to perform these computations in practice. Specifically, sharp asymptotic estimates are derived for expectations, probabilities, and mean first passage times in a form that is geared towards numerical purposes: they require solving well-posed matrix Riccati equations involving the minimizer of the Freidlin-Wentzell action as input, either forward or backward in time with appropriate initial or final conditions tailored to the estimate at hand. The usefulness of our approach is illustrated on several examples. In particular, invariant measure probabilities and mean first passage times are calculated in models involving stochastic partial differential equations of reaction-advection-diffusion type.

cond-mat.stat-mech↗

arXiv2021

Efficient Bayesian Sampling Using Normalizing Flows to Assist Markov Chain Monte Carlo Methods

Normalizing flows can generate complex target distributions and thus show promise in many applications in Bayesian statistics as an alternative or complement to MCMC for sampling posteriors. Since no data set from the target posterior distribution is available beforehand, the flow is typically trained using the reverse Kullback-Leibler (KL) divergence that only requires samples from a base distribution. This strategy may perform poorly when the posterior is complicated and hard to sample with an untrained normalizing flow. Here we explore a distinct training strategy, using the direct KL divergence as loss, in which samples from the posterior are generated by (i) assisting a local MCMC algorithm on the posterior with a normalizing flow to accelerate its mixing rate and (ii) using the data generated this way to train the flow. The method only requires a limited amount of \textit{a~priori} input about the posterior, and can be used to estimate the evidence required for model validation, as we illustrate on examples.

stat.ML↗

arXiv2021

On Energy-Based Models with Overparametrized Shallow Neural Networks

Energy-based models (EBMs) are a simple yet powerful framework for generative modeling. They are based on a trainable energy function which defines an associated Gibbs measure, and they can be trained and sampled from via well-established statistical tools, such as MCMC. Neural networks may be used as energy function approximators, providing both a rich class of expressive models as well as a flexible device to incorporate data structure. In this work we focus on shallow neural networks. Building from the incipient theory of overparametrized neural networks, we show that models trained in the so-called "active" regime provide a statistical advantage over their associated "lazy" or kernel regime, leading to improved adaptivity to hidden low-dimensional structure in the data distribution, as already observed in supervised learning. Our study covers both maximum likelihood and Stein Discrepancy estimators, and we validate our theoretical results with numerical experiments on synthetic data.

cs.LG↗

arXiv2021

Active Importance Sampling for Variational Objectives Dominated by Rare Events: Consequences for Optimization and Generalization

Deep neural networks, when optimized with sufficient data, provide accurate representations of high-dimensional functions; in contrast, function approximation techniques that have predominated in scientific computing do not scale well with dimensionality. As a result, many high-dimensional sampling and approximation problems once thought intractable are being revisited through the lens of machine learning. While the promise of unparalleled accuracy may suggest a renaissance for applications that require parameterizing representations of complex systems, in many applications gathering sufficient data to develop such a representation remains a significant challenge. Here we introduce an approach that combines rare events sampling techniques with neural network optimization to optimize objective functions that are dominated by rare events. We show that importance sampling reduces the asymptotic variance of the solution to a learning problem, suggesting benefits for generalization. We study our algorithm in the context of learning dynamical transition pathways between two states of a system, a problem with applications in statistical physics and implications in machine learning theory. Our numerical experiments demonstrate that we can successfully learn even with the compounding difficulties of high-dimension and rare data.

physics.data-an↗

arXiv2020

Optimization and Generalization of Shallow Neural Networks with Quadratic Activation Functions

We study the dynamics of optimization and the generalization properties of one-hidden layer neural networks with quadratic activation function in the over-parametrized regime where the layer width $m$ is larger than the input dimension $d$. We consider a teacher-student scenario where the teacher has the same structure as the student with a hidden layer of smaller width $m^*\le m$. We describe how the empirical loss landscape is affected by the number $n$ of data samples and the width $m^*$ of the teacher network. In particular we determine how the probability that there be no spurious minima on the empirical loss depends on $n$, $d$, and $m^*$, thereby establishing conditions under which the neural network can in principle recover the teacher. We also show that under the same conditions gradient descent dynamics on the empirical loss converges and leads to small generalization error, i.e. it enables recovery in practice. Finally we characterize the time-convergence rate of gradient descent in the limit of a large number of samples. These results are confirmed by numerical experiments.

cs.LG↗

arXiv2019

Experimental Evidence of Hydrodynamic Instantons: The Universal Route to Rogue Waves

A statistical theory of rogue waves is proposed and tested against experimental data collected in a long water tank where random waves with different degrees of nonlinearity are mechanically generated and free to propagate along the flume. Strong evidence is given that the rogue waves observed in the tank are hydrodynamic instantons, that is, saddle point configurations of the action associated with the stochastic model of the wave system. As shown here, these hydrodynamic instantons are complex spatio-temporal wave field configurations, which can be defined using the mathematical framework of Large Deviation Theory and calculated via tailored numerical methods. These results indicate that the instantons describe equally well rogue waves that originate from a simple linear superposition mechanism (in weakly nonlinear conditions) or from a nonlinear focusing one (in strongly nonlinear conditions), paving the way for the development of a unified explanation to rogue wave formation.

physics.flu-dyn↗

arXiv2019

Infinite Switch Simulated Tempering in Force (FISST)

Many proteins in cells are capable of sensing and responding to piconewton scale forces, a regime in which conformational changes are small but significant for biological processes. In order to efficiently and effectively sample the response of these proteins to small forces, enhanced sampling techniques will be required. In this work, we derive, implement, and evaluate an efficient method to simultaneously sample the result of applying any constant pulling force within a specified range to a molecular system of interest. We start from Simulated Tempering in Force, whereby force is applied as a linear bias on a collective variable to the system's Hamiltonian, and the coefficient is taken as a continuous auxiliary degree of freedom. We derive a formula for an average collective-variable-dependent force, which depends on a set of weights, learned on-the-fly throughout a simulation, that reflect the limit where force varies infinitely quickly. These weights can then be used to retroactively compute averages of any observable at any force within the specified range. This technique is based on recent work deriving similar equations for Infinite Switch Simulated Tempering in Temperature, that showed the infinite switch limit is the most efficient for sampling. Here, we demonstrate that our method accurately and simultaneously samples molecular systems at all forces within a user defined force range, and show how it can serve as an enhanced sampling tool for cases where the pulling direction destabilizes states of low free-energy at zero-force. This method is implemented in, and will be freely-distributed with, the PLUMED open-source sampling library, and hence can be readily applied to problems using a wide range of molecular dynamics software packages.

cond-mat.stat-mech↗

arXiv2019

Trainability and Accuracy of Neural Networks: An Interacting Particle System Approach

Neural networks, a central tool in machine learning, have demonstrated remarkable, high fidelity performance on image recognition and classification tasks. These successes evince an ability to accurately represent high dimensional functions, but rigorous results about the approximation error of neural networks after training are few. Here we establish conditions for global convergence of the standard optimization algorithm used in machine learning applications, stochastic gradient descent (SGD), and quantify the scaling of its error with the size of the network. This is done by reinterpreting SGD as the evolution of a particle system with interactions governed by a potential related to the objective or "loss" function used to train the network. We show that, when the number $n$ of units is large, the empirical distribution of the particles descends on a convex landscape towards the global minimum at a rate independent of $n$, with a resulting approximation error that universally scales as $O(n^{-1})$. These properties are established in the form of a Law of Large Numbers and a Central Limit Theorem for the empirical distribution. Our analysis also quantifies the scale and nature of the noise introduced by SGD and provides guidelines for the step size and batch size to use when training a neural network. We illustrate our findings on examples in which we train neural networks to learn the energy function of the continuous 3-spin model on the sphere. The approximation error scales as our analysis predicts in as high a dimension as $d=25$.

stat.ML↗

arXiv2019

Global convergence of neuron birth-death dynamics

Neural networks with a large number of parameters admit a mean-field description, which has recently served as a theoretical explanation for the favorable training properties of "overparameterized" models. In this regime, gradient descent obeys a deterministic partial differential equation (PDE) that converges to a globally optimal solution for networks with a single hidden layer under appropriate assumptions. In this work, we propose a non-local mass transport dynamics that leads to a modified PDE with the same minimizer. We implement this non-local dynamics as a stochastic neuronal birth-death process and we prove that it accelerates the rate of convergence in the mean-field limit. We subsequently realize this PDE with two classes of numerical schemes that converge to the mean-field equation, each of which can easily be implemented for neural networks with finite numbers of parameters. We illustrate our algorithms with two models to provide intuition for the mechanism through which convergence is accelerated.

stat.ML↗

arXiv2019

Simulated Tempering Method in the Infinite Switch Limit with Adaptive Weight Learning

We investigate the theoretical foundations of the simulated tempering method and use our findings to design efficient algorithms. Employing a large deviation argument first used for replica exchange molecular dynamics [Plattner et al., J. Chem. Phys. 135:134111 (2011)], we demonstrate that the most efficient approach to simulated tempering is to vary the temperature infinitely rapidly. In this limit, we can replace the equations of motion for the temperature and physical variables by averaged equations for the latter alone, with the forces rescaled according to a position-dependent function defined in terms of temperature weights. The averaged equations are similar to those used in Gao's integrated-over-temperature method, except that we show that it is better to use a continuous rather than a discrete set of temperatures. We give a theoretical argument for the choice of the temperature weights as the reciprocal partition function, thereby relating simulated tempering to Wang-Landau sampling. Finally, we describe a self-consistent algorithm for simultaneously sampling the canonical ensemble and learning the weights during simulation. This algorithm is tested on a system of harmonic oscillators as well as a continuous variant of the Curie-Weiss model, where it is shown to perform well and to accurately capture the second-order phase transition observed in this model.

physics.chem-ph↗

arXiv2018

Numerical computation of rare events via large deviation theory

An overview of rare events algorithms based on large deviation theory (LDT) is presented. It covers a range of numerical schemes to compute the large deviation minimizer in various setups, and discusses best practices, common pitfalls, and implementation trade-offs. Generalizations, extensions, and improvements of the minimum action methods are proposed. These algorithms are tested on example problems which illustrate several common difficulties which arise e.g. when the forcing is degenerate or multiplicative, or the systems are infinite-dimensional. Generalizations to processes driven by non-Gaussian noises or random initial data and parameters are also discussed, along with the connection between the LDT-based approach reviewed here and other methods, such as stochastic field theory and optimal control. Finally, the integration of this approach in importance sampling methods using e.g. genealogical algorithms is explored.

cond-mat.stat-mech↗

arXiv2018

Extreme event quantification in dynamical systems with random components

A central problem in uncertainty quantification is how to characterize the impact that our incomplete knowledge about models has on the predictions we make from them. This question naturally lends itself to a probabilistic formulation, by making the unknown model parameters random with given statistics. Here this approach is used in concert with tools from large deviation theory (LDT) and optimal control to estimate the probability that some observables in a dynamical system go above a large threshold after some time, given the prior statistical information about the system's parameters and/or its initial conditions. Specifically, it is established under which conditions such extreme events occur in a predictable way, as the minimizer of the LDT action functional. It is also shown how this minimization can be numerically performed in an efficient way using tools from optimal control. These findings are illustrated on the examples of a rod with random elasticity pulled by a time-dependent force, and the nonlinear Schrödinger equation (NLSE) with random initial conditions.

cond-mat.stat-mech↗

arXiv2018

Longtime convergence of the Temperature-Accelerated Molecular Dynamics Method

The equations of the temperature-accelerated molecular dynamics (TAMD) method for the calculations of free energies and partition functions are analyzed. Specifically, the exponential convergence of the law of these stochastic processes is established, with a convergence rate close to the one of the limiting, effective dynamics at higher temperature obtained with infinite acceleration. It is also shown that the invariant measures of TAMD are close to a known reference measure, with an error that can be quantified precisely. Finally, a Central Limit Theorem is proven, which allows the estimation of errors on properties calculated by ergodic time averages. These results not only demonstrate the usefulness and validity range of the TAMD equations, but they also permit in principle to adjust the parameter in these equations to optimize their efficiency.

math.PR↗

arXiv2018

Rogue Waves and Large Deviations in Deep Sea

The appearance of rogue waves in deep sea is investigated using the modified nonlinear Schrödinger (MNLS) equation in one spatial-dimension with random initial conditions that are assumed to be normally distributed, with a spectrum approximating realistic conditions of a uni-directional sea state. It is shown that one can use the incomplete information contained in this spectrum as prior and supplement this information with the MNLS dynamics to reliably estimate the probability distribution of the sea surface elevation far in the tail at later times. Our results indicate that rogue waves occur when the system hits unlikely pockets of wave configurations that trigger large disturbances of the surface height. The rogue wave precursors in these pockets are wave patterns of regular height but with a very specific shape that is identified explicitly, thereby allowing for early detection. The method proposed here combines Monte Carlo sampling with tools from large deviations theory that reduce the calculation of the most likely rogue wave precursors to an optimization problem that can be solved efficiently. This approach is transferable to other problems in which the system's governing equations contain random initial conditions and/or parameters.

physics.flu-dyn↗

arXiv2017

Methodological and computational aspects of parallel tempering methods in the infinite swapping limit

A variant of the parallel tempering method is proposed in terms of a stochastic switching process for the coupled dynamics of replica configuration and temperature permutation. This formulation is shown to facilitate the analysis of the convergence properties of parallel tempering by large deviation theory, which indicates that the method should be operated in the infinite swapping limit to maximize sampling efficiency. The effective equation for the replica alone that arises in this infinite swapping limit simply involves replacing the original potential by a mixture potential. The analysis of the geometric properties of this potential offers a new perspective on the issues of how to choose of temperature ladder, and why many temperatures should typically be introduced to boost the sampling efficiency. It is also shown how to simulate the effective equation in this many temperature regime using multiscale integrators. Finally, similar ideas are also used to discuss extensions of the infinite swapping limits to the technique of simulated tempering.

physics.chem-ph↗

arXiv2017

Long Term Effects of Small Random Perturbations on Dynamical Systems: Theoretical and Computational Tools

Small random perturbations may have a dramatic impact on the long time evolution of dynamical systems, and large deviation theory is often the right theoretical framework to understand these effects. At the core of the theory lies the minimization of an action functional, which in many cases of interest has to be computed by numerical means. Here we review the theoretical and computational aspects behind these calculations, and propose an algorithm that simplifies the geometric minimum action method to minimize the action in the space of arc-length parametrized curves. We then illustrate this algorithm's capabilities by applying it to various examples from material sciences, fluid dynamics, atmosphere/ocean sciences, and reaction kinetics. In terms of models, these examples involve stochastic (ordinary or partial) differential equations with multiplicative or degenerate noise, Markov jump processes, and systems with fast and slow degrees of freedom, which all violate detailed balance, so that simpler computational methods are not applicable.

math.NA↗

arXiv2017

Non-equilibrium transitions in multiscale systems with a bifurcating slow manifold

Noise-induced transitions between metastable fixed points in systems evolving on multiple time scales are analyzed in situations where the time scale separation gives rise to a slow manifold with bifurcation. This analysis is performed within the realm of large deviation theory. It is shown that these non-equilibrium transitions make use of a reaction channel created by the bifurcation structure of the slow manifold, leading to vastly increased transition rates. Several examples are used to illustrate these findings, including an insect outbreak model, a system modeling phase separation in the presence of evaporation, and a system modeling transitions in active matter self-assembly. The last example involves a spatially extended system modeled by a stochastic partial differential equation.

cond-mat.stat-mech↗

arXiv2017

Spatiotemporal Self-Organization of Fluctuating Bacterial Colonies

We model an enclosed system of bacteria, whose motility-induced phase separation is coupled to slow population dynamics. Without noise, the system shows both static phase separation and a limit cycle, in which a rising global population causes a dense bacterial colony to form, which then declines by local cell death, before dispersing to re-initiate the cycle. Adding fluctuations, we find that static colonies are now metastable, moving between spatial locations via rare and strongly nonequilibrium pathways, whereas the limit cycle becomes almost periodic such that after each redispersion event the next colony forms in a random location. These results, which hint at some aspects of the biofilm-planktonic life cycle, can be explained by combining tools from large deviation theory with a bifurcation analysis in which the global population density plays the role of control parameter.

cond-mat.stat-mech↗