arXiv subjects

Eric Vanden-Eijnden

Publications and source records attributed to Eric Vanden-Eijnden.

At least 55 records · Page 3Linked to original sources

arXiv2023

Efficient Training of Energy-Based Models Using Jarzynski Equality

Energy-based models (EBMs) are generative models inspired by statistical physics with a wide range of applications in unsupervised learning. Their performance is best measured by the cross-entropy (CE) of the model distribution relative to the data distribution. Using the CE as the objective for training is however challenging because the computation of its gradient with respect to the model parameters requires sampling the model distribution. Here we show how results for nonequilibrium thermodynamics based on Jarzynski equality together with tools from sequential Monte-Carlo sampling can be used to perform this computation efficiently and avoid the uncontrolled approximations made using the standard contrastive divergence algorithm. Specifically, we introduce a modification of the unadjusted Langevin algorithm (ULA) in which each walker acquires a weight that enables the estimation of the gradient of the cross-entropy at any step during GD, thereby bypassing sampling biases induced by slow mixing of ULA. We illustrate these results with numerical experiments on Gaussian mixture distributions as well as the MNIST dataset. We show that the proposed approach outperforms methods based on the contrastive divergence algorithm in all the considered situations.

cs.LG↗

arXiv2023

Computing equilibrium free energies through a nonequilibrium quench

Many methods to accelerate sampling of molecular configurations are based on the idea that temperature can be used to accelerate rare transitions. These methods typically compute equilibrium properties at a target temperature using reweighting or through Monte Carlo exchanges between replicas at higher temperatures. A recent paper demonstrated that accurate equilibrium densities of states can also be computed through a nonequilibrium "quench" process, where sampling is performed at a higher temperature to encourage rapid mixing and then quenched to lower energy states with dissipative dynamics. Here we provide an implementation of the quench dynamics in LAMMPS and evaluate a new formulation of nonequilibrium estimators for the computation of partition functions or free energy surfaces (FESs) of molecular systems. We show that the method is exact for a minimal model of $N$-independent harmonic springs, and use these analytical results to develop heuristics for the amount of quenching required to obtain accurate sampling.= We then test the quench approach on alanine dipeptide, where we show that it gives an FES that is accurate near the most stable configurations using the quench approach, but disagrees with a reference umbrella sampling calculation in high FE regions. We then show that combining quenching with umbrella sampling allows the efficient calculation of the free energy in all regions. Moreover, by using this combined scheme, we obtain the FES across a range of temperatures at no additional cost, making it much more efficient than standard umbrella sampling if this information is required. Finally, we discuss how this approach can be extended to solute tempering and demonstrate that it is highly accurate for the case of solvated alanine dipeptide without any additional modifications.

physics.chem-ph↗

arXiv2023

Learning to Sample Better

These lecture notes provide an introduction to recent advances in generative modeling methods based on the dynamical transportation of measures, by means of which samples from a simple base measure are mapped to samples from a target measure of interest. Special emphasis is put on the applications of these methods to Monte-Carlo (MC) sampling techniques, such as importance sampling and Markov Chain Monte-Carlo (MCMC) schemes. In this context, it is shown how the maps can be learned variationally using data generated by MC sampling, and how they can in turn be used to improve such sampling in a positive feedback loop.

cs.LG↗

arXiv2023

Multimarginal generative modeling with stochastic interpolants

Given a set of $K$ probability densities, we consider the multimarginal generative modeling problem of learning a joint distribution that recovers these densities as marginals. The structure of this joint distribution should identify multi-way correspondences among the prescribed marginals. We formalize an approach to this task within a generalization of the stochastic interpolant framework, leading to efficient learning algorithms built upon dynamical transport of measure. Our generative models are defined by velocity and score fields that can be characterized as the minimizers of simple quadratic objectives, and they are defined on a simplex that generalizes the time variable in the usual dynamical transport framework. The resulting transport on the simplex is influenced by all marginals, and we show that multi-way correspondences can be extracted. The identification of such correspondences has applications to style transfer, algorithmic fairness, and data decorruption. In addition, the multimarginal perspective enables an efficient algorithm for reducing the dynamical transport cost in the ordinary two-marginal setting. We demonstrate these capacities with several numerical examples.

cs.LG↗

arXiv2023

Coupling parameter and particle dynamics for adaptive sampling in Neural Galerkin schemes

Training nonlinear parametrizations such as deep neural networks to numerically approximate solutions of partial differential equations is often based on minimizing a loss that includes the residual, which is analytically available in limited settings only. At the same time, empirically estimating the training loss is challenging because residuals and related quantities can have high variance, especially for transport-dominated and high-dimensional problems that exhibit local features such as waves and coherent structures. Thus, estimators based on data samples from un-informed, uniform distributions are inefficient. This work introduces Neural Galerkin schemes that estimate the training loss with data from adaptive distributions, which are empirically represented via ensembles of particles. The ensembles are actively adapted by evolving the particles with dynamics coupled to the nonlinear parametrizations of the solution fields so that the ensembles remain informative for estimating the training loss. Numerical experiments indicate that few dynamic particles are sufficient for obtaining accurate empirical estimates of the training loss, even for problems with local features and with high-dimensional spatial domains.

math.NA↗

arXiv2023

Estimating earthquake-induced tsunami height probabilities without sampling

Given a distribution of earthquake-induced seafloor elevations, we present a method to compute the probability of the resulting tsunamis reaching a certain size on shore. Instead of sampling, the proposed method relies on optimization to compute the most likely fault slips that result in a seafloor deformation inducing a large tsunami wave. We model tsunamis induced by bathymetry change using the shallow water equations on an idealized slice through the sea. The earthquake slip model is based on a sum of multivariate log-normal distributions, and follows the Gutenberg-Richter law for moment magnitudes 7--9. For a model problem inspired by the Tohoku-Oki 2011 earthquake and tsunami, we quantify annual probabilities of differently sized tsunami waves. Our method also identifies the most effective tsunami mechanisms. These mechanisms have smoothly varying fault slip patches that lead to an expansive but moderately large bathymetry change. The resulting tsunami waves are compressed as they approach shore and reach close-to-vertical leading wave edge close to shore.

physics.geo-ph↗

arXiv2023

Building Normalizing Flows with Stochastic Interpolants

A generative model based on a continuous-time normalizing flow between any pair of base and target probability densities is proposed. The velocity field of this flow is inferred from the probability current of a time-dependent density that interpolates between the base and the target in finite time. Unlike conventional normalizing flow inference methods based the maximum likelihood principle, which require costly backpropagation through ODE solvers, our interpolant approach leads to a simple quadratic loss for the velocity itself which is expressed in terms of expectations that are readily amenable to empirical estimation. The flow can be used to generate samples from either the base or target, and to estimate the likelihood at any time along the interpolant. In addition, the flow can be optimized to minimize the path length of the interpolant density, thereby paving the way for building optimal transport maps. In situations where the base is a Gaussian density, we also show that the velocity of our normalizing flow can also be used to construct a diffusion model to sample the target as well as estimate its score. However, our approach shows that we can bypass this diffusion completely and work at the level of the probability flow with greater simplicity, opening an avenue for methods based solely on ordinary differential equations as an alternative to those based on stochastic differential equations. Benchmarking on density estimation tasks illustrates that the learned flow can match and surpass conventional continuous flows at a fraction of the cost, and compares well with diffusions on image generation on CIFAR-10 and ImageNet $32\times32$. The method scales ab-initio ODE flows to previously unreachable image resolutions, demonstrated up to $128\times128$.

cs.LG↗

arXiv2023

Probability flow solution of the Fokker-Planck equation

The method of choice for integrating the time-dependent Fokker-Planck equation in high-dimension is to generate samples from the solution via integration of the associated stochastic differential equation. Here, we study an alternative scheme based on integrating an ordinary differential equation that describes the flow of probability. Acting as a transport map, this equation deterministically pushes samples from the initial density onto samples from the solution at any later time. Unlike integration of the stochastic dynamics, the method has the advantage of giving direct access to quantities that are challenging to estimate from trajectories alone, such as the probability current, the density itself, and its entropy. The probability flow equation depends on the gradient of the logarithm of the solution (its "score"), and so is a-priori unknown. To resolve this dependence, we model the score with a deep neural network that is learned on-the-fly by propagating a set of samples according to the instantaneous probability current. We show theoretically that the proposed approach controls the KL divergence from the learned solution to the target, while learning on external samples from the stochastic differential equation does not control either direction of the KL divergence. Empirically, we consider several high-dimensional Fokker-Planck equations from the physics of interacting particle systems. We find that the method accurately matches analytical solutions when they are available as well as moments computed via Monte-Carlo when they are not. Moreover, the method offers compelling predictions for the global entropy production rate that out-perform those obtained from learning on stochastic trajectories, and can effectively capture non-equilibrium steady-state probability currents over long time intervals.

cs.LG↗

arXiv2022

Learning Optimal Flows for Non-Equilibrium Importance Sampling

Many applications in computational sciences and statistical inference require the computation of expectations with respect to complex high-dimensional distributions with unknown normalization constants, as well as the estimation of these constants. Here we develop a method to perform these calculations based on generating samples from a simple base distribution, transporting them by the flow generated by a velocity field, and performing averages along these flowlines. This non-equilibrium importance sampling (NEIS) strategy is straightforward to implement and can be used for calculations with arbitrary target distributions. On the theory side, we discuss how to tailor the velocity field to the target and establish general conditions under which the proposed estimator is a perfect estimator with zero-variance. We also draw connections between NEIS and approaches based on mapping a base distribution onto a target via a transport map. On the computational side, we show how to use deep learning to represent the velocity field by a neural network and train it towards the zero variance optimum. These results are illustrated numerically on benchmark examples (with dimension up to $10$), where after training the velocity field, the variance of the NEIS estimator is reduced by up to $6$ orders of magnitude than that of a vanilla estimator. We also compare the performances of NEIS with those of Neal's annealed importance sampling (AIS).

math.ST↗

arXiv2022

Learning sparse features can lead to overfitting in neural networks

It is widely believed that the success of deep networks lies in their ability to learn a meaningful representation of the features of the data. Yet, understanding when and how this feature learning improves performance remains a challenge: for example, it is beneficial for modern architectures trained to classify images, whereas it is detrimental for fully-connected networks trained for the same task on the same data. Here we propose an explanation for this puzzle, by showing that feature learning can perform worse than lazy training (via random feature kernel or the NTK) as the former can lead to a sparser neural representation. Although sparsity is known to be essential for learning anisotropic data, it is detrimental when the target function is constant or smooth along certain directions of input space. We illustrate this phenomenon in two settings: (i) regression of Gaussian random functions on the d-dimensional unit sphere and (ii) classification of benchmark datasets of images. For (i), we compute the scaling of the generalization error with number of training points, and show that methods that do not learn features generalize better, even when the dimension of the input space is large. For (ii), we show empirically that learning features can indeed lead to sparse and thereby less smooth representations of the image predictors. This fact is plausibly responsible for deteriorating the performance, which is known to be correlated with smoothness along diffeomorphisms.

stat.ML↗

arXiv2022

Minimum Action Method for Nonequilibrium Phase Transitions

First-order nonequilibrium phase transitions observed in active matter, fluid dynamics, biology, climate science, and other systems with irreversible dynamics are challenging to analyze because they cannot be inferred from a simple free energy minimization principle. Rather the mechanism of these transitions depends crucially on the system's dynamics, which requires us to analyze them in trajectory space rather than in phase space. Here we consider situations where the path of these transitions can be characterized as the minimizer of an action, whose minimum value can be used in a nonequilibrium generalization of the Arrhenius law to calculate the system's phase diagram. We also develop efficient numerical tools for the minimization of this action. These tools are general enough to be transportable to many situations of interest, in particular when the fluctuations present in the microscopic system are non-Gaussian and its dynamics is not governed by the standard Langevin equation. As an illustration, first-order phase transitions in two spatially-extended nonequilibrium systems are analyzed: a modified Ginzburg-Landau equation with a chemical potential which is non-gradient, and a reaction-diffusion network based on the Schlögl model. The phase diagrams of both systems are calculated as a function of their control parameters, and the paths of the transitions, including their critical nuclei, are identified. These results clearly demonstrate the nonequilibrium nature of the transitions, with differing forward and backward paths.

cond-mat.stat-mech↗

arXiv2022

On Feature Learning in Neural Networks with Global Convergence Guarantees

We study the optimization of wide neural networks (NNs) via gradient flow (GF) in setups that allow feature learning while admitting non-asymptotic global convergence guarantees. First, for wide shallow NNs under the mean-field scaling and with a general class of activation functions, we prove that when the input dimension is no less than the size of the training set, the training loss converges to zero at a linear rate under GF. Building upon this analysis, we study a model of wide multi-layer NNs whose second-to-last layer is trained via GF, for which we also prove a linear-rate convergence of the training loss to zero, but regardless of the input dimension. We also show empirically that, unlike in the Neural Tangent Kernel (NTK) regime, our multi-layer model exhibits feature learning and can achieve better generalization performance than its NTK counterpart.

cs.LG↗

arXiv2022

A Dynamical Central Limit Theorem for Shallow Neural Networks

Recent theoretical works have characterized the dynamics of wide shallow neural networks trained via gradient descent in an asymptotic mean-field limit when the width tends towards infinity. At initialization, the random sampling of the parameters leads to deviations from the mean-field limit dictated by the classical Central Limit Theorem (CLT). However, since gradient descent induces correlations among the parameters, it is of interest to analyze how these fluctuations evolve. Here, we use a dynamical CLT to prove that the asymptotic fluctuations around the mean limit remain bounded in mean square throughout training. The upper bound is given by a Monte-Carlo resampling error, with a variance that that depends on the 2-norm of the underlying measure, which also controls the generalization error. This motivates the use of this 2-norm as a regularization term during training. Furthermore, if the mean-field dynamics converges to a measure that interpolates the training data, we prove that the asymptotic deviation eventually vanishes in the CLT scaling. We also complement these results with numerical experiments.

math.PR↗

arXiv2022

Dual Training of Energy-Based Models with Overparametrized Shallow Neural Networks

Energy-based models (EBMs) are generative models that are usually trained via maximum likelihood estimation. This approach becomes challenging in generic situations where the trained energy is non-convex, due to the need to sample the Gibbs distribution associated with this energy. Using general Fenchel duality results, we derive variational principles dual to maximum likelihood EBMs with shallow overparametrized neural network energies, both in the feature-learning and lazy linearized regimes. In the feature-learning regime, this dual formulation justifies using a two time-scale gradient ascent-descent (GDA) training algorithm in which one updates concurrently the particles in the sample space and the neurons in the parameter space of the energy. We also consider a variant of this algorithm in which the particles are sometimes restarted at random samples drawn from the data set, and show that performing these restarts at every iteration step corresponds to score matching training. These results are illustrated in simple numerical experiments, which indicates that GDA performs best when features and particles are updated using similar time scales.

cs.LG↗

arXiv2022

Adaptive Monte Carlo augmented with normalizing flows

Many problems in the physical sciences, machine learning, and statistical inference necessitate sampling from a high-dimensional, multi-modal probability distribution. Markov Chain Monte Carlo (MCMC) algorithms, the ubiquitous tool for this task, typically rely on random local updates to propagate configurations of a given system in a way that ensures that generated configurations will be distributed according to a target probability distribution asymptotically. In high-dimensional settings with multiple relevant metastable basins, local approaches require either immense computational effort or intricately designed importance sampling strategies to capture information about, for example, the relative populations of such basins. Here we analyze an adaptive MCMC which augments MCMC sampling with nonlocal transition kernels parameterized with generative models known as normalizing flows. We focus on a setting where there is no preexisting data, as is commonly the case for problems in which MCMC is used. Our method uses: (i) a MCMC strategy that blends local moves obtained from any standard transition kernel with those from a generative model to accelerate the sampling and (ii) the data generated this way to adapt the generative model and improve its efficacy in the MCMC algorithm. We provide a theoretical analysis of the convergence properties of this algorithm, and investigate numerically its efficiency, in particular in terms of its propensity to equilibrate fast between metastable modes whose rough location is known \textit{a~priori} but respective probability weight is not. We show that our algorithm can sample effectively across large free energy barriers, providing dramatic accelerations relative to traditional MCMC algorithms.

physics.data-an↗

arXiv2022

Averaged equation for energy diffusion on a graph reveals bifurcation diagram and thermally assisted reversal times in spin-torque driven nanomagnets

Driving nanomagnets by spin-polarized currents offers exciting prospects in magnetoelectronics, but the response of the magnets to such currents remains poorly understood. We show that an averaged equation describing the diffusion of energy on a graph captures the low-damping dynamics of these systems. From this equation we obtain the bifurcation diagram of the magnets, including the critical currents to induce stable precessional states and magnetization switching, as well as the mean times of thermally assisted magnetization reversal in situations where the standard reaction rate theory of Kramers is no longer valid. These results match experimental observations and give a theoretical basis for a Néel-Brown-type formula with an effective energy barrier for the reversal times.

cond-mat.mes-hall↗

arXiv2021

Anomalous Statistics and Large Deviations of Turbulent Water Waves past a Step

A computational strategy based on large deviation theory (LDT) is used to study the anomalous statistical features of turbulent surface waves propagating past an abrupt depth change created via a step in the bottom topography. The dynamics of the outgoing waves past the step are modeled using the truncated Korteweg-de Vries (TKdV) equation with random initial conditions at the step drawn from the system's Gibbs invariant measure of the incoming waves. Within the LDT framework, the probability distributions of the wave height can be obtained via the solution of a deterministic optimization problem. Detailed numerical tests show that this approach accurately captures the non-Gaussian features of the wave height distributions, in particular their asymmetric tails leading to high skewness. These calculations also give the spatio-temporal pattern of the anomalous waves most responsible for these non-Gaussian features. The strategy shows potential for a general class of nonlinear Hamiltonian systems with highly non-Gaussian statistics.

physics.flu-dyn↗

arXiv2021

Dynamical computation of the density of states and Bayes factors using nonequilibrium importance sampling

Nonequilibrium sampling is potentially much more versatile than its equilibrium counterpart, but it comes with challenges because the invariant distribution is not typically known when the dynamics breaks detailed balance. Here, we derive a generic importance sampling technique that leverages the statistical power of configurations transported by nonequilibrium trajectories, and can be used to compute averages with respect to arbitrary target distributions. As a dissipative reweighting scheme, the method can be viewed in relation to the annealed importance sampling (AIS) method and the related Jarzynski equality. Unlike AIS, our approach gives an unbiased estimator, with provably lower variance than directly estimating the average of an observable. We also establish a direct relation between a dynamical quantity, the dissipation, and the volume of phase space, from which we can compute quantities such as the density of states and Bayes factors. We illustrate the properties of estimators relying on this sampling technique in the context of density of state calculations, showing that it scales favorable with dimensionality -- in particular, we show that it can be used to compute the phase diagram of the mean-field Ising model from a single nonequilibrium trajectory. We also demonstrate the robustness and efficiency of the approach with an application to a Bayesian model comparison problem of the type encountered in astrophysics and machine learning.

cond-mat.stat-mech↗