arXiv subjects

Konstantinos Spiliopoulos

Publications and source records attributed to Konstantinos Spiliopoulos.

At least 37 records · Page 2Linked to original sources

arXiv2024

Transport map unadjusted Langevin algorithms: learning and discretizing perturbed samplers

Langevin dynamics are widely used in sampling high-dimensional, non-Gaussian distributions whose densities are known up to a normalizing constant. In particular, there is strong interest in unadjusted Langevin algorithms (ULA), which directly discretize Langevin dynamics to estimate expectations over the target distribution. We study the use of transport maps that approximately normalize a target distribution as a way to precondition and accelerate the convergence of Langevin dynamics. We show that in continuous time, when a transport map is applied to Langevin dynamics, the result is a Riemannian manifold Langevin dynamics (RMLD) with metric defined by the transport map. We also show that applying a transport map to an irreversibly-perturbed ULA results in a geometry-informed irreversible perturbation (GiIrr) of the original dynamics. These connections suggest more systematic ways of learning metrics and perturbations, and also yield alternative discretizations of the RMLD described by the map, which we study. Under appropriate conditions, these discretized processes can be endowed with non-asymptotic bounds describing convergence to the target distribution in 2-Wasserstein distance. Illustrative numerical results complement our theoretical claims.

stat.ME↗

arXiv2024

Mean field limits of particle-based stochastic reaction-drift-diffusion models

We consider particle-based stochastic reaction-drift-diffusion models where particles move via diffusion and drift induced by one- and two-body potential interactions. The dynamics of the particles are formulated as measure-valued stochastic processes (MVSPs), which describe the evolution of the singular, stochastic concentration fields of each chemical species. The mean field large population limit of such models is derived and proven, giving coarse-grained deterministic partial integro-differential equations (PIDEs) for the limiting deterministic concentration fields' dynamics. We generalize previous studies on the mean field limit of models involving only diffusive motion, with care to formulating the MVSP representation to ensure detailed balance of reversible reactions in the presence of potentials. Our work illustrates the more general set of PIDEs that arise in the mean field limit, demonstrating that the limiting macroscopic reactive interaction terms for reversible reactions obtain additional nonlinear concentration-dependent coefficients compared to the purely diffusive case. Numerical studies are presented which illustrate that two-body repulsive potential interactions can have a significant impact on the reaction dynamics, and also demonstrate the empirical numerical convergence of solutions to the PBSRDD model to the derived mean field PIDEs as the population size increases.

math.PR↗

arXiv2023

Importance sampling for stochastic reaction-diffusion equations in the moderate deviation regime

We develop a provably efficient importance sampling scheme that estimates exit probabilities of solutions to small-noise stochastic reaction-diffusion equations from scaled neighborhoods of a stable equilibrium. The moderate deviation scaling allows for a local approximation of the nonlinear dynamics by their linearized version. In addition, we identify a finite-dimensional subspace where exits take place with high probability. Using stochastic control and variational methods we show that our scheme performs well both in the zero noise limit and pre-asymptotically. Simulation studies for stochastically perturbed bistable dynamics illustrate the theoretical results.

math.PR↗

arXiv2023

PDE-constrained Models with Neural Network Terms: Optimization and Global Convergence

Recent research has used deep learning to develop partial differential equation (PDE) models in science and engineering. The functional form of the PDE is determined by a neural network, and the neural network parameters are calibrated to available data. Calibration of the embedded neural network can be performed by optimizing over the PDE. Motivated by these applications, we rigorously study the optimization of a class of linear elliptic PDEs with neural network terms. The neural network parameters in the PDE are optimized using gradient descent, where the gradient is evaluated using an adjoint PDE. As the number of parameters become large, the PDE and adjoint PDE converge to a non-local PDE system. Using this limit PDE system, we are able to prove convergence of the neural network-PDE to a global minimum during the optimization. Finally, we use this adjoint method to train a neural network model for an application in fluid mechanics, in which the neural network functions as a closure model for the Reynolds-averaged Navier--Stokes (RANS) equations. The RANS neural network model is trained on several datasets for turbulent channel flow and is evaluated out-of-sample at different Reynolds numbers.

cs.LG↗

arXiv2023

Fluctuation analysis for particle-based stochastic reaction-diffusion models

Recent works have derived and proven the large-population mean-field limit for several classes of particle-based stochastic reaction-diffusion (PBSRD) models. These limits correspond to systems of partial integral-differential equations (PIDEs) that generalize standard mass-action reaction-diffusion PDE models. In this work we derive and prove the next order fluctuation corrections to such limits, which we show satisfy systems of stochastic PIDEs with Gaussian noise. Numerical examples are presented to illustrate how including the fluctuation corrections can enable the accurate estimation of higher order statistics of the underlying PBSRD model.

math.PR↗

arXiv2023

Moderate deviations for fully coupled multiscale weakly interacting particle systems

We consider a collection of fully coupled weakly interacting diffusion processes moving in a two-scale environment. We study the moderate deviations principle of the empirical distribution of the particles' positions in the combined limit as the number of particles grow to infinity and the time-scale separation parameter goes to zero simultaneously. We make use of weak convergence methods, which provide a convenient representation for the moderate deviations rate function in terms of an effective mean field control problem. We rigorously obtain equivalent representations for the moderate deviations rate function in an appropriate "negative Sobolev" form, which is reminiscent of the large deviations rate function for the empirical measure of weakly interacting diffusions obtained in the 1987 seminal paper by Dawson-Gärtner. In the course of the proof we obtain related ergodic theorems and we rigorously study the regularity of Poisson type of equations associated to McKean-Vlasov problems, both of which are topics of independent interest.

math.PR↗

arXiv2023

Moderate deviation principle for multiscale systems driven by fractional Brownian motion

In this paper we study the moderate deviations principle (MDP) for slow-fast stochastic dynamical systems where the slow motion is governed by small fractional Brownian motion (fBm) with Hurst parameter $H\in(1/2,1)$. We derive conditions on the moderate deviations scaling and on the Hurst parameter $H$ under which the MDP holds. In addition, we show that in typical situations the resulting action functional is discontinuous in $H$ at $H=1/2$, suggesting that the tail behavior of stochastic dynamical systems perturbed by fBm can have different characteristics than the tail behavior of such systems that are perturbed by standard Brownian motion.

math.PR↗

arXiv2022

Large deviations for interacting multiscale particle systems

We consider a collection of weakly interacting diffusion processes moving in a two-scale locally periodic environment. We study the large deviations principle of the empirical distribution of the particles' positions in the combined limit as the number of particles grow to infinity and the time-scale separation parameter goes to zero simultaneously. We make use of weak convergence methods providing a convenient representation for the large deviations rate function, which allow us to characterize the effective controlled mean field dynamics. In addition, we obtain equivalent representations for the large deviations rate function of the form of Dawson-Gärtner which hold even in the case where the diffusion matrix depends on the empirical measure and when the particles undergo averaging in addition to the propagation of chaos.

math.PR↗

arXiv2022

Normalization effects on deep neural networks

We study the effect of normalization on the layers of deep neural networks of feed-forward type. A given layer $i$ with $N_{i}$ hidden units is allowed to be normalized by $1/N_{i}^{γ_{i}}$ with $γ_{i}\in[1/2,1]$ and we study the effect of the choice of the $γ_{i}$ on the statistical behavior of the neural network's output (such as variance) as well as on the test accuracy on the MNIST data set. We find that in terms of variance of the neural network's output and test accuracy the best choice is to choose the $γ_{i}$'s to be equal to one, which is the mean-field scaling. We also find that this is particularly true for the outer layer, in that the neural network's behavior is more sensitive in the scaling of the outer layer as opposed to the scaling of the inner layers. The mechanism for the mathematical analysis is an asymptotic expansion for the neural network's output. An important practical consequence of the analysis is that it provides a systematic and mathematically informed way to choose the learning rate hyperparameters. Such a choice guarantees that the neural network behaves in a statistically robust way as the $N_i$ grow to infinity.

cs.LG↗

arXiv2022

Geometry-informed irreversible perturbations for accelerated convergence of Langevin dynamics

We introduce a novel geometry-informed irreversible perturbation that accelerates convergence of the Langevin algorithm for Bayesian computation. It is well documented that there exist perturbations to the Langevin dynamics that preserve its invariant measure while accelerating its convergence. Irreversible perturbations and reversible perturbations (such as Riemannian manifold Langevin dynamics (RMLD)) have separately been shown to improve the performance of Langevin samplers. We consider these two perturbations simultaneously by presenting a novel form of irreversible perturbation for RMLD that is informed by the underlying geometry. Through numerical examples, we show that this new irreversible perturbation can improve estimation performance over irreversible perturbations that do not take the geometry into account. Moreover we demonstrate that irreversible perturbations generally can be implemented in conjunction with the stochastic gradient version of the Langevin algorithm. Lastly, while continuous-time irreversible perturbations cannot impair the performance of a Langevin estimator, the situation can sometimes be more complicated when discretization is considered. To this end, we describe a discrete-time example in which irreversibility increases both the bias and variance of the resulting estimator.

stat.ME↗

arXiv2022

Rate of homogenization for fully-coupled McKean-Vlasov SDEs

We consider a fully-coupled slow-fast system of McKean-Vlasov SDEs with full dependence on the slow and fast component and on the law of the slow component and derive convergence rates to its homogenized limit. We do not make periodicity assumptions, but we impose conditions on the fast motion to guarantee ergodicity. In the course of the proof we obtain related ergodic theorems and we gain results on the regularity of Poisson type of equations and of the associated Cauchy-Problem on the Wasserstein space that are of independent interest.

math.PR↗

arXiv2022

Disentangling positive and negative partisanship in social media interactions using a coevolving latent space network with attractors model

We develop a broadly applicable class of coevolving latent space network with attractors (CLSNA) models, where nodes represent individual social actors assumed to lie in an unknown latent space, edges represent the presence of a specified interaction between actors, and attractors are added in the latent level to capture the notion of attractive and repulsive forces. We apply the CLSNA models to understand the dynamics of partisan polarization on social media, where we expect Republicans and Democrats to increasingly interact with their own party and disengage with the opposing party. Using longitudinal social networks from the social media platforms Twitter and Reddit, we investigate the relative contributions of positive (attractive) and negative (repulsive) forces among political elites and the public, respectively. Our goals are to disentangle the positive and negative forces within and between parties and explore if and how they change over time. Our analysis confirms the existence of partisan polarization in social media interactions among both political elites and the public. Moreover, while positive partisanship is the driving force of interactions across the full periods of study for both the public and Democratic elites, negative partisanship has come to dominate Republican elites' interactions since the run-up to the 2016 presidential election.

stat.AP↗

arXiv2022

Normalization effects on shallow neural networks and related asymptotic expansions

We consider shallow (single hidden layer) neural networks and characterize their performance when trained with stochastic gradient descent as the number of hidden units $N$ and gradient descent steps grow to infinity. In particular, we investigate the effect of different scaling schemes, which lead to different normalizations of the neural network, on the network's statistical output, closing the gap between the $1/\sqrt{N}$ and the mean-field $1/N$ normalization. We develop an asymptotic expansion for the neural network's statistical output pointwise with respect to the scaling parameter as the number of hidden units grows to infinity. Based on this expansion, we demonstrate mathematically that to leading order in $N$, there is no bias-variance trade off, in that both bias and variance (both explicitly characterized) decrease as the number of hidden units increases and time grows. In addition, we show that to leading order in $N$, the variance of the neural network's statistical output decays as the implied normalization by the scaling parameter approaches the mean field normalization. Numerical studies on the MNIST and CIFAR10 datasets show that test and train accuracy monotonically improve as the neural network's normalization gets closer to the mean field normalization.

stat.ML↗

arXiv2022

Scaling Limit of Neural Networks with the Xavier Initialization and Convergence to a Global Minimum

We analyze single-layer neural networks with the Xavier initialization in the asymptotic regime of large numbers of hidden units and large numbers of stochastic gradient descent training steps. The evolution of the neural network during training can be viewed as a stochastic system and, using techniques from stochastic analysis, we prove the neural network converges in distribution to a random ODE with a Gaussian distribution. The limit is completely different than in the typical mean-field results for neural networks due to the $\frac{1}{\sqrt{N}}$ normalization factor in the Xavier initialization (versus the $\frac{1}{N}$ factor in the typical mean-field framework). Although the pre-limit problem of optimizing a neural network is non-convex (and therefore the neural network may converge to a local minimum), the limit equation minimizes a (quadratic) convex objective function and therefore converges to a global minimum. Furthermore, under reasonable assumptions, the matrix in the limiting quadratic objective function is positive definite and thus the neural network (in the limit) will converge to a global minimum with zero loss on the training set.

math.PR↗

arXiv2022

Moderate deviations for systems of slow-fast stochastic reaction-diffusion equations

The goal of this paper is to study the Moderate Deviation Principle (MDP) for a system of stochastic reaction-diffusion equations with a time-scale separation in slow and fast components and small noise in the slow component. Based on weak convergence methods in infinite dimensions and related stochastic control arguments, we obtain an exact form for the moderate deviations rate function in different regimes as the small noise and time-scale separation parameters vanish. Many issues that appear due to the infinite dimensionality of the problem are completely absent in their finite-dimensional counterpart. In comparison to corresponding Large Deviation Principles, the moderate deviation scaling necessitates a more delicate approach to establishing tightness and properly identifying the limiting behavior of the underlying controlled problem. The latter involves regularity properties of a solution of an associated elliptic Kolmogorov equation on Hilbert space along with a finite-dimensional approximation argument.

math.PR↗

arXiv2022

Online Adjoint Methods for Optimization of PDEs

We present and mathematically analyze an online adjoint algorithm for the optimization of partial differential equations (PDEs). Traditional adjoint algorithms would typically solve a new adjoint PDE at each optimization iteration, which can be computationally costly. In contrast, an online adjoint algorithm updates the design variables in continuous-time and thus constantly makes progress towards minimizing the objective function. The online adjoint algorithm we consider is similar in spirit to the the pseudo-time-stepping, one-shot method which has been previously proposed. Motivated by the application of such methods to engineering problems, we mathematically study the convergence of the online adjoint algorithm. The online adjoint algorithm relies upon a time-relaxed adjoint PDE which provides an estimate of the direction of steepest descent. The algorithm updates this estimate continuously in time, and it asymptotically converges to the exact direction of steepest descent as $t \rightarrow \infty$. We rigorously prove that the online adjoint algorithm converges to a critical point of the objective function for optimizing the PDE. Under appropriate technical conditions, we also prove a convergence rate for the algorithm. A crucial step in the convergence proof is a multi-scale analysis of the coupled system for the forward PDE, adjoint PDE, and the gradient descent ODE for the design variables.

math.OC↗

arXiv2021

Mean Field Limits of Particle-Based Stochastic Reaction-Diffusion Models

Particle-based stochastic reaction-diffusion (PBSRD) models are a popular approach for studying biological systems involving both noise in the reaction process and diffusive transport. In this work we derive coarse-grained deterministic partial integro-differential equation (PIDE) models that provide a mean field approximation to the volume reactivity PBSRD model, a model commonly used for studying cellular processes. We formulate a weak measure-valued stochastic process (MVSP) representation for the volume reactivity PBSRD model, demonstrating for a simplified but representative system that it is consistent with the commonly used Doi Fock Space representation of the corresponding forward equation. We then prove the convergence of the general volume reactivity model MVSP to the mean field PIDEs in the large-population (i.e. thermodynamic) limit.

math.PR↗

arXiv2021

How reaction-diffusion PDEs approximate the large-population limit of stochastic particle models

Reaction-diffusion PDEs and particle-based stochastic reaction-diffusion (PBSRD) models are commonly-used approaches for modeling the spatial dynamics of chemical and biological systems. Standard reaction-diffusion PDE models ignore the underlying stochasticity of spatial transport and reactions, and are often described as appropriate in regimes where there are large numbers of particles in a system. Recent studies have proven the rigorous large-population limit of PBSRD models, showing the resulting mean-field models (MFM) correspond to non-local systems of partial-integro differential equations. In this work we explore the rigorous relationship between standard reaction-diffusion PDE models and the derived MFM. We prove that the former can be interpreted as an asymptotic approximation to the later in the limit that bimolecular reaction kernels are short-range and averaging. As the reactive interaction length scale approaches zero, we prove the MFMs converge at second order to standard reaction-diffusion PDE models. In proving this result we also establish local well-posedness of the MFM model in time for general systems, and global well-posedness for specific reaction systems and kernels. Finally, we illustrate the agreement and disagreement between the MFM, SM and the underlying particle model for several numerical examples.

math.AP↗