arXiv subjects

Wuchen Li

Publications and source records attributed to Wuchen Li.

At least 19 recordsLinked to original sources

arXiv2026

A Kernel Formula for Kinetic Fokker-Planck Equations

We derive an explicit one-step kernel formula for the kinetic Fokker-Planck equation. The construction begins with a reference dynamics admitting an explicit transition kernel and uses a carefully chosen auxiliary weight to encode the drift perturbation. The resulting kernel formula provides an explicit approximation of the density evolution. In a weighted phase-space setting, we establish local weak consistency and first-order finite-time weak convergence under suitable regularity and moment assumptions. We also give a formal variational interpretation of the formula in terms of a regularized Wasserstein-type proximal operator and present analogous constructions for a broader class of parabolic equations. Numerical experiments illustrate the accuracy of the kernel approximation and its application to deterministic particle schemes for kinetic sampling.

math.NA↗

arXiv2026

Deep Kinetic JKO schemes for Vlasov-Fokker-Planck Equations

We introduce a deep neural network-based numerical method for solving kinetic Fokker Planck equations, including both linear and nonlinear cases. Building upon the conservative dissipative structure of Vlasov-type equations, we formulate a class of generalized minimizing movement schemes as iterative constrained minimization problems: the conservative part determines the constraint set, while the dissipative part defines the objective functional. This leads to an analog of the classical Jordan-Kinderlehrer-Otto (JKO) scheme for Wasserstein gradient flows, and we refer to it as the kinetic JKO scheme. To compute each step of the kinetic JKO iteration, we introduce a particle-based approximation in which the velocity field is parameterized by deep neural networks. The resulting algorithm can be interpreted as a kinetic-oriented neural differential equation that enables the representation of high-dimensional kinetic dynamics while preserving the essential variational and structural properties of the underlying PDE. We validate the method with extensive numerical experiments and demonstrate that the proposed kinetic JKO-neural ODE framework is effective for high-dimensional numerical simulations.

math.NA↗

arXiv2026

SympFormer: Accelerated attention blocks via Inertial Dynamics on Density Manifolds

Transformers owe much of their empirical success in natural language processing to the self-attention blocks. Recent perspectives interpret attention blocks as interacting particle systems, whose mean-field limits correspond to gradient flows of interaction energy functionals on probability density spaces equipped with Wasserstein-$2$-type metrics. We extend this viewpoint by introducing accelerated attention blocks derived from inertial Nesterov-type dynamics on density spaces. In our proposed architecture, tokens carry both spatial (feature) and velocity variables. The time discretization and the approximation of accelerated density dynamics yield Hamiltonian momentum attention blocks, which constitute the proposed accelerated attention architectures. In particular, for linear self-attention, we show that the attention blocks approximate a Stein variational gradient flow, using a bilinear kernel, of a potential energy. In this setting, we prove that elliptically contoured probability distributions are preserved by the accelerated attention blocks. We present implementable particle-based algorithms and demonstrate that the proposed accelerated attention blocks converge faster than the classical attention blocks while preserving the number of oracle calls.

cs.LG↗

arXiv2026

Accelerated Regularized Wasserstein Proximal Sampling Algorithms

We consider sampling from a Gibbs distribution by evolving a finite number of particles using a particular score estimator rather than Brownian motion. To accelerate the particles, we consider a second-order score-based ODE, similar to Nesterov acceleration. In contrast to traditional kernel density score estimation, we use the recently proposed regularized Wasserstein proximal method, yielding the Accelerated Regularized Wasserstein Proximal method (ARWP). We provide a detailed analysis of continuous- and discrete-time non-asymptotic and asymptotic mixing rates for Gaussian initial and target distributions, using techniques from Euclidean acceleration and accelerated information gradients. Compared with the kinetic Langevin sampling algorithm, the proposed algorithm exhibits a higher contraction rate in the asymptotic time regime. Numerical experiments are conducted across various low-dimensional experiments, including multi-modal Gaussian mixtures and ill-conditioned Rosenbrock distributions. ARWP exhibits structured and convergent particles, accelerated discrete-time mixing, and faster tail exploration than the non-accelerated regularized Wasserstein proximal method and kinetic Langevin methods. Additionally, ARWP particles exhibit better generalization properties for some non-log-concave Bayesian neural network tasks.

stat.ML↗

arXiv2025

Sparse Transformer Architectures via Regularized Wasserstein Proximal Operator with $L_1$ Prior

In this work, we propose a sparse transformer architecture that incorporates prior information about the underlying data distribution directly into the transformer structure of the neural network. The design of the model is motivated by a special optimal transport problem, namely the regularized Wasserstein proximal operator, which admits a closed-form solution and turns out to be a special representation of transformer architectures. Compared with classical flow-based models, the proposed approach improves the convexity properties of the optimization problem and promotes sparsity in the generated samples. Through both theoretical analysis and numerical experiments, including applications in generative modeling and Bayesian inverse problems, we demonstrate that the sparse transformer achieves higher accuracy and faster convergence to the target distribution than classical neural ODE-based methods.

cs.LG↗

arXiv2025

Towards understanding Accelerated Stein Variational Gradient Flow -- Analysis of Generalized Bilinear Kernels for Gaussian target distributions

Stein variational gradient descent (SVGD) is a kernel-based and non-parametric particle method for sampling from a target distribution, such as in Bayesian inference and other machine learning tasks. Different from other particle methods, SVGD does not require estimating the score, which is the gradient of the log-density. However, in practice, SVGD can be slow compared to score-estimation-based sampling algorithms. To design a fast and efficient high-dimensional sampling algorithm with the advantages of SVGD, we introduce accelerated SVGD (ASVGD), based on an accelerated gradient flow in a metric space of probability densities following Nesterov's method. We then derive a momentum-based discrete-time sampling algorithm, which evolves a set of particles deterministically. To stabilize the particles' position update, we also include a Wasserstein metric regularization. This paper extends the conference version \cite{SL2025}. For the bilinear kernel and Gaussian target distributions, we study the kernel parameter and damping parameters with an optimal convergence rate of the proposed dynamics. This is achieved by analyzing the linearized accelerated gradient flows at the equilibrium. Interestingly, the optimal parameter is a constant, which does not depend on the covariance of the target distribution. For the generalized kernel functions, such as the Gaussian kernel, numerical examples with varied target distributions demonstrate the effectiveness of ASVGD compared to SVGD and other popular sampling methods. Furthermore, we show that in the setting of Bayesian neural networks, ASVGD outperforms SVGD significantly in terms of log-likelihood and total iteration times.

math.OC↗

arXiv2025

Preconditioned Regularized Wasserstein Proximal Sampling

We consider sampling from a Gibbs distribution by evolving finitely many particles. We propose a preconditioned version of a recently proposed noise-free sampling method, governed by approximating the score function with the numerically tractable score of a regularized Wasserstein proximal operator. This is derived by a Cole--Hopf transformation on coupled anisotropic heat equations, yielding a kernel formulation for the preconditioned regularized Wasserstein proximal. The diffusion component of the proposed method is also interpreted as a modified self-attention block, as in transformer architectures. For quadratic potentials, we provide a discrete-time non-asymptotic convergence analysis and explicitly characterize the bias, which is dependent on regularization and independent of step-size. Experiments demonstrate acceleration and particle-level stability on various log-concave and non-log-concave toy examples to Bayesian total-variation regularized image deconvolution, and competitive/better performance on non-convex Bayesian neural network training when utilizing variable preconditioning matrices.

stat.ML↗

arXiv2025

Expansive Natural Neural Gradient Flows for Energy Minimization

This paper develops expansive gradient dynamics in deep neural network-induced mapping spaces. Specifically, we generate tools and concepts for minimizing a class of energy functionals in an abstract Hilbert space setting covering a wide scope of applications such as PDEs-based inverse problems and supervised learning. The approach hinges on a Hilbert space metric in the full diffeomorphism mapping space, which could be viewed as a generalized Wasserstein-2 metric. We then study a projection gradient descent method within deep neural network parameterized sets. More importantly, we develop an adaptation and expanding strategy to step-by-step enlarge the deep neural network structures. In particular, the expansion mechanism aims to enhance the alignment of the neural manifold induced natural gradient direction as well as possible with the ideal Hilbert space gradient descent direction leveraging the fact that we can evaluate projections of the Hilbert space gradient. We demonstrate the efficacy of the proposed strategy for several simple model problems for energies arising in the context of supervised learning, model reduction, or inverse problems. In particular, we highlight the importance of assembling the neural flow matrix based on the inner product for the ambient Hilbert space. The actual algorithms are the simplest specifications of a broader spectrum based on a correspondingly wider discussion, postponing a detailed analysis to forthcoming work.

math.OC↗

arXiv2025

Simulating Fokker-Planck equations via mean field control of score-based normalizing flows

The Fokker--Planck (FP) equation governs the evolution of densities for stochastic dynamics of physical systems, such as the Langevin dynamics and the Lorenz system. This work simulates FP equations through a mean field control (MFC) problem. We first formulate the FP equation as a continuity equation, where the velocity field consists of the drift function and the score function, i.e., the gradient of the logarithm of the density function. Next, we design a MFC problem that matches the velocity fields in a continuity equation with the ones in the FP equation. The score functions along deterministic trajectories are computed efficiently through the score-based normalizing flow, which only relies on the derivatives of the parameterized velocity fields. Numerical results, including Langevin dynamics, underdamped Langevin dynamics, chaotic systems, and high-dimensional interacting particle systems validate the effectiveness and scalability of our proposed algorithm. A convergence analysis is conducted for our algorithm on the FP equation of Ornstein--Uhlenbeck processes.

math.OC↗

arXiv2025

Accelerated Markov Chain Monte Carlo Algorithms on Discrete States

We propose a class of discrete state sampling algorithms based on Nesterov's accelerated gradient method, which extends the classical Metropolis-Hastings (MH) algorithm. The evolution of the discrete states probability distribution governed by MH can be interpreted as a gradient descent direction of the Kullback--Leibler (KL) divergence, via a mobility function and a score function. Specifically, this gradient is defined on a probability simplex equipped with a discrete Wasserstein-2 metric with a mobility function. This motivates us to study a momentum-based acceleration framework using damped Hamiltonian flows on the simplex set, whose stationary distribution matches the discrete target distribution. Furthermore, we design an interacting particle system to approximate the proposed accelerated sampling dynamics. The extension of the algorithm with a general choice of potentials and mobilities is also discussed. In particular, we choose the accelerated gradient flow of the relative Fisher information, demonstrating the advantages of the algorithm in estimating discrete score functions without requiring the normalizing constant and keeping positive probabilities. Numerical examples, including sampling on a Gaussian mixture supported on lattices or a distribution on a hypercube, demonstrate the effectiveness of the proposed discrete-state sampling algorithm.

math.OC↗

arXiv2025

Geometric calculations on probability manifolds from reciprocal relations in Master equations

Onsager reciprocal relations model physical irreversible processes from complex systems. Recently, it has been shown that Onsager principles for master equations on finite states introduce a class of Riemannian metrics on the probability simplex, leading to probability manifolds or finite-state Wasserstein--2 spaces. In this paper, we study geometric calculations on probability manifolds, deriving the Levi-Civita connection, gradient, Hessian operators of energies, parallel transport, and calculating both the Riemannian and sectional curvatures. We present two examples of geometric quantities in probability manifolds. One example is the Levi-Civita connection from the chemical monomolecular triangle reaction. The other example is the sectional, Ricci, and scalar curvatures in Wasserstein space on a three-point lattice graph.

math-ph↗

arXiv2025

Transport f divergences

We define a class of divergences to measure differences between probability density functions in one-dimensional sample space. The construction is based on the convex function with the Jacobi operator of mapping function that pushforwards one density to the other. We call these information measures transport f-divergences. We present several properties of transport $f$-divergences, including invariances, convexities, variational formulations, and Taylor expansions in terms of mapping functions. Examples of transport f-divergences in generative models are provided.

math.ST↗

arXiv2025

Transport alpha divergences

We derive a class of divergences measuring the difference between probability density functions on the one-dimensional sample space. This divergence is a one-parameter variation of the Itakura--Saito divergence between quantile density functions. We prove that the proposed divergence is a one-parameter variation of the transport Kullback-Leibler divergence and the Hessian distance of negative Boltzmann entropy with respect to the Wasserstein-$2$ metric. From Taylor expansions, we also formulate the $3$-symmetric tensor in Wasserstein-$2$ space, which is given by an iterative Gamma three operator. The alpha--geodesic on Wasserstein space is also derived. From these properties, we name the proposed divergences transport alpha divergences. We provide several examples of transport alpha divergences on one dimensional distributions, such as generative models and Cauchy distributions.

cs.IT↗

arXiv2025

Accelerated Stein Variational Gradient Flow

Stein variational gradient descent (SVGD) is a kernel-based particle method for sampling from a target distribution, e.g., in generative modeling and Bayesian inference. SVGD does not require estimating the gradient of the log-density, which is called score estimation. In practice, SVGD can be slow compared to score-estimation based sampling algorithms. To design fast and efficient high-dimensional sampling algorithms, we introduce ASVGD, an accelerated SVGD, based on an accelerated gradient flow in a metric space of probability densities following Nesterov's method. We then derive a momentum-based discrete-time sampling algorithm, which evolves a set of particles deterministically. To stabilize the particles' momentum update, we also study a Wasserstein metric regularization. For the generalized bilinear kernel and the Gaussian kernel, toy numerical examples with varied target distributions demonstrate the effectiveness of ASVGD compared to SVGD and other popular sampling methods.

stat.ML↗

arXiv2025

Variational conditional normalizing flows for computing second-order mean field control problems

Mean field control (MFC) problems have vast applications in artificial intelligence, engineering, and economics, while solving MFC problems accurately and efficiently in high-dimensional spaces remains challenging. This work introduces variational conditional normalizing flow (VCNF), a neural network-based variational algorithm for solving general MFC problems based on flow maps. Formulating MFC problems as optimal control of Fokker-Planck (FP) equations with suitable constraints and cost functionals, we use VCNF to model the Lagrangian formulation of the MFC problems. In particular, VCNF builds upon conditional normalizing flows and neural spline flows, allowing efficient calculations of the inverse push-forward maps and score functions in MFC problems. We demonstrate the effectiveness of VCNF through extensive numerical examples, including optimal transport, regularized Wasserstein proximal operators, and flow matching problems for FP equations.

math.OC↗

arXiv2025

Splitting Regularized Wasserstein Proximal Algorithms for Nonsmooth Sampling Problems

Sampling from nonsmooth target probability distributions is essential in various applications, including the Bayesian Lasso. We propose a splitting-based sampling algorithm for the time-implicit discretization of the probability flow for the Fokker-Planck equation, where the score function, defined as the gradient logarithm of the current probability density function, is approximated by the regularized Wasserstein proximal. When the prior distribution is the Laplace prior, our algorithm is explicitly formulated as a deterministic interacting particle system, incorporating softmax operators and shrinkage operations to efficiently compute the gradient drift vector field and the score function. We verify the convergence towards target distributions regarding R\'enyi divergences and Wasserstein-2 distance under suitable conditions. Numerical experiments in high-dimensional nonsmooth sampling problems, such as sampling from mixed Gaussian and Laplace distributions, logistic regressions, image restoration with $L_1$-TV regularization, and Bayesian neural networks, demonstrate the efficiency and robust performance of the proposed method.

stat.CO↗

arXiv2025

Geometric calculations on density manifolds from reciprocal relations in hydrodynamics

Hydrodynamics describes the evolution of macroscopic states in non--equilibrium thermodynamics. Following Onsager reciprocal relations, one can formulate a large class of hydrodynamic equations as gradient flows of free energies. In recent years, such Onsager gradient flows have been extensively investigated on optimal transport type metric spaces with nonlinear mobilities, known as hydrodynamical density manifolds. A typical example is the gradient--drift Fokker--Planck equation, which can be characterized as the gradient flow of the free energy in the Wasserstein-2 metric space. This paper studies geometric calculations on general hydrodynamical density manifolds. We first formulate the associated Levi--Civita connections, gradients, Hessians, and parallel transports, and then derive the corresponding Riemannian and sectional curvatures. Finally, we obtain closed-form formulas for sectional curvatures in one dimensional density manifolds, where the signs of the curvatures are determined by the convexity of the mobility functions. As illustrations, we present density manifolds and their sectional curvatures for several zero range models, including independent particle systems, simple exclusion processes, and Kipnis--Marchioro--Presutti models.

math-ph↗

arXiv2024

A Natural Primal-Dual Hybrid Gradient Method for Adversarial Neural Network Training on Solving Partial Differential Equations

We propose a scalable preconditioned primal-dual hybrid gradient algorithm for solving partial differential equations (PDEs). We multiply the PDE with a dual test function to obtain an inf-sup problem whose loss functional involves lower-order differential operators. The Primal-Dual Hybrid Gradient (PDHG) algorithm is then leveraged for this saddle point problem. By introducing suitable precondition operators to the proximal steps in the PDHG algorithm, we obtain an alternative natural gradient ascent-descent optimization scheme for updating the neural network parameters. We apply the Krylov subspace method (MINRES) to evaluate the natural gradients efficiently. Such treatment readily handles the inversion of precondition matrices via matrix-vector multiplication. An \textit{a posteriori} convergence analysis is established for the time-continuous version of the proposed algorithm for general linear PDEs. By incorporating appropriate boundary loss terms, we further obtain a refined \textit{a priori} convergence result for elliptic equations in divergence form. The algorithm is tested on various types of PDEs with dimensions ranging from $1$ to $50$, including linear and nonlinear elliptic equations, reaction-diffusion equations, and Monge-Amp\`ere equations stemming from the $L^2$ optimal transport problems. We compare the performance of the proposed method with several commonly used deep learning algorithms such as physics-informed neural networks (PINNs), the DeepRitz method and weak adversarial networks (WANs) using either the Adam or the L-BFGS optimizer. The numerical results suggest that the proposed method performs efficiently and robustly and converges more stably with higher accuracy.

math.NA↗