arXiv subjects

Yiqi Gu

Publications and source records attributed to Yiqi Gu.

16 recordsLinked to original sources

arXiv2026

Kernel Localization and Whole-Trajectory Generalization for Linear Multistep Methods in Deep Learning-Based Discovery of Dynamical Systems

Linear multistep methods (LMMs) combined with neural-network approximation provide a high-order framework for learning governing vector fields of dynamical systems from discrete trajectory data. This paper studies two issues in LMM-based discovery that are not resolved by existing grid-level convergence theory. First, in non-auxiliary Adams--Bashforth (A-B) and Adams--Moulton (A-M) discovery systems, we observe that zero-residual grid solutions are nonunique but consistent along the trajectory, with differences limited to the boundary layer. We explain this phenomenon through a kernel analysis of the non-auxiliary discovery matrices. Under the corresponding discovery-stability conditions, the differences between zero-residual grid solutions are exponentially localized near the initial indices for A-B schemes, whereas they form two-sided boundary layers near the initial and terminal indices for A-M schemes. Second, we derive whole-trajectory generalization estimates for both auxiliary and non-auxiliary formulations. Once the learned vector field is restricted to a fixed observed trajectory, each component of the error becomes a scalar function of time. For the auxiliary formulation, the trajectory error is $O(h^p)$ under the corresponding grid accuracy, approximation, and trace regularity assumptions. For non-auxiliary formulations, the global estimates contain additional boundary-layer terms. On fixed interior subintervals, these terms are exponentially damped. Numerical experiments illustrate the convergence behavior.

math.NA↗

arXiv2026

A Layer Separation Optimization Framework for Cross-Entropy Training in Deep Learning

This paper investigates the deep learning optimization problem with softmax cross-entropy loss. We propose a layer separation strategy to alleviate the strong nonconvexity encountered during training deep networks. For cross-entropy models with fully connected and convolutional neural networks, we introduce auxiliary variables associated with hidden layer outputs and construct corresponding layer separation models, which decompose the original deeply nested optimization problem into a sequence of more manageable subproblems. We also conduct theoretical analyses, proving that the new layer separation loss provides an upper bound for the original cross-entropy loss. Moreover, we design alternating minimization algorithms and prove that, under appropriate conditions, these algorithms exhibit decreasing properties of the loss function. Numerical experiments validate the effectiveness of the proposed methods and indicate improved optimization behavior, especially for fully connected and convolutional neural networks.

cs.LG↗

arXiv2026

Operator splitting based diffusion samplers and improved convergence analysis

In this paper, we develop a class of samplers for the diffusion model using the operator-splitting technique. The linear drift term and the nonlinear score-driven drift of the probability flow ordinary differential equation are split and applied by flow maps alternatively. Moreover, we conduct detailed analyses for the second-order sampler, establishing a non-asymptotic total variation distance error bound of order $O(d/T^2+\sqrt{d}\varepsilon_{\mathrm{score}}+d\varepsilon_{\mathrm{Jac}})$, where $d$ is the data dimension; $T$ is the number of sampling steps; $\varepsilon_{\mathrm{score}}$ and $\varepsilon_{\mathrm{Jac}}$ measure the discrepancy between the actual score function and learned score function. Our bound is sharper than existing works, yielding bounds of $O(d^p/T^2)$ with some $p>1$ for specific second-order samplers. Numerical experiments on a two-dimensional synthetic dataset corroborate the established quadratic dependence on the step size $1/T$ in the error bound.

math.NA↗

arXiv2025

General transformation neural networks: A class of parametrized functions for high-dimensional function approximation

We propose a novel class of neural network-like parametrized functions, i.e., general transformation neural networks (GTNNs), for high-dimensional approximation. Conventional deep neural networks sometimes perform less accurately on learning problems trained with gradient descent, especially when the target function is oscillatory. To improve accuracy, we generalize the neuron's affine transformation to a broader class of functions that can capture complex shapes and offer greater capacity. Specifically, we discuss three types of GTNNs in detail: the cubic, quadratic and trigonometric transformation neural networks (CTNNs, QTNNs and TTNNs). We perform an approximation error analysis of GTNNs, presenting their universal approximation properties for continuous functions, error bounds for Barron-type functions and error bounds of deep architectures. Several numerical examples of regression problems are presented, demonstrating that CTNNs/QTNNs/TTNNs achieve higher accuracy than conventional fully connected neural networks.

math.NA↗

arXiv2025

Data-driven optimized high-order WENO schemes with low-dissipation and low-dispersion

Classical high-order weighted essentially non-oscillatory (WENO) schemes are designed to achieve optimal convergence order for smooth solutions and to maintain non-oscillatory behaviors for discontinuities. However, their spectral properties are not optimal, which limits the ability to capture high-frequency waves and small-scale features. In this paper, we propose a data-driven optimized method to improve the spectral properties of the WENO schemes. By analyzing the approximate dispersion relation (ADR), the spectral error of the schemes can be bounded by the reconstructed errors of a series of trigonometric functions with different wavenumbers. Therefore, we propose the new schemes WENO5-JS/Z-NN that introduce a compensation term parameterized by a neural network to the weight function of the WENO5-JS/Z schemes. The neural network is trained such that the generated weights can minimize the reconstructed errors over a large number of spatial stencils, and furthermore, improve the spectral accuracy. Meanwhile, the Total Variation Diminishing (TVD) constraint and anti-dissipation penalization are incorporated into the loss function to enhance the shock-capturing capability and preserve stability in simulating high-frequency waves. Compared to WENO5-JS/Z, our schemes maintain the ability to capture discontinuities while providing higher resolution for fine-scale flow features. The ADR indicates that the new schemes can match the exact spectrum more accurately over a broader range of wavenumbers.

math.NA↗

arXiv2025

Optimization and generalization analysis for two-layer physics-informed neural networks without over-parametrization

This work focuses on the behavior of stochastic gradient descent (SGD) in solving least-squares regression with physics-informed neural networks (PINNs). Past work on this topic has been based on the over-parameterization regime, whose convergence may require the network width to increase vastly with the number of training samples. So, the theory derived from over-parameterization may incur prohibitive computational costs and is far from practical experiments. We perform new optimization and generalization analysis for SGD in training two-layer PINNs, making certain assumptions about the target function to avoid over-parameterization. Given $\epsilon>0$, we show that if the network width exceeds a threshold that depends only on $\epsilon$ and the problem, then the training loss and expected loss will decrease below $O(\epsilon)$.

cs.LG↗

arXiv2025

Layer Separation Deep Learning Model with Auxiliary Variables for Partial Differential Equations

In this paper, we propose a new optimization framework, the layer separation (LySep) model, to improve the deep learning-based methods in solving partial differential equations. Due to the highly non-convex nature of the loss function in deep learning, existing optimization algorithms often converge to suboptimal local minima or suffer from gradient explosion or vanishing, resulting in poor performance. To address these issues, we introduce auxiliary variables to separate the layers of deep neural networks. Specifically, the output and its derivatives of each layer are represented by auxiliary variables, effectively decomposing the deep architecture into a series of shallow architectures. New loss functions with auxiliary variables are established, in which only variables from two neighboring layers are coupled. Corresponding algorithms based on alternating directions are developed, where many variables can be updated optimally in closed forms. Moreover, we provide theoretical analyses demonstrating the consistency between the LySep model and the original deep model. High-dimensional numerical results validate our theory and demonstrate the advantages of LySep in minimizing loss and reducing solution error.

cs.LG↗

arXiv2025

Deep Learning Optimization Using Self-Adaptive Weighted Auxiliary Variables

In this paper, we develop a new optimization framework for the least squares learning problem via fully connected neural networks or physics-informed neural networks. The gradient descent sometimes behaves inefficiently in deep learning because of the high non-convexity of loss functions and the vanishing gradient issue. Our idea is to introduce auxiliary variables to separate the layers of the deep neural networks and reformulate the loss functions for ease of optimization. We design the self-adaptive weights to preserve the consistency between the reformulated loss and the original mean squared loss, which guarantees that optimizing the new loss helps optimize the original problem. Numerical experiments are presented to verify the consistency and show the effectiveness and robustness of our models over gradient descent.

cs.LG↗

arXiv2025

Learning high-accuracy numerical schemes for hyperbolic equations on coarse meshes

When solving partial differential equations using classical schemes such as finite difference or finite volume methods, sufficiently fine meshes and carefully designed schemes are required to achieve high-order accuracy of numerical solutions, leading to a significant increase in computational costs, especially for three-dimensional (3D) time-dependent problems. Recently, machine learning-assisted numerical methods have been proposed to enhance accuracy or efficiency. In this paper, we propose a data-driven finite difference numerical method to solve the hyperbolic equations with smooth solutions on coarse grids, which can achieve higher accuracy than classical numerical schemes based on the same mesh size. In addition, the data-driven schemes have better spectrum properties than the classical schemes, although the spectrum properties are not explicitly optimized during the training process. Numerical examples are presented to demonstrate the accuracy and efficiency of the proposed method, as well as its good performance on dispersion and dissipation.

math.NA↗

arXiv2023

Deep neural networks for solving large linear systems arising from high-dimensional problems

This paper studies deep neural networks for solving extremely large linear systems arising from highdimensional problems. Because of the curse of dimensionality, it is expensive to store both the solution and right-hand side vector in such extremely large linear systems. Our idea is to employ a neural network to characterize the solution with much fewer parameters than the size of the solution under a matrix-free setting. We present an error analysis of the proposed method, indicating that the solution error is bounded by the condition number of the matrix and the neural network approximation error. Several numerical examples from partial differential equations, queueing problems, and probabilistic Boolean networks are presented to demonstrate that the solutions of linear systems can be learned quite accurately.

math.NA↗

arXiv2022

The Discovery of Dynamics via Linear Multistep Methods and Deep Learning: Error Estimation

Identifying hidden dynamics from observed data is a significant and challenging task in a wide range of applications. Recently, the combination of linear multistep methods (LMMs) and deep learning has been successfully employed to discover dynamics, whereas a complete convergence analysis of this approach is still under development. In this work, we consider the deep network-based LMMs for the discovery of dynamics. We put forward error estimates for these methods using the approximation property of deep networks. It indicates, for certain families of LMMs, that the $\ell^2$ grid error is bounded by the sum of $O(h^p)$ and the network approximation error, where $h$ is the time step size and $p$ is the local truncation error order. Numerical results of several physically relevant examples are provided to demonstrate our theory.

math.NA↗

arXiv2022

Deep adaptive basis Galerkin method for high-dimensional evolution equations with oscillatory solutions

In this paper, we study deep neural networks (DNNs) for solving high-dimensional evolution equations with oscillatory solutions. Different from deep least-squares methods that deal with time and space variables simultaneously, we propose a deep adaptive basis Galerkin (DABG) method, which employs the spectral-Galerkin method for the time variable of oscillatory solutions and the deep neural network method for high-dimensional space variables. The proposed method can lead to a linear system of differential equations having unknown DNNs that can be trained via the loss function. We establish a posterior estimates of the solution error, which is bounded by the minimal loss function and the term $O(N^{-m})$, where $N$ is the number of basis functions and $m$ characterizes the regularity of the e'quation. We also show that if the true solution is a Barron-type function, the error bound converges to zero as $M=O(N^p)$ approaches to infinity, where $M$ is the width of the used networks, and $p$ is a positive constant. Numerical examples, including high-dimensional linear evolution equations and the nonlinear Allen-Cahn equation, are presented to demonstrate the performance of the proposed DABG method is better than that of existing DNNs.

math.NA↗

arXiv2021

Deep Ritz method for the spectral fractional Laplacian equation using the Caffarelli-Silvestre extension

In this paper, we propose a novel method for solving high-dimensional spectral fractional Laplacian equations. Using the Caffarelli-Silvestre extension, the $d$-dimensional spectral fractional equation is reformulated as a regular partial differential equation of dimension $d+1$. We transform the extended equation as a minimal Ritz energy functional problem and search for its minimizer in a special class of deep neural networks. Moreover, based on the approximation property of networks, we establish estimates on the error made by the deep Ritz method. Numerical results are reported to demonstrate the effectiveness of the proposed method for solving fractional Laplacian equations up to ten dimensions. Technically, in this method, we design a special network-based structure to adapt to the singularity and exponential decaying of the true solution. Also, A hybrid integration technique combining Monte Carlo method and sinc quadrature is developed to compute the loss function with higher accuracy.

math.NA↗

arXiv2021

Stationary Density Estimation of Itô Diffusions Using Deep Learning

In this paper, we consider the density estimation problem associated with the stationary measure of ergodic Itô diffusions from a discrete-time series that approximate the solutions of the stochastic differential equations. To take an advantage of the characterization of density function through the stationary solution of a parabolic-type Fokker-Planck PDE, we proceed as follows. First, we employ deep neural networks to approximate the drift and diffusion terms of the SDE by solving appropriate supervised learning tasks. Subsequently, we solve a steady-state Fokker-Plank equation associated with the estimated drift and diffusion coefficients with a neural-network-based least-squares method. We establish the convergence of the proposed scheme under appropriate mathematical assumptions, accounting for the generalization errors induced by regressing the drift and diffusion coefficients, and the PDE solvers. This theoretical study relies on a recent perturbation theory of Markov chain result that shows a linear dependence of the density estimation to the error in estimating the drift term, and generalization error results of nonparametric regression and of PDE regression solution obtained with neural-network models. The effectiveness of this method is reflected by numerical simulations of a two-dimensional Student's t distribution and a 20-dimensional Langevin dynamics.

math.NA↗

arXiv2021

SelectNet: Self-paced Learning for High-dimensional Partial Differential Equations

The least squares method with deep neural networks as function parametrization has been applied to solve certain high-dimensional partial differential equations (PDEs) successfully; however, its convergence is slow and might not be guaranteed even within a simple class of PDEs. To improve the convergence of the network-based least squares model, we introduce a novel self-paced learning framework, SelectNet, which quantifies the difficulty of training samples, treats samples equally in the early stage of training, and slowly explores more challenging samples, e.g., samples with larger residual errors, mimicking the human cognitive process for more efficient learning. In particular, a selection network and the PDE solution network are trained simultaneously; the selection network adaptively weighting the training samples of the solution network achieving the goal of self-paced learning. Numerical examples indicate that the proposed SelectNet model outperforms existing models on the convergence speed and the convergence robustness, especially for low-regularity solutions.

math.NA↗

arXiv2021

Structure Probing Neural Network Deflation

Deep learning is a powerful tool for solving nonlinear differential equations, but usually, only the solution corresponding to the flattest local minimizer can be found due to the implicit regularization of stochastic gradient descent. This paper proposes a network-based structure probing deflation method to make deep learning capable of identifying multiple solutions that are ubiquitous and important in nonlinear physical models. First, we introduce deflation operators built with known solutions to make known solutions no longer local minimizers of the optimization energy landscape. Second, to facilitate the convergence to the desired local minimizer, a structure probing technique is proposed to obtain an initial guess close to the desired local minimizer. Together with neural network structures carefully designed in this paper, the new regularized optimization can converge to new solutions efficiently. Due to the mesh-free nature of deep learning, the proposed method is capable of solving high-dimensional problems on complicated domains with multiple solutions, while existing methods focus on merely one or two-dimensional regular domains and are more expensive in operation counts. Numerical experiments also demonstrate that the proposed method could find more solutions than exiting methods.

math.NA↗