arXiv subjects

Jinchao Xu

Publications and source records attributed to Jinchao Xu.

At least 55 records · Page 3Linked to original sources

arXiv2023

FV-MgNet: Fully Connected V-cycle MgNet for Interpretable Time Series Forecasting

By investigating iterative methods for a constrained linear model, we propose a new class of fully connected V-cycle MgNet for long-term time series forecasting, which is one of the most difficult tasks in forecasting. MgNet is a CNN model that was proposed for image classification based on the multigrid (MG) methods for solving discretized partial differential equations (PDEs). We replace the convolutional operations with fully connected operations in the existing MgNet and then apply them to forecasting problems. Motivated by the V-cycle structure in MG, we further propose the FV-MgNet, a V-cycle version of the fully connected MgNet, to extract features hierarchically. By evaluating the performance of FV-MgNet on popular data sets and comparing it with state-of-the-art models, we show that the FV-MgNet achieves better results with less memory usage and faster inference speed. In addition, we develop ablation experiments to demonstrate that the structure of FV-MgNet is the best choice among the many variants.

cs.AI↗

arXiv2022

An Interpretive Constrained Linear Model for ResNet and MgNet

We propose a constrained linear data-feature-mapping model as an interpretable mathematical model for image classification using a convolutional neural network (CNN). From this viewpoint, we establish detailed connections between the traditional iterative schemes for linear systems and the architectures of the basic blocks of ResNet- and MgNet-type models. Using these connections, we present some modified ResNet models that compared with the original models have fewer parameters and yet can produce more accurate results, thereby demonstrating the validity of this constrained learning data-feature-mapping assumption. Based on this assumption, we further propose a general data-feature iterative scheme to show the rationality of MgNet. We also provide a systematic numerical study on MgNet to show its success and advantages in image classification problems and demonstrate its advantages in comparison with established networks.

cs.CV↗

arXiv2022

On the Activation Function Dependence of the Spectral Bias of Neural Networks

Neural networks are universal function approximators which are known to generalize well despite being dramatically overparameterized. We study this phenomenon from the point of view of the spectral bias of neural networks. Our contributions are two-fold. First, we provide a theoretical explanation for the spectral bias of ReLU neural networks by leveraging connections with the theory of finite element methods. Second, based upon this theory we predict that switching the activation function to a piecewise linear B-spline, namely the Hat function, will remove this spectral bias, which we verify empirically in a variety of settings. Our empirical studies also show that neural networks with the Hat activation function are trained significantly faster using stochastic gradient descent and ADAM. Combined with previous work showing that the Hat activation function also improves generalization accuracy on image classification tasks, this indicates that using the Hat activation provides significant advantages over the ReLU on certain problems.

cs.LG↗

arXiv2022

A Priori Analysis of Stable Neural Network Solutions to Numerical PDEs

Methods for solving PDEs using neural networks have recently become a very important topic. We provide an a priori error analysis for such methods which is based on the $\mathcal{K}_1(\mathbb{D})$-norm of the solution. We show that the resulting constrained optimization problem can be efficiently solved using a greedy algorithm, which replaces stochastic gradient descent. Following this, we show that the error arising from discretizing the energy integrals is bounded both in the deterministic case, i.e. when using numerical quadrature, and also in the stochastic case, i.e. when sampling points to approximate the integrals. In the later case, we use a Rademacher complexity analysis, and in the former we use standard numerical quadrature bounds. This extends existing results to methods which use a general dictionary of functions to learn solutions to PDEs and importantly gives a consistent analysis which incorporates the optimization, approximation, and generalization aspects of the problem. In addition, the Rademacher complexity analysis is simplified and generalized, which enables application to a wide range of problems.

math.NA↗

arXiv2022

Extended Regularized Dual Averaging Methods for Stochastic Optimization

We introduce a new algorithm, extended regularized dual averaging (XRDA), for solving regularized stochastic optimization problems, which generalizes the regularized dual averaging (RDA) method. The main novelty of the method is that it allows a flexible control of the backward step size. For instance, the backward step size used in RDA grows without bound, while for XRDA the backward step size can be kept bounded. We demonstrate experimentally that additional control over the backward step size can significantly improve the convergence rate of the algorithm while preserving desired properties of the iterates, such as sparsity. Theoretically, we show that the XRDA method achieves the same convergence rate as RDA for general convex objectives.

math.OC↗

arXiv2022

A sharp Korn's inequality for piecewise $H^1$ space and its application

In this paper, we revisit Korn's inequality for the piecewise $H^1$ space based on general polygonal or polyhedral decompositions of the domain. Our Korn's inequality is expressed with minimal jump terms. These minimal jump terms are identified by characterizing the restriction of rigid body mode to edge/face of the partitions. Such minimal jump conditions are shown to be sharp for achieving the Korn's inequality as well. The sharpness of our result and explicitly given minimal conditions can be used to test whether any given finite element spaces satisfy Korn's inequality, immediately as well as to build or modify nonconforming finite elements for Korn's inequality to hold.

math.NA↗

arXiv2022

An efficient iterative method for dynamical Ginzburg-Landau equations

In this paper, we propose a new finite element approach to simulate the time-dependent Ginzburg-Landau equations under the temporal gauge, and design an efficient preconditioner for the Newton iteration of the resulting discrete system. The new approach solves the magnetic potential in H(curl) space by the lowest order of the second kind Nedelec element. This approach offers a simple way to deal with the boundary condition, and leads to a stable and reliable performance when dealing with the superconductor with reentrant corners. The comparison in numerical simulations verifies the efficiency of the proposed preconditioner, which can significantly speed up the simulation in large-scale computations.

math.NA↗

arXiv2022

ReLU Deep Neural Networks from the Hierarchical Basis Perspective

We study ReLU deep neural networks (DNNs) by investigating their connections with the hierarchical basis method in finite element methods. First, we show that the approximation schemes of ReLU DNNs for $x^2$ and $xy$ are composition versions of the hierarchical basis approximation for these two functions. Based on this fact, we obtain a geometric interpretation and systematic proof for the approximation result of ReLU DNNs for polynomials, which plays an important role in a series of recent exponential approximation results of ReLU DNNs. Through our investigation of connections between ReLU DNNs and the hierarchical basis approximation for $x^2$ and $xy$, we show that ReLU DNNs with this special structure can be applied only to approximate quadratic functions. Furthermore, we obtain a concise representation to explicitly reproduce any linear finite element function on a two-dimensional uniform mesh by using ReLU DNNs with only two hidden layers.

math.NA↗

arXiv2022

Approximation Properties of Deep ReLU CNNs

This paper focuses on establishing $L^2$ approximation properties for deep ReLU convolutional neural networks (CNNs) in two-dimensional space. The analysis is based on a decomposition theorem for convolutional kernels with a large spatial size and multi-channels. Given the decomposition result, the property of the ReLU activation function, and a specific structure for channels, a universal approximation theorem of deep ReLU CNNs with classic structure is obtained by showing its connection with one-hidden-layer ReLU neural networks (NNs). Furthermore, approximation properties are obtained for one version of neural networks with ResNet, pre-act ResNet, and MgNet architecture based on connections between these networks.

cs.LG↗

arXiv2022

Characterization of the Variation Spaces Corresponding to Shallow Neural Networks

We study the variation space corresponding to a dictionary of functions in $L^2(Ω)$ for a bounded domain $Ω\subset \mathbb{R}^d$. Specifically, we compare the variation space, which is defined in terms of a convex hull with related notions based on integral representations. This allows us to show that three important notions relating to the approximation theory of shallow neural networks, the Barron space, the spectral Barron space, and the Radon BV space, are actually variation spaces with respect to certain natural dictionaries.

stat.ML↗

arXiv2022

Optimal Convergence Rates for the Orthogonal Greedy Algorithm

We analyze the orthogonal greedy algorithm when applied to dictionaries $\mathbb{D}$ whose convex hull has small entropy. We show that if the metric entropy of the convex hull of $\mathbb{D}$ decays at a rate of $O(n^{-\frac{1}{2}-α})$ for $α> 0$, then the orthogonal greedy algorithm converges at the same rate on the variation space of $\mathbb{D}$. This improves upon the well-known $O(n^{-\frac{1}{2}})$ convergence rate of the orthogonal greedy algorithm in many cases, most notably for dictionaries corresponding to shallow neural networks. These results hold under no additional assumptions on the dictionary beyond the decay rate of the entropy of its convex hull. In addition, they are robust to noise in the target function and can be extended to convergence rates on the interpolation spaces of the variation norm. We show empirically that the predicted rates are obtained for the dictionary corresponding to shallow neural networks with Heaviside activation function in two dimensions. Finally, we show that these improved rates are sharp and prove a negative result showing that the iterates generated by the orthogonal greedy algorithm cannot in general be bounded in the variation norm of $\mathbb{D}$.

math.ST↗

arXiv2021

Extensible Structure-Informed Prediction of Formation Energy with Improved Accuracy and Usability employing Neural Networks

In the present paper, we introduce a new neural network-based tool for the prediction of formation energies of atomic structures based on elemental and structural features of Voronoi-tessellated materials. We provide a concise overview of the connection between the machine learning and the true material-property relationship, how to improve the generalization accuracy by reducing overfitting, how new data can be incorporated into the model to tune it to a specific material system, and preliminary results on using models to preform local structure relaxations. The present work resulted in three final models optimized for (1) highest test accuracy on the Open Quantum Materials Database (OQMD), (2) performance in the discovery of new materials, and (3) performance at a low computational cost. On a test set of 21,800 compounds randomly selected from OQMD, they achieve a mean absolute error (MAE) of 28, 40, and 42 meV/atom, respectively. The second model provides better predictions in a test case of interest not present in the OQMD, while the third reduces the computational cost by a factor of 8. We collect our results in a new open-source tool called SIPFENN (Structure-Informed Prediction of Formation Energy using Neural Networks). SIPFENN not only improves the accuracy beyond existing models but also ships in a ready-to-use form with pre-trained neural networks and a GUI interface. By virtue of this, it can be included in DFT calculations routines at nearly no cost.

cond-mat.mtrl-sci↗

arXiv2021

High-Order Approximation Rates for Shallow Neural Networks with Cosine and ReLU$^k$ Activation Functions

We study the approximation properties of shallow neural networks with an activation function which is a power of the rectified linear unit. Specifically, we consider the dependence of the approximation rate on the dimension and the smoothness in the spectral Barron space of the underlying function $f$ to be approximated. We show that as the smoothness index $s$ of $f$ increases, shallow neural networks with ReLU$^k$ activation function obtain an improved approximation rate up to a best possible rate of $O(n^{-(k+1)}\log(n))$ in $L^2$, independent of the dimension $d$. The significance of this result is that the activation function ReLU$^k$ is fixed independent of the dimension, while for classical methods the degree of polynomial approximation or the smoothness of the wavelets used would have to increase in order to take advantage of the dimension dependent smoothness of $f$. In addition, we derive improved approximation rates for shallow neural networks with cosine activation function on the spectral Barron space. Finally, we prove lower bounds showing that the approximation rates attained are optimal under the given assumptions.

math.NA↗

arXiv2021

New Discontinuous Galerkin Algorithms and Analysis for Linear Elasticity with Symmetric Stress Tensor

This paper presents a new and unified approach to the derivation and analysis of many existing, as well as new discontinuous Galerkin methods for linear elasticity problems. The analysis is based on a unified discrete formulation for the linear elasticity problem consisting of four discretization variables: strong symmetric stress tensor $\dsig$ and displacement $\du$ inside each element, and the modifications of these two variables $\hsig$ and $\hu$ on elementary boundaries of elements. Motivated by many relevant methods in the literature, this formulation can be used to derive most existing discontinuous, nonconforming and conforming Galerkin methods for linear elasticity problems and especially to develop a number of new discontinuous Galerkin methods. Many special cases of this four-field formulation are proved to be hybridizable and can be reduced to some known hybridizable discontinuous Galerkin, weak Galerkin and local discontinuous Galerkin methods by eliminating one or two of the four fields. As certain stabilization parameter tends to zero, this four-field formulation is proved to converge to some conforming and nonconforming mixed methods for linear elasticity problems. Two families of inf-sup conditions, one known as $H^1$-based and the other known as $H({\rm div})$-based, are proved to be uniformly valid with respect to different choices of discrete spaces and parameters. These inf-sup conditions guarantee the well-posedness of the new proposed methods and also offer a new and unified analysis for many existing methods in the literature as a by-product. Some numerical examples are provided to verify the theoretical analysis including the optimal convergence of the new proposed methods.

math.NA↗

arXiv2021

Sharp Lower Bounds on the Approximation Rate of Shallow Neural Networks

We consider the approximation rates of shallow neural networks with respect to the variation norm. Upper bounds on these rates have been established for sigmoidal and ReLU activation functions, but it has remained an important open problem whether these rates are sharp. In this article, we provide a solution to this problem by proving sharp lower bounds on the approximation rates for shallow neural networks, which are obtained by lower bounding the $L^2$-metric entropy of the convex hull of the neural network basis functions. In addition, our methods also give sharp lower bounds on the Kolmogorov $n$-widths of this convex hull, which show that the variation spaces corresponding to shallow neural networks cannot be efficiently approximated by linear methods. These lower bounds apply to both sigmoidal activation functions with bounded variation and to activation functions which are a power of the ReLU. Our results also quantify how much stronger the Barron spectral norm is than the variation norm and, combined with previous results, give the asymptotics of the $L^\infty$-metric entropy up to logarithmic factors in the case of the ReLU activation function.

stat.ML↗

arXiv2021

Robust BPX Preconditioner for Fractional Laplacians on Bounded Lipschitz Domains

We propose and analyze a robust BPX preconditioner for the integral fractional Laplacian on bounded Lipschitz domains. For either quasi-uniform grids or graded bisection grids, we show that the condition numbers of the resulting systems remain uniformly bounded with respect to both the number of levels and the fractional power. The results apply also to the spectral and censored fractional Laplacians.

math.NA↗

arXiv2021

Training Sparse Neural Networks using Compressed Sensing

Pruning the weights of neural networks is an effective and widely-used technique for reducing model size and inference complexity. We develop and test a novel method based on compressed sensing which combines the pruning and training into a single step. Specifically, we utilize an adaptively weighted $\ell^1$ penalty on the weights during training, which we combine with a generalization of the regularized dual averaging (RDA) algorithm in order to train sparse neural networks. The adaptive weighting we introduce corresponds to a novel regularizer based on the logarithm of the absolute value of the weights. We perform a series of ablation studies demonstrating the improvement provided by the adaptive weighting and generalized RDA algorithm. Furthermore, numerical experiments on the CIFAR-10, CIFAR-100, and ImageNet datasets demonstrate that our method 1) trains sparser, more accurate networks than existing state-of-the-art methods; 2) can be used to train sparse networks from scratch, i.e. from a random initialization, as opposed to initializing with a well-trained base model; 3) acts as an effective regularizer, improving generalization accuracy.

cs.CV↗

arXiv2021

Helicity-conservative finite element discretization for incompressible MHD systems

We construct finite element methods for the incompressible magnetohydrodynamics (MHD) system that precisely preserve magnetic and cross helicity, the energy law and the magnetic Gauss law at the discrete level. The variables are discretized as discrete differential forms in a de Rham complex. We present numerical tests to show the performance of the algorithm.

math.NA↗