arXiv subjects

Tengchao Yu

Publications and source records attributed to Tengchao Yu.

7 recordsLinked to original sources

arXiv2024

Numerical Approximation Capacity of Neural Networks with Bounded Parameters: Do Limits Exist, and How Can They Be Measured?

The Universal Approximation Theorem posits that neural networks can theoretically possess unlimited approximation capacity with a suitable activation function and a freely chosen or trained set of parameters. However, a more practical scenario arises when these neural parameters, especially the nonlinear weights and biases, are bounded. This leads us to question: \textbf{Does the approximation capacity of a neural network remain universal, or does it have a limit when the parameters are practically bounded? And if it has a limit, how can it be measured?} Our theoretical study indicates that while universal approximation is theoretically feasible, in practical numerical scenarios, Deep Neural Networks (DNNs) with any analytic activation functions (such as Tanh and Sigmoid) can only be approximated by a finite-dimensional vector space under a bounded nonlinear parameter space (NP space), whether in a continuous or discrete sense. Based on this study, we introduce the concepts of \textit{$\epsilon$ outer measure} and \textit{Numerical Span Dimension (NSdim)} to quantify the approximation capacity limit of a family of networks both theoretically and practically. Furthermore, drawing on our new theoretical study and adopting a fresh perspective, we strive to understand the relationship between back-propagation neural networks and random parameter networks (such as the Extreme Learning Machine (ELM)) with both finite and infinite width. We also aim to provide fresh insights into regularization, the trade-off between width and depth, parameter space, width redundancy, condensation, and other related important issues.

cs.LG↗

arXiv2022

Discontinuity Computing using Physics-Informed Neural Network

Simulating discontinuities is a long standing problem especially for shock waves with strong nonlinear feather. Despite being a promising method, the recently developed physics-informed neural network (PINN) is still weak for calculating discontinuities compared with traditional shock-capturing methods. In this paper, we intend to improve the shock-capturing ability of the PINN. The primary strategy of this work is to weaken the expression of the network near discontinuities by adding a gradient-weight into the governing equations locally at each residual point. This strategy allows the network to focus on training smooth parts of the solutions. Then, automatically affected by the compressible property near shock waves, a sharp discontinuity appears with wrong inside shock transition-points compressed into well-trained smooth regions as passive particles. We study the solutions of one-dimensional Burgers equation and one- and two-dimensional Euler equations. Compared with the traditional high-order WENO-Z method in numerical examples, the proposed method can substantially improve discontinuity computing.

physics.flu-dyn↗

arXiv2021

Inverse Gaussian Process regression for likelihood-free inference

In this work we consider Bayesian inference problems with intractable likelihood functions. We present a method to compute an approximate of the posterior with a limited number of model simulations. The method features an inverse Gaussian Process regression (IGPR), i.e., one from the output of a simulation model to the input of it. Within the method, we provide an adaptive algorithm with a tempering procedure to construct the approximations of the marginal posterior distributions. With examples we demonstrate that IGPR has a competitive performance compared to some commonly used algorithms, especially in terms of statistical stability and computational efficiency, while the price to pay is that it can only compute a weighted Gaussian approximation of the marginal posteriors.

stat.CO↗

arXiv2019

Bayesian optimization with local search

Global optimization finds applications in a wide range of real world problems. The multi-start methods are a popular class of global optimization techniques, which are based on the ideas of conducting local searches at multiple starting points. In this work we propose a new multi-start algorithm where the starting points are determined in a Bayesian optimization framework. Specifically, the method can be understood as to construct a new function by conducting local searches of the original objective function, where the new function attains the same global optima as the original one. Bayesian optimization is then applied to find the global optima of the new local search defined function.

stat.ML↗

arXiv2019

Maximum conditional entropy Hamiltonian Monte Carlo sampler

The performance of Hamiltonian Monte Carlo (HMC) sampler depends critically on some algorithm parameters such as the total integration time and the numerical integration stepsize. The parameter tuning is particularly challenging when the mass matrix of the HMC sampler is adapted. We propose in this work a Kolmogorov-Sinai entropy (KSE) based design criterion to optimize these algorithm parameters, which can avoid some potential issues in the often used jumping-distance based measures. For near-Gaussian distributions, we are able to derive the optimal algorithm parameters with respect to the KSE criterion analytically. As a byproduct the KSE criterion also provides a theoretical justification for the need to adapt the mass matrix in HMC sampler. Based on the results, we propose an adaptive HMC algorithm, and we then demonstrate the performance of the proposed algorithm with numerical examples.

stat.CO↗

arXiv2019

Bayesian inference and uncertainty quantification for image reconstruction with Poisson data

We provide a complete framework for performing infinite-dimensional Bayesian inference and uncertainty quantification for image reconstruction with Poisson data. In particular, we address the following issues to make the Bayesian framework applicable in practice. We first introduce a positivity-preserving reparametrization, and we prove that under the reparametrization and a hybrid prior, the posterior distribution is well-posed in the infinite dimensional setting. Second we provide a dimension-independent MCMC algorithm, based on the preconditioned Crank-Nicolson Langevin method, in which we use a primal-dual scheme to compute the offset direction. Third we give a method combining the model discrepancy method and maximum likelihood estimation to determine the regularization parameter in the hybrid prior. Finally we propose to use the obtained posterior distribution to detect artifacts in a recovered image. We provide an example to demonstrate the effectiveness of the proposed method.

math.NA↗

arXiv2018

A weight-bounded importance sampling method for variance reduction

Importance sampling (IS) is an important technique to reduce the estimation variance in Monte Carlo simulations. In many practical problems, however, the use of IS method may result in unbounded variance, and thus fail to provide reliable estimates. To address the issue, we propose a method which can prevent the risk of unbounded variance; the proposed method performs the standard IS for the integral of interest in a region only in which the IS weight is bounded and use the result as an approximation to the original integral. It can be verified that the resulting estimator has a finite variance. Moreover, we also provide a normality test based method to identify the region with bounded IS weight (termed as the safe region) from the samples drawn from the standard IS distribution. With numerical examples, we demonstrate that the proposed method can yield rather reliable estimate when the standard IS fails, and it also outperforms the defensive IS, a popular method to prevent unbounded variance.

stat.CO↗