Estimation and inference for Deep Neuronal Networks

Abstract

Nonlinear regression problem is one of the most popular and important statistical tasks. The first methods like least squares estimation go back to Gauss and Legendre. Recent models and developments in statistics and machine learning like Deep Neuronal Networks (DNN) or nonlinear PDE stimulate new research in this direction which has to address the important issues and challenges of modern statistical inference such as huge complexity and parameter dimension of the model, limited sample size, lack of convexity and identifiability, among many others. Classical results of nonparametric statistics in terms of rate of convergence do not really address the mentioned issues. This paper offers a general approach to studying a nonlinear regression problem based on the notion of effective dimension. First, a special case of models with stochastically linear structure (SLS) is studied. The results provide finite sample expansions for the loss of the penalized maximum likelihood estimation (MLE). The leading term of such expansions as well as the corresponding remainder are given via the effective dimension and the effective sample size. The obtained expansions can be used to obtain sharp risk bounds and for statistical inference. Despite generality, all the presented bounds are nearly sharp and the classical asymptotic results can be obtained as simple corollaries. Although the basic SLS assumptions are not fulfilled for nonlinear smooth regression, we explain how the stochastic linearity can be achieved by extending the parameter space. The obtained general results are specified to nonlinear smooth regression and to a DNN with one hidden layer.