Investigating Statistical Inference and Covariate Effects in Shallow Neural Networks

Abstract

Feedforward neural networks (FNNs) are typically viewed as pure prediction algorithms, and their strong predictive performance has led to their use in many machine-learning applications. However, their flexibility comes with an interpretability trade-off; thus, FNNs have been historically less popular among statisticians. Nevertheless, for suitably parsimonious shallow FNNs, classical statistical theory, such as significance testing and uncertainty quantification, may still provide useful regression-style summaries. Supplementing FNNs with methods of statistical inference, and covariate-effect visualisations, can shift the focus away from black-box prediction and move FNNs towards traditional statistical models. This can allow for more inferential analysis, and, hence, make FNNs more accessible within the statistical-modelling context. We investigate covariate-level Wald testing in the context of penalised FNNs, and also propose covariate-effect plots that emulate regression coefficients. Simulation studies are used to extensively investigate the performance of Wald-based inference, with particular emphasis on when this approach performs well. This statistical-based approach to neural networks is demonstrated through an application to insurance data.