Smooth Reduced Rank Regression with P-splines

Abstract

Linear regression is one of the core statistical tools used for analysis of data. In the era of statistical learning, linear regression has been expanded into two directions. The first is regularisation, where penalties are added to the loss function to obtain more stable or sparse solutions. The second direction is basis expansion, such as with spline or kernel functions, where the linearity assumption is dropped. In practice, empirical researchers often collect multiple outcome variables. Regression models, either linear, regularized, or expanded, can be fitted to each of these outcome variables, but such an approach does not take into account the associations among the response variables. Reduced rank regression is a multivariate regression tool that takes into account and models the association among the response variables. In this paper, we develop and test a B-spline basis expansion for reduced rank regression, a regression model for multiple outcome variables where we penalize the coefficients to obtain a smooth fit, as in P-splines. This approach is useful for the analysis of data sets with multiple outcomes and a relatively small number of predictors. A block-relaxation algorithm is developed for parameter estimation and we discuss ways for penalty parameter tuning. We develop visualizations tools for model interpretation based on biplot methodology for ``all predictor - all response'' relationships and partial dependence plots for interpreting ``single predictor - single response'' relationships. With show several experimental gauges and we analyze three empirical data sets. We conclude this paper with a discussion.