The Equivariance Criterion in a Linear Model for Fixed-X Cases

Abstract

The field of machine have seen rising applications of equivariance criterion. However, there is no systematic way to justify its usage, including why it works, whether there is an optimal solution and if so, what form it carries. In this article, we explored the usage of equivariance criterion in a normal linear model with fixed-$X$ and extended the model to allow multiple populations, which, in turn, leads to a multivariate invariant location-scale transformation group, compared than the commonly used univariate one. The minimum risk equivariant estimators of the coefficient vector and the diagonal covariance matrix were derived, which were consistent with literature works. This work serves as an early exploration of the usage of equivariance criterion in machine learning, where we confirmed that the least square approach widely used in machine learning indeed carries optimality in some sense at least in the framework of estimation. Meanwhile, the problems can be shown to be equivalent to a mixture from $p$ independent normal samples and via the principle of functional equivariance, an alternative proof can be derived. However, such an approach carries its own limitation with a strong tie to equivariance criterion.