Basic Inequalities for First-Order Optimization with Applications to Statistical Risk Analysis
In this work, we introduce $\textit{basic inequalities}$ for first-order iterative optimization algorithms, forming a simple yet versatile framework which connects implicit and explicit regularization. Building on related comparison inequalities for optimization iterates that already exist in the literature, we extend and unify these arguments to produce a general framework, which can be used as a tool for statistical analysis. In more detail, let $f$ denote the objective function to be optimized. Given a first-order iterative algorithm initialized at $θ_0$, with current iterate $θ_T$, the basic inequality upper bounds $f(θ_T) - f(z)$ for any reference point $z$ in terms of the accumulated step sizes, and the distances between $θ_0$, $θ_T$, and $z$. These distances are measured in a geometry inherent to the optimization algorithm, which then translates into a notion of regularization being applied across the path of iterates. In addition to refining existing results on gradient descent, we provide new results for mirror descent and other first-order methods. We then show how to use these basic inequalities to derive elementary yet useful bounds on the prediction risk of early-stopped gradient descent and exponentiated gradient descent iterates in generalized linear models. We also supplement these findings with numerical experiments.