Hinge-Proximal Stochastic Gradient Methods for Convex Optimization with Functional Constraints
This paper considers stochastic convex optimization problems with smooth functional constraints arising in constrained estimation and robust signal recovery. We operate in the high-dimensional and highly-constrained setting, where oracle access is restricted to one or a few objective and constraint gradients per-iteration, as in streaming or online estimation. Existing approaches to solve such problems are based on either the stochastic primal-dual or stochastic subgradient methods, and require globally Lipschitz continuous constraint functions. In this work, we develop a hinge-proximal framework that utilizes an exact penalty reformulation to yield updates involving only one linearized constraint (and hence accessing one constraint gradient) per-iteration. The updates also admit a novel hinge-proximal three-point inequality relying on smoothness rather than global Lipschitz continuity of the constraint functions. The framework leads to three algorithms: a baseline hinge-proximal SGD (HPS), a variance-reduced HPS version for finite-sum settings, and a nested HPS version whose performance depends on a geometric regularity constant of the constraint region rather than explicitly on the number of constraints, while achieving near-SGD sample complexity. The superior empirical performance of the proposed algorithms is demonstrated on a robust regression problem with noisy features, representative of errors-in-variables estimation.