A Generalized Energy-Based Adaptive Gradient Method for Optimization

Abstract

Adaptive Gradient Descent with Energy (AEGD) is a variant of Gradient Descent (GD) designed to address step size sensitivity through an energy-based formulation. AEGD is notable for its unconditional energy stability, ensuring convergence in energy regardless of the initial step size. In this work, we propose the Generalized Energy-Based Adaptive Gradient (gAEGD) method, which extends AEGD by generalizing the energy function beyond the square root form to a broader class of functions. We prove that gAEGD retains the unconditional energy stability property, remains robust to step size selection, and exhibits a two-phase adaptive dynamic: the effective step size first adjusts adaptively, then stabilizes within a range that guarantees decay of the objective function values. We establish an optimal convergence rate of $O(1/k)$ for finding an $\epsilon$-stationary point, along with improved convergence rates for the objective gap under a local Kurdyka-{\L}ojasiewicz (KL) condition. Empirical results support the theoretical analysis and indicate that the generalized energy-based approach preforms effectively and reliably for a broad range of optimization problems.