Lyapunov Constructions for System Interconnections Arising from Adaptation in Some Optimization Methods
Interconnected systems have been widely studied, with a focus on interconnected systems whose subsystems are solely input-to-state stable (ISS) or passive systems. The focus of this work is on the interconnected systems that appear in adaptive gradient methods. In adaptive gradient methods, one subsystem seeks to move parameters of a cost function towards a minimizer while the other subsystem works to estimate some derivative information about the cost function to help determine the direction of the parameter update. This work studies instances of such interconnected systems and gives various Lyapunov function constructions for them using different techniques. Global asymptotic stability (GAS) of adaptive gradient optimizers follows from small-gain theory; the Lyapunov functions constructed here certify it explicitly.