Generalization of Zeroth-Order Method for Quotients of Quadratic Functions

Abstract

Optimization of quadratic functions and their quotients is relevant in subspace and iterative optimization methods. In this paper, we consider the matrix-free computation of the generalized operator norm and the maximization of a generalized Rayleigh quotient when only forward evaluations of two linear operators $A$ and $B$ are available. The proposed method samples search directions uniformly from the full unit sphere, thereby avoiding tangent-space sampling and explicit access to the metric matrix $B^{\mathrm T}B$. The exact line search along each sampled direction reduces to a $2\times2$ generalized eigenvalue problem on specific Gram matrices. In the generic case its maximizing step has a closed form. We prove that the objective values converge almost surely to the largest generalized eigenvalue and that the distance of the iterates to the leading generalized eigenspace converges to zero. We also relate full-sphere moment estimators to the Riemannian gradient and Hessian. The numerical experiments on synthetic Gau{\ss}ian operators illustrate the behavior of the methods, and further studies of the proposed algorithms indicate favorable empirical convergence and weak dependence on the tested problem dimensions.