Multilevel Monte Carlo for Zeroth-Order Stochastic Bilevel Optimization with Inexact Lower Level Solves

Abstract

Variational regularization techniques are dominant in the field of mathematical imaging. The expressiveness of these methods heavily depends on the adopted regularization formulation which usually involves a number of parameters which have to be fine-tuned by the user. To circumvent the heavy computational costs associated with this parameter tuning, data-driven learning strategies have turned out to be a useful strategy. Selecting parameters by learning from data gives rise to a hierarchical (bilevel) optimization problem, which is in general computationally hard to solve. Our aim in this paper is to develop a data-efficient solution strategy for the bilevel learning problem. To that end, we propose a multilevel Monte Carlo derivative-free optimization method that couples Gaussian smoothing of the upper-level objective (parameter selection) with a hierarchy of lower-level (variational regularization) solution accuracies. Being agnostic towards the specific way the lower-level problem is solved, we derive explicit oracle complexity and computational cost statements for approximately solving the entire bilevel learning problem. Our statements are parameterized by the employed lower-level solution method, thus allowing for a direct cost accounting from the variational reconstruction method to the parameter selection problem. This enables us to give a universal end-to-end complexity analysis of bilevel learning, valid for the most important optimization routines in inverse problems, rather than for a specific choice of methods.