arXiv subjects

Christian Varner

Publications and source records attributed to Christian Varner.

4 recordsLinked to original sources

arXiv2025

A Novel First-order Method with Event-driven Objective Evaluations

Arising in semi-parametric statistics, control applications, and as sub-problems in global optimization methods, certain optimization problems can have objective functions requiring numerical integration to evaluate, yet gradient function evaluations that are relatively cheap. For such problems, typical optimization methods that require multiple evaluations of the objective for each new iterate become computationally expensive. In light of this, optimization methods that avoid objective function evaluations are attractive, yet we show anti-convergence behavior for these methods on the problem class of interest. To address this gap, we develop a novel gradient algorithm that only evaluates the objective function when specific events are triggered and propose a step size scheme that greedily takes advantage of the properties induced by these triggering events. We prove that our methodology has global convergence guarantees under the most general smoothness conditions, and show through extensive numerical results that our method performs favorably on optimization problems arising in semi-parametric statistics.

math.OC↗

arXiv2024

Recent Advances in Non-convex Smoothness Conditions and Applicability to Deep Linear Neural Networks

The presence of non-convexity in smooth optimization problems arising from deep learning have sparked new smoothness conditions in the literature and corresponding convergence analyses. We discuss these smoothness conditions, order them, provide conditions for determining whether they hold, and evaluate their applicability to training a deep linear neural network for binary classification.

cs.LG↗

arXiv2024

The Challenges of Optimization For Data Science

Optimization problems arising in data science have given rise to a number of new derivative-based optimization methods. Such methods often use standard smoothness assumptions -- namely, global Lipschitz continuity of the gradient function -- to establish a convergence theory. Unfortunately, in this work, we show that common optimization problems from data science applications are not globally Lipschitz smooth, nor do they satisfy some more recently developed smoothness conditions in literature. Instead, we show that such optimization problems are better modeled as having locally Lipschitz continuous gradients. We then construct explicit examples satisfying this assumption on which existing classes of optimization methods are either unreliable or experience an explosion in evaluation complexity. In summary, we show that optimization problems arising in data science are particularly difficult to solve, and that there is a need for methods that can reliably and practically solve these problems.

math.OC↗

arXiv2023

A Novel Gradient Methodology with Economical Objective Function Evaluations for Data Science Applications

Gradient methods are experiencing a growth in methodological and theoretical developments owing to the challenges posed by optimization problems arising in data science. However, such gradient methods face diverging optimality gaps or exploding objective evaluations when applied to optimization problems with realistic properties for data science applications. In this work, we address this gap by developing a generic methodology that economically uses objective function evaluations in a problem-driven manner to prevent optimality gap divergence and avoid explosions in objective evaluations. Our methodology allows for a variety of step size routines and search direction strategies. Furthermore, we develop a particular, novel step size selection methodology that is well-suited to our framework. We show that our specific procedure is highly competitive with standard optimization methods on CUTEst test problems. We then show our specific procedure is highly favorable relative to standard optimization methods on a particularly tough data science problem: learning the parameters in a generalized estimating equation model. Thus, we provide a novel gradient methodology that is better suited to optimization problems from this important class of data science applications.

math.OC↗