Searcharxiv⌕ Search

arXiv subjects

Victor S. Portella

Publications and source records attributed to Victor S. Portella.

6 recordsLinked to original sources
arXiv2026

Mirror Polyak and a Primal-Dual Lifting

First-order methods typically require a specific step-size that depends on the regularity conditions of the objective function, such as the smoothness, Lipschitz continuity, or strong convexity constants. The Polyak step-size is a classical alternative for subgradient descent on convex functions that only uses knowledge of the optimal value of the objective function and automatically adapts to the above-mentioned regimes. However, many optimization problems are better described by non-Euclidean geometries and are more amenable to mirror descent. Extending this adaptivity to mirror descent is subtle. Some existing generalizations of the Polyak step-size rely on norms instead of purely on relative geometry, excluding many of the use cases of mirror descent. In this work, we revisit a variant of the Polyak step-size based on Bregman projections due to Kiwiel (1997), which we call mirror Polyak. This method is known to converge asymptotically, but its convergence rate is not known. We show that mirror Polyak enjoys guarantees similar to its Euclidean counterpart, automatically adapting to relative notions of smoothness, Lipschitz continuity, or strong convexity. We then leverage mirror Polyak to avoid having to know the optimal value in some structured optimization problems such as regularized linear and logistic regression. We propose a lifted formulation based on convex duality with optimal value exactly zero and a natural mirror map given by the problem's structure. Mirror Polyak applied to the lifted problem enjoys the same worst-case guarantees as the Polyak step-size in the original problem if we knew the optimal value.

math.OC↗
arXiv2026

A second order regret bound for NormalHedge

We consider the problem of prediction with expert advice for ``easy'' sequences. We show that a variant of NormalHedge enjoys a second-order $ε$-quantile regret bound of $O\big(\sqrt{V_T \log(V_T/ε)}\big) $ when $V_T > \log N$, where $V_T$ is the cumulative second moment of instantaneous per-expert regret averaged with respect to a natural distribution determined by the algorithm. The algorithm is motivated by a continuous time limit using Stochastic Differential Equations. The discrete time analysis uses self-concordance techniques.

cs.LG↗
arXiv2023

Searching for Optimal Per-Coordinate Step-sizes with Multidimensional Backtracking

The backtracking line-search is an effective technique to automatically tune the step-size in smooth optimization. It guarantees similar performance to using the theoretically optimal step-size. Many approaches have been developed to instead tune per-coordinate step-sizes, also known as diagonal preconditioners, but none of the existing methods are provably competitive with the optimal per-coordinate stepsizes. We propose multidimensional backtracking, an extension of the backtracking line-search to find good diagonal preconditioners for smooth convex problems. Our key insight is that the gradient with respect to the step-sizes, also known as hypergradients, yields separating hyperplanes that let us search for good preconditioners using cutting-plane methods. As black-box cutting-plane approaches like the ellipsoid method are computationally prohibitive, we develop an efficient algorithm tailored to our setting. Multidimensional backtracking is provably competitive with the best diagonal preconditioner and requires no manual tuning.

math.OC↗
arXiv2021

Online mirror descent and dual averaging: keeping pace in the dynamic case

Online mirror descent (OMD) and dual averaging (DA) -- two fundamental algorithms for online convex optimization -- are known to have very similar (and sometimes identical) performance guarantees when used with a fixed learning rate. Under dynamic learning rates, however, OMD is provably inferior to DA and suffers a linear regret, even in common settings such as prediction with expert advice. We modify the OMD algorithm through a simple technique that we call stabilization. We give essentially the same abstract regret bound for OMD with stabilization and for DA by modifying the classical OMD convergence analysis in a careful and modular way that allows for straightforward and flexible proofs. Simple corollaries of these bounds show that OMD with stabilization and DA enjoy the same performance guarantees in many applications -- even under dynamic learning rates. We also shed light on the similarities between OMD and DA and show simple conditions under which stabilized-OMD and DA generate the same iterates.

cs.LG↗
arXiv2020

Regret Bounds without Lipschitz Continuity: Online Learning with Relative-Lipschitz Losses

In online convex optimization (OCO), Lipschitz continuity of the functions is commonly assumed in order to obtain sublinear regret. Moreover, many algorithms have only logarithmic regret when these functions are also strongly convex. Recently, researchers from convex optimization proposed the notions of "relative Lipschitz continuity" and "relative strong convexity". Both of the notions are generalizations of their classical counterparts. It has been shown that subgradient methods in the relative setting have performance analogous to their performance in the classical setting. In this work, we consider OCO for relative Lipschitz and relative strongly convex functions. We extend the known regret bounds for classical OCO algorithms to the relative setting. Specifically, we show regret bounds for the follow the regularized leader algorithms and a variant of online mirror descent. Due to the generality of these methods, these results yield regret bounds for a wide variety of OCO algorithms. Furthermore, we further extend the results to algorithms with extra regularization such as regularized dual averaging.

cs.LG↗