SearcharxivSearch

arXiv subjects

Giampaolo Liuzzi

Publications and source records attributed to Giampaolo Liuzzi.

At least 19 recordsLinked to original sources

Worst-Case Complexity of High-Order Algorithms for Pareto-Front Reconstruction

In this paper, we are concerned with a worst-case complexity analysis of a-posteriori algorithms for unconstrained multiobjective optimization. Specifically, we propose an algorithmic framework that generates sets of points by means of $p$th-order models regularized with a power $p+1$ of the norm of the step. Through a tailored search procedure, several trial points are generated at each iteration and they can be added to the current set if a decrease is obtained for at least one objective function. Building upon this idea, we devise two algorithmic versions: at every iteration, the first tries to update all points in the current set, while the second tries to update only one point. Under Lipschitz continuity of the derivatives of the objective functions, we derive worst-case complexity bounds for both versions. For the first one, we show that at most $\mathcal O(ε^{-m(p+1)/p})$ iterations are needed to generate a set where all points are $ε$-approximate Pareto-stationary, with $m$ being the number of objective functions, requiring at most $\mathcal O(|X(ε)|ε^{-m(p+1)/p})$ function evaluations, where $X(ε)$ is the largest set of points computed by the algorithm. Additionally, at most $\mathcal O(ε^{-(p+1)/p})$ iterations are needed to generate at least one $ε$-approximate Pareto-stationary point, requiring at most $\mathcal O(|X(ε)|ε^{-(p+1)/p})$ function evaluations. For the second version, we get $\mathcal O(ε^{-m(p+1)/p})$ worst-case complexity bounds on the number of iterations and function evaluations for generating at least one $ε$-approximate Pareto-stationary point. Our results align with those for single objective optimization and generalize those obtained for methods that produce a single Pareto-stationary point.

math.OC

Projected Gradient Methods with Momentum

We focus on the optimization problem with smooth, possibly nonconvex objectives and a convex constraint set for which the Euclidean projection operation is practically available. Focusing on this setting, we carry out a general convergence and complexity analysis for algorithmic frameworks. Consequently, we discuss theoretically sound strategies to integrate momentum information within classical projected gradient type algorithms. One of these approaches is then developed in detail, up to the definition of a tailored algorithm with both theoretical guarantees and reasonable per-iteration cost. The proposed method is finally shown to outperform the standard (spectral) projected gradient method in two different experimental benchmarks, indicating that the addition of momentum terms is as beneficial in the constrained setting as it is in the unconstrained scenario.

math.OC

Derivative-Free Bilevel Optimization with Inexact Lower-Level Solutions

In this work, we propose derivative-free framework for bilevel optimization. We consider both the upper and lower-level problems with bound constraints on the variables, as well as general nonlinear constraints, assuming that first-order information (in the upper-level) is not available or it is impractical to obtain. The lower-level problem is solved with an accuracy that is progressively refined throughout the optimization process. We first analyze the case in which the upper-level problem is subject only to bound constraints, establishing convergence to Clarke-Jahn stationary points when the refinement process is allowed to reach its maximum precision. When a limitation is imposed on this refinement process, we prove convergence to approximate stationary points using an extended notion of Goldstein stationarity. Finally, we extend the proposed framework to handle more complex constraints via an exact penalty function approach, proving convergence to stationary points under suitable assumptions. A comprehensive numerical study on 160 problems from the BOLIB collection shows that the adaptive accuracy strategy consistently yields better results than fixed-precision solves, with its benefits becoming more pronounced as the required lower-level accuracy becomes more stringent.

math.OC

Nonlinear Derivative-free Constrained Optimization with a Penalty-Interior Point Method and Direct Search

In this work, the joint use of a mixed penalty-interior point method and direct search is proposed, to address {nonlinear} constrained derivative-free optimization problems. A merit function is considered, wherein the set of nonlinear inequality constraints is divided into two groups: one treated with a logarithmic barrier approach, and another, along with the equality constraints, addressed using a penalization term. This strategy is adapted and incorporated into a direct search method, enabling the effective handling of general nonlinear constraints. Convergence to KKT-stationary points is established under continuous differentiability assumptions, without requiring any kind of convexity. Computational experiments on analytical problems and an engineering application demonstrate the robustness, efficiency, and overall effectiveness of the proposed method, when compared with state-of-the-art solvers.

math.OC

Complexity results and active-set identification of a derivative-free method for bound-constrained problems

In this paper, we analyze a derivative-free line search method designed for bound-constrained problems. Our analysis demonstrates that this method exhibits a worst-case complexity comparable to other derivative-free methods for unconstrained and linearly constrained problems. In particular, when minimizing a function with $n$ variables, we prove that at most ${\cal O(nε^{-2})}$ iterations are needed to drive a criticality measure below a predefined threshold $ε$, requiring at most ${\cal O(n^2ε^{-2})}$ function evaluations. We also show that the total number of iterations where the criticality measure is not below $ε$ is upper bounded by ${\cal O(n^2ε^{-2})}$. Moreover, we investigate the method capability to identify active constraints at the final solutions. We show that, after a finite number of iterations, all the active constraints satisfying the strict complementarity condition are correctly identified.

math.OC

A Globally Convergent Gradient Method with Momentum

In this work, we consider smooth unconstrained optimization problems and we deal with the class of gradient methods with momentum, i.e., descent algorithms where the search direction is defined as a linear combination of the current gradient and the preceding search direction. This family of algorithms includes nonlinear conjugate gradient methods and Polyak's heavy-ball approach, and is thus of high practical and theoretical interest in large-scale nonlinear optimization. We propose a general framework where the scalars of the linear combination defining the search direction are computed simultaneously by minimizing the approximate quadratic model in the 2 dimensional subspace. This strategy allows us to define a class of gradient methods with momentum enjoying global convergence guarantees and an optimal worst-case complexity bound in the nonconvex setting. Differently than all related works in the literature, the convergence conditions are stated in terms of the Hessian matrix of the bi-dimensional quadratic model. To the best of our knowledge, these results are novel to the literature. Moreover, extensive computational experiments show that the gradient method with momentum here presented is a solid choice to tackle some classes of nonconvex unconstrained problems.

math.OC

Nonlinear Derivative-free Constrained Optimization with a Penalty-Interior Point Method and Direct Search

In this work, we propose the joint use of a mixed penalty-interior point method and direct search, for addressing nonlinearly constrained derivative-free optimization problems. A merit function is considered, wherein the set of nonlinear inequality constraints is divided into two groups: one treated with a logarithmic barrier approach, and another, along with the equality constraints, addressed using a penalization term. This strategy, is adapted and incorporated into a direct search method, enabling the effective handling of general nonlinear constraints. Convergence to KKT-stationary points is established under continuous differentiability assumptions, without requiring any kind of convexity. Using CUTEst test problems, numerical experiments demonstrate the robustness, efficiency, and overall effectiveness of the proposed method, when compared with state-of-the-art solvers

math.OC

A linesearch-based derivative-free method for noisy black-box problems

In this work we consider unconstrained optimization problems. The objective function is known through a zeroth order stochastic oracle that gives an estimate of the true objective function. To solve these problems, we propose a derivative-free algorithm based on extrapolation techniques. Under reasonable assumptions we are able to prove convergence properties for the proposed algorithms. Furthermore, we also give a worst-case complexity result stating that the total number of iterations where the expected value of the norm of the objective function gradient is above a prefixed $ε>0$ is ${\cal O}(n^2ε^{-2}/β^2)$ in the worst case.

math.OC

Worst-case complexity analysis of derivative-free methods for multi-objective optimization

In this work, we are concerned with the worst case complexity analysis of "a posteriori" methods for unconstrained multi-objective optimization problems where objective function values can only be obtained by querying a black box. We present two main algorithms, namely DFMOnew and DFMOlight which are based on a linesearch expansion technique. In particular, \DFMOnew, requires a complete exploration of the points in the current set of non-dominated solutions, whereas DFMOlight only requires the exploration around a single point in the set of non-dominated solutions. For these algorithms, we derive worst case iteration and evaluation complexity results. In particular, the complexity results for DFMOlight aligns with those recently proved in the literature for a directional multisearch method. Furthermore, exploiting an expansion technique of the step, we are also able to give further complexity results concerning the number of iterations with a measure of stationarity above a prefixed tolerance.

math.OC

Combining Gradient Information and Primitive Directions for High-Performance Mixed-Integer Optimization

In this paper we consider bound-constrained mixed-integer optimization problems where the objective function is differentiable w.r.t.\ the continuous variables for every configuration of the integer variables. We mainly suggest to exploit derivative information when possible in these scenarios: concretely, we propose an algorithmic framework that carries out local optimization steps, alternating searches along gradient-based and primitive directions. The algorithm is shown to match the convergence properties of a derivative-free counterpart. Most importantly, the results of thorough computational experiments show that the proposed method clearly outperforms not only the derivative-free approach but also the main alternatives available from the literature to be used in the considered setting, both in terms of efficiency and effectiveness.

math.OC

CMA Light: a novel Minibatch Algorithm for large-scale non convex finite sum optimization

The supervised training of a deep neural network on a given dataset consists in the unconstrained minimization of the finite sum of continuously differentiable functions, commonly referred to as loss with respect to the samples. These functions depend on the network parameters and most of the times are non-convex. We develop CMA Light, a globally convergent mini-batch gradient method to tackle this problem. We consider the recently introduced Controlled Minibatch Algorithm (CMA) framework and we overcome its main bottleneck, removing the need for at least one evaluation of the whole objective function per iteration. We prove globally convergence of CMA Light under mild assumptions and we discuss extensive computational results on the same experimental test-bed used for CMA, showing that CMA Light requires less computational effort than most of the state-of-the-art optimizers. Eventually, we present early results on a large-scale Image Classification task.

math.OC

Convergence of ease-controlled Random Reshuffling gradient Algorithms under Lipschitz smoothness

In this work, we consider minimizing the average of a very large number of smooth and possibly non-convex functions, and we focus on two widely used minibatch frameworks to tackle this optimization problem: Incremental Gradient (IG) and Random Reshuffling (RR). We define ease-controlled modifications of the IG/RR schemes, which require a light additional computational effort {but} can be proved to converge under {weak} and standard assumptions. In particular, we define two algorithmic schemes in which the IG/RR iteration is controlled by using a watchdog rule and a derivative-free linesearch that activates only sporadically to guarantee convergence. The two schemes differ in the watchdog and the linesearch, which are performed using either a monotonic or a non-monotonic rule. The two schemes also allow controlling the updating of the stepsize used in the main IG/RR iteration, avoiding the use of pre-set rules that may drive the stepsize to zero too fast, reducing the effort in designing effective updating rules of the stepsize. We prove convergence under the mild assumption of Lipschitz continuity of the gradients of the component functions and perform extensive computational analysis using different deep neural architectures and a benchmark of varying-size datasets. We compare our implementation with both a full batch gradient method (i.e. L-BFGS) and an implementation of IG/RR methods, proving that our algorithms require a similar computational effort compared to the other online algorithms and that the control on the learning rate may allow a faster decrease of the objective function.

math.OC

Worst case complexity bounds for linesearch-type derivative-free algorithms

This paper is devoted to the analysis of worst case complexity bounds for linesearch-type derivative-free algorithms for the minimization of general non-convex smooth functions. We prove that two linesearch-type algorithms enjoy the same complexity properties which have been proved for pattern and direct search algorithms. In particular, we consider two derivative-free algorithms based on two different linesearch techniques and manage to prove that the number of iterations and of function evaluations required to drive the norm of the gradient of the objective function below a given threshold $ε$ is ${\cal O}(ε^{-2})$ in the worst case.

math.OC

A clustering heuristic to improve a derivative-free algorithm for nonsmooth optimization

In this paper we propose an heuristic to improve the performances of the recently proposed derivative-free method for nonsmooth optimization CS-DFN. The heuristic is based on a clustering-type technique to compute a direction { which relies on an estimate of Clarke's generalized gradient} of the objective function. As such, this direction (as it is shown by the numerical experiments) is a good descent direction for the objective function. We report some numerical results and comparison with the original CS-DFN method to show the utility of the proposed improvement on a set of well-known test problems.

math.OC

An interior point method for nonlinear constrained derivative-free optimization

In this paper we consider constrained optimization problems where both the objective and constraint functions are of the black-box type. Furthermore, we assume that the nonlinear inequality constraints are non-relaxable, i.e. their values and that of the objective function cannot be computed outside of the feasible region. This situation happens frequently in practice especially in the black-box setting where function values are typically computed by means of complex simulation programs which may fail to execute if the considered point is outside of the feasible region. For such problems, we propose a new derivative-free optimization method which is based on the use of a merit function that handles inequality constraints by means of a log-barrier approach and equality constraints by means of a quadratic penalty approach. We prove convergence of the proposed method to KKT stationary points of the problem under quite mild assumptions. Furthermore, we also carry out a preliminary numerical experience on standard test problems and comparison with a state-of-the-art solver which shows efficiency of the proposed method.

math.OC

A derivative-free approach to mixed integer constrained multiobjective nonsmooth optimization

In this work, we consider multiobjective optimization problems with both bound constraints on the variables and general nonlinear constraints, where objective and constraint function values can only be obtained by querying a black box. Furthermore, we consider the case where a subset of the variables can only take integer values. We propose a new linesearch-based solution method and show that it converges to a set of stationary points for the problem. For what concerns the continuous variables, we employ a strategy for the estimation of the Pareto frontier recently proposed in the literature and which takes advantage of dense sequences of search directions. The subset of variables that must assume discrete values are dealt with using primitive directions appropriately modified to take into account the presence of more than one objective functions. Numerical results obtained with the proposed method on a set of test problems and comparison with other solution methods show the viability and efficiency of the proposed approach.

math.OC

An augmented Lagrangian method exploiting an active-set strategy and second-order information

In this paper, we consider nonlinear optimization problems with nonlinear equality constraints and bound constraints on the variables. For the solution of such problems, many augmented Lagrangian methods have been defined in the literature. Here, we propose to modify one of these algorithms, namely ALGENCAN by Andreani et al., in such a way to incorporate second-order information into the augmented Lagrangian framework, using an active-set strategy. We show that the overall algorithm has the same convergence properties as ALGENCAN and an asymptotic quadratic convergence rate under suitable assumptions. The numerical results confirm that the proposed algorithm is a viable alternative to ALGENCAN with greater robustness.

math.OC

Derivative-free methods for mixed-integer nonsmooth constrained optimization

In this paper, we consider mixed-integer nonsmooth constrained optimization problems whose objective/constraint functions are available only as the output of a black-box zeroth-order oracle (i.e., an oracle that does not provide derivative information) and we propose a new derivative-free linesearch-based algorithmic framework to suitably handle those problems. We first describe a scheme for bound constrained problems that combines a dense sequence of directions (to handle the nonsmoothness of the objective function) with primitive directions (to handle discrete variables). Then, we embed an exact penalty approach in the scheme to suitably manage nonlinear (possibly nonsmooth) constraints. We analyze the global convergence properties of the proposed algorithms toward stationary points and we report the results of an extensive numerical experience on a set of mixed-integer test problems.

math.OC