arXiv subjects

Masoud Ahookhosh

Publications and source records attributed to Masoud Ahookhosh.

36 records · Page 2Linked to original sources

arXiv2025

ItsDEAL: Inexact two-level smoothing descent algorithms for weakly convex optimization

This paper deals with nonconvex optimization problems via a two-level smoothing framework in which the high-order Moreau envelope (HOME) is applied to generate a smooth approximation of weakly convex cost functions. As such, the differentiability and weak smoothness of HOME are further studied, as is necessary for developing inexact first-order methods for finding its critical points. Building on the concept of the inexact two-level smoothing optimization (ItsOPT), the proposed scheme offers a versatile setting, called Inexact two-level smoothing DEscent ALgorithm (ItsDEAL), for developing inexact first-order methods: (i) solving the proximal subproblem approximately to provide an inexact first-order oracle of HOME at the lower-level; (ii) developing an upper inexact first-order method at the upper-level. In particular, parameter-free inexact descent methods (i.e., dynamic step-sizes and an inexact nonmonotone Armijo line search) are studied that effectively leverage the weak smooth property of HOME. Although the subsequential convergence of these methods is investigated under some mild inexactness assumptions, the global convergence and the linear rates are studied under the extra Kurdyka-Łojasiewicz (KL) property. In order to validate the theoretical foundation, preliminary numerical experiments for robust sparse recovery problems are provided which reveal a promising behavior of the proposed methods.

math.OC↗

arXiv2022

Bregman Finito/MISO for nonconvex regularized finite sum minimization without Lipschitz gradient continuity

We introduce two algorithms for nonconvex regularized finite sum minimization, where typical Lipschitz differentiability assumptions are relaxed to the notion of relative smoothness. The first one is a Bregman extension of Finito/MISO, studied for fully nonconvex problems when the sampling is randomized, or under convexity of the nonsmooth term when it is essentially cyclic. The second algorithm is a low-memory variant, in the spirit of SVRG and SARAH, that also allows for fully nonconvex formulations. Our analysis is made remarkably simple by employing a Bregman Moreau envelope as Lyapunov function. In the randomized case, linear convergence is established when the cost function is strongly convex, yet with no convexity requirements on the individual functions in the sum. For the essentially cyclic and low-memory variants, global and linear convergence results are established when the cost function satisfies the Kurdyka-Łojasiewicz property.

math.OC↗

arXiv2021

High-order methods beyond the classical complexity bounds, II: inexact high-order proximal-point methods with segment search

A bi-level optimization framework (BiOPT) was proposed in [3] for convex composite optimization, which is a generalization of bi-level unconstrained minimization framework (BLUM) given in [20]. In this continuation paper, we introduce a $p$th-order proximal-point segment search operator which is used to develop novel accelerated methods. Our first algorithm combines the exact element of this operator with the estimating sequence technique to derive new iteration points, which are shown to attain the convergence rate $\mathcal{O}(k^{-(3p+1)/2})$, for the iteration counter $k$. We next consider inexact elements of the high-order proximal-point segment search operator to be employed in the BiOPT framework. We particularly apply the accelerated high-order proximal-point method at the upper level, while we find approximate solutions of the proximal-point segment search auxiliary problem by a combination of non-Euclidean composite gradient and bisection methods. For $q=\lfloor p/2 \rfloor$, this amounts to a $2q$th-order method with the convergence rate $\mathcal{O}(k^{-(6q+1)/2})$ (the same as the optimal bound of $2q$th-order methods) for even $p$ ($p=2q$) and the superfast convergence rate $\mathcal{O}(k^{-(3q+1)})$ for odd $p$ ($p=2q+1$).

math.OC↗

arXiv2021

High-order methods beyond the classical complexity bounds, I: inexact high-order proximal-point methods

In this paper, we introduce a \textit{Bi-level OPTimization} (BiOPT) framework for minimizing the sum of two convex functions, where both can be nonsmooth. The BiOPT framework involves two levels of methodologies. At the upper level of BiOPT, we first regularize the objective by a $(p+1)$th-order proximal term and then develop the generic inexact high-order proximal-point scheme and its acceleration using the standard estimation sequence technique. At the lower level, we solve the corresponding $p$th-order proximal auxiliary problem inexactly either by one iteration of the $p$th-order tensor method or by a lower-order non-Euclidean composite gradient scheme with the complexity $\mathcal{O}(\log \tfrac{1}{\varepsilon})$, for the accuracy parameter $\varepsilon>0$. Ultimately, if the accelerated proximal-point method is applied at the upper level, and the auxiliary problem is handled by a non-Euclidean composite gradient scheme, then we end up with a $2q$-order method with the convergence rate $\mathcal{O}(k^{-(p+1)})$, for $q=\lfloor p/2 \rfloor$, where $k$ is the iteration counter.

math.OC↗

arXiv2021

Block Alternating Bregman Majorization Minimization with Extrapolation

In this paper, we consider a class of nonsmooth nonconvex optimization problems whose objective is the sum of a block relative smooth function and a proper and lower semicontinuous block separable function. Although the analysis of block proximal gradient (BPG) methods for the class of block $L$-smooth functions have been successfully extended to Bregman BPG methods that deal with the class of block relative smooth functions, accelerated Bregman BPG methods are scarce and challenging to design. Taking our inspiration from Nesterov-type acceleration and the majorization-minimization scheme, we propose a block alternating Bregman Majorization-Minimization framework with Extrapolation (BMME). We prove subsequential convergence of BMME to a first-order stationary point under mild assumptions, and study its global convergence under stronger conditions. We illustrate the effectiveness of BMME on the penalized orthogonal nonnegative matrix factorization problem.

math.OC↗

arXiv2020

A block inertial Bregman proximal algorithm for nonsmooth nonconvex problems with application to symmetric nonnegative matrix tri-factorization

We propose BIBPA, a block inertial Bregman proximal algorithm for minimizing the sum of a block relatively smooth function (that is, relatively smooth concerning each block) and block separable nonsmooth nonconvex functions. We prove that the sequence generated by BIBPA subsequentially converges to critical points of the objective under standard assumptions, and globally converges when the objective function is additionally assumed to satisfy the Kurdyka-Łojasiewicz (KŁ) property. We also provide the convergence rate when the objective satisfies the Łojasiewicz inequality. We apply BIBPA to the symmetric nonnegative matrix tri-factorization (SymTriNMF) problem, where we propose kernel functions for SymTriNMF and provide closed-form solutions for subproblems of BIBPA.

math.OC↗

arXiv2020

A Bregman forward-backward linesearch algorithm for nonconvex composite optimization: superlinear convergence to nonisolated local minima

We introduce Bella, a locally superlinearly convergent Bregman forward backward splitting method for minimizing the sum of two nonconvex functions, one of which satisfying a relative smoothness condition and the other one possibly nonsmooth. A key tool of our methodology is the Bregman forward-backward envelope (BFBE), an exact and continuous penalty function with favorable first- and second-order properties, and enjoying a nonlinear error bound when the objective function satisfies a Lojasiewicz-type property. The proposed algorithm is of linesearch type over the BFBE along candidate update directions, and converges subsequentially to stationary points, globally under a KL condition, and owing to the given nonlinear error bound can attain superlinear convergence rates even when the limit point is a nonisolated minimum, provided the directions are suitably selected.

math.OC↗

arXiv2019

Multi-block Bregman proximal alternating linearized minimization and its application to orthogonal nonnegative matrix factorization

We introduce and analyze BPALM and A-BPALM, two multi-block proximal alternating linearized minimization algorithms using Bregman distances for solving structured nonconvex problems. The objective function is the sum of a multi-block relatively smooth function (i.e., relatively smooth by fixing all the blocks except one) and block separable (nonsmooth) nonconvex functions. It turns out that the sequences generated by our algorithms are subsequentially convergent to critical points of the objective function, while they are globally convergent under KL inequality assumption. Further, the rate of convergence is further analyzed for functions satisfying the Łojasiewicz's gradient inequality. We apply this framework to orthogonal nonnegative matrix factorization (ONMF) that satisfies all of our assumptions and the related subproblems are solved in closed forms, where some preliminary numerical results is reported.

math.OC↗

arXiv2019

Local convergence of the Levenberg-Marquardt method under Hölder metric subregularity

We describe and analyse Levenberg-Marquardt methods for solving systems of nonlinear equations. More specifically, we propose an adaptive formula for the Levenberg-Marquardt parameter and analyse the local convergence of the method under Hölder metric subregularity of the function defining the equation and Hölder continuity of its gradient mapping. Further, we analyse the local convergence of the method under the additional assumption that the Łojasiewicz gradient inequality holds. We finally report encouraging numerical results confirming the theoretical findings for the problem of computing moiety conserved steady states in biochemical reaction networks. This problem can be cast as finding a solution of a system of nonlinear equations, where the associated mapping satisfies the Łojasiewicz gradient inequality assumption.

q-bio.MN↗

arXiv2018

Finding Zeros of Hölder Metrically Subregular Mappings via Globally Convergent Levenberg-Marquardt Methods

We present two globally convergent Levenberg-Marquardt methods for finding zeros of Hölder metrically subregular mappings that may have non-isolated zeros. The first method unifies the Levenberg- Marquardt direction and an Armijo-type line search, while the second incorporates this direction with a nonmonotone trust-region technique. For both methods, we prove the global convergence to a first-order stationary point of the associated merit function. Furthermore, the worst-case global complexity of these methods are provided, indicating that an approximate stationary point can be computed in at most $\mathcal{O}(\varepsilon^{-2})$ function and gradient evaluations, for an accuracy parameter $\varepsilon>0$. We also study the conditions for the proposed methods to converge to a zero of the associated mappings. Computing a moiety conserved steady state for biochemical reaction networks can be cast as the problem of finding a zero of a Hölder metrically subregular mapping. We report encouraging numerical results for finding a zero of such mappings derived from real-world biological data, which supports our theoretical foundations.

math.OC↗

arXiv2018

On the acceleration of forward-backward splitting via an inexact Newton method

We propose a Forward-Backward Truncated-Newton method (FBTN) for minimizing the sum of two convex functions, one of which smooth. Unlike other proximal Newton methods, our approach does not involve the employment of variable metrics, but is rather based on a reformulation of the original problem as the unconstrained minimization of a continuously differentiable function, the forward-backward envelope (FBE). We introduce a generalized Hessian for the FBE that symmetrizes the generalized Jacobian of the nonlinear system of equations representing the optimality conditions for the problem. This enables the employment of conjugate gradient method (CG) for efficiently solving the resulting (regularized) linear systems, which can be done inexactly. The employment of CG prevents the computation of full (generalized) Jacobians, as it requires only (generalized) directional derivatives. The resulting algorithm is globally (subsequentially) convergent, $Q$-linearly under an error bound condition, and up to $Q$-superlinearly and $Q$-quadratically under regularity assumptions at the possibly non-isolated limit point.

math.OC↗

arXiv2018

Creation and analysis of biochemical constraint-based models: the COBRA Toolbox v3.0

COnstraint-Based Reconstruction and Analysis (COBRA) provides a molecular mechanistic framework for integrative analysis of experimental data and quantitative prediction of physicochemically and biochemically feasible phenotypic states. The COBRA Toolbox is a comprehensive software suite of interoperable COBRA methods. It has found widespread applications in biology, biomedicine, and biotechnology because its functions can be flexibly combined to implement tailored COBRA protocols for any biochemical network. Version 3.0 includes new methods for quality controlled reconstruction, modelling, topological analysis, strain and experimental design, network visualisation as well as network integration of chemoinformatic, metabolomic, transcriptomic, proteomic, and thermochemical data. New multi-lingual code integration also enables an expansion in COBRA application scope via high-precision, high-performance, and nonlinear numerical optimisation solvers for multi-scale, multi-cellular and reaction kinetic modelling, respectively. This protocol can be adapted for the generation and analysis of a constraint-based model in a wide variety of molecular systems biology scenarios. This protocol is an update to the COBRA Toolbox 1.0 and 2.0. The COBRA Toolbox 3.0 provides an unparalleled depth of constraint-based reconstruction and analysis methods.

q-bio.QM↗

arXiv2016

Accelerated first-order methods for large-scale convex minimization

This paper discusses several (sub)gradient methods attaining the optimal complexity for smooth problems with Lipschitz continuous gradients, nonsmooth problems with bounded variation of subgradients, weakly smooth problems with Hölder continuous gradients. The proposed schemes are optimal for smooth strongly convex problems with Lipschitz continuous gradients and optimal up to a logarithmic factor for nonsmooth problems with bounded variation of subgradients. More specifically, we propose two estimation sequences of the objective and give two iterative schemes for each of them. In both cases, the first scheme requires the smoothness parameter and the Hölder constant, while the second scheme is parameter-free (except for the strong convexity parameter which we set zero if it is not available) at the price of applying a nonmonotone backtracking line search. A complexity analysis for all the proposed schemes is given. Numerical results for some applications in sparse optimization and machine learning are reported, which confirm the theoretical foundations.

math.OC↗

arXiv2015

Two globally convergent nonmonotone trust-region methods for unconstrained optimization

This paper addresses some trust-region methods equipped with nonmonotone strategies for solving nonlinear unconstrained optimization problems. More specifically, the importance of using nonmonotone techniques in nonlinear optimization is motivated, then two new nonmonotone terms are proposed, and their combinations into the traditional trust-region framework are studied. The global convergence to first- and second-order stationary points and local superlinear and quadratic convergence rates for both algorithms are established. Numerical experiments on the \textsf{CUTEst} test collection of unconstrained problems and some highly nonlinear test functions are reported, where a comparison among state-of-the-art nonmonotone trust-region methods show the efficiency of the proposed nonmonotne schemes.

math.OC↗

arXiv2015

An optimal subgradient algorithm for large-scale convex optimization in simple domains

This paper shows that the optimal subgradient algorithm, OSGA, proposed in \cite{NeuO} can be used for solving structured large-scale convex constrained optimization problems. Only first-order information is required, and the optimal complexity bounds for both smooth and nonsmooth problems are attained. More specifically, we consider two classes of problems: (i) a convex objective with a simple closed convex domain, where the orthogonal projection on this feasible domain is efficiently available; (ii) a convex objective with a simple convex functional constraint. If we equip OSGA with an appropriate prox-function, the OSGA subproblem can be solved either in a closed form or by a simple iterative scheme, which is especially important for large-scale problems. We report numerical results for some applications to show the efficiency of the proposed scheme. A software package implementing OSGA for above domains is available.

math.OC↗

arXiv2015

An optimal subgradient algorithm for large-scale bound-constrained convex optimization

This paper shows that the OSGA algorithm -- which uses first-order information to solve convex optimization problems with optimal complexity -- can be used to efficiently solve arbitrary bound-constrained convex optimization problems. This is done by constructing an explicit method as well as an inexact scheme for solving the bound-constrained rational subproblem required by OSGA. This leads to an efficient implementation of OSGA on large-scale problems in applications arising signal and image processing, machine learning and statistics. Numerical experiments demonstrate the promising performance of OSGA on such problems. ions to show the efficiency of the proposed scheme. A software package implementing OSGA for bound-constrained convex problems is available.

math.OC↗

arXiv2014

On efficiency of nonmonotone Armijo-type line searches

Monotonicity and nonmonotonicity play a key role in studying the global convergence and the efficiency of iterative schemes employed in the field of nonlinear optimization, where globally convergent and computationally efficient schemes are explored. This paper addresses some features of descent schemes and the motivation behind nonmonotone strategies and investigates the efficiency of an Armijo-type line search equipped with some popular nonmonotone terms. More specifically, we propose two novel nonmonotone terms, combine them into Armijo's rule and establish the global convergence of sequences generated by these schemes. Furthermore, we report extensive numerical results and comparisons indicating the performance of the nonmonotone Armijo-type line searches using the most popular search directions for solving unconstrained optimization problems. Finally, we exploit the considered nonmonotone schemes to solve an important inverse problem arising in signal and image processing.

math.OC↗

arXiv2014

Optimal subgradient algorithms with application to large-scale linear inverse problems

This study addresses some algorithms for solving structured unconstrained convex optimiza- tion problems using first-order information where the underlying function includes high-dimensional data. The primary aim is to develop an implementable algorithmic framework for solving problems with multi- term composite objective functions involving linear mappings using the optimal subgradient algorithm, OSGA, proposed by Neumaier in [49]. To this end, we propose some prox-functions for which the cor- responding subproblem of OSGA is solved in a closed form. Considering various inverse problems arising in signal and image processing, machine learning, statistics, we report extensive numerical and compar- isons with several state-of-the-art solvers proposing favourably performance of our algorithm. We also compare with the most widely used optimal first-order methods for some smooth and nonsmooth con- vex problems. Surprisingly, when some Nesterov-type optimal methods originally proposed for smooth problems are adapted for solving nonsmooth problems by simply passing a subgradient instead of the gradient, the results of these subgradient-based algorithms are competitive and totally interesting for solving nonsmooth problems. Finally, the OSGA software package is available.

math.OC↗