arXiv subjects

Mohammad Alkousa

Publications and source records attributed to Mohammad Alkousa.

At least 19 recordsLinked to original sources

arXiv2026

Mirror Descent Methods for Quasar Convex Optimization Problems With Non-Smooth Inequality Constraints

In this paper, we consider constraint optimization problems subject to non-smooth convex functional (inequality-type) constraints, wherein the objective function is non-smooth and quasar convex. We propose and analyze two groups of algorithms, each consisting of a standard version and a modified variant, that operate by switching between two types of iteration points: productive and non-productive. Within each group, we develop distinct mirror descent-type algorithms for both deterministic and stochastic settings, and we establish their convergence rates.

math.OC↗

arXiv2025

Speeding up the Goemans-Williamson randomized procedure by difference-of-convex optimization

We present a novel approach to accelerate the Goemans-Williamson (GW) randomized rounding procedure for quadratic unconstrained binary optimization (QUBO) problems. Instead of solving the conventional semi-definite programming (SDP) relaxation, which is computationally expensive, we employ a difference-of-convex (DC) optimization framework to efficiently approximate the SDP solution. The DC optimization produces candidate vectors that are then used within the GW randomized rounding scheme to generate high-quality binary solutions. Furthermore, we perform direct expectation minimization over manifolds of matrices with limited rank to further enhance the solution quality. Our method is benchmarked on real-world QUBO instances, including inverse kinematics problems, and compared against state-of-the-art solvers, such as quantum-inspired algorithms, demonstrating competitive approximation guarantees alongside substantial computational gains.

math.OC↗

arXiv2025

Optimal Convergence Rate for Mirror Descent Methods with special Time-Varying Step Sizes Rules

In this paper, the optimal convergence rate $O\left(N^{-1/2}\right)$ (where $N$ is the total number of iterations performed by the algorithm), without the presence of a logarithmic factor, is proved for mirror descent algorithms with special time-varying step sizes, for solving classical constrained non-smooth problems, problems with the composite model and problems with non-smooth functional (inequality types) constraints. The proven result is an improvement on the well-known rate $O\left(\log (N) N^{-1/2}\right)$ for the mirror descent algorithms with the time-varying step sizes under consideration. It was studied a new weighting scheme assigns smaller weights to the initial points and larger weights to the most recent points. This scheme improves the convergence rate of the considered mirror descent methods, which in the conducted numerical experiments outperform the other methods providing a better solution in all the considered test problems.

math.OC↗

arXiv2025

On Solving Minimization and Min-Max Problems by First-Order Methods with Relative Error in Gradients

First-order methods for minimization and saddle point (min-max) problems are widely used for solving large-scale problems, in particular arising in machine learning. The majority of works obtain favorable complexity guarantees of such methods, assuming that exact gradient information is available. At the same time, even the use of floating-point representation of real numbers already leads to relative error in all the computations. Relative errors also arise in such applications as bilevel optimization, inverse problems, derivative-free optimization, and inexact proximal methods. This paper answers several theoretical open questions on first-order optimization methods under relative errors in the first-order oracle. We propose an explicit single-loop accelerated gradient method that preserves optimal linear convergence rate under maximal possible relative error in the gradient, and explore the tradeoff between the relative error and deterioration in the linear convergence rate. We further explore similar questions for saddle point problems and nonlinear equations, showing, for the first time in the literature, that a variant of gradient descent-ascent and the extragradient method are robust to such errors and providing estimates for the maximum level of noise that does not break linear convergence.

math.OC↗

arXiv2024

Optimization in complex spaces with the Mixed Newton Method

We propose a second-order method for unconditional minimization of functions $f(z)$ of complex arguments. We call it the Mixed Newton Method due to the use of the mixed Wirtinger derivative $\frac{\partial^2f}{\partial\bar z\partial z}$ for computation of the search direction, as opposed to the full Hessian $\frac{\partial^2f}{\partial(z,\bar z)^2}$ in the classical Newton method. The method has been developed for specific applications in wireless network communications, but its global convergence properties are shown to be superior on a more general class of functions $f$, namely sums of squares of absolute values of holomorphic functions. In particular, for such objective functions minima are surrounded by attraction basins, while the iterates are repelled from other types of critical points. We provide formulas for the asymptotic convergence rate and show that in the scalar case the method reduces to the well-known complex Newton method for the search of zeros of holomorphic functions. In this case, it exhibits generically fractal global convergence patterns.

math.OC↗

arXiv2024

Mixed Newton Method for Optimization in Complex Spaces

In this paper, we modify and apply the recently introduced Mixed Newton Method, which is originally designed for minimizing real-valued functions of complex variables, to the minimization of real-valued functions of real variables by extending the functions to complex space. We show that arbitrary regularizations preserve the favorable local convergence properties of the method, and construct a special type of regularization used to prevent convergence to complex minima. We compare several variants of the method applied to training neural networks with real and complex parameters.

math.OC↗

arXiv2024

Higher Degree Inexact Model for Optimization problems

In this paper, it was proposed a new concept of the inexact higher degree $(δ, L, q)$-model of a function that is a generalization of the inexact $(δ, L)$-model, $(δ, L)$-oracle and $(δ, L)$-oracle of degree $q \in [0,2)$. Some examples were provided to illustrate the proposed new model. Adaptive inexact gradient and fast gradient methods for convex and strongly convex functions were constructed and analyzed using the new proposed inexact model. A universal fast gradient method that allows solving optimization problems with a weaker level of smoothness, among them non-smooth problems was proposed. For convex optimization problems it was proved that the proposed gradient and fast gradient methods could be converged with rates $O\left(\frac{1}{k} + \fracδ{k^{q/2}}\right)$ and $O\left(\frac{1}{k^2} + \fracδ{k^{(3q-2)/2}}\right)$, respectively. For the gradient method, the coefficient of $δ$ diminishes with $k$, and for the fast gradient method, there is no error accumulation for $q \geq 2/3$. It proposed a definition of an inexact higher degree oracle for strongly convex functions and a projected gradient method using this inexact oracle. For variational inequalities and saddle point problems, a higher degree inexact model and an adaptive method called Generalized Mirror Prox to solve such class of problems using the proposed inexact model were proposed. Some numerical experiments were conducted to demonstrate the effectiveness of the proposed inexact model, we test the universal fast gradient method to solve some non-smooth problems with a geometrical nature.

math.OC↗

arXiv2024

Universal methods for variational inequalities: deterministic and stochastic cases

In this paper, we propose universal proximal mirror methods to solve the variational inequality problem with Holder continuous operators in both deterministic and stochastic settings. The proposed methods automatically adapt not only to the oracle's noise (in the stochastic setting of the problem) but also to the Holder continuity of the operator without having prior knowledge of either the problem class or the nature of the operator information. We analyzed the proposed algorithms in both deterministic and stochastic settings and obtained estimates for the required number of iterations to achieve a given quality of a solution to the variational inequality. We showed that, without knowing the Holder exponent and Holder constant of the operators, the proposed algorithms have the least possible in the worst case sense complexity for the considered class of variational inequalities. We also compared the resulting stochastic algorithm with other popular optimizers for the task of image classification.

math.OC↗

arXiv2024

Gradient-Type Methods For Decentralized Optimization Problems With Polyak-Łojasiewicz Condition Over Time-Varying Networks

This paper focuses on the decentralized optimization (minimization and saddle point) problems with objective functions that satisfy Polyak-Łojasiewicz condition (PL-condition). The first part of the paper is devoted to the minimization problem of the sum-type cost functions. In order to solve a such class of problems, we propose a gradient descent type method with a consensus projection procedure and the inexact gradient of the objectives. Next, in the second part, we study the saddle-point problem (SPP) with a structure of the sum, with objectives satisfying the two-sided PL-condition. To solve such SPP, we propose a generalization of the Multi-step Gradient Descent Ascent method with a consensus procedure, and inexact gradients of the objective function with respect to both variables. Finally, we present some of the numerical experiments, to show the efficiency of the proposed algorithm for the robust least squares problem.

math.OC↗

arXiv2024

Adaptive Algorithms for Relatively Lipschitz Continuous Convex Optimization Problems

Recently there were proposed some innovative convex optimization concepts, namely, relative smoothness [1] and relative strong convexity [2,3]. These approaches have significantly expanded the class of applicability of gradient-type methods with optimal estimates of the convergence rate, which are invariant regardless of the dimensionality of the problem. Later Yu. Nesterov and H. Lu introduced some modifications of the Mirror Descent method for convex minimization problems with the corresponding analogue of the Lipschitz condition (so-called relative Lipschitz continuity). By introducing an artificial inaccuracy to the optimization model, we propose adaptive methods for minimizing a convex Lipschitz continuous function, as well as for the corresponding class of variational inequalities. We also consider an adaptive "universal" method, applicable to convex minimization problems both on the class of relatively smooth and relatively Lipschitz continuous functionals with optimal estimates of the convergence rate. The universality of the method makes it possible to justify the applicability of the obtained theoretical results to a wider class of convex optimization problems. We also present the results of numerical experiments.

math.OC↗

arXiv2023

Algorithms for solving variational inequalities and saddle point problems with some generalizations of Lipschitz property for operators

The article is devoted to the development of numerical methods for solving saddle point problems and variational inequalities with simplified requirements for the smoothness conditions of functionals. Recently there were proposed some notable methods for optimization problems with strongly monotone operators. Our focus here is on newly proposed techniques for solving strongly convex-concave saddle point problems. One of the goals of the article is to improve the obtained estimates of the complexity of introduced algorithms by using accelerated methods for solving auxiliary problems. The second focus of the article is introducing an analogue of the boundedness condition for the operator in the case of arbitrary (not necessarily Euclidean) prox structure. We propose an analogue of the mirror descent method for solving variational inequalities with such operators, which is optimal in the considered class of problems.

math.OC↗

arXiv2023

Intermediate Gradient Methods with Relative Inexactness

This paper is devoted to first-order algorithms for smooth convex optimization with inexact gradients. Unlike the majority of the literature on this topic, we consider the setting of relative rather than absolute inexactness. More precisely, we assume that an additive error in the gradient is proportional to the gradient norm, rather than being globally bounded by some small quantity. We propose a novel analysis of the accelerated gradient method under relative inexactness and strong convexity and improve the bound on the maximum admissible error that preserves the linear convergence of the algorithm. In other words, we analyze how robust is the accelerated gradient method to the relative inexactness of the gradient information. Moreover, based on the Performance Estimation Problem (PEP) technique, we show that the obtained result is optimal for the family of accelerated algorithms we consider. Motivated by the existing intermediate methods with absolute error, i.e., the methods with convergence rates that interpolate between slower but more robust non-accelerated algorithms and faster, but less robust accelerated algorithms, we propose an adaptive variant of the intermediate gradient method with relative error in the gradient.

math.OC↗

arXiv2023

Online Optimization Problems with Functional Constraints under Relative Lipschitz Continuity and Relative Strong Convexity Conditions

Recently, there were introduced important classes of relatively smooth, relatively continuous, and relatively strongly convex optimization problems. These concepts have significantly expanded the class of problems for which optimal complexity estimates of gradient-type methods in high-dimensional spaces take place. Basing on some recent works devoted to online optimization (regret minimization) problems with both relatively Lipschitz continuous and relatively strongly convex objective function, we introduce algorithms for solving the strongly convex optimization problem with inequality constraints in the online setting. We propose a scheme with switching between productive and nonproductive steps for such types of problems and prove its convergence rate for the class of relatively Lipschitz and strongly convex minimization problems. We also provide an experimental comparison between the proposed method and AdaMirr, recently proposed for relatively Lipschitz convex problems.

math.OC↗

arXiv2022

Solving strongly convex-concave composite saddle point problems with a small dimension of one of the variables

The article is devoted to the development of algorithmic methods ensuring efficient complexity bounds for strongly convex-concave saddle point problems in the case when one of the groups of variables is high-dimensional, and the other is relatively low-dimensional (up to a hundred). The proposed technique is based on reducing problems of this type to a problem of minimizing a convex (maximizing a concave) functional in one of the variables, for which it is possible to find an approximate gradient at an arbitrary point with the required accuracy using an auxiliary optimization subproblem with another variable. In this case, the ellipsoid method is used for low-dimensional problems (if necessary, with an inexact $δ$-subgradient), and accelerated gradient methods are used for high-dimensional problems. For the case of a very small dimension of one of the groups of variables (up to 5), an approach based on a new version of the multidimensional analog of the Yu. E. Nesterov's method on the square (multidimensional dichotomy) is proposed with the possibility of using inexact values of the gradient of the objective functional.

math.OC↗

arXiv2022

Generalized Mirror Prox for Monotone Variational Inequalities: Universality and Inexact Oracle

We introduce an inexact oracle model for variational inequalities (VI) with monotone operator, propose a numerical method which solves such VI's and analyze its convergence rate. As a particular case, we consider VI's with Hölder-continuous operator and show that our algorithm is universal. This means that without knowing the Hölder parameter $ν$ and Hölder constant $L_ν$ it has the best possible complexity for this class of VI's, namely our algorithm has complexity $O\left( \inf_{ν\in[0,1]}\left(\frac{L_ν}{\varepsilon} \right)^{\frac{2}{1+ν}}R^2 \right)$, where $R$ is the size of the feasible set and $\varepsilon$ is the desired accuracy of the solution. We also consider the case of VI's with strongly monotone operator and generalize our method for VI's with inexact oracle and our universal method for this class of problems. Finally, we show, how our method can be applied to convex-concave saddle point problems with Hölder-continuous partial subgradients.

math.OC↗

arXiv2022

An Approach for Non-Convex Uniformly Concave Structured Saddle Point Problem

Recently, saddle point problems have received much attention due to their powerful modeling capability for a lot of problems from diverse domains. Applications of these problems occur in many applied areas, such as robust optimization, distributed optimization, game theory, and many applications in machine learning such as empirical risk minimization and generative adversarial networks training. Therefore, many researchers have actively worked on developing numerical methods for solving saddle point problems in many different settings. This paper is devoted to developing a numerical method for solving saddle point problems in the non-convex uniformly-concave setting. We study a general class of saddle point problems with composite structure and Hölder-continuous higher-order derivatives. To solve the problem under consideration, we propose an approach in which we reduce the problem to a combination of two auxiliary optimization problems separately for each group of variables, outer minimization problem w.r.t. primal variables, and inner maximization problem w.r.t the dual variables. For solving the outer minimization problem, we use the \textit{Adaptive Gradient Method}, which is applicable for non-convex problems and also works with an inexact oracle that is generated by approximately solving the inner problem. For solving the inner maximization problem, we use the \textit{Restarted Unified Acceleration Framework}, which is a framework that unifies the high-order acceleration methods for minimizing a convex function that has Hölder-continuous higher-order derivatives. Separate complexity bounds are provided for the number of calls to the first-order oracles for the outer minimization problem and higher-order oracles for the inner maximization problem. Moreover, the complexity of the whole proposed approach is then estimated.

math.OC↗

arXiv2021

Mirror Descent and Constrained Online Optimization Problems

We consider the following class of online optimization problems with functional constraints. Assume, that a finite set of convex Lipschitz-continuous non-smooth functionals are given on a closed set of $n$-dimensional vector space. The problem is to minimize the arithmetic mean of functionals with a convex Lipschitz-continuous non-smooth constraint. In addition, it is allowed to calculate the (sub)gradient of each functional only once. Using some recently proposed adaptive methods of Mirror Descent the method is suggested to solve the mentioned constrained online optimization problem with optimal estimate of accuracy. For the corresponding non-Euclidean prox-structure the case of a set of $n$-dimensional vectors lying on the standard $n$-dimensional simplex is considered.

math.OC↗

arXiv2021

Inexact Relative Smoothness and Strong Convexity for Optimization and Variational Inequalities by Inexact Model

In this paper, we propose a general algorithmic framework for first-order methods in optimization in a broad sense, including minimization problems, saddle-point problems, and variational inequalities. This framework allows obtaining many known methods as a special case, the list including accelerated gradient method, composite optimization methods, level-set methods, Bregman proximal methods. The idea of the framework is based on constructing an inexact model of the main problem component, i.e. objective function in optimization or operator in variational inequalities. Besides reproducing known results, our framework allows constructing new methods, which we illustrate by constructing a universal conditional gradient method and a universal method for variational inequalities with a composite structure. This method works for smooth and non-smooth problems with optimal complexity without a priori knowledge of the problem's smoothness. As a particular case of our general framework, we introduce relative smoothness for operators and propose an algorithm for variational inequalities (VIs) with such operators. We also generalize our framework for relatively strongly convex objectives and strongly monotone variational inequalities. This paper is an extended and updated version of [arXiv:1902.00990]. In particular, we add an extension of relative strong convexity for optimization and variational inequalities.

math.OC↗