Convex optimization in Hilbert space with applications to inverse problems
In this paper we propose accelerated gradient descent schemes for convex optimization problems in Hilbert space. We consider inexact oracle case.
math.OC↗
arXiv subjects
Alexander Gasnikov
Publications and source records attributed to Alexander Gasnikov.
Composite optimization for the resource allocation problem
In this paper we consider resource allocation problem stated as a convex minimization problem with linear constraints. To solve this problem, we use gradient and accelerated gradient descent applied to the dual problem and prove the convergence rate both for the primal iterates and the dual iterates. We obtain faster convergence rates than the ones known in the literature. We also provide economic interpretation for these two methods. This means that iterations of the algorithms naturally correspond to the process of price and production adjustment in order to obtain the desired production volume in the economy. Overall, we show how these actions of the economic agents lead the whole system to the equilibrium.
math.OC↗
Algorithms for local optimization of OPLS energy for large protein structures
Many problems arise in computational biology can be reduced to the minimization of energy function, that determines on the geometry of considered molecule. The solution of this problem allows in particular to solve folding and docking problems in structural biology. For the small molecules this problem is well solved. But for the large molecules ($10^4$ atoms and more) this is still an open problem. In this work we consider energy minimization problem (OPLS force field) for the large molecules but with good enough initial (starting) point. In the paper one can find a biological explanation of this assumption. Due to this assumption we reduce the global optimization problem to the local one. We compare different methods: gradient-free methods, gradient type methods (gradient method, fast gradient method, conjugate gradients (CG), LBFGS), high-order (tensor) methods. We observe that the most convenient ones in GPU realization are fast gradient descent with special line-search and CG (Polak--Ribiere--Polyak), LBFGS (memory = 3 iteration). Finally, we demonstrate how all these method work on real data set provided by BIOCAD.
math.OC↗
Fast gradient descent method for convex optimization problems with an oracle that generates a $(δ,L)$-model of a function in a requested point
In this article we propose a new concept of a $(δ,L)$-model of a function which generalizes the concept of the $(δ,L)$-oracle (Devolder-Glineur-Nesterov). Using this concept we describe the gradient descent method and the fast gradient descent method and we show that many popular methods (composite optimization methods, level methods, conjugate gradient methods, proximal methods) are special cases of proposed methods in this article.
math.OC↗
Primal-dual accelerated gradient methods with small-dimensional relaxation oracle
In this paper, a new variant of accelerated gradient descent is proposed. The pro-posed method does not require any information about the objective function, usesexact line search for the practical accelerations of convergence, converges accordingto the well-known lower bounds for both convex and non-convex objective functions,possesses primal-dual properties and can be applied in the non-euclidian set-up. Asfar as we know this is the rst such method possessing all of the above properties atthe same time. We also present a universal version of the method which is applicableto non-smooth problems. We demonstrate how in practice one can efficiently use thecombination of line-search and primal-duality by considering a convex optimizationproblem with a simple structure (for example, linearly constrained).
math.OC↗
Gradient Methods for Problems with Inexact Model of the Objective
We consider optimization methods for convex minimization problems under inexact information on the objective function. We introduce inexact model of the objective, which as a particular cases includes $(δ,L)$ inexact oracle and relative smoothness condition. We analyze gradient method which uses this inexact model and obtain convergence rates for convex and strongly convex problems. To show potential applications of our general framework we consider three particular problems. The first one is clustering by electorial model introduced in [Nesterov, 2018]. The second one is approximating optimal transport distance, for which we propose a Proximal Sinkhorn algorithm. The third one is devoted to approximating optimal transport barycenter and we propose a Proximal Iterative Bregman Projections algorithm. We also illustrate the practical performance of our algorithms by numerical experiments.
math.OC↗
Accelerated Directional Search with non-Euclidean prox-structure
In the paper we propose an accelerated directional search method with non-euclidian prox-structure. We consider convex unconstraint optimization problem in $\mathbb{R}^n$. For simplicity we start from the zero point. We expect in advance that 1-norm of the solution is close enough to its 2-norm. In this case the standard accelerated Nesterov's directional search method can be improved. In the paper we show how to make Nesterov's method $n$-times faster (up to a $\log n$-factor) in this case. The basic idea is to use linear coupling, proposed by Allen-Zhu & Orecchia in 2014, and to make Grad-step in 2-norm, but Mirr-step in 1-norm. We show that for constrained optimization problems this approach stable upon an obstacle.
math.OC↗
Walrasian Equilibrium and Centralized Distributed Optimization from the point of view of Modern Convex Optimization Methods on the Example of Resource Allocation Problem
We consider the resource allocation problem and its numerical solution. The following constructions are demonstrated: 1) Walrasian price-adjustment mechanism for determining the equilibrium; 2) Decentralized role of the prices; 3) Slater's method for price restrictions (dual Lagrange multipliers); 4) A new mechanism for determining equilibrium prices, in which prices are fully controlled not by Center (Government), but by economic agents -- nodes (factories). In economic literature the convergence of the considered methods is only proved. In contrast, this paper provides an accurate analysis of the convergence rate of the described procedures for determining the equilibrium. The analysis is based on the primal-dual nature of the suggested algorithms. More precisely, in this article we propose the economic interpretation of the following numerical primal-dual methods of convex optimization: dichotomy and subgradient projection method. Numerical experiments conclude the paper.
math.OC↗
On the upper bound for the mathematical expectation of the norm of a vector uniformly distributed on the sphere and the phenomenon of concentration of uniform measure on the sphere
We considered the problem of obtaining upper bounds for the mathematical expectation of the $q$-norm ($2\leqslant q \leqslant \infty$) of the vector which is uniformly distributed on the unit Euclidean sphere. We finish the paper with numerical experiments illustrating our results.
math.OC↗
Advances in Low-Memory Subgradient Optimization
This chapter is devoted to the black-box subgradient algorithms with the minimal requirements for the storage of auxiliary results, which are necessary to execute these algorithms. It starts with the original result of N.Z. Shor which open this field with the application to the classical transportation problem. To discuss the fundamentals of non-smooth optimization the theoretical complexity bounds for smooth and non-smooth convex and quasi-convex optimization problems are briefly exposed with the special attention given to adaptive step-size policy. Than this chapter contains descriptions of different modern techniques that allow to solve non-smooth convex optimization problems faster then lower complexity bounds: Netserov smoothing technique, Netserov Universal approach, Legendre (saddle point) representation approach. We also describe recent Universal Mirror Prox algorithm for variational inequalities. To demonstrate application of non-smooth convex optimization algorithms for solution of huge-scale extremal problems we consider convex optimization problems with non-smooth functional constraints and describe two adaptive Mirror Descent methods. The first method is of primal-dual nature and proved to be optimal in terms of lower oracle bounds for the class of Lipschitz-continuous convex objective and constraints. The advantages of application of this method to sparse Truss Topology Design problem are discussed in some details. The second method can be applied for solution of convex and quasi-convex optimization problems and is optimal in a sense of complexity bounds. The conclusion part of the survey contains the important references that characterize recent developments of non-smooth convex optimization.
math.OC↗
The global rate of convergence for optimal tensor methods in smooth convex optimization
We consider convex optimization problems with the objective function having Lipshitz-continuous $p$-th order derivative, where $p\geq 1$. We propose a new tensor method, which closes the gap between the lower $O\left(\varepsilon^{-\frac{2}{3p+1}} \right)$ and upper $O\left(\varepsilon^{-\frac{1}{p+1}} \right)$ iteration complexity bounds for this class of optimization problems. We also consider uniformly convex functions, and show how the proposed method can be accelerated under this additional assumption. Moreover, we introduce a $p$-th order condition number which naturally arises in the complexity analysis of tensor methods under this assumption. Finally, we make a numerical study of the proposed optimal method and show that in practice it is faster than the best known accelerated tensor method. We also compare the performance of tensor methods for $p=2$ and $p=3$ and show that the 3rd-order method is superior to the 2nd-order method in practice.
math.OC↗
Adaptive Proximal Method for Variational Inequalities
A new adaptive approach is proposed for variational inequalities with a Lipschitz-continuous field. Estimates of the necessary number of iterations are obtained to achieve a given quality of the variational inequality solution. A generalization of the method under consideration to the case of a Holder-continuous field is considered.
math.OC↗
Optimal Algorithms for Distributed Optimization
In this paper, we study the optimal convergence rate for distributed convex optimization problems in networks. We model the communication restrictions imposed by the network as a set of affine constraints and provide optimal complexity bounds for four different setups, namely: the function $F(\xb) \triangleq \sum_{i=1}^{m}f_i(\xb)$ is strongly convex and smooth, either strongly convex or smooth or just convex. Our results show that Nesterov's accelerated gradient descent on the dual problem can be executed in a distributed manner and obtains the same optimal rates as in the centralized version of the problem (up to constant or logarithmic factors) with an additional cost related to the spectral gap of the interaction matrix. Finally, we discuss some extensions to the proposed setup such as proximal friendly functions, time-varying graphs, improvement of the condition numbers.
math.OC↗
Algorithmic models of human behavior and stochastic optimization
In this article we describe the solutions of three problems posed at different time by Yurii Nesterov. First problem is 'Mage vs Experts': Assume that we live in some enviroment that characterized by unobservable convex function $f(x)$. Each day we can observe only the stochastic gradient of $f(x)$ (with heavy tails). We want to know where is a minimum of $f(x)$. What is better to live one long live or to live in parallel many short lives? The second problem is a variant of the previous one (we'd like to find the minimum of convex function $f(x)$) but we can observe only the (noise) value of the function at each day. The additional difficulties here is $f(x)$ isn't necessarily smooth and $x$ belongs to unit simplex. The third problem is to interprete equillibrium in Stable dynamic traffic assignment model (Nesterov-dePalma, 2000) as atractor of the best response dynamic. All these problems required proper generalizations of mirror descent (MD) algorithm. In this paper we collect different special versions of MD to solve the described problems.
math.OC↗
Distributed Computation of Wasserstein Barycenters over Networks
We propose a new \cu{class-optimal} algorithm for the distributed computation of Wasserstein Barycenters over networks. Assuming that each node in a graph has a probability distribution, we prove that every node can reach the barycenter of all distributions held in the network by using local interactions compliant with the topology of the graph. We provide an estimate for the minimum number of communication rounds required for the proposed method to achieve arbitrary relative precision both in the optimality of the solution and the consensus among all agents for undirected fixed networks.
math.OC↗
Primal-dual method for searching equillibriums in mixed traffic assignment problems
We consider mixed model of traffic flow distribution in large networks (BMW model, 1954 & Stable Dynamic model, 1999). We build dual problem and consider primal-dual mirror descent method for the dual problem. There are two ways to recover the solution of the primal problem. First technique is based on "models" approach and second one is based on Lagrange multiplyers technique. Both approaches have its own advantages and drawbacks. We discuss them.
math.OC↗
Markov models of macrosystems
We collect different examples reflect Bolzman--Jaynes theory of maximum entropy principle. This principle proposed that equillibrium of macrosystem (most probable macrostate of the invariant measure of macrosystem) can be find as a solution of the entropy linear programming problem. Among this examples one can find applications to the economics, sociology, linguistics, traffic flow theory, biology e.t.c. We also describe on examples the connection between Nash's equillibrium, Darwin's principle of natural selection and conception of equillibrium of macrosystem.
math.PR↗
Computational Optimal Transport: Complexity by Accelerated Gradient Descent Is Better Than by Sinkhorn's Algorithm
We analyze two algorithms for approximating the general optimal transport (OT) distance between two discrete distributions of size $n$, up to accuracy $\varepsilon$. For the first algorithm, which is based on the celebrated Sinkhorn's algorithm, we prove the complexity bound $\widetilde{O}\left({n^2/\varepsilon^2}\right)$ arithmetic operations. For the second one, which is based on our novel Adaptive Primal-Dual Accelerated Gradient Descent (APDAGD) algorithm, we prove the complexity bound $\widetilde{O}\left(\min\left\{n^{9/4}/\varepsilon, n^{2}/\varepsilon^2 \right\}\right)$ arithmetic operations. Both bounds have better dependence on $\varepsilon$ than the state-of-the-art result given by $\widetilde{O}\left({n^2/\varepsilon^3}\right)$. Our second algorithm not only has better dependence on $\varepsilon$ in the complexity bound, but also is not specific to entropic regularization and can solve the OT problem with different regularizers.
cs.DS↗