Searcharxiv⌕ Search

arXiv subjects

Akiko Takeda

Publications and source records attributed to Akiko Takeda.

At least 19 recordsLinked to original sources

Zeroth-Order Gradient Estimators under Decision-Dependent Noise

We study zeroth-order optimization of a generally nonconvex optimization problem that is accessible only through a noisy black-box oracle, where the observation noise has zero mean and its distribution depends on the queried decision vector. Since the gradient of the objective function is inaccessible, various zeroth-order methods have been developed to solve the problem. However, it remains unclear how the choice of search directions affects the sample complexity of zeroth-order methods and how their algorithmic parameters should be selected. In this paper, we conduct a unified sample complexity analysis of zeroth-order methods across gradient estimators with different search directions. As a result, we show that gradient estimators that average over multiple directions, either uniformly from the unit sphere or from a Gaussian distribution, achieve the lowest sample complexity. The resulting sample-complexity bounds improve upon those of existing zeroth-order methods for nonconvex objective functions that are not bounded above. Moreover, through simulation experiments on multiple products pricing and strategic classification applications, we demonstrate the practical performance of zeroth-order methods with various gradient estimators.

math.OC↗

Approximate Bregman proximal gradient algorithm with variable metric Armijo line search

We propose a variant of the approximate Bregman proximal gradient (ABPG) algorithm for minimizing the sum of a smooth nonconvex function and a nonsmooth convex function. ABPG is known to converge globally to a stationary point even when the smooth part of the objective function does not have a globally Lipschitz continuous gradient, and its iterates can often be expressed in closed form. However, ABPG relies on an Armijo line search to guarantee global convergence, which can slow down its practical performance. To address this issue, we propose a variant of ABPG with a variable metric Armijo line search, which relaxes the Armijo condition. We prove that the line search terminates in finitely many steps for a general nonsmooth convex function. Under the variable metric Armijo condition, we establish global subsequential convergence of the algorithm. Moreover, assuming the Kurdyka--Łojasiewicz property, we also prove that the algorithm globally converges to a stationary point. Numerical experiments on $\ell_p$-regularized least squares problems, nonnegative linear inverse problems and phase retrieval demonstrate that the proposed algorithm outperforms existing algorithms.

math.OC↗

Practical Regularized Quasi-Newton Methods with Inexact Function Values

Many practical optimization problems involve objective function values that are corrupted by unavoidable numerical errors. In smooth nonconvex optimization, quasi-Newton methods combined with line search are widely used due to their efficiency and scalability. These methods implicitly assume accurate function evaluations and thus may fail to converge in noisy settings. Developing fast and stable quasi-Newton methods for such scenarios is therefore crucial. To address this issue, we propose a novel noise-tolerant regularized quasi-Newton method equipped with a relaxed Armijo-type line search, designed to remain stable under inaccurate function evaluations. By combining a regularization parameter update rule inspired by Objective-Function-Free Optimization and the AdaGrad-Norm method, we prove an order-optimal $\mathcal{O}(\varepsilon^{-2})$ complexity bound under errors in function values. We performed extensive experiments on the CUTEst benchmark collection with artificially noisy objective function evaluations, as well as with low-precision floating-point arithmetic (64-, 32-, and 16-bit). The results demonstrate stable performance under reduced precision and function-value noise while retaining practical convergence speed and computational cost.

math.OC↗

Accelerated Proximal Gradient Method for Non-Strongly Convex Time-Varying Optimization

This paper studies time-varying composite optimization problems with convex but not necessarily strongly convex objective functions, on compact domains and with bounded temporal variability. As a baseline, we first analyze the time-varying proximal gradient (TV-PG) method. We show that its asymptotic function value gap is bounded by $O(δ^{1/2})$ as $δ\to0$, where $δ$ measures the temporal variability of the objective function. This dependence on $δ$ is tight in the worst case for TV-PG. Our main contribution is a time-varying accelerated proximal gradient method that improves the asymptotic upper bound on the function value gap to $O(δ^{2/3})$. The method builds on a novel acceleration scheme whose momentum parameter is determined by the current iterates, with no explicit dependence on the iteration index. We establish the bounds for both the baseline and accelerated methods through a unified analysis based on potential functions. Numerical experiments on a synthetic problem and autoregression problems with real-world datasets demonstrate the tracking performance of the proposed algorithm.

math.OC↗

A Simple yet Highly Accurate Prediction-Correction Algorithm for Time-Varying Optimization

This paper proposes a simple yet highly accurate prediction-correction algorithm, SHARP, for unconstrained time-varying optimization problems. Its prediction is based on an extrapolation derived from the Lagrange interpolation of past solutions. Since this extrapolation can be computed without Hessian matrices or even gradients, the computational cost is low. To ensure the stability of the prediction, the algorithm includes an acceptance condition that rejects the prediction when the update is excessively large. The proposed method achieves a tracking error of $O(h^{p})$, where $h$ is the sampling period, assuming that the $p$-th derivative of the target trajectory is bounded and the convergence of the correction step is locally linear. We also prove that the method can track a trajectory of stationary points even if the objective function is non-convex. Numerical experiments demonstrate the high accuracy of the proposed algorithm.

math.OC↗

Douglas--Rachford for multioperator comonotone inclusions with applications to multiblock optimization

We study the doubly relaxed Douglas--Rachford (DR) algorithm for solving a multioperator inclusion problem involving the sum of maximally comonotone operators. To address such problems, we adopt a product space reformulation that accommodates nonconvex-valued operators, which is essential when dealing with weakly comonotone mappings. We establish the convergence of the doubly relaxed DR algorithm under comonotonicity assumptions, subject to suitable conditions on the algorithm parameters and the comonotonicity moduli of the operators. Our analysis relies on the Attouch--Théra duality framework, which enables the study of convergence through the corresponding dual inclusion problem. As an application, we derive a multiblock ADMM-type algorithm for structured convex and nonconvex optimization problems by applying the doubly relaxed DR algorithm to the operator inclusion formulation of the KKT system. The resulting method extends the classical duality between the DR algorithm and the alternating direction method of multipliers from the convex two-block case to multiblock and nonconvex settings. Moreover, we establish convergence guarantees in both the fully convex and strongly convex-weakly convex regimes.

math.OC↗

Improved KKT Complexity for First-Order Bilevel Optimization under Weak Lower-Level Convexity

We study deterministic first-order bilevel optimization under weak lower-level convexity, allowing nonconvex lower-level objectives and without assuming strong convexity, the Polyak-Łojasiewicz condition, or an error-bound property. We consider a $δ$-relaxed Moreau-gap constraint, with $δ>0$, for the lower-level stationarity condition and propose an inexact variable-smoothing penalty method (IVSP) for computing its approximate Karush-Kuhn-Tucker (KKT) points. For any fixed relaxation level $δ$, under a standard extended no-nonzero-abnormal-multiplier constraint qualification (ENNAMCQ), we prove finite stabilization of the adaptive penalty parameter and an overall $\widetilde O(\varepsilon^{-3})$ first-order complexity for computing an $\varepsilon$-KKT point. Notably, we give verifiable sufficient conditions for ENNAMCQ covering convex lower-level objectives without nonconstant affine segments (including the strictly convex case), and a class of nonconvex sample-reweighting models. The positive relaxation avoids the intrinsic constraint-qualification degeneracy of the exact Moreau-gap constraint while achieving an $\mathcal O(\sqrtδ)$ lower-level near-stationarity guarantee. Numerical experiments on synthetic and real-world bilevel learning problems illustrate the practical performance of IVSP.

math.OC↗

Linesearch-Free Nonlinearly Preconditioned GD with Semiautomatic Geometry Design

Nonlinear preconditioning makes it possible to adapt gradient updates to the growth of the objective's curvature, but combining its design with suitable stepsize selection is challenging. We propose adaptive dual preconditioned gradient descent (adaptive DPGD), which combines curvature-based preconditioner design with a linesearch-free stepsize rule based on local information. For convex objectives, we establish linear convergence under local strong convexity of the objective. By adapting the stepsize rule, we extend the method to weakly convex objectives and establish asymptotic stationarity without global Lipschitz smoothness. Moreover, our local assumptions give greater freedom in preconditioner design. We exploit this freedom to develop a semiautomatic recipe guided by the objective's curvature, yielding a family of preconditioners that includes those underlying normalized and hyperbolic gradient descent. Experiments on convex and weakly convex problems with real data demonstrate the effectiveness of the preconditioner design and adaptive stepsize selection.

math.OC↗

Beyond Polytopes: Damped Newton Frank--Wolfe Methods over Compact Convex Sets

We study convex optimization over compact convex sets for twice differentiable objective functions with positive definite Hessians, assuming access to the feasible set through a linear minimization oracle. Existing second-order Frank--Wolfe methods either provide only a local linear rate over general convex sets or achieve global linear and local quadratic convergence only over polytopes. We propose damped Newton Frank--Wolfe methods that approximately solve constrained damped Newton subproblems using Frank--Wolfe or away-step Frank--Wolfe inner iterations. We develop residual backtracking, using a root Newton stepsize as a safe reference, together with a switching rule that eventually takes full steps. Under Hölder smoothness assumptions for $p$th derivatives, we establish global linear convergence and local convergence with Q-order at least $1+ν$ ($p=2$) and local quadratic convergence ($p=3$). Experiments on matrix sensing and ridge-regularized logistic regression show that the residual-backtracking variant is consistently robust and often achieves the best performance among the tested first- and second-order baselines.

math.OC↗

Accelerated Bregman Proximal Gradient Methods from Dual Geometric Perspectives

We study Bregman proximal gradient (BPG) algorithms under relative smoothness for convex, relatively strongly convex, and nonconvex objectives. Existing accelerated BPG algorithms for convex objectives typically require additional assumptions on Bregman divergences, most notably triangle-scaling conditions, which can lead to slower convergence rates. We propose a family of geometry-accelerated BPG algorithms that exploit Bregman geometry in both proximal-gradient and mirror-space updates, without imposing additional geometric conditions such as triangle-scaling conditions. Our methods adapt the stepsizes and mirror-space updates through local backtracking and computable acceptance criteria, without a global relative-smoothness constant as input. We derive convergence bounds in terms of the parameters accepted during the iterations. These bounds yield an $Ø(k^{-2})$ rate for convex objectives and a linear rate for relatively strongly convex objectives when the geometry-acceleration parameters remain uniformly bounded. For nonconvex objectives, we establish an $Ø(k^{-1})$ rate for a stationarity measure without requiring a lower Bregman bound or a full-domain Bregman divergence. Numerical experiments on inverse problems, entropy-regularized least squares, D-optimal design, and nonnegative matrix factorization demonstrate faster practical convergence than established Bregman baselines.

math.OC↗

Gradient Descent with Stochastic Subspaces via Persistence of Memory

Stochastic subspace methods have gained popularity as gradient descent based techniques for large scale optimisation problems, especially in distributed settings. In this paper, we introduce the technique of "persistence of memory" to greatly extend and improve the random subspace methods. To this end, we leverage a vector that is only weakly correlated with the gradient in order to provide a guiding structure to the generative process of the random subspace along which the descent is going to take place. This guidance vector may be fixed for a large number of iterations, only to be refreshed at wide intervals (on whose size we can provide guarantees in terms of problem parameters). In important machine learning settings, such as optimisation problems embodying sparsity or a minibatch structure, we show that the guidance vector can be obtained in an effective and computationally inexpensive manner by leveraging the structured properties of the problem. En route, we establish to our knowledge the first theoretical analysis of classical SSD methods for sparse functions. In a local neighbourhood of the optimum, we demonstrate an alignment phenomenon of our gradient estimates with a low-lying eigenvector of the Hessian, allowing a once-for-all computation of the guidance vector which renders the method computationally favourable even in scenarios with unstructured objectives.

math.OC↗

A Splitting Framework for Composite Semimonotone Inclusions

We introduce a general framework for composite inclusion problems with affine constraints, covering both monotone and semimonotone regimes. The central idea is a new interpretation of the constrained inclusion through an operator-vector pair that separates the implicit inclusion from the affine constraint: the former is handled through possibly preconditioned resolvent evaluations, while the latter is handled through an explicit forward step in an auxiliary variable. This yields a single abstract iteration applicable to multioperator inclusions, linearly coupled inclusions, and block-separable inclusions with affine constraints. The freedom in choosing the operator-vector pair enables the systematic construction of problem-adapted splitting algorithms, including new schemes for several important problem classes. Exploiting the orthogonal decomposition induced by the constraint subspace, we develop a unified and streamlined convergence analysis and establish weak and strong convergence guarantees under semimonotonicity assumptions. When specialized to multioperator inclusions, the framework permits general bounded linear operator coefficients, rather than only scalar coefficients, and therefore accommodates preconditioned resolvents. The resulting schemes recover several existing methods while extending them to previously uncovered regimes, and in several important cases, require weaker assumptions and admit provably larger admissible parameter ranges.

math.OC↗

Adaptive Gradient-Based Methods for a Broader Class of Optimization Problems under Performative Prediction

We study optimization under performative prediction, where deploying a model affects the future data distribution. For this setting, several gradient-based approaches have been proposed. However, they typically assume specific data distributions or loss functions, which limit their practical applicability. To overcome these limitations, we propose a gradient-based optimization method with convergence guarantees under substantially weaker assumptions. Our method explicitly estimates the induced distribution shift through finite differences. It enables higher-dimensional optimization across broader classes of loss functions and data distributions. We also propose a practical variant that reduces the number of samples required. Numerical experiments demonstrate that our proposed algorithms converge faster and more consistently than existing ones.

math.OC↗

A New Dynamic Pricing Model based on Convex Hull Pricing

This paper presents a new dynamic pricing model (a.k.a. real-time pricing) that reflects startup costs of generators. Dynamic pricing, which is a method to control demand by pricing electricity at hourly (or more often) intervals, has been studied by many researchers. They assume that the cost functions of suppliers are convex, although they may be nonconvex because of the startup costs of generators in practice. We provide a dynamic pricing model that takes into account such cost functions within the settings of unit commitment problems (UCPs). Our model gives convex hull price (CHP), which has not been used in the context of dynamic pricing, though it is known that the CHP minimizes the uplift payment which is disadvantageous to suppliers for a given demand. In addition, we apply an iterative algorithm based on the subgradient method to solve our model. Numerical experiments show the efficiency of our model on reducing uplift payments. The prices determined by our algorithm give sufficiently small uplift payments in a realistic computational time.

math.OC↗

Parameter-Free Cubic-Regularized Newton Method: Sharp Complexity and Generalized Smoothness

We analyze a variant of the cubic-regularized Newton method for nonconvex optimization. This variant is parameter-free in that it requires no prior knowledge of problem-dependent parameters. Under the generalized smoothness condition $\|\nabla^3 f(x)\| \leq L_0 + L_1 \|\nabla f(x)\|$, we derive an oracle complexity bound for finding an $(\varepsilon, δ)$-second-order stationary point. This assumption is weaker than the generalized smoothness conditions used in existing analyses of second-order methods, while the complexity bound improves upon existing guarantees for parameter-free second-order methods. In particular, when $L_1 = 0$, the bound matches the optimal dependence on $L_0$ as well as on $\varepsilon$, $δ$, and the initial function value gap, up to additive logarithmic terms. To establish this bound, we derive Taylor-type inequalities and prove their equivalence to the generalized smoothness condition.

math.OC↗

Change Point Detection in Precision Matrices with D-trace Loss

We consider the problem of estimating a time-varying sparse precision matrix, which is assumed to evolve in a piecewise constant manner. Building upon the Group Fused LASSO and LASSO penalty functions, we estimate both the precision matrix and the change points. We propose an alternative estimator to the commonly employed Gaussian likelihood loss, namely the D-trace loss. We provide the conditions for the consistency of the estimated change points and of the sparse estimators in each block. We show that the solutions to the corresponding estimation problem exist when some conditions relating to the tuning parameters of the penalty functions are satisfied. Unfortunately, these conditions are not verifiable in general, posing challenges for tuning the parameters in practice. To address this issue, we introduce a modified regularizer and develop a revised problem that always admits solutions: these solutions can be used for detecting possible unsolvability of the original problem or obtaining a solution of the original problem otherwise. An alternating direction method of multipliers (ADMM) is then proposed to solve the revised problem. The relevance of the method is illustrated through numerical experiments.

math.ST↗

Initial Placement for Fruchterman--Reingold Force Model With Coordinate Newton Direction

The Fruchterman--Reingold (FR) force model is widely used in force-directed graph drawing, and multilevel approaches such as sfdp in Graphviz scale these methods effectively. A crucial step in multilevel schemes is refinement, which improves the graph layout, typically performed with the simulation-based algorithm. For this refinement, we can utilize optimization-based methods such as L-BFGS, or combine them with initial placement methods such as Simulated Annealing to achieve better layouts. However, they have several limitations, such as suffering from high per-iteration costs for large graphs or having difficulty with weighted and structurally complex graphs, leaving room for improvement. In this research, we propose a new initial placement based on stochastic coordinate descent to accelerate the optimization process. We first reformulate the problem as a discrete optimization problem using a hexagonal lattice and then iteratively update a randomly selected vertex along the coordinate Newton direction with low per-iteration costs. We demonstrate the effectiveness of our method through numerical experiments, showing that our initial placement leads to faster convergence and higher-quality layouts compared to naive optimization approaches. We also discuss applications of our method, such as drawing for Hooke--Coulomb and Eades force models.

cs.CG↗

Projection onto hyperbolicity cones and beyond: a dual Frank-Wolfe approach

We discuss the problem of projecting a point onto an arbitrary hyperbolicity cone from both theoretical and numerical perspectives. While hyperbolicity cones are furnished with a generalization of the notion of eigenvalues, obtaining closed form expressions for the projection operator as in the case of semidefinite matrices is an elusive endeavour. To address that we propose a Frank-Wolfe method to handle this task and, more generally, strongly convex optimization over closed convex cones. One of our innovations is that the Frank-Wolfe method is actually applied to the dual problem and, by doing so, subproblems can be solved in closed-form using minimum eigenvalue functions and conjugate vectors. To test the validity of our proposed approach, we present numerical experiments where we check the performance of alternative approaches including interior point methods and an earlier accelerated gradient method proposed by Renegar. We also show numerical examples where the hyperbolic polynomial has millions of monomials. Finally, we also discuss the problem of projecting onto p-cones which, although not hyperbolicity cones in general, are still amenable to our techniques.

math.OC↗