arXiv subjects

Akiko Takeda

Publications and source records attributed to Akiko Takeda.

At least 37 records · Page 2Linked to original sources

arXiv2026

A primal-dual interior point trust region method for second-order stationary points of Riemannian inequality-constrained optimization problems

We consider Riemannian inequality-constrained optimization problems. Such problems inherit the benefits of Riemannian approach developed in the unconstrained setting and naturally arise from applications in control, machine learning, and other fields. We propose a Riemannian primal-dual interior point trust region method (RIPTRM) for solving them. We prove its global convergence to an approximate Karush-Kuhn-Tucker point and a weak second-order stationary point. Under the strict complementarity condition, this result reduces to global convergence to a second-order stationary point. To the best of our knowledge, this is the first algorithm that incorporates the trust region strategy for constrained optimization on Riemannian manifolds, and has the second-order convergence property for optimization problems on Riemannian manifolds with nonlinear inequality constraints. We conduct numerical experiments in which we introduce a truncated conjugate gradient method and an eigenvalue-based subsolver for RIPTRM to approximately and exactly solve the trust region subproblems, respectively. Empirical results show that RIPTRMs consistently find solutions with high accuracy. Additionally, we observe that RIPTRM with the exact search direction shows promising performance in an instance where the Hessian of the Lagrangian has a large negative eigenvalue.

math.OC↗

arXiv2026

Local near-quadratic convergence of Riemannian interior point methods

We consider Riemannian optimization problems with inequality and equality constraints and analyze a class of Riemannian interior point methods for solving them. The algorithm of interest consists of outer and inner iterations. We show that, under standard assumptions, the algorithm achieves local superlinear convergence by solving a linear system at each outer iteration, removing the need for further computations in the inner iterations. We also provide a specific update for the barrier parameter that achieves local near-quadratic convergence of the algorithm. We apply our results to the method proposed by Obara, Okuno, and Takeda (2026) and show its local superlinear and near-quadratic convergence with an analysis of the second-order stationarity. To our knowledge, this is the first algorithm for constrained optimization on Riemannian manifolds that achieves both local convergence and global convergence to a second-order stationary point. Numerical results support the theoretical analyses of the proposed methods.

math.OC↗

arXiv2026

Randomized Subspace Nesterov Accelerated Gradient

Randomized-subspace methods reduce the cost of first-order optimization by using only low-dimensional projected-gradient information, a feature that is attractive in forward-mode automatic differentiation and communication-limited settings. While Nesterov acceleration is well understood for full-gradient and coordinate-based methods, obtaining accelerated methods for general subspace sketches that use only projected-gradient information and can improve over full-dimensional Nesterov acceleration in oracle complexity is technically nontrivial. We develop randomized-subspace Nesterov accelerated gradient methods for smooth convex and smooth strongly convex optimization under matrix smoothness and generic sketch moment assumptions. The key technical ingredient is a three-sequence formulation tailored to matrix smoothness, which recovers the corresponding classical Nesterov methods in the full-dimensional case. The resulting theory establishes accelerated oracle-complexity guarantees and makes explicit how matrix smoothness and the sketch distribution enter the complexity. It also provides a unified basis for comparing sketch families and identifying when randomized-subspace acceleration improves over full-dimensional Nesterov acceleration in oracle complexity.

math.OC↗

arXiv2026

Inexact subgradient algorithm with a non-asymptotic convergence guarantee for copositive programming problems

In this paper, we propose a subgradient algorithm with a non-asymptotic convergence guarantee to solve copositive programming problems. The subproblem to be solved at each iteration is a standard quadratic programming problem, which is NP-hard in general. However, the proposed algorithm allows this subproblem to be solved inexactly. For a prescribed accuracy $ε> 0$ for both the objective function and the constraint arising from the copositivity condition, the proposed algorithm yields an approximate solution after $O(ε^{-2})$ iterations, even when the subproblems are solved inexactly. We also discuss exact and inexact approaches for solving standard quadratic programming problems and compare their performance through numerical experiments. In addition, we apply the proposed algorithm to the problem of testing complete positivity of a matrix and derive a sufficient condition for certifying that a matrix is not completely positive. Experimental results demonstrate that we can detect the lack of complete positivity in various doubly nonnegative matrices that are not completely positive.

math.OC↗

arXiv2026

Approximate Bregman proximal gradient algorithm with variable metric Armijo--Wolfe line search

We propose a variant of the approximate Bregman proximal gradient (ABPG) algorithm for minimizing the sum of a smooth nonconvex function and a nonsmooth convex function. ABPG is known to converge globally to a stationary point even when the smooth part of the objective function does not have a globally Lipschitz continuous gradient, and its iterates can often be expressed in closed form. However, ABPG relies on an Armijo line search to guarantee global convergence, which can slow down its practical performance. To address this issue, we propose a variant of ABPG with a variable metric Armijo--Wolfe line search. Under the variable metric Armijo--Wolfe condition, we establish global subsequential convergence of the algorithm. Moreover, assuming the Kurdyka--Łojasiewicz property, we also prove that the algorithm globally converges to a stationary point. Numerical experiments on $\ell_p$-regularized least squares problems and nonnegative linear inverse problems demonstrate that the proposed algorithm outperforms existing algorithms.

math.OC↗

arXiv2026

Fairness in Robust Unit Commitment Problem Considering Suppression of Renewable Energy

Power company operators make power generation plans one day in advance, in what is known as the Unit Commitment (UC) problem. UC is exposed to uncertainties, such as unknown electricity load and disturbances caused by renewable energy sources, especially PVs. In previous research, we proposed the Renewable Energy Robust Optimization Problem (RE-RP), which solves these uncertainties by considering suppression. In this paper, we propose a new model called RE-RP with fairness (RE-RPfair), which aims to achieve fair allocation among PVs allocation. This model is an expansion of the original RE-RP, and we prove its effectiveness through simulation. To measure the degree of fairness, we use the Gini Index, which is well-known in social science.

math.OC↗

arXiv2026

Fast Frank--Wolfe Algorithms with Adaptive Bregman Step-Size for Weakly Convex Functions

We propose Frank--Wolfe (FW) algorithms with an adaptive Bregman step-size strategy for smooth adaptable (also called: relatively smooth) (weakly-) convex functions. This means that the gradient of the objective function is not necessarily Lipschitz continuous, and we only require the smooth adaptable property. Compared with existing FW algorithms, our assumptions are less restrictive. We establish convergence guarantees in various settings, including convergence rates ranging from sublinear to linear, depending on the assumptions for convex and nonconvex objective functions. Assuming that the objective function is weakly convex and satisfies the local quadratic growth condition, we provide both local sublinear and local linear convergence with respect to the primal gap. We also propose a variant of the away-step FW algorithm using Bregman distances over polytopes. We establish faster global convergence (up to a linear rate) for convex optimization under the Hölder error bound condition and local linear convergence for nonconvex optimization under the local quadratic growth condition. Numerical experiments demonstrate that our proposed FW algorithms outperform existing methods.

math.OC↗

arXiv2026

Convergence Analysis of Randomized Subspace Normalized SGD under Heavy-Tailed Noise

Randomized subspace methods reduce per-iteration cost; however, in nonconvex optimization, most analyses are expectation-based, and high-probability bounds remain scarce even under sub-Gaussian noise. We first prove that randomized subspace SGD (RS-SGD) admits a high-probability convergence bound under sub-Gaussian noise, achieving the same order of oracle complexity as prior in-expectation results. Motivated by the prevalence of heavy-tailed gradients in modern machine learning, we then propose randomized subspace normalized SGD (RS-NSGD), which integrates direction normalization into subspace updates. Assuming the noise has bounded $p$-th moments, we establish both in-expectation and high-probability convergence guarantees, and show that RS-NSGD can achieve better oracle complexity than full-dimensional normalized SGD.

math.OC↗

arXiv2025

Complexity and convergence analysis of a single-loop SDCAM for Lipschitz composite optimization and beyond

We develop and analyze a single-loop algorithm for minimizing the sum of a Lipschitz differentiable function $f$, a prox-friendly proper closed function $g$ (with a closed domain on which $g$ is continuous) and the composition of another prox-friendly proper closed function $h$ (whose domain is closed on which $h$ is continuous) with a continuously differentiable mapping $c$ (that is Lipschitz continuous and Lipschitz differentiable on the convex closure of the domain of $g$). Such models arise naturally in many contemporary applications, where $f$ is the loss function for data misfit, and $g$ and $h$ are nonsmooth functions for inducing desirable structures in $x$ and $c(x)$. Existing single-loop algorithms mainly focus either on the case where $h$ is Lipschitz continuous or the case where $h$ is an indicator function of a closed convex set. In this paper, we develop a single-loop algorithm for more general possibly non-Lipschitz $h$. Our algorithm is a single-loop variant of the successive difference-of-convex approximation method (SDCAM) proposed in [22]. We show that when $h$ is Lipschitz, our algorithm exhibits an iteration complexity that matches the best known complexity result for obtaining an $(ε_1,ε_2,0)$-stationary point. Moreover, we show that, by assuming additionally that dom $g$ is compact, our algorithm exhibits an iteration complexity of $\tilde{O}(ε^{-4})$ for obtaining an $(ε,ε,ε)$-stationary point when $h$ is merely continuous and real-valued. Furthermore, we consider a scenario where $h$ does not have full domain and establish vanishing bounds on successive changes of iterates. Finally, in all three cases mentioned above, we show that one can construct a subsequence such that any accumulation point $x^*$ satisfies $c(x^*)\in$ dom $h$, and if a standard constraint qualification holds at $x^*$, then $x^*$ is a stationary point.

math.OC↗

arXiv2025

A Regression-Based Prediction-Correction Method for Stochastic Time-Varying Optimization Problems

In many real-world applications, optimization problems evolve continuously over time and are often subject to stochastic noise. We consider a stochastic time-varying optimization problem in which the objective function $f(x;t)$ changes continuously and only noisy gradient observations are available. In deterministic settings, the prediction-correction method that exploits the time derivative of the solution is effective for accurately tracking the solution trajectory. However, a straightforward extension to stochastic problems requires an estimate of $\nabla_{xt} f(x;t)$ and the computation of a Hessian inverse at each step--requirements that are difficult or costly in practice. To address these issues, we propose a prediction-correction algorithm that uses a regression-based prediction step: the prediction is formed as a linear combination of recent iterates, which can be computed efficiently without estimating $\nabla_{xt}f(x;t)$ or computing Hessian inversions. We prove a tracking-error bound for the proposed method under standard smoothness and stochastic assumptions. Numerical experiments show that the regression-based prediction improves tracking accuracy while reducing computational cost compared with existing methods.

math.OC↗

arXiv2025

Douglas--Rachford algorithm for nonmonotone multioperator inclusion problems

The Douglas--Rachford algorithm is a classic splitting method for finding a zero of the sum of two maximal monotone operators. It has also been applied to settings that involve one weakly and one strongly monotone operator. In this work, we extend the Douglas--Rachford algorithm to address multioperator inclusion problems involving $m$ ($m\geq 2$) weakly and strongly monotone operators, reformulated as a two-operator inclusion in a product space. By selecting appropriate parameters, we establish the convergence of the algorithm to a fixed point, from which solutions can be extracted. Furthermore, we illustrate its applicability to sum-of-$m$-functions minimization problems characterized by weakly convex and strongly convex functions. For general nonconvex problems in finite-dimensional spaces, comprising Lipschitz continuously differentiable functions and a proper closed function, we provide global subsequential convergence guarantees.

math.OC↗

arXiv2025

Zeroth-order gradient estimators for stochastic problems with decision-dependent distributions

Stochastic optimization problems with unknown decision-dependent distributions have attracted increasing attention in recent years due to its importance in applications. Since the gradient of the objective function is inaccessible as a result of the unknown distribution, various zeroth-order methods have been developed to solve the problem. However, it remains unclear which search direction to construct a gradient estimator is more appropriate and how to set the algorithmic parameters. In this paper, we conduct a unified sample complexity analysis of zeroth-order methods across gradient estimators with different search directions. As a result, we show that gradient estimators that average over multiple directions, either uniformly from the unit sphere or from a Gaussian distribution, achieve the lowest sample complexity. The attained sample complexities improve those of existing zeroth-order methods in the problem setting that allows nonconvexity and unboundedness of the objective function. Moreover, by simulation experiments on multiple products pricing and strategic classification applications, we show practical performance of zeroth-order methods with various gradient estimators.

math.OC↗

arXiv2025

Theoretical analysis of the randomized subspace regularized Newton method for non-convex optimization

While there already exist randomized subspace Newton methods that restrict the search direction to a random subspace for a convex function, we propose a randomized subspace regularized Newton method for a non-convex function {and more generally we investigate thoroughly the local convergence rate of the randomized subspace Newton method}. In our proposed algorithm using a modified Hessian of the function restricted to some random subspace, with high probability, the function value decreases even when the objective function is non-convex. In this paper, we show that our method has global convergence under appropriate assumptions and its convergence rate is the same as that of the full regularized Newton method. %We also prove that Furthermore, we can obtain a local linear convergence rate under some additional assumptions, and prove that this rate is the best we can hope, in general, when using random subspace. We furthermore prove that if the Hessian at the local optimum is rank defficient then superlienar convergence holds.

math.OC↗

arXiv2025

Univariate representations of solutions to generic polynomial complementarity problems

By using the squared slack variables technique, we demonstrate that the solution set of a general polynomial complementarity problem is the image, under a specific projection, of the set of real zeroes of a system of polynomials. This paper points out that, generically, this polynomial system has finitely many complex zeroes. In such a case, we use symbolic computation techniques to compute a univariate representation of the solution set. Consequently, univariate representations of special solutions, such as least-norm and sparse solutions, are obtained. After that, enumerating solutions boils down to solving problems governed by univariate polynomials. We also provide some experiments on small-scale problems with worst-case scenarios. At the end of the paper, we propose a method for computing approximate solutions to copositive polynomial complementarity problems that may have infinitely many solutions.

math.OC↗

arXiv2025

Modified K-means Algorithm with Local Optimality Guarantees

The K-means algorithm is one of the most widely studied clustering algorithms in machine learning. While extensive research has focused on its ability to achieve a globally optimal solution, there still lacks a rigorous analysis of its local optimality guarantees. In this paper, we first present conditions under which the K-means algorithm converges to a locally optimal solution. Based on this, we propose simple modifications to the K-means algorithm which ensure local optimality in both the continuous and discrete sense, with the same computational complexity as the original K-means algorithm. As the dissimilarity measure, we consider a general Bregman divergence, which is an extension of the squared Euclidean distance often used in the K-means algorithm. Numerical experiments confirm that the K-means algorithm does not always find a locally optimal solution in practice, while our proposed methods provide improved locally optimal solutions with reduced clustering loss. Our code is available at https://github.com/lmingyi/LO-K-means.

cs.LG↗

arXiv2025

Theoretical smoothing frameworks for nonsmooth simple bilevel problems

Bilevel programming has recently received a great deal of attention due to its abundant applications in many areas. The optimal value function approach provides a useful reformulation of the bilevel problem, but its utility is often limited due to the nonsmoothness of the value function even in cases when the associated lower-level function is smooth. In this paper, we present two smoothing strategies for the value function associated with lower-level functions that are not necessarily smooth but are Lipschitz continuous. The first method employs quadratic regularization for partially convex lower-level functions, while the second utilizes entropic regularization for general lower-level objective functions. Meanwhile, the property known as gradient consistency is crucial in ensuring that a designed smoothing algorithm is globally subsequentially convergent to stationary points of the value function reformulation. With this motivation, we prove that the proposed smooth approximations satisfy the gradient consistent property under certain conditions on the lower-level function.

math.OC↗

arXiv2025

On the Role of Label Noise in the Feature Learning Process

Deep learning with noisy labels presents significant challenges. In this work, we theoretically characterize the role of label noise from a feature learning perspective. Specifically, we consider a signal-noise data distribution, where each sample comprises a label-dependent signal and label-independent noise, and rigorously analyze the training dynamics of a two-layer convolutional neural network under this data setup, along with the presence of label noise. Our analysis identifies two key stages. In Stage I, the model perfectly fits all the clean samples (i.e., samples without label noise) while ignoring the noisy ones (i.e., samples with noisy labels). During this stage, the model learns the signal from the clean samples, which generalizes well on unseen data. In Stage II, as the training loss converges, the gradient in the direction of noise surpasses that of the signal, leading to overfitting on noisy samples. Eventually, the model memorizes the noise present in the noisy samples and degrades its generalization ability. Furthermore, our analysis provides a theoretical basis for two widely used techniques for tackling label noise: early stopping and sample selection. Experiments on both synthetic and real-world setups validate our theory.

stat.ML↗

arXiv2025

Efficient Optimization with Orthogonality Constraint: a Randomized Riemannian Submanifold Method

Optimization with orthogonality constraints frequently arises in various fields such as machine learning. Riemannian optimization offers a powerful framework for solving these problems by equipping the constraint set with a Riemannian manifold structure and performing optimization intrinsically on the manifold. This approach typically involves computing a search direction in the tangent space and updating variables via a retraction operation. However, as the size of the variables increases, the computational cost of the retraction can become prohibitively high, limiting the applicability of Riemannian optimization to large-scale problems. To address this challenge and enhance scalability, we propose a novel approach that restricts each update on a random submanifold, thereby significantly reducing the per-iteration complexity. We introduce two sampling strategies for selecting the random submanifolds and theoretically analyze the convergence of the proposed methods. We provide convergence results for general nonconvex functions and functions that satisfy Riemannian Polyak-Lojasiewicz condition as well as for stochastic optimization settings. Additionally, we demonstrate how our approach can be generalized to quotient manifolds derived from the orthogonal manifold. Extensive experiments verify the benefits of the proposed method, across a wide variety of problems.

math.OC↗