arXiv subjects

Isao Yamada

Publications and source records attributed to Isao Yamada.

31 records · Page 2Linked to original sources

arXiv2024

A variable smoothing for Nonconvexly constrained nonsmooth optimization with application to sparse spectral clustering

We propose a variable smoothing algorithm for solving nonconvexly constrained nonsmooth optimization problems. The target problem has two issues that need to be addressed: (i) the nonconvex constraint and (ii) the nonsmooth term. To handle the nonconvex constraint, we translate the target problem into an unconstrained problem by parameterizing the nonconvex constraint in terms of a Euclidean space. We show that under a certain condition, these problems are equivalent in view of finding a stationary point. To find a stationary point of the parameterized problem, the proposed algorithm performs the gradient descent update for the smoothed version of the parameterized problem with replacement of the nonsmooth function by the Moreau envelope, inspired by a variable smoothing algorithm [Böhm-Wright, J. Optim. Theory Appl., 2021] specialized for unconstrained nonsmooth optimization. We also present a convergence analysis of the proposed algorithm as well as its application to a nonconvex reformulation of the sparse spectral clustering.

math.OC↗

arXiv2023

Generalized Left-Localized Cayley Parametrization for Optimization with Orthogonality Constraints

We present a reformulation of optimization problems over the Stiefel manifold by using a Cayley-type transform, named the generalized left-localized Cayley transform, for the Stiefel manifold. The reformulated optimization problem is defined over a vector space, whereby we can apply directly powerful computational arts designed for optimization over a vector space. The proposed Cayley-type transform enjoys several key properties which are useful to (i) study relations between the original problem and the proposed problem; (ii) check the conditions to guarantee the global convergence of optimization algorithms. Numerical experiments demonstrate that the proposed algorithm outperforms the standard algorithms designed with a retraction on the Stiefel manifold.

math.OC↗

arXiv2023

A Unified Framework for Solving a General Class of Nonconvexly Regularized Convex Models

Recently, several nonconvex sparse regularizers which can preserve the convexity of the cost function have received increasing attention. This paper proposes a general class of such convexity-preserving (CP) regularizers, termed partially smoothed difference-of-convex (pSDC) regularizer. The pSDC regularizer is formulated as a structured difference-of-convex (DC) function, where the landscape of the subtrahend function can be adjusted by a parameterized smoothing function so as to attain overall-convexity. Assigned with proper building blocks, the pSDC regularizer reproduces existing CP regularizers and opens the way to a large number of promising new ones. With respect to the resultant nonconvexly regularized convex (NRC) model, we derive a series of overall-convexity conditions which naturally embrace the conditions in previous works. Moreover, we develop a unified framework based on DC programming for solving the NRC model. Compared to previously reported proximal splitting type approaches, the proposed framework makes less stringent assumptions. We establish the convergence of the proposed framework to a global minimizer. Numerical experiments demonstrate the power of the pSDC regularizers and the efficiency of the proposed DC algorithm.

eess.SP↗

arXiv2023

Adaptive Localized Cayley Parametrization for Optimization over Stiefel Manifold

We present an adaptive parametrization strategy for optimization problems over the Stiefel manifold by using generalized Cayley transforms to utilize powerful Euclidean optimization algorithms efficiently. The generalized Cayley transform can translate an open dense subset of the Stiefel manifold into a vector space, and the open dense subset is determined according to a tunable parameter called a center point. With the generalized Cayley transform, we recently proposed the naive Cayley parametrization, which reformulates the optimization problem over the Stiefel manifold as that over the vector space. Although this reformulation enables us to transplant powerful Euclidean optimization algorithms, their convergences may become slow by a poor choice of center points. To avoid such a slow convergence, in this paper, we propose to estimate adaptively 'good' center points so that the reformulated problem can be solved faster. We also present a unified convergence analysis, regarding the gradient, in cases where fairly standard Euclidean optimization algorithms are employed in the proposed adaptive parametrization strategy. Numerical experiments demonstrate that (i) the proposed strategy succeeds in escaping from the slow convergence observed in the naive Cayley parametrization strategy; (ii) the proposed strategy outperforms the standard strategy which employs a retraction.

math.OC↗

arXiv2023

Linearly-involved Moreau-Enhanced-over-Subspace Model: Debiased Sparse Modeling and Stable Outlier-Robust Regression

We present an efficient mathematical framework based on the linearly-involved Moreau-enhanced-over-subspace (LiMES) model. Two concrete applications are considered: sparse modeling and robust regression. The popular minimax concave (MC) penalty for sparse modeling subtracts, from the $\ell_1$ norm, its Moreau envelope, inducing nearly unbiased estimates and thus yielding remarkable performance enhancements. To extend it to underdetermined linear systems, we propose the projective minimax concave penalty using the projection onto the input subspace, where the Moreau-enhancement effect is restricted to the subspace for preserving the overall convexity. We also present a novel concept of stable outlier-robust regression which distinguishes noise and outlier explicitly. The LiMES model encompasses those two specific examples as well as two other applications: stable principal component pursuit and robust classification. The LiMES function involved in the model is an ``additively nonseparable'' weakly convex function but is defined with the Moreau envelope returning the minimum of a ``separable'' convex function. This mixed nature of separability and nonseparability allows an application of the LiMES model to the underdetermined case with an efficient algorithmic implementation. Two linear/affine operators play key roles in the model: one corresponds to the projection mentioned above and the other takes care of robust regression/classification. A necessary and sufficient condition for convexity of the smooth part of the objective function is studied. Numerical examples show the efficacy of LiMES in applications to sparse modeling and robust regression.

eess.SP↗

arXiv2022

Hierarchical Convex Optimization by the Hybrid Steepest Descent Method with Proximal Splitting Operators -- Enhancements of SVM and Lasso

The breakthrough ideas in the modern proximal splitting methodologies allow us to express the set of all minimizers of a superposition of multiple nonsmooth convex functions as the fixed point set of computable nonexpansive operators. In this paper, we present practical algorithmic strategies for the hierarchical convex optimization problems which require further strategic selection of a most desirable vector from the solution set of the standard convex optimization. The proposed algorithms are established by applying the hybrid steepest descent method to special nonexpansive operators designed through the art of proximal splitting. We also present applications of the proposed strategies to certain unexplored hierarchical enhancements of the support vector machine and the Lasso estimator.

math.OC↗

arXiv2021

A Convexly Constrained LiGME Model and Its Proximal Splitting Algorithm

For the sparsity-rank-aware least squares estimations, the LiGME (Linearly involved Generalized Moreau Enhanced) model was established recently in [Abe, Yamagishi, Yamada, 2020] to use certain nonconvex enhancements of linearly involved convex regularizers without losing their overall convexities. In this paper, for further advancement of the LiGME model by incorporating multiple a priori knowledge as hard convex constraints, we newly propose a convexly constrained LiGME (cLiGME) model. The cLiGME model can utilize multiple convex constraints while preserving benefits achieved by the LiGME model. We also present a proximal splitting type algorithm for the proposed cLiGME model. Numerical experiments demonstrate the efficacy of the proposed model and the proposed optimization algorithm in a scenario of signal processing application.

math.OC↗

arXiv2021

Linearly Involved Generalized Moreau Enhanced Models and Their Proximal Splitting Algorithm under Overall Convexity Condition

The convex envelopes of the direct discrete measures, for the sparsity of vectors or for the low-rankness of matrices, have been utilized extensively as practical penalties in order to compute a globally optimal solution of the corresponding regularized least-squares models. Motivated mainly by the ideas in [Zhang'10, Selesnick'17, Yin, Parekh, Selesnick'19] to exploit nonconvex penalties in the regularized least-squares models without losing their overall convexities, this paper presents the Linearly involved Generalized Moreau Enhanced (LiGME) model as a unified extension of such utilizations of nonconvex penalties. The proposed model can admit multiple nonconvex penalties without losing its overall convexity and thus is applicable to much broader scenarios in the sparsity-rank-aware signal processing. Under the general overall-convexity condition of the LiGME model, we also present a novel proximal splitting type algorithm of guaranteed convergence to a globally optimal solution. Numerical experiments in typical examples of the sparsity-rank-aware signal processing demonstrate the effectiveness of the LiGME models and the proposed proximal splitting algorithm.

math.OC↗

arXiv2020

Approximate Simultaneous Diagonalization of Matrices via Structured Low-Rank Approximation

Approximate Simultaneous Diagonalization (ASD) is a problem to find a common similarity transformation which approximately diagonalizes a given square-matrix tuple. Many data science problems have been reduced into ASD through ingenious modelling. For ASD, the so-called Jacobi-like methods have been extensively used. However, the methods have no guarantee to suppress the magnitude of off-diagonal entries of the transformed tuple even if the given tuple has a common exact diagonalizer, i.e., the given tuple is simultaneously diagonalizable. In this paper, to establish an alternative powerful strategy for ASD, we present a novel two-step strategy, called Approximate-Then-Diagonalize-Simultaneously (ATDS) algorithm. The ATDS algorithm decomposes ASD into (Step 1) finding a simultaneously diagonalizable tuple near the given one; and (Step 2) finding a common similarity transformation which diagonalizes exactly the tuple obtained in Step 1. The proposed approach to Step 1 is realized by solving a Structured Low-Rank Approximation (SLRA) with Cadzow's algorithm. In Step 2, by exploiting the idea in the constructive proof regarding the conditions for the exact simultaneous diagonalizability, we obtain a common exact diagonalizer of the obtained tuple in Step 1 as a solution for the original ASD. Unlike the Jacobi-like methods, the ATDS algorithm has a guarantee to find a common exact diagonalizer if the given tuple happens to be simultaneously diagonalizable. Numerical experiments show that the ATDS algorithm achieves better performance than the Jacobi-like methods.

math.NA↗

arXiv2020

A Hierarchical Convex Optimization for Multiclass SVM Achieving Maximum Pairwise Margins with Least Empirical Hinge-Loss

In this paper, we formulate newly a hierarchical convex optimization for multiclass SVM achieving maximum pairwise margins with least empirical hinge-loss. This optimization problem is a most faithful as well as robust multiclass extension of an NP-hard hierarchical optimization appeared for the first time in the seminal paper by C.~Cortes and V.~Vapnik almost 25 years ago. By extending the very recent fixed point theoretic idea [Yamada-Yamagishi 2019] with the generalized hinge loss function [Crammer-Singer 2001], we show that the hybrid steepest descent method [Yamada 2001] in the computational fixed point theory is applicable to this much more complex hierarchical convex optimization problem.

math.OC↗

arXiv2018

Fejér-monotone hybrid steepest descent method for affinely constrained and composite convex minimization tasks

This paper introduces the Fejér-monotone hybrid steepest descent method (FM-HSDM), a new member to the HSDM family of algorithms, for solving affinely constrained minimization tasks in real Hilbert spaces, where convex smooth and non-smooth losses compose the objective function. FM-HSDM offers sequences of estimates which converge weakly and, under certain hypotheses, strongly to solutions of the task at hand. Fixed-point theory, variational inequalities and affine-nonexpansive mappings are utilized to devise a scheme that accommodates affine constraints in a more versatile way than state-of-the-art primal-dual techniques and the alternating direction method of multipliers do. Recursions can be tuned to score low computational footprints, well-suited for large-scale optimization tasks, without compromising convergence guarantees. In contrast to its HSDM's precursors, FM-HSDM enjoys Fejér monotonicity, the step-size parameter stays constant across iterations to promote convergence speed-ups of the sequence of estimates to a minimizer, while only Lipschitzian continuity, and not strong monotonicity, of the derivative of the smooth-loss function is needed to ensure convergence. Results on the rate of convergence to an optimal point are also presented.

math.OC↗

arXiv2014

Compositions and Convex Combinations of Averaged Nonexpansive Operators

Properties of compositions and convex combinations of averaged nonexpansive operators are investigated and applied to the design of new fixed point algorithms in Hilbert spaces. An extended version of the forward-backward splitting algorithm for finding a zero of the sum of two monotone operators is obtained.

math.FA↗

arXiv2011

The adaptive projected subgradient method constrained by families of quasi-nonexpansive mappings and its application to online learning

Many online, i.e., time-adaptive, inverse problems in signal processing and machine learning fall under the wide umbrella of the asymptotic minimization of a sequence of non-negative, convex, and continuous functions. To incorporate a-priori knowledge into the design, the asymptotic minimization task is usually constrained on a fixed closed convex set, which is dictated by the available a-priori information. To increase versatility towards the usage of the available information, the present manuscript extends the Adaptive Projected Subgradient Method (APSM) by introducing an algorithmic scheme which incorporates a-priori knowledge in the design via a sequence of strongly attracting quasi-nonexpansive mappings in a real Hilbert space. In such a way, the benefits offered to online learning tasks by the proposed method unfold in two ways: 1) the rich class of quasi-nonexpansive mappings provides a plethora of ways to cast a-priori knowledge, and 2) by introducing a sequence of such mappings, the proposed scheme is able to capture the time-varying nature of a-priori information. The convergence properties of the algorithm are studied, several special cases of the method with wide applicability are shown, and the potential of the proposed scheme is demonstrated by considering an increasingly important, nowadays, online sparse system/signal recovery task.

math.OC↗