Searcharxiv⌕ Search

arXiv subjects

Deren Han

Publications and source records attributed to Deren Han.

At least 37 records · Page 2Linked to original sources
arXiv2025

Randomized iterative methods for generalized absolute value equations: Solvability and error bounds

Randomized iterative methods, such as the Kaczmarz method and its variants, have gained growing attention due to their simplicity and efficiency in solving large-scale linear systems. Meanwhile, absolute value equations (AVE) have attracted increasing interest due to their connection with the linear complementarity problem. In this paper, we investigate the application of randomized iterative methods to generalized AVE (GAVE). Our approach differs from most existing works in that we tackle GAVE with non-square coefficient matrices. We establish more comprehensive sufficient and necessary conditions for characterizing the solvability of GAVE and propose precise error bound conditions. Furthermore, we introduce a flexible and efficient randomized iterative algorithmic framework for solving GAVE, which employs randomized sketching matrices drawn from user-specified distributions. This framework is capable of encompassing many well-known methods, including the Picard iteration method and the randomized Kaczmarz method. Leveraging our findings on solvability and error bounds, we establish both almost sure convergence and linear convergence rates for this versatile algorithmic framework. Finally, we present numerical examples to illustrate the advantages of the new algorithms.

math.NA↗
arXiv2025

A Single-loop Proximal Subgradient Algorithm for A Class Structured Fractional Programs

In this paper, we investigate a class of nonconvex and nonsmooth fractional programming problems, where the numerator composed of two parts: a convex, nonsmooth function and a differentiable, nonconvex function, and the denominator consists of a convex, nonsmooth function composed of a linear operator. These structured fractional programming problems have broad applications, including CT reconstruction, sparse signal recovery, the single-period optimal portfolio selection problem and standard Sharpe ratio minimization problem. We develop a single-loop proximal subgradient algorithm that alleviates computational complexity by decoupling the evaluation of the linear operator from the nonsmooth component. We prove the global convergence of the proposed single-loop algorithm to an exact lifted stationary point under the Kurdyka-Łojasiewicz assumption. Additionally, we present a practical variant incorporating a nonmonotone line search to improve computational efficiency. Finally, through extensive numerical simulations, we showcase the superiority of the proposed approach over the existing state-of-the-art methods for three applications: $L_{1}/S_κ$ sparse signal recovery, limited-angle CT reconstruction, and optimal portfolio selection.

math.OC↗
arXiv2025

An Accelerated Alternating Partial Bregman Algorithm for ReLU-based Matrix Decomposition

Despite the remarkable success of low-rank estimation in data mining, its effectiveness diminishes when applied to data that inherently lacks low-rank structure. To address this limitation, in this paper, we focus on non-negative sparse matrices and aim to investigate the intrinsic low-rank characteristics of the rectified linear unit (ReLU) activation function. We first propose a novel nonlinear matrix decomposition framework incorporating a comprehensive regularization term designed to simultaneously promote useful structures in clustering and compression tasks, such as low-rankness, sparsity, and non-negativity in the resulting factors. This formulation presents significant computational challenges due to its multi-block structure, non-convexity, non-smoothness, and the absence of global gradient Lipschitz continuity. To address these challenges, we develop an accelerated alternating partial Bregman proximal gradient method (AAPB), whose distinctive feature lies in its capability to enable simultaneous updates of multiple variables. Under mild and theoretically justified assumptions, we establish both sublinear and global convergence properties of the proposed algorithm. Through careful selection of kernel generating distances tailored to various regularization terms, we derive corresponding closed-form solutions while maintaining the $L$-smooth adaptable property always holds for any $L\ge 1$. Numerical experiments, on graph regularized clustering and sparse NMF basis compression confirm the effectiveness of our model and algorithm.

cs.LG↗
arXiv2025

A generalization of the relaxation-based matrix splitting iterative method for solving the system of generalized absolute value equations

By incorporating a new matrix splitting and the momentum acceleration into the relaxed-based matrix splitting (RMS) method \cite{soso2023}, a generalization of the RMS (GRMS) iterative method for solving the generalized absolute value equations (GAVEs) is proposed. On the one hand, unlike some existing methods, by using the Cauchy's convergence principle we give some sufficient conditions for the existence and uniqueness of the solution to GAVEs and prove that our method can converge to the unique solution of GAVEs. On the other hand, we obtain a few new and weaker convergence conditions for some existing methods. Moreover, we establish comparison theorems between GRMS method and some existing methods. Preliminary numerical experiments show that the proposed method is efficient.

math.NA↗
arXiv2024

SOR-like iteration and FPI are consistent when they are equipped with certain optimal iterative parameters

Two common methods for solving absolute value equations (AVE) are SOR-like iteration method and fixed point iteration (FPI) method. In this paper, novel convergence analysis, which result wider convergence range, of the SOR-like iteration and the FPI are given. Based on the new analysis, a new optimal iterative parameter with a analytical form is obtained for the SOR-like iteration. In addition, an optimal iterative parameter with a analytical form is also obtained for FPI. Surprisingly, the SOR-like iteration and the FPI are the same whenever they are equipped with our optimal iterative parameters. As a by product, we give two new constructive proof for a well known sufficient condition such that AVE has a unique solution for any right hand side. Numerical results demonstrate our claims.

math.NA↗
arXiv2024

Fair Primal Dual Splitting Method for Image Inverse Problems

Image inverse problems have numerous applications, including image processing, super-resolution, and computer vision, which are important areas in image science. These application models can be seen as a three-function composite optimization problem solvable by a variety of primal dual-type methods. We propose a fair primal dual algorithmic framework that incorporates the smooth term not only into the primal subproblem but also into the dual subproblem. We unify the global convergence and establish the convergence rates of our proposed fair primal dual method. Experiments on image denoising and super-resolution reconstruction demonstrate the superiority of the proposed method over the current state-of-the-art.

cs.CV↗
arXiv2024

Noisy phase retrieval from subgaussian measurements

This paper aims to address the phase retrieval problem from subgaussian measurements with arbitrary noise, with a focus on devising robust and efficient algorithms for solving non-convex problems. To ensure uniqueness of solutions in the subgaussian setting, we explore two commonly used assumptions: either the subgaussian measurements satisfy a fourth-moment condition or the target signals exhibit non-peakiness. For each scenario, we introduce a novel spectral initialization method that yields robust initial estimates. Building on this, we employ leave-one-out arguments to show that the classical Wirtinger flow algorithm achieves a linear rate of convergence for both real-valued and complex-valued cases, provided the sampling complexity $m\ge O(n \log^3 m)$, where $n$ is the dimension of the underlying signals. In contrast to existing work, our algorithms are regularization-free, requiring no truncation, trimming, or additional penalty terms, and they permit the algorithm step sizes as large as $O(1)$, compared to the $O(1/n)$ in previous literature. Furthermore, our results accommodate arbitrary noise vectors that meet certain statistical conditions, covering a wide range of noise scenarios, with sub-exponential noise as a notable special case. The effectiveness of our algorithms is validated through various numerical experiments. We emphasize that our findings provide the first theoretical guarantees for recovering non-peaky signals using non-convex methods from Bernoulli measurements, which is of independent interest.

math.OC↗
arXiv2024

A distributed Douglas-Rachford splitting method for solving linear constrained multi-block weakly convex problems

In recent years, a distributed Douglas-Rachford splitting method (DDRSM) has been proposed to tackle multi-block separable convex optimization problems. This algorithm offers relatively easier subproblems and greater efficiency for large-scale problems compared to various augmented-Lagrangian-based parallel algorithms. Building upon this, we explore the extension of DDRSM to weakly convex cases. By assuming weak convexity of the objective function and introducing an error bound assumption, we demonstrate the linear convergence rate of DDRSM. Some promising numerical experiments involving compressed sensing and robust alignment of structures across images (RASL) show that DDRSM has advantages over augmented-Lagrangian-based algorithms, even in weakly convex scenarios.

math.OC↗
arXiv2024

On greedy multi-step inertial randomized Kaczmarz method for solving linear systems

The multi-step inertial randomized Kaczmarz (MIRK) method is an iterative method for solving large-scale linear systems. In this paper, we enhance the MIRK method by incorporating the greedy probability criterion, coupled with the introduction of a tighter threshold parameter for this criterion. We prove that the proposed greedy MIRK (GMIRK) method enjoys an improved deterministic linear convergence compared to both the MIRK method and the greedy randomized Kaczmarz method. Furthermore, we exhibit that the multi-step inertial extrapolation approach can be geometrically interpreted as an orthogonal projection method, and establish its relationship with the sketch-and-project method in (SIAM J. Matrix Anal. Appl. 36(4):1660-1690, 2015) and the oblique projection technique in (Results Appl. Math. 16:100342, 2022). Numerical experiments are provided to confirm our results.

math.NA↗
arXiv2024

A Momentum Accelerated Algorithm for ReLU-based Nonlinear Matrix Decomposition

Recently, there has been a growing interest in the exploration of Nonlinear Matrix Decomposition (NMD) due to its close ties with neural networks. NMD aims to find a low-rank matrix from a sparse nonnegative matrix with a per-element nonlinear function. A typical choice is the Rectified Linear Unit (ReLU) activation function. To address over-fitting in the existing ReLU-based NMD model (ReLU-NMD), we propose a Tikhonov regularized ReLU-NMD model, referred to as ReLU-NMD-T. Subsequently, we introduce a momentum accelerated algorithm for handling the ReLU-NMD-T model. A distinctive feature, setting our work apart from most existing studies, is the incorporation of both positive and negative momentum parameters in our algorithm. Our numerical experiments on real-world datasets show the effectiveness of the proposed model and algorithm. Moreover, the code is available at https://github.com/nothing2wang/NMD-TM.

cs.LG↗
arXiv2024

An accelerated preconditioned proximal gradient algorithm with a generalized Nesterov momentum for PET image reconstruction

This paper presents an Accelerated Preconditioned Proximal Gradient Algorithm (APPGA) for effectively solving a class of Positron Emission Tomography (PET) image reconstruction models with differentiable regularizers. We establish the convergence of APPGA with the Generalized Nesterov (GN) momentum scheme, demonstrating its ability to converge to a minimizer of the objective function with rates of $o(1/k^{2ω})$ and $o(1/k^ω)$ in terms of the function value and the distance between consecutive iterates, respectively, where $ω\in(0,1]$ is the power parameter of the GN momentum. To achieve an efficient algorithm with high-order convergence rate for the higher-order isotropic total variation (ITV) regularized PET image reconstruction model, we replace the ITV term by its smoothed version and subsequently apply APPGA to solve the smoothed model. Numerical results presented in this work indicate that as $ω\in(0,1]$ increase, APPGA converges at a progressively faster rate. Furthermore, APPGA exhibits superior performance compared to the preconditioned proximal gradient algorithm and the preconditioned Krasnoselskii-Mann algorithm. The extension of the GN momentum technique for solving a more complex optimization model with multiple nondifferentiable terms is also discussed.

math.OC↗
arXiv2024

Non-convex Pose Graph Optimization in SLAM via Proximal Linearized Riemannian ADMM

Pose graph optimization (PGO) is a well-known technique for solving the pose-based simultaneous localization and mapping (SLAM) problem. In this paper, we represent the rotation and translation by a unit quaternion and a three-dimensional vector, and propose a new PGO model based on the von Mises-Fisher distribution. The constraints derived from the unit quaternions are spherical manifolds, and the projection onto the constraints can be calculated by normalization. Then a proximal linearized Riemannian alternating direction method of multipliers (PieADMM) is developed to solve the proposed model, which not only has low memory requirements, but also can update the poses in parallel. Furthermore, we establish the iteration complexity of $O(1/ε^{2})$ of PieADMM for finding an $ε$-stationary solution of our model. The efficiency of our proposed algorithm is demonstrated by numerical experiments on two synthetic and four 3D SLAM benchmark datasets.

math.OC↗
arXiv2024

Novel clustered federated learning based on local loss

This paper proposes LCFL, a novel clustering metric for evaluating clients' data distributions in federated learning. LCFL aligns with federated learning requirements, accurately assessing client-to-client variations in data distribution. It offers advantages over existing clustered federated learning methods, addressing privacy concerns, improving applicability to non-convex models, and providing more accurate classification results. LCFL does not require prior knowledge of clients' data distributions. We provide a rigorous mathematical analysis, demonstrating the correctness and feasibility of our framework. Numerical experiments with neural network instances highlight the superior performance of LCFL over baselines on several clustered federated learning benchmarks.

cs.LG↗
arXiv2024

Extended alternating structure-adapted proximal gradient algorithm for nonconvex nonsmooth problems

Alternating structure-adapted proximal (ASAP) gradient algorithm (M. Nikolova and P. Tan, SIAM J Optim, 29:2053-2078, 2019) has drawn much attention due to its efficiency in solving nonconvex nonsmooth optimization problems. However, the multiblock nonseparable structure confines the performance of ASAP to far-reaching practical problems, e.g., coupled tensor decomposition. In this paper, we propose an extended ASAP (eASAP) algorithm for nonconvex nonsmooth optimization whose objective is the sum of two nonseperable functions and a coupling one. By exploiting the blockwise restricted prox-regularity, eASAP is capable of minimizing the objective whose coupling function is multiblock nonseparable. Moreover, we analyze the global convergence of eASAP by virtue of the Aubin property on partial subdifferential mapping and the Kurdyka-Łojasiewicz property on the objective. Furthermore, the sublinear convergence rate of eASAP is built upon the proximal point algorithmic framework under some mild conditions. Numerical simulations on multimodal data fusion demonstrate the compelling performance of the proposed method.

math.OC↗
arXiv2024

On adaptive stochastic heavy ball momentum for solving linear systems

The stochastic heavy ball momentum (SHBM) method has gained considerable popularity as a scalable approach for solving large-scale optimization problems. However, one limitation of this method is its reliance on prior knowledge of certain problem parameters, such as singular values of a matrix. In this paper, we propose an adaptive variant of the SHBM method for solving stochastic problems that are reformulated from linear systems using user-defined distributions. Our adaptive SHBM (ASHBM) method utilizes iterative information to update the parameters, addressing an open problem in the literature regarding the adaptive learning of momentum parameters. We prove that our method converges linearly in expectation, with a better convergence bound compared to the basic method. Notably, we demonstrate that the deterministic version of our ASHBM algorithm can be reformulated as a variant of the conjugate gradient (CG) method, inheriting many of its appealing properties, such as finite-time convergence. Consequently, the ASHBM method can be further generalized to develop a brand-new framework of the stochastic CG (SCG) method for solving linear systems. Our theoretical results are supported by numerical experiments.

math.OC↗
arXiv2024

Randomized Douglas-Rachford methods for linear systems: Improved accuracy and efficiency

The Douglas-Rachford (DR) method is a widely used method for finding a point in the intersection of two closed convex sets (feasibility problem). However, the method converges weakly and the associated rate of convergence is hard to analyze in general. In addition, the direct extension of the DR method for solving more-than-two-sets feasibility problems, called the $r$-sets-DR method, is not necessarily convergent. To improve the efficiency of the optimization algorithms, the introduction of randomization and the momentum technique has attracted increasing attention. In this paper, we propose the randomized $r$-sets-DR (RrDR) method for solving the feasibility problem derived from linear systems, showing the benefit of the randomization as it brings linear convergence in expectation to the otherwise divergent $r$-sets-DR method. Furthermore, the convergence rate does not depend on the dimension of the coefficient matrix. We also study RrDR with heavy ball momentum and establish its accelerated rate. Numerical experiments are provided to confirm our results and demonstrate the notable improvements in accuracy and efficiency of the DR method, brought by the randomization and the momentum technique.

math.OC↗
arXiv2024

A Bregman Proximal Stochastic Gradient Method with Extrapolation for Nonconvex Nonsmooth Problems

In this paper, we explore a specific optimization problem that involves the combination of a differentiable nonconvex function and a nondifferentiable function. The differentiable component lacks a global Lipschitz continuous gradient, posing challenges for optimization. To address this issue and accelerate the convergence, we propose a Bregman proximal stochastic gradient method with extrapolation (BPSGE), which only requires smooth adaptivity of the differentiable part. Under the variance reduction framework, we not only analyze the subsequential and global convergence of the proposed algorithm under certain conditions, but also analyze the sublinear convergence rate of the subsequence, and the complexity of the algorithm, revealing that the BPSGE algorithm requires at most O(epsilon\^\,(-2)) iterations in expectation to attain an epsilon-stationary point. To validate the effectiveness of our proposed algorithm, we conduct numerical experiments on three real-world applications: graph regularized nonnegative matrix factorization (NMF), matrix factorization with weakly-convex regularization, and NMF with nonconvex sparsity constraints. These experiments demonstrate that BPSGE is faster than the baselines without extrapolation.

math.OC↗