arXiv subjects

Zheng Peng

Publications and source records attributed to Zheng Peng.

At least 19 recordsLinked to original sources

arXiv2025

Accelerated Tensor Completion via Trace-Regularized Fully-Connected Tensor Network

The fully-connected tensor network (FCTN) decomposition has gained prominence in the field of tensor completion owing to its powerful capacity to capture the low-rank characteristics of tensors. Nevertheless, the recovery of local details in the reconstructed tensor still leaves scope for enhancement. In this paper, we propose efficient tensor completion model that incorporates trace regularization within the FCTN decomposition framework. The trace regularization is constructed based on the mode-$k$ unfolding of the FCTN factors combined with periodically modified negative laplacian. The trace regularization promotes the smoothness of the FCTN factors through discrete second-order derivative penalties, thereby enhancing the continuity and local recovery performance of the reconstructed tensor. To solve the proposed model, we develop an efficient algorithm within the proximal alternating minimization (PAM) framework and theoretically prove its convergence. To reduce the runtime of the proposed algorithm, we design an intermediate tensor reuse mechanism that can decrease runtime by 10\%-30\% without affecting image recovery, with more significant improvements for larger-scale data. A comprehensive complexity analysis reveals that the mechanism attains a reduced computational complexity. Numerical experiments demonstrate that the proposed method outperforms existing approaches.

math.NA↗

arXiv2025

Accelerated Proximal Dogleg Majorization for Sparse Regularized Quadratic Optimization Problem

This paper addresses the problems of minimizing the sum of a quadratic function and a proximal-friendly nonconvex nonsmooth function. While the existing Proximal Dogleg Opportunistic Majorization (PDOM) algorithm for these problems offers computational efficiency by minimizing opportunistic majorization subproblems along mixed Newton directions and requiring only a single Hessian inversion, its convergence rate is limited due to the nonconvex nonsmooth regularization term, and its theoretical analysis is restricted to local convergence. To overcome these limitations, we firstly propose a novel algorithm named PDOM with extrapolation (PDOME). Its core innovations lie in two key aspects: (1) the integration of an extrapolation strategy into the construction of the hybrid Newton direction, and (2) the enhancement of the line search mechanism. Furthermore, we establish the global convergence of the entire sequence generated by PDOME to a critical point and derive its convergence rate under the Kurdyka-Lojasiewicz (KL) property. Numerical experiments demonstrate that PDOME achieves faster convergence and tends to converge to a better local optimum compared to the original PDOM.

math.OC↗

arXiv2025

Neural Network Training via Stochastic Alternating Minimization with Trainable Step Sizes

The training of deep neural networks is inherently a nonconvex optimization problem, yet standard approaches such as stochastic gradient descent (SGD) require simultaneous updates to all parameters, often leading to unstable convergence and high computational cost. To address these issues, we propose a novel method, Stochastic Alternating Minimization with Trainable Step Sizes (SAMT), which updates network parameters in an alternating manner by treating the weights of each layer as a block. By decomposing the overall optimization into sub-problems corresponding to different blocks, this block-wise alternating strategy reduces per-step computational overhead and enhances training stability in nonconvex settings. To fully leverage these benefits, inspired by meta-learning, we proposed a novel adaptive step size strategy to incorporate into the sub-problem solving steps of alternating updates. It supports different types of trainable step sizes, including but not limited to scalar, element-wise, row-wise, and column-wise, enabling adaptive step size selection tailored to each block via meta-learning. We further provide a theoretical convergence guarantee for the proposed algorithm, establishing its optimization soundness. Extensive experiments for multiple benchmarks demonstrate that SAMT achieves better generalization performance with fewer parameter updates compared to state-of-the-art methods, highlighting its effectiveness and potential in neural network optimization.

cs.LG↗

arXiv2025

Faster stochastic cubic regularized Newton methods with momentum

Cubic regularized Newton (CRN) methods have attracted signiffcant research interest because they offer stronger solution guarantees and lower iteration complexity. With the rise of the big-data era, there is growing interest in developing stochastic cubic regularized Newton (SCRN) methods that do not require exact gradient and Hessian evaluations. In this paper, we propose faster SCRN methods that incorporate gradient estimation with small, controlled errors and Hessian estimation with momentum-based variance reduction. These methods are particularly effective for problems where the gradient can be estimated accurately and at low cost, whereas accurate estimation of the Hessian is expensive. Under mild assumptions, we establish the iteration complexity of our SCRN methods by analyzing the descent of a novel potential sequence. Finally, numerical experiments show that our SCRN methods can achieve comparable performance to deterministic CRN methods and vastly outperform ffrst-order methods in terms of both iteration counts and solution quality.

math.OC↗

arXiv2025

A new insight on augmented Lagrangian method with applications in machine learning

By exploiting double-penalty terms for the primal subproblem, we develop a novel relaxed augmented Lagrangian method for solving a family of convex optimization problems subject to equality or inequality constraints. The method is then extended to solve a general multi-block separable convex optimization problem, and two related primal-dual hybrid gradient algorithms are also discussed. Convergence results about the sublinear and linear convergence rates are established by variational characterizations for both the saddle-point of the problem and the first-order optimality conditions of involved subproblems. A large number of experiments on testing the linear support vector machine problem and the robust principal component analysis problem arising from machine learning indicate that our proposed algorithms perform much better than several state-of-the-art algorithms.

math.NA↗

arXiv2025

Single-loop $\mathcal{O}(ε^{-3})$ stochastic smoothing algorithms for nonsmooth Riemannian optimization

In this paper, we develop two Riemannian stochastic smoothing algorithms for nonsmooth optimization problems on Riemannian manifolds, addressing distinct forms of the nonsmooth term $ h $. Both methods combine dynamic smoothing with a momentum-based variance reduction scheme in a fully online manner. When $ h $ is Lipschitz continuous, we propose an stochastic algorithm under adaptive parameter that achieves the optimal iteration complexity of $ \mathcal{O}(ε^{-3}) $, improving upon the best-known rates for exist algorithms. When $ h $ is the indicator function of a convex set, we design a new algorithm using truncated momentum, and under a mild error bound condition with parameter $ θ\geq 1 $, we establish a complexity of $ \tilde{\mathcal{O}}(ε^{-\max\{θ+2, 2θ\}}) $, in line with the best-known results in the Euclidean setting. Both algorithms feature a single-loop design with low per-iteration cost and require only $ \mathcal{O}(1) $ samples per iteration, ensuring that sample and iteration complexities coincide. Our framework encompasses a broad class of problems and recovers or matches optimal complexity guarantees in several important settings, including smooth stochastic Riemannian optimization, composite problems in Euclidean space, and constrained optimization via indicator functions.

math.OC↗

arXiv2025

Efficient QR-Based CP Decomposition Acceleration via Restructured Dimension Tree and Customized Extrapolation

The canonical polyadic (CP) decomposition is one of the most widely used tensor decomposition techniques. The conventional CP decomposition algorithm combines alternating least squares (ALS) with the normal equation. However, the normal equation is susceptible to numerical ill-conditioning, which can adversely affect the decomposition results. To mitigate this issue, ALS combined with QR decomposition has been proposed as a more numerically stable alternative. Although this method enhances stability, its iterative process involves tensor-times-matrix (TTM) operations, which typically result in higher computational costs. To reduce this cost, we propose restructured dimension tree, which increases the reuse of intermediate tensors and reduces the number of TTM operations. Compared with the standard dimension tree structure, this dimension tree structure can reduce the computational complexity of TTM operations for tensors of any order by 33\%. Additionally, we introduce a customized extrapolation strategy in the CP-ALS-QR algorithm, leveraging the unique structure of the matrix $\mathbf{Q}_0$ to further accelerate convergence. By integrating these two techniques, we propose a novel CP decomposition algorithm that significantly improves iteration efficiency, achieving up to twofold acceleration on datasets with certain specific structures. Numerical experiments on five real-world datasets show that, compared with the baseline algorithm, our proposed algorithm improves iteration efficiency while simultaneously enhancing fitting accuracy.

math.NA↗

arXiv2025

A Triple-Inertial Accelerated Alternating Optimization Method for Deep Learning Training

The stochastic gradient descent (SGD) algorithm has achieved remarkable success in training deep learning models. However, it has several limitations, including susceptibility to vanishing gradients, sensitivity to input data, and a lack of robust theoretical guarantees. In recent years, alternating minimization (AM) methods have emerged as a promising alternative for model training by employing gradient-free approaches to iteratively update model parameters. Despite their potential, these methods often exhibit slow convergence rates. To address this challenge, we propose a novel Triple-Inertial Accelerated Alternating Minimization (TIAM) framework for neural network training. The TIAM approach incorporates a triple-inertial acceleration strategy with a specialized approximation method, facilitating targeted acceleration of different terms in each sub-problem optimization. This integration improves the efficiency of convergence, achieving superior performance with fewer iterations. Additionally, we provide a convergence analysis of the TIAM algorithm, including its global convergence properties and convergence rate. Extensive experiments validate the effectiveness of the TIAM method, showing significant improvements in generalization capability and computational efficiency compared to existing approaches, particularly when applied to the rectified linear unit (ReLU) and its variants.

cs.LG↗

arXiv2024

The Rank-1 Completion Problem for Cubic Tensors

This paper studies the rank-$1$ tensor completion problem for cubic tensors. First of all, we show that this problem is equivalent to a special rank-$1$ matrix recovery problem. When the tensor is strongly rank-$1$ completable, we show that the problem is equivalent to a rank-$1$ matrix completion problem and it can be solved by an iterative formula. For other cases, we propose both nuclear norm relaxation and moment relaxation methods for solving the resulting rank-$1$ matrix recovery problem. The nuclear norm relaxation sometimes returns a rank-$1$ tensor completion, while sometimes it does not. When it fails, we apply the moment hierarchy of semidefinite programming relaxations to solve the rank-$1$ matrix recovery problem. The moment hierarchy can always get a rank-$1$ tensor completion, or detect its nonexistence. Numerical experiments are shown to demonstrate the efficiency of these proposed methods.

math.OC↗

arXiv2024

Inexact Riemannian Gradient Descent Method for Nonconvex Optimization

Gradient descent methods are fundamental first-order optimization algorithms in both Euclidean spaces and Riemannian manifolds. However, the exact gradient is not readily available in many scenarios. This paper proposes a novel inexact Riemannian gradient descent algorithm for nonconvex problems, accompanied by a convergence guarantee. In particular, we establish two inexact gradient conditions on Riemannian manifolds for the first time, enabling precise gradient approximations. Our method demonstrates strong convergence results for both gradient sequences and function values. The global convergence with constructive convergence rates for the sequence of iterates is ensured under the Riemannian Kurdyka-Łojasiewicz property. Furthermore, our algorithm encompasses two specific applications: Riemannian sharpness-aware minimization and Riemannian extragradient algorithm, both of which inherit the global convergence properties of the inexact gradient methods. Numerical experiments on low-rank matrix completion and principal component analysis problems validate the efficiency and practical relevance of the proposed approaches.

math.OC↗

arXiv2024

New vector transport operators extending a Riemannian CG algorithm to generalized Stiefel manifold with low-rank applications

This paper proposes two innovative vector transport operators, leveraging the Cayley transform, for the generalized Stiefel manifold embedded with a non-standard metric. Specifically, it introduces the differentiated retraction and an approximation of the Cayley transform to the differentiated matrix exponential. These vector transports are demonstrated to satisfy the Ring-Wirth non-expansive condition under non-standard metrics, and one of them is also isometric. Building upon the novel vector transport operators, we extend the modified Polak-Ribi$\grave{e}$re-Polyak (PRP) conjugate gradient method to the generalized Stiefel manifold. Under a non-monotone line search condition, we prove our algorithm globally converges to a stationary point. The efficiency of the proposed vector transport operators is empirically validated through numerical experiments involving generalized eigenvalue problems and canonical correlation analysis.

math.OC↗

arXiv2023

Riemannian Smoothing Gradient Type Algorithms]{Riemannian Smoothing Gradient Type Algorithms for Nonsmooth Optimization Problem on Compact Riemannian Submanifold Embedded in Euclidean Space

In this paper, we introduce the notion of generalized $ε$-stationarity for a class of nonconvex and nonsmooth composite minimization problems on compact Riemannian submanifold embedded in Euclidean space. To find a generalized $ε$-stationarity point, we develop a family of Riemannian gradient-type methods based on the Moreau envelope technique with a decreasing sequence of smoothing parameters, namely Riemannian smoothing gradient and Riemannian smoothing stochastic gradient methods. We prove that the Riemannian smoothing gradient method has the iteration complexity of $\mathcal{O}(ε^{-3})$ for driving a generalized $ε$-stationary point. To our knowledge, this is the best-known iteration complexity result for the nonconvex and nonsmooth composite problem on manifolds. For the Riemannian smoothing stochastic gradient method, one can achieve the iteration complexity of $\mathcal{O}(ε^{-5})$ for driving a generalized $ε$-stationary point. Numerical experiments are conducted to validate the superiority of our algorithms.

math.OC↗

arXiv2023

Accelerating Stochastic Recursive and Semi-stochastic Gradient Methods with Adaptive Barzilai-Borwein Step Sizes

The mini-batch versions of StochAstic Recursive grAdient algoritHm and Semi-Stochastic Gradient Descent method, employed the random Barzilai-Borwein step sizes (shorted as MB-SARAH-RBB and mS2GD-RBB), have surged into prominence through timely step size sequence. Inspired by modern adaptors and variance reduction techniques, we propose two new variant rules in the paper, referred to as RHBB and RHBB+, thereby leading to four algorithms MB-SARAH-RHBB, MB-SARAH-RHBB+, mS2GD-RHBB and mS2GD-RHBB+ respectively. RHBB+ is an enhanced version that additionally incorporates the importance sampling technique. They are aggressive in updates, robust in performance and self-adaptive along iterative periods. We analyze the flexible convergence structures and the corresponding complexity bounds in strongly convex cases. Comprehensive tuning guidance is theoretically provided for reference in practical implementations. Experiments show that the proposed methods consistently outperform the original and various state-of-the-art methods on frequently tested data sets. In particular, tests on the RHBB+ verify the efficacy of applying the importance sampling technique to the step size level. Numerous explorations display the promising scalability of our iterative adaptors.

math.OC↗

arXiv2023

Stochastic Conjugate Frameworks for Nonconvex and Nonsmooth Optimization

We introduce two new stochastic conjugate frameworks for a class of nonconvex and possibly also nonsmooth optimization problems. These frameworks are built upon Stochastic Recursive Gradient Algorithm (SARAH) and we thus refer to them as Acc-Prox-CG-SARAH and Acc-Prox-CG-SARAH-RS, respectively. They are efficiently accelerated, easy to implement, tune free and can be smoothly extended and modified. We devise a deterministic restart scheme for stochastic optimization and apply it in our second stochastic conjugate framework, which serves the key difference between the two approaches. In addition, we apply the ProbAbilistic Gradient Estimator (PAGE) and further develop a practical variant, denoted as Acc-Prox-CG-SARAH-ST, in order to reduce potential computational overhead. We provide comprehensive and rigorous convergence analysis for all three approaches and establish linear convergence rates for unconstrained minimization problem with nonconvex and nonsmooth objective functions. Experiments have demonstrated that Acc-Prox-CG-SARAH and Acc-Prox-CG-SARAH-RS both outperform state-of-art methods consistently and Acc-Prox-CG-SARAH-ST can as well achieve comparable convergence speed. In terms of theory and experiments, we verify the strong computational efficiency of the deterministic restart scheme in stochastic optimization methods.

math.OC↗

arXiv2022

Epitaxial growth of high quality $Mn_3Sn$ thin films by pulsed laser deposition

Non-collinear antiferromagnet Weyl semimetal $Mn_3Sn$ have attracted great research interest recently. Although large anomalous Hall effect, anomalous Nernst effect and magneto-optical effect have been observed in $Mn_3Sn$, most studies are based on single crystals. So far, it is still challenging to grow high quality epitaxial $Mn_3Sn$ thin films with transport and optical properties comparable to their single crystal counterparts. Here, we report the structure, magneto-optical and transport properties of epitaxial $Mn_3Sn$ thin films fabricated by pulsed laser deposition (PLD). Highly oriented $Mn_{3+x}Sn_{1-x}$ (0001) and (11$\bar2$0) epitaxial films are successfully growth on single crystalline $Al_2O_3$ and MgO substrates. Large anomalous Hall effect (AHE) up to $\left| ΔR_H\right|$=3.02 $μΩ\cdot cm$, and longitudinal magneto-optical Kerr effect (LMOKE) with $θ_K$ = 38.1 mdeg at 633 nm wavelength are measured at 300 K temperature, which are comparable to $Mn_3Sn$ single crystals. Our work demonstrates that high quality $Mn_3Sn$ epitaxial thin films can be fabricated by PLD, paving the way for future device applications.

cond-mat.mtrl-sci↗

arXiv2021

External pressure dependence of Granular Orifice Flow: transition to Beverloo flow

In this paper, we have designed and employed a suspended-wall silo to remove Janssen effect in order to explore directly the local pressure dependence of Granular Orifice Flow (GOF) systematically. We find that as Janssen effect is removed, the flow rate Q changes linearly with the external pressure. The slope α of the linear change decays exponentially with the ratio of the silo size and the size of the orifice Φ/D, which suggests the existence of a characteristic ratio λ (~2.4). When Φ/D > λ, α gradually decays to zero, and the effect of external pressure on the GOF becomes negligible, where the Beverloo law retrieves. Our results show that Janssen effect is not a determining factor of the constant rate of GOF, although it may contribute to shield the top load. The key parameter in GOF is Φ/D. In small Φ/D, the flow rate of GOF can be directly adjusted by the external pressure via our suspended-wall setup, which may be useful to the transportation of granules in microgravity environment where the gravity-driven Beverloo law is disabled.

cond-mat.soft↗

arXiv2021

Electrically Tunable Four-Wave-Mixing in Graphene Heterogeneous Fiber for Individual Gas Molecule Detection

Detection of individual molecules is the ultimate goal of any chemical sensor. In the case of gas detection, such resolution has been achieved in advanced nanoscale electronic solid-state sensors, but it has not been possible so far in integrated photonic devices, where the weak light-molecule interaction is typically hidden by noise. Here, we demonstrate a scheme to generate ultrasensitive down-conversion four-wave-mixing (FWM) in a graphene bipolar-junction-transistor heterogeneous D-shaped fiber. In the communication band, the FWM conversion efficiency can change steeply when the graphene Fermi level approaches 0.4 eV. In this condition, we exploit our unique two-step optoelectronic heterodyne detection scheme, and we achieve real-time individual gas molecule detection in vacuum. Such combination of graphene strong nonlinearities, electrical tunability, and all-fiber integration paves the way toward the design of versatile high-performance graphene photonic devices.

cond-mat.mes-hall↗

arXiv2020

An Inexact Manifold Augmented Lagrangian Method for Adaptive Sparse Canonical Correlation Analysis with Trace Lasso Regularization

Canonical correlation analysis (CCA for short) describes the relationship between two sets of variables by finding some linear combinations of these variables that maximizing the correlation coefficient. However, in high-dimensional settings where the number of variables exceeds sample size, or in the case of that the variables are highly correlated, the traditional CCA is no longer appropriate. In this paper, an adaptive sparse version of CCA (ASCCA for short) is proposed by using the trace Lasso regularization. The proposed ASCCA reduces the instability of the estimator when the covariates are highly correlated, and thus improves its interpretation. The ASCCA is further reformulated to an optimization problem on Riemannian manifolds, and an manifold inexact augmented Lagrangian method is then proposed for the resulting optimization problem. The performance of the ASCCA is compared with the other sparse CCA techniques in different simulation settings, which illustrates that the ASCCA is feasible and efficient.

math.OC↗