arXiv subjects

Uijeong Jang

Publications and source records attributed to Uijeong Jang.

7 recordsLinked to original sources

arXiv2026

Higher-Order Derivatives Do Not Accelerate the Computation of Fixed Points

The Picard iteration converges to the unique fixed point of a $q$-contractive operator at a linear rate $q^N$, and a lower bound with an affine construction shows that no deterministic method querying only operator values can do better. But what about higher-order methods that query derivatives? A single Jacobian evaluation reveals an affine map entirely, so the affine construction says nothing about higher-order methods. In this work, we show that finite-order derivative information still does not accelerate the worst-case complexity for smooth contractive fixed-point computation. This contrasts with higher-order smooth minimization, where higher-order derivatives do improve worst-case rates for convex and non-convex minimization.

math.OC↗

arXiv2026

Optimal Acceleration for Proximal Minimization of the Sum of Convex and Strongly Convex Functions

When minimizing the sum of a convex and a strongly convex function, or when finding the zero of the sum of a monotone operator and a strongly monotone operator, Chambolle and Pock (2010) and Davis and Yin (2015) proposed accelerated mechanisms that achieve an $\mathcal{O}(1/N^2)$ convergence rate for the squared distance to the solution, but the optimality of this rate was not established. In this work, we present Fast Douglas--Rachford Splitting (FDR), an accelerated method that improves the constants established in the prior works, and provide a complexity lower bound establishing that both the $\mathcal{O}(1/N^2)$ convergence rate and the leading-order constant of FDR's rate are optimal.

math.OC↗

arXiv2026

ALiA: Adaptive Linearized ADMM

We propose ALiA, a novel adaptive variant of the alternating direction method of multipliers (ADMM). Specifically, ALiA is a variant of function-linearized proximal ADMM (FLiP ADMM), which generalizes the classical ADMM by leveraging the differentiable structure of the objective function, making it highly versatile. Notably, ALiA features an adaptive stepsize selection scheme that eliminates the need for backtracking linesearch. Motivated by recent advances in adaptive gradient and proximal methods, we establish point convergence of ALiA for convex and differentiable objectives. Furthermore, by introducing negligible computational overhead, we develop an alternative stepsize selection scheme for ALiA that improves the convergence speed both theoretically and empirically. Extensive numerical experiments on practical datasets confirm the accelerated performance of ALiA compared to standard FLiP ADMM. Additionally, we demonstrate that ALiA either outperforms or matches the practical performance of existing adaptive methods across problem classes where it is applicable.

math.OC↗

arXiv2026

Sharpness-Aware Minimization Can Hallucinate Minimizers

Sharpness-Aware Minimization (SAM) is widely used to seek flatter minima -- often linked to better generalization. In its standard implementation, SAM updates the current iterate using the loss gradient evaluated at a point perturbed by distance $ρ$ along the normalized gradient direction. We show that, for some choices of $ρ$, SAM can stall at points where this shifted (perturbed-point) gradient vanishes despite a nonzero original gradient, and therefore, they are not stationary points of the original loss. We call these points hallucinated minimizers, prove their existence under simple nonconvex landscape conditions (e.g., the presence of a local minimizer and a local maximizer), and establish sufficient conditions for local convergence of the SAM iterates to them. We corroborate this failure mode in neural network training and observe that it aligns with SAM's performance degradation often seen at large $ρ$. Finally, as a practical safeguard, we find that a short initial SGD warm-start before enabling SAM mitigates this failure mode and reduces sensitivity to the choice of $ρ$.

cs.LG↗

arXiv2026

Point Convergence of Nesterov's Accelerated Gradient Method: An AI-Assisted Proof

The Nesterov accelerated gradient method, introduced in 1983, has been a cornerstone of optimization theory and practice. Yet the question of its point convergence had remained open. In this work, we resolve this longstanding open problem in the affirmative. The discovery of the proof was heavily assisted by ChatGPT, a proprietary large language model, and we describe the process through which its assistance was elicited.

math.OC↗

arXiv2025

Computer-Assisted Design of Accelerated Composite Optimization Methods: OptISTA

The accelerated composite optimization method FISTA (Beck, Teboulle 2009) is suboptimal by a constant factor, and we present a new method OptISTA that improves FISTA by a constant factor of 2. The performance estimation problem (PEP) has recently been introduced as a new computer-assisted paradigm for designing optimal first-order methods. In this work, we present a double-function stepsize-optimization PEP methodology that poses the optimization over fixed-step first-order methods for composite optimization as a finite-dimensional nonconvex QCQP, which can be practically solved through spatial branch-and-bound algorithms, and use it to design the exact optimal method OptISTA for the composite optimization setup. We then establish the exact optimality of OptISTA under the large-scale assumption with a lower-bound construction that extends the semi-interpolated zero-chain construction (Drori, Taylor 2022) to the double-function setup of composite optimization. By establishing exact optimality, our work concludes the search for the fastest first-order methods, with respect to the performance measure of worst-case function value suboptimality, for the proximal, projected-gradient, and proximal-gradient setups involving a smooth convex function and a closed proper convex function.

math.OC↗

arXiv2024

LoRA Training in the NTK Regime has No Spurious Local Minima

Low-rank adaptation (LoRA) has become the standard approach for parameter-efficient fine-tuning of large language models (LLM), but our theoretical understanding of LoRA has been limited. In this work, we theoretically analyze LoRA fine-tuning in the neural tangent kernel (NTK) regime with $N$ data points, showing: (i) full fine-tuning (without LoRA) admits a low-rank solution of rank $r\lesssim \sqrt{N}$; (ii) using LoRA with rank $r\gtrsim \sqrt{N}$ eliminates spurious local minima, allowing gradient descent to find the low-rank solutions; (iii) the low-rank solution found using LoRA generalizes well.

cs.LG↗