arXiv subjects

Shangzhi Zeng

Publications and source records attributed to Shangzhi Zeng.

At least 19 recordsLinked to original sources

arXiv2026

Single-Loop Gradient Algorithms for Pessimistic Bilevel Optimization Problems

Bilevel optimization has recently attracted growing attention, particularly in the development of efficient numerical methods. Despite substantial progress on optimistic bilevel optimization, pessimistic bilevel optimization (PBO) remains much less explored, especially the design of fully first-order, single-loop gradient-based methods. To address this gap, we propose a smooth approximation of PBO through reformulation, penalization and regularization, and establish convergence guarantees in terms of both minimizers and stationarity. Building on this framework, we then develop two single-loop algorithms for deterministic and stochastic PBOs, respectively. Both use only first-order gradient information and avoid second-order derivatives and inner-loop subproblem solves. Non-asymptotic convergence rates for the proposed algorithms are established to provide theoretical guarantees. Through a systematic empirical study of both synthetic and practical problem instances, we demonstrate that our algorithms are highly effective and efficient. In particular, our results on spam classification and Smart Predict-then-Optimize further illustrate PBO's advantages over its classical optimistic bilevel counterpart, highlighting its strong potential for practical modeling and the delivery of robust solutions.

math.OC↗

arXiv2026

Limiting Stationarity of Regularized Gap-Function Reformulations for Bilevel Optimization with Unbounded Multipliers

Value-function-type reformulations have generated a broad class of methods for bilevel optimization. However, the corresponding value-function-type constraints are inherently degenerate and generally fail to satisfy standard constraint qualifications, so the associated multiplier sequences may be unbounded and bounded-multiplier convergence analyses become inapplicable. We study this issue for the regularized gap-function reformulation of bilevel problems with constrained convex lower-level programs. We prove that accumulation points of approximate stationary sequences are C-stationary for the corresponding Karush-Kuhn-Tucker-based mathematical program with complementarity constraints (MPCC), even when the multiplier sequence associated with the regularized gap-function constraint is unbounded. The result holds under Mangasarian-Fromovitz constraint qualification (MFCQ) for the upper- and lower-level constraint systems and MPCC-MFCQ at the limiting MPCC point, without any constraint qualification on the regularized gap-function constraint itself. We further provide an example showing that approximate stationary points of the standard regularized gap-function reformulation may converge to a point that is C-stationary but not M-stationary. To guarantee M-stationarity, we introduce a slack-based two-parameter penalty formulation preserving exact multiplier-slack complementarity and establish M-stationarity under a domination condition on the penalty parameters. We develop an inexact slack-penalty method with adaptive penalty updates and feasibility correction, whose accumulation points are M-stationary under the stated assumptions.

math.OC↗

arXiv2026

Extended SQP Methods in Nonsmooth Difference Programming Applied to Problems with Variational Inequality Constraints

This paper explores a new class of constrained difference programming problems, where the objective and constraints are formulated as differences of functions, without requiring their convexity. To investigate such problems, novel variants of the extended sequential quadratic method are introduced. These algorithms iteratively solve strongly convex quadratic subproblems constructed via linear approximations of the given data by using their gradients and subgradients. The convergence of the proposed methods is rigorously analyzed by employing, in particular, the Polyak-Łojasiewicz-Kurdyka property that ensures global convergence for various classes of functions in the problem formulation, e.g., semialgebraic ones. The original framework is further extended to address difference programming problems with variational inequality (VI) constraints. By reformulating VI constraints via regularized gap functions, such problems are naturally embedded into constrained difference programming that leads us to direct applications of the proposed algorithms. Numerical experiments for the class of continuous network design problems demonstrate the efficiency of the new methods.

math.OC↗

arXiv2026

A Single-Loop Penalty-based Algorithm for Stochastic Minimax Optimization with Nonlinear Coupled Constraints

We study stochastic nonconvex-concave minimax optimization with nonlinear coupled constraints that are convex in the maximization variable. To address the nonsmoothness arising from such constraints, we develop a penalty-based smooth approximation that combines quadratic penalization of the coupled constraints with quadratic regularization of the inner maximization problem. Based on this approximation, we propose SPACO, a single-loop stochastic gradient algorithm that tracks the inner maximizer by one stochastic ascent step, updates the outer variable using an inexact stochastic descent direction, and adaptively updates the penalty and regularization parameters over the iterations. For the penalty-based smooth approximation, we establish convergence guarantees from both minimizer and stationarity perspectives. In particular, we introduce enhanced KKT conditions and show that stationary points of the smooth approximations can converge to points satisfying these conditions. An example illustrates that the enhanced KKT conditions can help exclude KKT points that are not local minimizers. For SPACO, we prove non-asymptotic complexity bounds for stationarity and feasibility, as well as asymptotic subsequential convergence to enhanced KKT points. Numerical experiments on synthetic examples, fairness-aware classification, and constrained generative adversarial network training demonstrate the effectiveness of the proposed method.

math.OC↗

arXiv2026

A Single-Loop Bilevel Deep Learning Method for Optimal Control of Obstacle Problems

Optimal control of obstacle problems arises in a wide range of applications and is computationally challenging due to its nonsmoothness, nonlinearity, and bilevel structure. Classical numerical approaches rely on mesh-based discretization and typically require solving a sequence of costly subproblems. In this work, we propose a single-loop bilevel deep learning method, which is mesh-free, scalable to high-dimensional and complex domains, and avoids repeated solution of discretized subproblems. The method employs constraint-embedding neural networks to approximate the state and control and preserves the bilevel structure. To train the neural networks efficiently, we propose a Single-Loop Stochastic First-Order Bilevel Algorithm (S2-FOBA), which eliminates nested optimization and does not rely on restrictive lower-level uniqueness assumptions. We analyze the convergence behavior of S2-FOBA under mild assumptions. Numerical experiments on benchmark examples, including distributed and obstacle control problems with regular and irregular obstacles on complex domains, demonstrate that the proposed method achieves satisfactory accuracy while reducing computational cost compared to classical numerical methods.

math.OC↗

arXiv2026

Alternating Gradient-Type Algorithm for Bilevel Optimization with Inexact Lower-Level Solutions via Moreau Envelope-based Reformulation

In this paper, we study a class of bilevel optimization problems where the lower-level problem is a convex composite optimization model, which arises in various applications, including bilevel hyperparameter selection for regularized regression models. To solve these problems, we propose an Alternating Gradient-type algorithm with Inexact Lower-level Solutions (AGILS) based on a Moreau envelope-based reformulation of the bilevel optimization problem. The proposed algorithm does not require exact solutions of the lower-level problem at each iteration, improving computational efficiency. We prove the convergence of AGILS to stationary points and, under the Kurdyka-Łojasiewicz (KL) property, establish its sequential convergence. Numerical experiments, including a toy example and a bilevel hyperparameter selection problem for the sparse group Lasso model, demonstrate the effectiveness of the proposed AGILS.

math.OC↗

arXiv2026

A Single-Loop Gradient Algorithm for Pessimistic Bilevel Optimization via Smooth Approximation

Bilevel optimization has garnered significant attention in the machine learning community recently, particularly regarding the development of efficient numerical methods. While substantial progress has been made in developing efficient algorithms for optimistic bilevel optimization, the study of methods for solving Pessimistic Bilevel Optimization (PBO) remains relatively less explored, especially the design of fully first-order, single-loop gradient-based algorithms. This paper aims to bridge this research gap. We first propose a novel smooth approximation to the PBO problem, using penalization and regularization techniques. Building upon this approximation, we then propose SiPBA (Single-loop Pessimistic Bilevel Algorithm), a new gradient-based method specifically designed for PBO which avoids second-order derivative information or inner-loop iterations for subproblem solving. We provide theoretical validation for the proposed smooth approximation scheme and establish theoretical convergence for the algorithm SiPBA. Numerical experiments on synthetic examples and practical applications demonstrate the effectiveness and efficiency of SiPBA.

math.OC↗

arXiv2025

Augmenting Iterative Trajectory for Bilevel Optimization: Methodology, Analysis and Extensions

In recent years, there has been a surge of machine learning applications developed with hierarchical structure, which can be approached from Bi-Level Optimization (BLO) perspective. However, most existing gradient-based methods overlook the interdependence between hyper-gradient calculation and Lower-Level (LL) iterative trajectory, focusing solely on the former. Consequently, convergence theory is constructed with restrictive LL assumptions, which are often challenging to satisfy in real-world scenarios. In this work, we thoroughly analyze the constructed iterative trajectory, and highlight two deficiencies, including empirically chosen initialization and default use of entire trajectory for hyper-gradient calculation. To address these issues, we introduce two augmentation techniques including Initialization Auxiliary (IA) and Pessimistic Trajectory Truncation (PTT), and investigate various extension strategies such as prior regularization, different iterative mapping schemes and acceleration dynamics to construct Augmented Iterative Trajectory (AIT) for corresponding BLO scenarios (e.g., LL convexity and LL non-convexity). Theoretically, we provide convergence analysis for AIT and its variations under different LL assumptions, and establish the convergence analysis for BLOs with non-convex LL subproblem. Finally, we demonstrate the effectiveness of AIT through three numerical examples, typical learning and vision applications (e.g., data hyper-cleaning and few-shot learning) and more challenging tasks such as neural architecture search.

math.OC↗

arXiv2025

Prox-PINNs: A Deep Learning Algorithmic Framework for Elliptic Variational Inequalities

Elliptic variational inequalities (EVIs) present significant challenges in numerical computation due to their inherent non-smoothness, nonlinearity, and inequality formulations. Traditional mesh-based methods often struggle with complex geometries and high computational costs, while existing deep learning approaches lack generality for diverse EVIs. To alleviate these issues, this paper introduces Prox-PINNs, a novel deep learning algorithmic framework that integrates proximal operators with physics-informed neural networks (PINNs) to solve a broad class of EVIs. The Prox-PINNs reformulate EVIs as nonlinear equations using proximal operators and then approximate the solutions via neural networks that enforce boundary conditions as hard constraints. Then the neural networks are trained by minimizing physics-informed residuals. The Prox-PINNs framework advances the state-of-the-art by unifying the treatment of diverse EVIs within a mesh-free and scalable computational architecture. The framework is demonstrated on several prototypical applications, including obstacle problems, elasto-plastic torsion, Bingham visco-plastic flows, and simplified friction problems. Numerical experiments validate the method's accuracy, efficiency, robustness, and flexibility across benchmark examples.

math.OC↗

arXiv2024

Gradient-based algorithms for multi-objective bi-level optimization

Multi-Objective Bi-Level Optimization (MOBLO) addresses nested multi-objective optimization problems common in a range of applications. However, its multi-objective and hierarchical bilevel nature makes it notably complex. Gradient-based MOBLO algorithms have recently grown in popularity, as they effectively solve crucial machine learning problems like meta-learning, neural architecture search, and reinforcement learning. Unfortunately, these algorithms depend on solving a sequence of approximation subproblems with high accuracy, resulting in adverse time and memory complexity that lowers their numerical efficiency. To address this issue, we propose a gradient-based algorithm for MOBLO, called gMOBA, which has fewer hyperparameters to tune, making it both simple and efficient. Additionally, we demonstrate the theoretical validity by accomplishing the desirable Pareto stationarity. Numerical experiments confirm the practical efficiency of the proposed method and verify the theoretical results. To accelerate the convergence of gMOBA, we introduce a beneficial L2O neural network (called L2O-gMOBA) implemented as the initialization phase of our gMOBA algorithm. Comparative results of numerical experiments are presented to illustrate the performance of L2O-gMOBA.

math.OC↗

arXiv2024

Overcoming Lower-Level Constraints in Bilevel Optimization: A Novel Approach with Regularized Gap Functions

Constrained bilevel optimization tackles nested structures present in constrained learning tasks like constrained meta-learning, adversarial learning, and distributed bilevel optimization. However, existing bilevel optimization methods mostly are typically restricted to specific constraint settings, such as linear lower-level constraints. In this work, we overcome this limitation and develop a new single-loop, Hessian-free constrained bilevel algorithm capable of handling more general lower-level constraints. We achieve this by employing a doubly regularized gap function tailored to the constrained lower-level problem, transforming constrained bilevel optimization into an equivalent single-level optimization problem with a single smooth constraint. We rigorously establish the non-asymptotic convergence analysis of the proposed algorithm under the convexity of lower-level problem, avoiding the need for strong convexity assumptions on the lower-level objective or coupling convexity assumptions on lower-level constraints found in existing literature. Additionally, the generality of our method allows for its extension to bilevel optimization with minimax lower-level problem. We evaluate the effectiveness and efficiency of our algorithm on various synthetic problems, typical hyperparameter learning tasks, and generative adversarial network.

math.OC↗

arXiv2024

Moreau Envelope for Nonconvex Bi-Level Optimization: A Single-loop and Hessian-free Solution Strategy

This work focuses on addressing two major challenges in the context of large-scale nonconvex Bi-Level Optimization (BLO) problems, which are increasingly applied in machine learning due to their ability to model nested structures. These challenges involve ensuring computational efficiency and providing theoretical guarantees. While recent advances in scalable BLO algorithms have primarily relied on lower-level convexity simplification, our work specifically tackles large-scale BLO problems involving nonconvexity in both the upper and lower levels. We simultaneously address computational and theoretical challenges by introducing an innovative single-loop gradient-based algorithm, utilizing the Moreau envelope-based reformulation, and providing non-asymptotic convergence analysis for general nonconvex BLO problems. Notably, our algorithm relies solely on first-order gradient information, enhancing its practicality and efficiency, especially for large-scale BLO learning tasks. We validate our approach's effectiveness through experiments on various synthetic problems, two typical hyper-parameter learning tasks, and a real-world neural architecture search application, collectively demonstrating its superior performance.

math.OC↗

arXiv2024

A modularized algorithmic framework for interface related optimization problems using characteristic functions

In this paper, we consider the algorithms and convergence for a general optimization problem, which has a wide range of applications in image segmentation, topology optimization, flow network formulation, and surface reconstruction. In particular, the problem focuses on interface related optimization problems where the interface is implicitly described by characteristic functions of the corresponding domains. Under such representation and discretization, the problem is then formulated into a discretized optimization problem where the objective function is concave with respect to characteristic functions and convex with respect to state variables. We show that under such structure, the iterative scheme based on alternative minimization can converge to a local minimizer. Extensive numerical examples are performed to support the theory.

math.OC↗

arXiv2024

Optimality conditions for bilevel programs via Moreau envelope reformulation

For bilevel programs with a convex lower level program, the classical approach replaces the lower level program with its Karush-Kuhn-Tucker condition and solve the resulting mathematical program with complementarity constraint (MPCC). It is known that when the set of lower level multipliers is not unique, MPCC may not be equivalent to the original bilevel problem, and many MPCC-tailored constraint qualifications do not hold. In this paper, we study bilevel programs where the lower level is generalized convex. Applying the equivalent reformulation via Moreau envelope, we derive new directional optimality conditions. Even in the nondirectional case, the new optimality condition is stronger than the strong stationarity for the corresponding MPCC.

math.OC↗

arXiv2024

Constrained Bi-Level Optimization: Proximal Lagrangian Value function Approach and Hessian-free Algorithm

This paper presents a new approach and algorithm for solving a class of constrained Bi-Level Optimization (BLO) problems in which the lower-level problem involves constraints coupling both upper-level and lower-level variables. Such problems have recently gained significant attention due to their broad applicability in machine learning. However, conventional gradient-based methods unavoidably rely on computationally intensive calculations related to the Hessian matrix. To address this challenge, we begin by devising a smooth proximal Lagrangian value function to handle the constrained lower-level problem. Utilizing this construct, we introduce a single-level reformulation for constrained BLOs that transforms the original BLO problem into an equivalent optimization problem with smooth constraints. Enabled by this reformulation, we develop a Hessian-free gradient-based algorithm-termed proximal Lagrangian Value function-based Hessian-free Bi-level Algorithm (LV-HBA)-that is straightforward to implement in a single loop manner. Consequently, LV-HBA is especially well-suited for machine learning applications. Furthermore, we offer non-asymptotic convergence analysis for LV-HBA, eliminating the need for traditional strong convexity assumptions for the lower-level problem while also being capable of accommodating non-singleton scenarios. Empirical results substantiate the algorithm's superior practical performance.

cs.LG↗

arXiv2024

Moreau Envelope Based Difference-of-weakly-Convex Reformulation and Algorithm for Bilevel Programs

Bilevel programming has emerged as a valuable tool for hyperparameter selection, a central concern in machine learning. In a recent study by Ye et al. (2023), a value function-based difference of convex algorithm was introduced to address bilevel programs. This approach proves particularly powerful when dealing with scenarios where the lower-level problem exhibits convexity in both the upper-level and lower-level variables. Examples of such scenarios include support vector machines and $\ell_1$ and $\ell_2$ regularized regression. In this paper, we significantly expand the range of applications, now requiring convexity only in the lower-level variables of the lower-level program. We present an innovative single-level difference of weakly convex reformulation based on the Moreau envelope of the lower-level problem. We further develop a sequentially convergent Inexact Proximal Difference of Weakly Convex Algorithm (iP-DwCA). To evaluate the effectiveness of the proposed iP-DwCA, we conduct numerical experiments focused on tuning hyperparameters for kernel support vector machines on simulated data.

math.OC↗

arXiv2023

Hierarchical Optimization-Derived Learning

In recent years, by utilizing optimization techniques to formulate the propagation of deep model, a variety of so-called Optimization-Derived Learning (ODL) approaches have been proposed to address diverse learning and vision tasks. Although having achieved relatively satisfying practical performance, there still exist fundamental issues in existing ODL methods. In particular, current ODL methods tend to consider model construction and learning as two separate phases, and thus fail to formulate their underlying coupling and depending relationship. In this work, we first establish a new framework, named Hierarchical ODL (HODL), to simultaneously investigate the intrinsic behaviors of optimization-derived model construction and its corresponding learning process. Then we rigorously prove the joint convergence of these two sub-tasks, from the perspectives of both approximation quality and stationary analysis. To our best knowledge, this is the first theoretical guarantee for these two coupled ODL components: optimization and learning. We further demonstrate the flexibility of our framework by applying HODL to challenging learning tasks, which have not been properly addressed by existing ODL methods. Finally, we conduct extensive experiments on both synthetic data and real applications in vision and other learning tasks to verify the theoretical properties and practical performance of HODL in various application scenarios.

cs.LG↗

arXiv2023

Averaged Method of Multipliers for Bi-Level Optimization without Lower-Level Strong Convexity

Gradient methods have become mainstream techniques for Bi-Level Optimization (BLO) in learning fields. The validity of existing works heavily rely on either a restrictive Lower-Level Strong Convexity (LLSC) condition or on solving a series of approximation subproblems with high accuracy or both. In this work, by averaging the upper and lower level objectives, we propose a single loop Bi-level Averaged Method of Multipliers (sl-BAMM) for BLO that is simple yet efficient for large-scale BLO and gets rid of the limited LLSC restriction. We further provide non-asymptotic convergence analysis of sl-BAMM towards KKT stationary points, and the comparative advantage of our analysis lies in the absence of strong gradient boundedness assumption, which is always required by others. Thus our theory safely captures a wider variety of applications in deep learning, especially where the upper-level objective is quadratic w.r.t. the lower-level variable. Experimental results demonstrate the superiority of our method.

math.OC↗