arXiv subjects

Cosmin G. Petra

Publications and source records attributed to Cosmin G. Petra.

At least 19 recordsLinked to original sources

arXiv2026

Porting the Nonlinear Optimization Library HiOp to Accelerator-Based Hardware Architectures

While interior point methods have been the centerpiece of nonlinear programming tools used in science and engineering, their reliance on linear solvers that can tackle sparse symmetric indefinite and highly ill-conditioned problems made it difficult to implement them effectively on hardware accelerators. At this time, there are few sparse linear solvers that can be used in this context. Here, we present a novel formulation of an interior point method implemented in our HiOp library, which is designed to be able to run entirely on hardware accelerators. This formulation avoids dependence on sparse solvers altogether, which is achieved by compressing the underlying sparse linear problem into a dense one of manageable size. We demonstrate feasibility of this approach and provide a baseline for future interior point method implementations on hardware accelerators. Our investigation is motivated by problems arising in optimal power flow analysis in power systems engineering and our approach is tailored to the broad class of problems arising in that important domain. We also demonstrate utility of modern programming models based on performance portability libraries, namely, Umpire and RAJA. We discuss trade-offs between performance, portability and development cost in the solution space for this non-linear optimization problem. As a result of this research, we demonstrate for the first time that interior point methods for sparse problems can be efficiently realized on modern computing systems where more than 90% of processing power is in GPUs.

cs.MS↗

arXiv2026

Practical Efficient Global Optimization is No-regret

Efficient global optimization (EGO) is one of the most widely used noise-free Bayesian optimization algorithms.It comprises the Gaussian process (GP) surrogate model and expected improvement (EI) acquisition function. In practice, when EGO is applied, a scalar matrix of a small positive value (also called a nugget or jitter) is usually added to the covariance matrix of the deterministic GP to improve numerical stability. We refer to this EGO with a positive nugget as the practical EGO. Despite its wide adoption and empirical success, to date, cumulative regret bounds for practical EGO have yet to be established. In this paper, we present for the first time the cumulative regret upper bound of practical EGO. In particular, we show that practical EGO has sublinear cumulative regret bounds and thus is a no-regret algorithm for commonly used kernels including the squared exponential (SE) and Matérn kernels ($ν>\frac{1}{2}$). Moreover, we analyze the effect of the nugget on the regret bound and discuss the theoretical implication on its choice. Numerical experiments are conducted to support and validate our findings.

stat.ML↗

arXiv2026

Convergence Rates of Constrained Expected Improvement

Constrained Bayesian optimization (CBO) methods have seen significant success in black-box optimization with constraints. One of the most commonly used CBO methods is the constrained expected improvement (CEI) algorithm. CEI is a natural extension of expected improvement (EI) when constraints are incorporated. However, the theoretical convergence rate of CEI has not been established. In this work, we study the convergence rate of CEI by analyzing its simple regret upper bound. First, we show that when the objective function $f$ and constraint function $c$ are assumed to each lie in a reproducing kernel Hilbert space (RKHS), CEI achieves the convergence rates of $\mathcal{O} \left(t^{-\frac{1}{2}}\log^{\frac{d+1}{2}}(t) \right) \ \text{and }\ \mathcal{O}\left(t^{\frac{-ν}{2ν+d}} \log^{\fracν{2ν+d}}(t)\right)$ for the commonly used squared exponential and Matérn kernels ($ν>\frac{1}{2}$), respectively. Second, we show that when $f$ is assumed to be sampled from Gaussian processes (GPs), CEI achieves similar convergence rates with a high probability. Numerical experiments are performed to validate the theoretical analysis.

stat.ML↗

arXiv2025

A Constrained Multi-Fidelity Bayesian Optimization Method

Recently, multi-fidelity Bayesian optimization (MFBO) has been successfully applied to many engineering design optimization problems, where the cost of high-fidelity simulations and experiments can be prohibitive. However, challenges remain for constrained optimization problems using the MFBO framework, particularly in efficiently identifying the feasible region defined by the constraints. In this paper, we propose a constrained multi-fidelity Bayesian optimization (CMFBO) method with novel acquisition functions. Specifically, we design efficient acquisition functions that 1) have analytically closed-form expressions; 2) are straightforward to implement; and 3) do not require feasible initial samples, an important feature often missing in commonly used acquisition functions such as expected constrained improvement (ECI). We demonstrate the effectiveness of our algorithms on synthetic test problems using different combinations of acquisition functions. Then, we apply the proposed method to a data-driven inertial confinement fusion (ICF) design problem, and a high-current joint design problem using finite element simulations with computational contact mechanics.

math.NA↗

arXiv2025

Bayesian Optimization with Expected Improvement: No Regret and the Choice of Incumbent

Expected improvement (EI) is one of the most widely used acquisition functions in Bayesian optimization (BO). Despite its proven empirical success in applications, the cumulative regret upper bound of EI remains an open question. In this paper, we analyze the classic noisy Gaussian process expected improvement (GP-EI) algorithm. We consider the Bayesian setting, where the objective is a sample from a GP. Three commonly used incumbents, namely the best posterior mean incumbent (BPMI), the best sampled posterior mean incumbent (BSPMI), and the best observation incumbent (BOI) are considered as the choices of the current best value in GP-EI. We present for the first time the cumulative regret upper bounds of GP-EI with BPMI and BSPMI. Importantly, we show that in both cases, GP-EI is a no-regret algorithm for both squared exponential (SE) and Matérn kernels. Further, we present for the first time that GP-EI with BOI either achieves a sublinear cumulative regret upper bound or has a fast converging noisy simple regret bound for SE and Matérn kernels. Our results provide theoretical guidance to the choice of incumbent when practitioners apply GP-EI in the noisy setting. Numerical experiments are conducted to validate our findings.

stat.ML↗

arXiv2025

On the convergence rate of noisy Bayesian Optimization with Expected Improvement

Expected improvement (EI) is one of the most widely used acquisition functions in Bayesian optimization (BO). Despite its proven success in applications for decades, important open questions remain on the theoretical convergence behaviors and rates for EI. In this paper, we contribute to the convergence theory of EI in three novel and critical areas. First, we consider objective functions that fit under the Gaussian process (GP) prior assumption, whereas existing works mostly focus on functions in the reproducing kernel Hilbert space (RKHS). Second, we establish for the first time the asymptotic error bound and its corresponding rate for GP-EI with noisy observations under the GP prior assumption. Third, by investigating the exploration and exploitation properties of the non-convex EI function, we establish improved error bounds of GP-EI for both the noise-free and noisy cases.

stat.ML↗

arXiv2024

On Improved Regret Bounds In Bayesian Optimization with Gaussian Noise

Bayesian optimization (BO) with Gaussian process (GP) surrogate models is a powerful black-box optimization method. Acquisition functions are a critical part of a BO algorithm as they determine how the new samples are selected. Some of the most widely used acquisition functions include upper confidence bound (UCB) and Thompson sampling (TS). The convergence analysis of BO algorithms has focused on the cumulative regret under both the Bayesian and frequentist settings for the objective. In this paper, we establish new pointwise bounds on the prediction error of GP under the frequentist setting with Gaussian noise. Consequently, we prove improved convergence rates of cumulative regret bound for both GP-UCB and GP-TS. Of note, the new prediction error bound under Gaussian noise can be applied to general BO algorithms and convergence analysis, e.g., the asymptotic convergence of expected improvement (EI) with noise.

cs.LG↗

arXiv2024

A Scalable Interior-Point Gauss-Newton Method for PDE-Constrained Optimization with Bound Constraints

We present a scalable approach to solve a class of elliptic partial differential equation (PDE)-constrained optimization problems with bound constraints. This approach utilizes a robust full-space interior-point (IP)-Gauss-Newton optimization method. To cope with the poorly-conditioned IP-Gauss-Newton saddle-point linear systems that need to be solved, once per optimization step, we propose two spectrally related preconditioners. These preconditioners leverage the limited informativeness of data in regularized PDE-constrained optimization problems. A block Gauss-Seidel preconditioner is proposed for the GMRES-based solution of the IP-Gauss-Newton linear systems. It is shown, for a large-class of PDE- and bound-constrained optimization problems, that the spectrum of the block Gauss-Seidel preconditioned IP-Gauss-Newton matrix is asymptotically independent of discretization and is not impacted by the ill-conditioning that notoriously plagues interior-point methods. We propose a regularization and log-barrier Hessian preconditioner for the preconditioned conjugate gradient (PCG)-based solution of the related IP-Gauss-Newton-Schur complement linear systems. The scalability of the approach is demonstrated on an example problem with bound and nonlinear elliptic PDE constraints. The numerical solution of the optimization problem is shown to require a discretization independent number of IP-Gauss-Newton linear solves. Furthermore, the linear systems are solved in a discretization and IP ill-conditioning independent number of preconditioned Krylov subspace iterations. The parallel scalability of preconditioner and linear system matrix applies, achieved with algebraic multigrid based solvers, and the aforementioned algorithmic scalability permits a parallel scalable means to compute solutions of a large class of PDE- and bound-constrained problems.

math.OC↗

arXiv2024

Design optimization in unilateral contact using pressure constraints and Bayesian optimization

Design optimization problems, e.g., shape optimization, that involve deformable bodies in unilateral contact are challenging as they require robust contact solvers, complex optimization methods that are typically gradient-based, and sensitivity derivations. Notably, the problems are nonsmooth, adding significant difficulty to the optimization process. We study design optimization problems in frictionless unilateral contact subject to pressure constraints, using both gradient-based and gradient-free optimization methods, namely Bayesian optimization. The contact simulation problem is solved via the mortar contact and finite element methods. For the gradient-based method, we use the direct differentiation method to compute the sensitivities of the cost and constraint function with respect to the design variables. Then, we use Ipopt to solve the optimization problems. For the gradient-free approach, we use a constrained Bayesian optimization algorithm based on the standard Gaussian Process surrogate model. We present numerical examples that control the contact pressure, inspired by real-life engineering applications, to demonstrate the effectiveness, strengths and shortcomings of both methods. Our results suggest that both optimization methods perform reasonably well for these nonsmooth problems.

math.NA↗

arXiv2023

Local convergence of a sequential quadratic programming method for a class of nonsmooth nonconvex objectives

A sequential quadratic programming (SQP) algorithm is designed for nonsmooth optimization problems with upper-C^2 objective functions. Upper-C^2 functions are locally equivalent to difference-of-convex (DC) functions with smooth convex parts. They arise naturally in many applications such as certain classes of solutions to parametric optimization problems, e.g., recourse of stochastic programming, and projection onto closed sets. The proposed algorithm conducts line search and adopts an exact penalty merit function. The potential inconsistency due to the linearization of constraints are addressed through relaxation, similar to that of Sl_1QP. We show that the algorithm is globally convergent under reasonable assumptions. Moreover, we study the local convergence behavior of the algorithm under additional assumptions of Kurdyka-Łojasiewicz (KL) properties, which have been applied to many nonsmooth optimization problems. Due to the nonconvex nature of the problems, a special potential function is used to analyze local convergence. We show that under acceptable assumptions, upper bounds on local convergence can be proven. Additionally, we show that for a large number of optimization problems with upper-C^2 objectives, their corresponding potential functions are indeed KL functions. Numerical experiment is performed with a power grid optimization problem that is consistent with the assumptions and analysis in this paper.

math.OC↗

arXiv2022

Compact representations of structured BFGS matrices

For general large-scale optimization problems compact representations exist in which recursive quasi-Newton update formulas are represented as compact matrix factorizations. For problems in which the objective function contains additional structure, so-called structured quasi-Newton methods exploit available second-derivative information and approximate unavailable second derivatives. This article develops the compact representations of two structured Broyden-Fletcher-Goldfarb-Shanno update formulas. The compact representations enable efficient limited memory and initialization strategies. Two limited memory line search algorithms are described and tested on a collection of problems, including a real world large scale imaging application.

math.OC↗

arXiv2022

Recent Developments in Security-Constrained AC Optimal Power Flow: Overview of Challenge 1 in the ARPA-E Grid Optimization Competition

The optimal power flow problem is central to many tasks in the design and operation of electric power grids. This problem seeks the minimum cost operating point for an electric power grid while satisfying both engineering requirements and physical laws describing how power flows through the electric network. By additionally considering the possibility of component failures and using an accurate AC power flow model of the electric network, the security-constrained AC optimal power flow (SC-AC-OPF) problem is of paramount practical relevance. To assess recent progress in solution algorithms for SC-AC-OPF problems and spur new innovations, the U.S. Department of Energy's Advanced Research Projects Agency--Energy (ARPA-E) organized Challenge 1 of the Grid Optimization (GO) competition. This paper describes the SC-AC-OPF problem formulation used in the competition, overviews historical developments and the state of the art in SC-AC-OPF algorithms, discusses the competition, and summarizes the algorithms used by the top three teams in Challenge 1 of the GO Competition (Teams gollnlp, GO-SNIP, and GMI-GO).

math.OC↗

arXiv2022

An Optimization algorithm for nonsmooth nonconvex problems with upper-C^2 objective

An optimization algorithm for nonsmooth nonconvex constrained optimization problems with upper-C2 objective functions is proposed and analyzed. Upper-C2 is a weakly concave property that exists in difference of convex (DC) functions and arises naturally in many applications, particularly certain classes of solutions to parametric optimization problems [34, 4], e.g., recourse of stochastic programming [36] and projection into closed sets [34]. The algorithm can be viewed as a bundle method specialized for upper-C2 problems and is globally convergent with bounded algorithm parameters. Compared to conventional bundle methods, the proposed method is both simpler and computationally more efficient. The algorithm handles general smooth constraints similarly to sequential quadratic programming (SQP) methods and uses a line search to ensure progress. The potential inconsistencies from the linearization of the constraints are addressed through a penalty method. The capabilities of the algorithm are demonstrated by solving both simple upper-C2 problems and real-world optimal power flow problems used in current power grid industry practices.

math.OC↗

arXiv2022

A simplified nonsmooth nonconvex bundle method with applications to security-constrained ACOPF problems

An optimization algorithm for a group of nonsmooth nonconvex problems inspired by two-stage stochastic programming problems is proposed. The main challenges for these problems include (1) the problems lack the popular lower-type properties such as prox-regularity assumed in many nonsmooth nonconvex optimization algorithms, (2) the objective can not be analytically expressed and (3) the evaluation of function values and subgradients are computationally expensive. To address these challenges, this report first examines the properties that exist in many two-stage problems, specifically upper-C^2 objectives. Then, we show that quadratic penalty method for security-constrained alternating current optimal power flow (SCACOPF) contingency problems can make the contingency solution functions upper-C^2 . Based on these observations, a simplified bundle algorithm that bears similarity to sequential quadratic programming (SQP) method is proposed. It is more efficient in implementation and computation compared to conventional bundle methods. Global convergence analysis of the algorithm is presented under novel and reasonable assumptions. The proposed algorithm therefore fills the gap of theoretical convergence for some smoothed SCACOPF problems. The inconsistency that might arise in our treatment of the constraints are addressed through a penalty algorithm whose convergence analysis is also provided. Finally, theoretical capabilities and numerical performance of the algorithm are demonstrated through numerical examples.

math.OC↗

arXiv2021

Another source of mesh dependence in topology optimization

The topology optimization community has regularly employed nonlinear programming (NLP) algorithms from the operations research community. However, these algorithms are implemented in the real vector space $\mathbb{R}^n$ instead of the proper function space where the design variable resides. In this article, we show how the volume fraction variable discretization on non-uniform meshes affects the convergence of $\mathbb{R}^n$ based NLP algorithms. We do so by first summarizing the functional analysis tools necessary to understand why convergence is affected by the mesh. Namely, the distinction between derivative definitions and the role of the mesh-dependent inner product within the NLP algorithm. These tools are then used to make the Globally Convergent Method of Moving Asymptotes (GCMMA), a popular NLP algorithm in the topology optimization community, converge in a mesh independent fashion when starting from the same initial design. We then benchmark our algorithms with three common problems in topology optimization.

math.OC↗

arXiv2021

A Hybrid Direct-Iterative Method for Solving KKT Linear Systems

We propose a solution strategy for linear systems arising in interior method optimization, which is suitable for implementation on hardware accelerators such as graphical processing units (GPUs). The current gold standard for solving these systems is the LDL^T factorization. However, LDL^T requires pivoting during factorization, which substantially increases communication cost and degrades performance on GPUs. Our novel approach solves a large indefinite system by solving multiple smaller positive definite systems, using an iterative solve for the Schur complement and an inner direct solve (via Cholesky factorization) within each iteration. Cholesky is stable without pivoting, thereby reducing communication and allowing reuse of the symbolic factorization. We demonstrate the practicality of our approach and show that on large systems it can efficiently utilize GPUs and outperform LDL^T factorization of the full system.

math.OC↗

arXiv2021

Solving realistic security-constrained optimal power flow problems

We present a decomposition approach for obtaining good feasible solutions for the security-constrained alternating-current optimal power flow (SCACOPF) problem at an industrial scale and under real-world time and computational limits. The approach aims at complementing the existing body of literature on bounding the problem via convex relaxations. It was designed for the participation in ARPA-E's Grid Optimization (GO) Competition Challenge 1. The challenge focused on a near-real-time version of the SCACOPF problem where a base case operating point is optimized taking into account possible single-element contingencies, after which the system adapts its operating point following the response of automatic frequency drop controllers and voltage regulators. Our solution approach for this problem relies on state-of-the-art nonlinear programming algorithms and employs nonconvex relaxations for complementarity constraints, a specialized two-stage decomposition technique with sparse approximations of recourse terms, and contingency ranking and pre-screening. The paper also outlines the salient features of our implementation, such as fast model functions and derivatives evaluation, warm-starting strategies, and asynchronous parallelism. We discuss the results of the independent benchmark of our approach done by ARPA-E's GO team in Challenge 1, which found that our methodology consistently produces high quality solutions across a wide range of network sizes and difficulty. Finally, we conclude by outlining potential extensions and improvements of our methodology.

math.OC↗

arXiv2021

Large-scale Optimization with Linear Equality Constraints using Reduced Compact Representation

For optimization problems with linear equality constraints, we prove that the (1,1) block of the inverse KKT matrix remains unchanged when projected onto the nullspace of the constraint matrix. We develop reduced compact representations of the limited-memory inverse BFGS Hessian to compute search directions efficiently when the constraint Jacobian is sparse. Orthogonal projections are implemented by a sparse QR factorization or a preconditioned LSQR iteration. In numerical experiments two proposed trust-region algorithms improve in computation times, often significantly, compared to previous implementations of related algorithms and compared to IPOPT.

math.OC↗