arXiv subjects

Warren Hare

Publications and source records attributed to Warren Hare.

At least 19 recordsLinked to original sources

arXiv2026

The cosine measure of a function at a point

The cosine measure of a set of vectors in $\mathbb{R}^n$ measures how well the set covers all directions in $\mathbb{R}^n$. It identifies the direction furthest, in angle, from the set. It is used in the convergence theory of various optimization algorithms, but also highlights interesting geometric properties of sets. For example, the cosine measure of a set $S$ is greater than zero if and only if given any $\mathcal{C}^1$ function $f$ at a point $\mathbf{x}$ where the gradient is nonzero, $S$ must contain a descent direction of $f$ at $\mathbf{x}$. In this paper, we examine the question of what can be said when the function $f$ is non-differentiable or if it has a gradient equal to the zero vector. To examine these cases, we introduce the novel concept of the {\em cosine measure of a function} at a point. This value provides an infimum on the value of the cosine measure that a set of vectors requires to guarantee it contains a descent direction of the function at the point of interest. We present mathematical theory around this concept, including examples showing that the cosine measure of a smooth function can have any value in $[-1,1]$. We further present algorithms to compute the cosine measure of a function, and examples demonstrating the algorithm on smooth and nonsmooth functions. These results also shed light on the the cosine measure of infinite sets and nonconvex cones.

math.OC↗

arXiv2026

Accuracy and Relationships of Quadratic Models in Derivative-free Optimization

We study three quadratic models in model-based derivative-free optimization: the minimum norm (MN), minimum Frobenius norm (MFN), and quadratic generalized simplex derivative (QS) models. Despite their widespread use, their approximation accuracy and relationships have not been systematically explored. We establish fully linear error bounds for all three models, removing the uniformly bounded model Hessian assumption required in existing MN analyses and deriving the first such results for the QS model. We further analyze Hessian approximation accuracy via directional error bounds, showing that all three models achieve fully quadratic accuracy along sample directions under a mild condition on the sample set. This reveals a form of directional fully quadratic accuracy not captured by existing theory. Finally, we characterize the relationships among these models, identifying conditions under which they coincide and clarifying their structural connections.

math.OC↗

arXiv2025

A unified vertical alignment and earthwork model in road design with a new convex optimization model for road networks

The vertical alignment optimization problem in road design seeks the optimal vertical alignment of a road at minimal cost, taking into account earthwork while meeting all safety and design requirements. In recent years, modelling techniques have been advanced to incorporate: side slopes, multiple material types, multiple hauling types, and road networks. However, the advancements were created disjointly with implementations that only made a single advancement to the basic model. Herein, we present a mixed-integer linear programming optimization model that unifies all previous advancements. The model further improves on previous work by maintaining convexity even in the multi-material setting. We compare our new model to previous models, validate it numerically, and demonstrate its capability in approximating material volumes. Our new model performs particularly well for determining the optimal vertical alignment for large road networks.

math.OC↗

arXiv2025

CLARSTA: A random subspace trust-region algorithm for convex-constrained derivative-free optimization

This paper proposes a random subspace trust-region algorithm for general convex-constrained derivative-free optimization (DFO) problems. Similar to previous random subspace DFO methods, the convergence of our algorithm requires a certain accuracy of models and a certain quality of subspaces. For model accuracy, we define a new class of models that is only required to provide reasonable accuracy on the projection of the constraint set onto the subspace. We provide a new geometry measure to make these models easy to analyze, construct, and manage. For subspace quality, we use a normalized random orthogonal projection matrix to provide a method to sample subspaces that preserve the first-order criticality measure by a certain fraction with a probability lower bound independent of the ambient dimension. Based on all these new theoretical results, we present an almost-sure liminf convergence result and an $\mathcal{O}(\epsilon^{-2})$ expected iteration complexity analysis of our algorithm. Numerical experiments on problems with dimensions up to 10000 demonstrate the reliable performance of our algorithm in high dimensions.

math.OC↗

arXiv2025

On the computation of the cosine measure in high dimensions

In derivative-free optimization, the cosine measure is a value that often arises in the convergence analysis of direct search methods. Given the increasing interest in high-dimensional derivative-free optimization problems, it is valuable to compute the cosine measure in this setting; however, it has recently been shown to be NP-hard. We propose a new formulation of the problem and heuristic to tackle this problem in higher dimensions and compare it with existing algorithms in the literature. In addition, new results are presented to facilitate the construction of sets with specific cosine measures, allowing for the creation of a test-set to benchmark the algorithms with.

math.OC↗

arXiv2025

Determining inscribability of polytopes via rank minimization based on slack matrices

A polytope is inscribable if there is a realization where all vertices lie on the sphere. In this paper, we provide a necessary and sufficient condition for a polytope to be inscribable. Based on this condition, we characterize the problem of determining inscribability as a minimum rank optimization problem using slack matrices. We propose an SDP approximation for the minimum rank optimization problem and prove that it is tight for certain classes of polytopes. Given a polytope, we provide three algorithms to determine its inscribability. All the optimization problems and algorithms we propose in this paper depend on the number of vertices and facets but are independent of the dimension of the polytope. Numerical results demonstrate our SDP approximation's efficiency, accuracy, and robustness for determining inscribability of simplicial polytopes of dimensions $4\le d\le 8$ with vertices $n\le 10$, revealing its potential in high dimensions.

math.CO↗

arXiv2024

A general framework for floating point error analysis of simplex derivatives

Gradient approximations are a class of numerical approximation techniques that are of central importance in numerical optimization. In derivative-free optimization, most of the gradient approximations, including the simplex gradient, centred simplex gradient, and adapted centred simplex gradient, are in the form of simplex derivatives. Owing to machine precision, the approximation accuracy of any numerical approximation technique is subject to the influence of floating point errors. In this paper, we provide a general framework for floating point error analysis of simplex derivatives. Our framework is independent of the choice of the simplex derivative as long as it satisfies a general form. We review the definition and approximation accuracy of the generalized simplex gradient and generalized centred simplex gradient. We define and analyze the accuracy of a generalized version of the adapted centred simplex gradient. As examples, we apply our framework to the generalized simplex gradient, generalized centred simplex gradient, and generalized adapted centred simplex gradient. Based on the results, we give suggestions on the minimal choice of approximate diameter of the sample set.

math.NA↗

arXiv2024

The cosine measure relative to a subspace

The cosine measure was introduced in 2003 to quantify the richness of a finite positive spanning sets of directions in the context of derivative-free directional methods. A positive spanning set is a set of vectors whose nonnegative linear combinations span the whole space. The present work extends the definition of cosine measure. In particular, the paper studies cosine measures relative to a subspace, and proposes a deterministic algorithm to compute it. The paper also studies the situation in which the set of vectors is infinite. The extended definition of the cosine measure might be useful for subspace decomposition methods.

math.OC↗

arXiv2023

$Q$-fully Quadratic Modeling and its Application in a Random Subspace Derivative-free Method

Model-based derivative-free optimization (DFO) methods are an important class of DFO methods that are known to struggle with solving high-dimensional optimization problems. Recent research has shown that incorporating random subspaces into model-based DFO methods has the potential to improve their performance on high-dimensional problems. However, most of the current theoretical and practical results are based on linear approximation models due to the complexity of quadratic approximation models. This paper proposes a random subspace trust-region algorithm based on quadratic approximations. Unlike most of its precursors, this algorithm does not require any special form of objective function. We study the geometry of sample sets, the error bounds for approximations, and the quality of subspaces. In particular, we provide a technique to construct $Q$-fully quadratic models, which is easy to analyze and implement. We present an almost-sure global convergence result of our algorithm and give an upper bound on the expected number of iterations to find a sufficiently small gradient. We also develop numerical experiments to compare the performance of our algorithm using both linear and quadratic approximation models. The numerical results demonstrate the strengths and weaknesses of using quadratic approximations.

math.OC↗

arXiv2023

Expected decrease for derivative-free algorithms using random subspaces

Derivative-free algorithms seek the minimum of a given function based only on function values queried at appropriate points. Although these methods are widely used in practice, their performance is known to worsen as the problem dimension increases. Recent advances in developing randomized derivative-free techniques have tackled this issue by working in low-dimensional subspaces that are drawn at random in an iterative fashion. The connection between the dimension of these random subspaces and the algorithmic guarantees has yet to be fully understood. In this paper, we develop an analysis for derivative-free algorithms (both direct-search and model-based approaches) employing random subspaces. Our results leverage linear local approximations of smooth functions to obtain understanding of the expected decrease achieved per function evaluation. Although the quantities of interest involve multidimensional integrals with no closed-form expression, a relative comparison for different subspace dimensions suggest that low dimension is preferable. Numerical computation of the quantities of interest confirm the benefit of operating in low-dimensional subspaces.

math.OC↗

arXiv2023

Using orthogonally structured positive bases for constructing positive $k$-spanning sets with cosine measure guarantees

Positive spanning sets span a given vector space by nonnegative linear combinations of their elements. These have attracted significant attention in recent years, owing to their extensive use in derivative-free optimization. In this setting, the quality of a positive spanning set is assessed through its cosine measure, a geometric quantity that expresses how well such a set covers the space of interest. In this paper, we investigate the construction of positive $k$-spanning sets with geometrical guarantees. Our results build on recently identified positive spanning sets, called orthogonally structured positive bases. We first describe how to identify such sets and compute their cosine measures efficiently. We then focus our study on positive $k$-spanning sets, for which we provide a complete description, as well as a new notion of cosine measure that accounts for the resilient nature of such sets. By combining our results, we are able to use orthogonally structured positive bases to create positive $k$-spanning sets with guarantees on the value of their cosine measures.

math.NA↗

arXiv2022

Detecting negative eigenvalues of exact and approximate Hessian matrices in optimization

Nonconvex minimization algorithms often benefit from the use of second-order information as represented by the Hessian matrix. When the Hessian at a critical point possesses negative eigenvalues, the corresponding eigenvectors can be used to search for further improvement in the objective function value. Computing such eigenpairs can be computationally challenging, particularly if the Hessian matrix itself cannot be built directly but must rather be sampled or approximated. In blackbox optimization, such derivative approximations are built at a significant cost in terms of function values. In this paper, we investigate practical approaches to detect negative eigenvalues in Hessian matrices without access to the full matrix. We propose a general framework that begins with the diagonal and gradually builds submatrices to detect negative curvature. Crucially,our approach works both when exact Hessian coordinate values are available and when Hessian coordinate values are approximated. We compare several instances of our framework on a test set of Hessian matrices from a popular optimization library, and finite-differences approximations thereof. Our experiments highlight the importance of the variable order in the problem description, and show that forming submatrices is often an efficient approach to detect negative curvature.

math.OC↗

arXiv2021

Nicely structured positive bases with maximal cosine measure

The properties of positive bases make them a useful tool in derivative-free optimization (DFO) and an interesting concept in mathematics. The notion of the \emph{cosine measure} helps to quantify the quality of a positive basis. It provides information on how well the vectors in the positive basis uniformly cover the space considered. The number of vectors in a positive basis is known to be between $n+1$ and $2n$ inclusively. When the number of vectors is strictly between $n+1$ and $2n$, we say that it is an intermediate positive basis. In this paper, the structure of intermediate positive bases with maximal cosine measure is investigated. The structure of an intermediate positive basis with maximal cosine measure over a certain subset of positive bases is provided. This type of positive bases has a simple structure that makes them easy to generate with a computer software.

math.OC↗

arXiv2021

Error Analysis of Surrogate Models Constructed through Operations on Sub-models

Model-based methods are popular in derivative-free optimization (DFO). In most of them, a single model function is built to approximate the objective function. This is generally based on the assumption that the objective function is one blackbox. However, some real-life and theoretical problems show that the objective function may consist of several blackboxes. In those problems, the information provided by each blackbox may not be equal. In this situation, one could build multiple sub-models that are then combined to become a final model. In this paper, we analyze the relation between the accuracy of those sub-models and the model constructed through their operations. We develop a broad framework that can be used as a theoretical tool in model error analysis and future research in DFO algorithms design.

math.OC↗

arXiv2021

Limiting behaviour of the generalized simplex gradient as the number of points tends to infinity on a fixed shape in R^n

This work investigates the asymptotic behaviour of the gradient approximation method called the generalized simplex gradient (GSG). This method has an error bound that at first glance seems to tend to infinity as the number of sample points increases, but with some careful construction, we show that this is not the case. For functions in finite dimensions, we present two new error bounds ad infinitum depending on the position of the reference point. The error bounds are not a function of the number of sample points and thus remain finite.

math.NA↗

arXiv2020

Hessian approximations

This work introduces the nested-set Hessian approximation, a second-order approximation method that can be used in any derivative-free optimization routine that requires such information. It is built on the foundation of the generalized simplex gradient and proved to have an error bound that is on the order of the maximal radius of the two sets used in its construction. We show that when the points used in the computation of the nested-set Hessian have a favourable structure, (n+1)(n+2)/2 function evaluations are sufficient to approximate the Hessian. However, the nested-set Hessian also allows for evaluation sets with more points without negating the error analysis. Two calculus-based approximation techniques of the Hessian are developed and some advantages of the same are demonstrated.

math.OC↗

arXiv2020

Error bounds for overdetermined and underdetermined generalized centred simplex gradients

Using the Moore--Penrose pseudoinverse, this work generalizes the gradient approximation technique called centred simplex gradient to allow sample sets containing any number of points. This approximation technique is called the \emph{generalized centred simplex gradient}. We develop error bounds and, under a full-rank condition, show that the error bounds have order $O(\Delta^2)$, where $\Delta$ is the radius of the sample set of points used. We establish calculus rules for generalized centred simplex gradients, introduce a calculus-based generalized centred simplex gradient and confirm that error bounds for this new approach are also order $O(\Delta^2)$. We provide several examples to illustrate the results and some benefits of these new methods.

math.NA↗

arXiv2020

A deterministic algorithm to compute the cosine measure of a finite positive spanning set

Originally developed in 1954, positive bases and positive spanning sets have been found to be a valuable concept in derivative-free optimization (DFO). The quality of a positive basis (or positive spanning set) can be quantified via the {\em cosine measure} and convergence properties of certain DFO algorithms are intimately linked to the value of this measure. However, it is unclear how to compute the cosine measure for a positive basis from the definition. In this paper, a deterministic algorithm to compute the cosine measure of any positive basis or finite positive spanning set is provided. The algorithm is proven to return the exact value of the cosine measure in finite time.

math.OC↗