arXiv subjects
Publications and source records attributed to Gabriel Ponte.
We establish new properties of the NLP-Id upper bound for the maxi\-mum-entropy sampling problem (MESP). In particular, we give a detailed look at the concavity of its objective function as a function of the scaling parameter employed for NLP bounds for MESP. This leads to more relaxed choices for the scaling parameter for NLP-Id and even improved upper bounds for MESP.
The constrained generalized maximum-entropy sampling problem (CGMESP) is to select an order-s principal submatrix from an order-n covariance matrix, subject to some linear side constraints, so as to maximize the product of its t greatest eigenvalues, 0 < t <= s <n. GMESP refers to the version with no side constraints. Introduced more than 25 years ago, CGMESP is a natural generalization of two fundamental problems in statistical design theory: (i) constrained maximum-entropy sampling problem (CMESP); (ii) binary D-optimality (D-Opt). In the general case, it can be motivated by a selection problem in the context of principal component analysis (PCA). We present novel non-convex extended variable formulations for CGMESP. Using these formulations as points of departure, we present, first non-convex and then convex, continuous relaxations for CGMESP. We demonstrate many relations between different upper bounds for CGMESP, including upper bounds from the literature and our new upper bounds. We investigate the behavior of our relaxations related to the constraints linking the natural variables with the extended variables. We propose and investigate a generalized scaling technique for bound improvement. In the context of branch-and-bound, we determine the better of two natural branching techniques for fixing variables to zero. Finally, we present numerical experiments illustrating the value of our methods.
The maximum-entropy remote sampling problem (MERSP) is to select a subset of $s$ random variables from a set of $n$ random variables, so as to maximize the information concerning a set of target random variables that are not directly observable. We assume that the set of all of these random variables follows a joint Gaussian distribution, and that we have the covariance matrix available. Finally, we measure information using Shannon's differential entropy. The main approach for exact solution of moderate-sized instances of MERSP has been branch-and-bound (B\&B), and so previous work concentrated on upper bounds. Prior to our work, there were two upper-bounding methods for MERSP: the so-called ``complementary NLP bound'' and the ``spectral bound'', both introduced 25 years ago. We are able now to establish domination results between these two upper bounds. Further, we propose a novel and effective ``hyper-scaled NLP bound'' (hNLP bound) based on a subtle convex relaxation. The ``complementary'' version of hNLP bound for MERSP generalizes the previous complementary NLP bound for MERSP. We provide theoretical guarantees, giving sufficient conditions under which the complementary hNLP bound strictly dominates the complementary NLP bound. In addition, the hNLP formulation allows us to derive upper bounds for rank-deficient covariance matrices when they satisfy a technical condition. This is in contrast to the previous NLP bound that worked with only positive definite covariance matrices (because it was wedded to a complementary formulation). Additionally, we describe procedures for calculating hyper-scaling parameters. Finally, for B\&B, we provide a variable-fixing methodology and results guiding the best way to construct subproblems. Numerical experiments on benchmark instances demonstrate the effectiveness of our approaches in advancing the algorithmic state-of-the-art for MERSP.
We establish strong connections between two fundamental nonlinear 0/1 optimization problems coming from the area of experimental design, namely maximum-entropy sampling and 0/1 D-optimality. The connections are based on maps between instances, and we analyze the behavior of these maps. Using these maps, we transport basic upper-bounding methods between these two problems. Further, we establish results relating how different branch-and-bound schemes based on these maps compare. Additionally, we observe some surprising numerical results, where bounding methods that did not seem promising in their direct application to real-data MESP instances, are now useful for MESP instances that come from 0/1 D-optimality.
The 0/1 D-optimality problem and the Maximum-Entropy Sampling problem are two well-known NP-hard discrete maximization problems in experimental design. Algorithms for exact optimization (of moderate-sized instances) are based on branch-and-bound. The best upper-bounding methods are based on convex relaxation. We present ADMM (Alternating Direction Method of Multipliers) algorithms for solving these relaxations and experimentally demonstrate their practical value.
In optimal experimental design, the objective is to select a limited set of experiments that maximizes information about unknown model parameters based on factor levels. This work addresses the generalized D-optimal design problem, allowing for nonlinear relationships in factor levels. We develop scalable algorithms suitable for cases where the number of candidate experiments grows exponentially with the factor dimension, focusing on both first- and second-order models under design constraints. Particularly, our approach integrates convex relaxation with pricing-based local search techniques, which can provide upper bounds and performance guarantees. Unlike traditional local search methods, such as the ``Fedorov exchange" and its variants, our method effectively accommodates arbitrary side constraints in the design space. Furthermore, it yields both a feasible solution and an upper bound on the optimal value derived from the convex relaxation. Numerical results highlight the efficiency and scalability of our algorithms, demonstrating superior performance compared to the state-of-the-art commercial software, JMP
The generalized maximum-entropy sampling problem (GMESP) is to select an order-$s$ principal submatrix from an order-$n$ covariance matrix, to maximize the product of its $t$ greatest eigenvalues, $0<t\leq s <n$. Introduced more than 25 years ago, GMESP is a natural generalization of two fundamental problems in statistical design theory: (i) maximum-entropy sampling problem (MESP); (ii) binary D-optimality (D-Opt). In the general case, it can be motivated by a selection problem in the context of principal component analysis (PCA). We introduce the first convex-optimization based relaxation for GMESP, study its behavior, compare it to an earlier spectral bound, and demonstrate its use in a branch-and-bound scheme. We find that such an approach is practical when $s-t$ is very small.
We present several algorithms aimed at constructing sparse and structured sparse (row-sparse) generalized inverses, with application to the efficient computation of least-squares solutions, for inconsistent systems of linear equations, in the setting of multiple right-hand sides and a rank-deficient constraint matrix. Leveraging our earlier formulations to minimize the 1- and 2,1- norms of generalized inverses that satisfy important properties of the Moore-Penrose pseudoinverse, we develop efficient and scalable ADMM algorithms to address these norm-minimization problems and to limit the number of nonzero rows in the solution. We establish a 2,1-norm approximation result for a local-search procedure that was originally designed for 1-norm minimization, and we compare the ADMM algorithms with the local-search procedure and with general-purpose optimization solvers.
The well-known M-P (Moore-Penrose) pseudoinverse is used in several linear-algebra applications; for example, to compute least-squares solutions of inconsistent systems of linear equations. It is uniquely characterized by four properties, but not all of them need to be satisfied for some applications. For computational reasons, it is convenient then, to construct sparse block-structured matrices satisfying relevant properties of the M-P pseudoinverse for specific applications. (Vector) 1-norm minimization has been used to induce sparsity in this context. Aiming at row-sparse generalized inverses motivated by the least-squares application, we consider 2,1-norm minimization (and generalizations). In particular, we show that a 2,1-norm minimizing generalized inverse satisfies two additional M-P pseudoinverse properties, including the one needed for computing least-squares solutions. We present mathematical-optimization formulations related to finding row-sparse generalized inverses that can be solved very efficiently, and compare their solutions numerically to generalized inverses constructed by other methodologies, also aiming at sparsity and row-sparse structure.
We consider algorithmic approaches to the D-optimality problem for cases where the input design matrix is large and highly structured, in particular implicitly specified as a full quadratic or linear response-surface model in several levels of several factors. Using row generation techniques of mathematical optimization, in the context of discrete local-search and continuous relaxation aimed at branch-and-bound solution, we are able to design practical algorithms.
We develop a branch-and-bound algorithm for the integer D-optimality problem, a central problem in statistical design theory, based on two convex relaxations, employing variable-bound tightening and fast local-search procedures, testing our ideas on various test problems.
We apply a branch-and-bound (B\&B) algorithm to the D-optimality problem based on a convex mixed-integer nonlinear formulation. We discuss possible methodologies to accelerate the convergence of the B\&B algorithm, by combining the use of different upper bounds, variable-bound tightening inequalities, and local-search procedures. Different methodologies to compute the determinant of a matrix after a rank-one update are investigated to accelerate the local-searches. We discuss our findings through numerical experiments with randomly generated test problem.
The M-P (Moore-Penrose) pseudoinverse has as a key application the computation of least-squares solutions of inconsistent systems of linear equations. Irrespective of whether a given input matrix is sparse, its M-P pseudoinverse can be dense, potentially leading to high computational burden, especially when we are dealing with high-dimensional matrices. The M-P pseudoinverse is uniquely characterized by four properties, but only two of them need to be satisfied for the computation of least-squares solutions. Fampa and Lee (2018) and Xu, Fampa, Lee, and Ponte (2019) propose local-search procedures to construct sparse block-structured generalized inverses that satisfy the two key M-P properties, plus one more (the so-called reflexive property). That additional M-P property is equivalent to imposing a minimum-rank condition on the generalized inverse. (Vector) 1-norm minimization is used to induce sparsity and, importantly, to keep the magnitudes of entries under control for the generalized-inverses constructed. Here, we investigate the trade-off between low 1-norm and low rank for generalized inverses that can be used in the computation of least-squares solutions. We propose several algorithmic approaches that start from a $1$-norm minimizing generalized inverse that satisfies the two key M-P properties, and gradually decrease its rank, by iteratively imposing the reflexive property. The algorithms iterate until the generalized inverse has the least possible rank. During the iterations, we produce intermediate solutions, trading off low 1-norm (and typically high sparsity) against low rank.
The well-known M-P (Moore-Penrose) pseudoinverse is used in several linear-algebra applications; for example, to compute least-squares solutions of inconsistent systems of linear equations. Irrespective of whether a given matrix is sparse, its M-P pseudoinverse can be completely dense, potentially leading to high computational burden and numerical difficulties, especially when we are dealing with high-dimensional matrices. The M-P pseudoinverse is uniquely characterized by four properties, but not all of them need to be satisfied for some applications. In this context, Fampa and Lee (Oper. Res. Letters, 46:605--610, 2018) and Xu, Fampa, Lee and Ponte (SIAM J. on Optimization, to appear) propose local-search procedures to construct sparse block-structured generalized inverses that satisfy only some of the M-P properties. (Vector) 1-norm minimization is used to induce sparsity and to keep the magnitude of the entries under control, and theoretical results limit the distance between the 1-norm of the solution of the local searches and the minimum 1-norm of generalized inverses with corresponding properties. We have implemented several local-search procedures based on results presented in these two papers and make here an experimental analysis of them, considering their application to randomly generated matrices of varied dimensions, ranks, and densities. Further, we carried out a case study on a real-world data set.
Fundamental in matrix algebra and its applications, a \emph{generalized inverse} of a real matrix $A$ is a matrix $H$ that satisfies the Moore-Penrose (M-P) property $AHA=A$. If $H$ also satisfies the additional useful M-P property, $HAH=H$, it is called a \emph{reflexive generalized inverse}. Reflexivity is equivalent to minimum rank, so we are particularly interested in reflexive generalized inverses. We consider aspects of symmetry related to the calculation of a \emph{sparse} reflexive generalized inverse of $A$. As is common, and following Lee and Fampa (2018) for calculating sparse generalized inverses, we use (vector) 1-norm minimization for inducing sparsity and for keeping the magnitude of entries under control. When $A$ is symmetric, we may naturally desire a symmetric $H$; while generally such a restriction on $H$ may not lead to a 1-norm minimizing reflexive generalized inverse. We investigate a block construction method to produce a symmetric reflexive generalized inverse that is structured and has guaranteed sparsity. We provide a theoretically-efficient and practical local-search algorithm to block-construct an approximate 1-norm minimizing symmetric reflexive generalized inverse. Another aspect of symmetry that we consider relates to another M-P property: $H$ is \emph{ah-symmetric} if $AH$ is symmetric. The ah-symmetry property is the key one for solving least-squares problems using $H$. Here we do not assume that $A$ is symmetric, and we do not impose symmetry on $H$. We investigate a column block construction method to produce an ah-symmetric reflexive generalized inverse that is structured and has guaranteed sparsity. We provide a theoretically-efficient and practical local-search algorithm to column block construct an approximate 1-norm minimizing ah-symmetric reflexive generalized inverse.