Searcharxiv⌕ Search

arXiv subjects

Yong Sheng Soh

Publications and source records attributed to Yong Sheng Soh.

At least 19 recordsLinked to original sources

Gradient Flows over Convex Bodies for Learning Optimal Regularizers

We consider the question of describing gradient flows over the collection of convex bodies. This is motivated by foundational questions arising in inverse problems. Suppose we wish to solve an ill-posed inverse problem using regularization techniques. Can we learn an optimal choice of a convex regularizer directly from prior data? Given a numerical algorithm that computes such an optimal regularizer, is there a principled manner for describing the evolution of its iterates? The objective of this paper is to lay the foundation towards answering these questions. To this end, we consider a criterion of an optimal regularizer based on seeking a convex body that minimizes the expected gauge evaluation with respect to a prior data distribution subject to a volume constraint. The formulation is initially nonconvex, but admits hidden convexity by passing to its polar. We characterize the Karush-Kuhn-Tucker (KKT) conditions that certify global optimality. We also analyze the $L^2$-gradient flow as well as a minimizing movement scheme with the Hausdorff penalty. Our results show that the objective gap decays exponentially along the gradient flow and we obtain dimension-dependent rates for both the objective gap and distance to the optimal body for the Hausdorff scheme. Finally, we apply the KKT conditions to identify explicit optimal convex bodies. For data where the optimal nonconvex regularizer is the $\ell_p$-quasinorm, $p \in (0,1)$, the optimal convex regularizer is the volume-normalized $\ell_1$-ball. For data described by paired basis atoms, the optimal convex regularizer is the polar of a shifted zonotope.

math.OC↗

The Star Geometry of Critic-Based Regularizer Learning

Variational regularization is a classical technique to solve statistical inference tasks and inverse problems, with modern data-driven approaches parameterizing regularizers via deep neural networks showcasing impressive empirical performance. Recent works along these lines learn task-dependent regularizers. This is done by integrating information about the measurements and ground-truth data in an unsupervised, critic-based loss function, where the regularizer attributes low values to likely data and high values to unlikely data. However, there is little theory about the structure of regularizers learned via this process and how it relates to the two data distributions. To make progress on this challenge, we initiate a study of optimizing critic-based loss functions to learn regularizers over a particular family of regularizers: gauges (or Minkowski functionals) of star-shaped bodies. This family contains regularizers that are commonly employed in practice and shares properties with regularizers parameterized by deep neural networks. We specifically investigate critic-based losses derived from variational representations of statistical distances between probability measures. By leveraging tools from star geometry and dual Brunn-Minkowski theory, we illustrate how these losses can be interpreted as dual mixed volumes that depend on the data distribution. This allows us to derive exact expressions for the optimal regularizer in certain cases. Finally, we identify which neural network architectures give rise to such star body gauges and when do such regularizers have favorable properties for optimization. More broadly, this work highlights how the tools of star geometry can aid in understanding the geometry of unsupervised regularizer learning.

cs.LG↗

Distributionally Robust Optimization over Incomparable Spaces

Distributionally robust optimization (DRO) provides a principled framework for making decisions under uncertainty when the underlying distribution may be incorrectly specified. A widely used approach builds on ideas from optimal transport (OT) as a means of measuring differences between distributions. However, this depends on these distributions residing in the same space. In certain settings, the reference data distribution we have and the space on which we make our decision may be different, which renders classical OT techniques impossible. In this paper, we provide a new DRO formulation in which ambiguity sets are defined through the Gromov-Wasserstein distance, which is an extension of OT to pairs of distributions residing in incomparable spaces. We show that several seemingly unrelated optimization formulations, including conditional value-at-risk minimization, group distributionally robust optimization, robust graph $k$-median, and $k$-median with outliers, can be recovered as special cases of our formulation. We describe the computational aspects of solving the proposed formulation based on the doubly-nonnegative relaxation. Last, we illustrate the framework through experiments on clustering over incomparable spaces and stochastic graph $k$-median.

math.OC↗

A Note on the NP-Hardness on Quadratic Optimization over Doubly Stochastic Matrices and the Gromov-Wasserstein Problem

The Gromov-Wasserstein problem is an extension of the regular optimal transport problem that is adapted for pairs of distributions residing in incomparable spaces. Broadly speaking, given a pair of metric measure spaces, the Gromov-Wasserstein problem seeks couplings between these spaces that minimize the distortion between the metric evaluations. In this note we record the fact that the Gromov-Wasserstein distance between finite metric spaces is NP-Hard to compute. Our argument relies on a reduction from the Hamiltonian Cycle Problem, and it builds on ideas in Vayer's thesis and Ryner et. al's work that draw connections between the 2-Gromov-Wasserstein and the Quadratic Assignment Problem. During the preparation of this manuscript we learned of the appearance of Breitung et. al's work, whose result essentially supersedes ours. We hope this note serves as a useful companion whose proof is, hopefully, more elementary and easier to understand.

math.OC↗

Quadratic Optimization over Probability Measures with Coupling Constraints

We consider solving an optimization instance in which the objective is quadratic and where the decision variable is a probability measure. Our class of problems are motivated by applications arising from optimal transport (with the Gromov-Wasserstein problem being a prominent example) as well as energy landscape minimization. Because the objective depends quadratically on the decision variable, our class of problems fall outside the standard modeling framework of the Generalized Moment Problems (which requires the objective to be linear). To this end, we propose a hierarchy of convex relaxations based on searching over probability measures over products of the base space. These have a natural interpretation with the moment Sum-of-squares hierarchy-a prominent framework for solving polynomial optimization instances, which we adapt to accommodate probability measures. A key conceptual contribution is to introduce a notion of positive-semidefiniteness that extends the usual notion over matrices. Under the assumption that the decision variables satisfy certain marginal constraints (as in the Kantorovich formulation of the optimal transport problem), we establish convergence of our hierarchy towards the globally optimal solution. Under the additional assumption that the objective is a polynomial, we propose a moment-SOS type hierarchy of finite dimensional semidefinite programs whose optimal solution converges to that of the original quadratic optimization over measures. We demonstrate our framework with numerical experiments. More generally, optimization over measures where the objective and/or constraint depends on the decision in a polynomial way is a fundamental problem. It is hoped that our work provides a road-map as to how the ideas of the SOS-ordinarily developed for polynomial optimization-may be applied to a broader class of non-linear problems involving measures.

math.OC↗

Sum-of-Squares Hierarchy for the Gromov Wasserstein Problem

The Gromov-Wasserstein (GW) problem is a variant of the classical optimal transport problem that allows one to compute meaningful transportation plans between incomparable spaces. At an intuitive level, it seeks plans that minimize the discrepancy between metric evaluations of pairs of points. The GW problem is typically cast as an instance of a non-convex quadratic program that is, unfortunately, intractable to solve. In this paper, we describe tractable semidefinite relaxations of the GW problem based on the Sum-of-Squares (SOS) hierarchy. We describe how the Putinar-type and the Schmüdgen-type moment hierarchies can be simplified using marginal constraints, and we prove convergence rates for these hierarchies towards computing global optimal solutions to the GW problem. The proposed SOS hierarchies naturally induce a distance measure analogous to the distortion metrics, and we show that these are genuine distances in that they satisfy the triangle inequality. In particular, the proposed SOS hierarchies provide computationally tractable proxies of the GW distance and the associated distortion distances (over metric measure spaces) that are otherwise intractable to compute.

math.OC↗

Dictionary Learning under Symmetries via Group Representations

The dictionary learning problem can be viewed as a data-driven process to learn a suitable transformation so that data is sparsely represented directly from example data. In this paper, we examine the problem of learning a dictionary that is invariant under a pre-specified group of transformations. Natural settings include Cryo-EM, multi-object tracking, synchronization, pose estimation, etc. We specifically study this problem under the lens of mathematical representation theory. Leveraging the power of non-abelian Fourier analysis for functions over compact groups, we prescribe an algorithmic recipe for learning dictionaries that obey such invariances. We relate the dictionary learning problem in the physical domain, which is naturally modelled as being infinite dimensional, with the associated computational problem, which is necessarily finite dimensional. We establish that the dictionary learning problem can be effectively understood as an optimization instance over certain matrix orbitopes having a particular block-diagonal structure governed by the irreducible representations of the group of symmetries. This perspective enables us to introduce a band-limiting procedure which obtains dimensionality reduction in applications. We provide guarantees for our computational ansatz to provide a desirable dictionary learning outcome. We apply our paradigm to investigate the dictionary learning problem for the groups SO(2) and SO(3). While the SO(2)-orbitope admits an exact spectrahedral description, substantially less is understood about the SO(3)-orbitope. We describe a tractable spectrahedral outer approximation of the SO(3)-orbitope, and contribute an alternating minimization paradigm to perform optimization in this setting. We provide numerical experiments to highlight the efficacy of our approach in learning SO(3)-invariant dictionaries, both on synthetic and on real world data.

math.OC↗

The Lovász Theta Function for Recovering Planted Clique Covers and Graph Colorings

The problems of computing graph colorings and clique covers are central challenges in combinatorial optimization. Both of these are known to be NP-hard, and thus computationally intractable in the worst-case instance. A prominent approach for computing approximate solutions to these problems is the celebrated Lovász theta function $\vartheta(G)$, which is specified as the solution of a semidefinite program (SDP), and hence tractable to compute. In this work, we move beyond the worst-case analysis and set out to understand whether the Lovász theta function recovers clique covers for random instances that have a latent clique cover structure, possibly obscured by noise. We answer this question in the affirmative and show that for graphs generated from the planted clique model we introduce in this work, the SDP formulation of $\vartheta(G)$ has a unique solution that reveals the underlying clique-cover structure with high-probability. The main technical step is an intermediate result where we prove a deterministic condition of recovery based on an appropriate notion of sparsity.

math.OC↗

Evaluating Policy Effects through Opinion Dynamics and Network Sampling

In the process of enacting or introducing a new policy, policymakers frequently consider the population's responses. These considerations are critical for effective governance. There are numerous methods to gauge the ground sentiment from a subset of the population; examples include surveys or listening to various feedback channels. Many conventional approaches implicitly assume that opinions are static; however, in reality, the population will discuss and debate these new policies among themselves, and reform new opinions in the process. In this paper, we pose the following questions: Can we quantify the effect of these social dynamics on the broader opinion towards a new policy? Given some information about the relationship network that underlies the population, how does overall opinion change post-discussion? We investigate three different settings in which the policy is revealed: respondents who do not know each other, groups of respondents who all know each other, and respondents chosen randomly. By controlling who the policy is revealed to, we control the degree of discussion among the population. We quantify how these factors affect the changes in policy beliefs via the Wasserstein distance between the empirically observed data post-discussion and its distribution pre-discussion. We also provide several numerical analyses based on generated network and real-life network datasets. Our work aims to address the challenges associated with network topology and social interactions, and provide policymakers with a quantitative lens to assess policy effectiveness in the face of resource constraints and network complexities.

cs.SI↗

Optimal Regularization Under Uncertainty: Distributional Robustness and Convexity Constraints

Regularization is a central tool for addressing ill-posedness in inverse problems and statistical estimation, with the choice of a suitable penalty often determining the reliability and interpretability of downstream solutions. While recent work has characterized optimal regularizers for well-specified data distributions, practical deployments are often complicated by distributional uncertainty and the need to enforce structural constraints such as convexity. In this paper, we introduce a framework for distributionally robust optimal regularization, which identifies regularizers that remain effective under perturbations of the data distribution. Our approach leverages convex duality to reformulate the underlying distributionally robust optimization problem, eliminating the inner maximization and yielding formulations that are amenable to numerical computation. We show how the resulting robust regularizers interpolate between memorization of the training distribution and uniform priors, providing insights into their behavior as robustness parameters vary. For example, we show how certain ambiguity sets, such as those based on the Wasserstein-1 distance, naturally induce regularity in the optimal regularizer by promoting regularizers with smaller Lipschitz constants. We further investigate the setting where regularizers are required to be convex, formulating a convex program for their computation and illustrating their stability with respect to distributional shifts. Taken together, our results provide both theoretical and computational foundations for designing regularizers that are reliable under model uncertainty and structurally constrained for robust deployment.

math.OC↗

Moment Sum-of-Squares Hierarchy for Gromov Wasserstein: Continuous Extensions and Sample Complexity

The Gromov-Wasserstein (GW) problem is an extension of the classical optimal transport problem to settings where the source and target distributions reside in incomparable spaces, and for which a cost function that attributes the price of moving resources is not available. The sum-of-squares (SOS) hierarchy is a principled method for deriving tractable semidefinite relaxations to generic polynomial optimization problems. In this work, we apply ideas from the moment-SOS hierarchy to solve the GW problem. More precisely, we identify extensions of the moment-SOS hierarchy, previously introduced for the discretized GW problem, such that they remain valid for general probability distributions. This process requires a suitable generalization of positive semidefiniteness over finite-dimensional vector spaces to the space of probability distributions. We prove the following properties concerning these continuous extensions: First, these relaxations form a genuine hierarchy in that the optimal value converges to the GW distance. Second, each of these relaxations induces a pseudo-metric over the collection of metric measure spaces. Crucially, unlike the GW problem, these induced instances are tractable to compute -- the discrete analogs are expressible as semidefinite programs and hence are tractable to solve. Separately from these properties, we also establish a statistical consistency result arising from sampling the source and target distributions. Our work suggests fascinating applications of the SOS hierarchy to optimization problems over probability distributions in settings where the objective and constraint depend on these distributions in a polynomial way.

math.OC↗

Semidefinite Relaxations of the Gromov-Wasserstein Distance

The Gromov-Wasserstein (GW) distance is an extension of the optimal transport problem that allows one to match objects between incomparable spaces. At its core, the GW distance is specified as the solution of a non-convex quadratic program and is not known to be tractable to solve. In particular, existing solvers for the GW distance are only able to find locally optimal solutions. In this work, we propose a semi-definite programming (SDP) relaxation of the GW distance. The relaxation can be viewed as the Lagrangian dual of the GW distance augmented with constraints that relate to the linear and quadratic terms of transportation plans. In particular, our relaxation provides a tractable (polynomial-time) algorithm to compute globally optimal transportation plans (in some instances) together with an accompanying proof of global optimality. Our numerical experiments suggest that the proposed relaxation is strong in that it frequently computes the globally optimal solution. Our Python implementation is available at https://github.com/tbng/gwsdp.

math.OC↗

Exactness Conditions for Semidefinite Relaxations of the Quadratic Assignment Problem

The Quadratic Assignment Problem (QAP) is an important discrete optimization instance that encompasses many well-known combinatorial optimization problems, and has applications in a wide range of areas such as logistics and computer vision. The QAP, unfortunately, is NP-hard to solve. To address this difficulty, a number of semidefinite relaxations of the QAP have been developed. These techniques are known to be powerful in that they compute globally optimal solutions in many instances, and are often deployed as sub-routines within enumerative procedures for solving QAPs. In this paper, we investigate the strength of these semidefinite relaxations. Our main result is a deterministic set of conditions on the input matrices -- specified via linear inequalities -- under which these semidefinite relaxations are exact. Our result is simple to state, in the hope that it serves as a foundation for subsequent studies on semidefinite relaxations of the QAP as well as other related combinatorial optimization problems. As a corollary, we show that the semidefinite relaxation is exact under a perturbation model whereby the input matrices differ by a suitably small perturbation term. One technical difficulty we encounter is that the set of dual feasible solutions is not closed. To circumvent these difficulties, our analysis constructs a sequence of dual feasible solutions whose objective value approaches the primal objective, but never attains it.

math.OC↗

Optimal Regularization for a Data Source

In optimization-based approaches to inverse problems and to statistical estimation, it is common to augment criteria that enforce data fidelity with a regularizer that promotes desired structural properties in the solution. The choice of a suitable regularizer is typically driven by a combination of prior domain information and computational considerations. Convex regularizers are attractive computationally but they are limited in the types of structure they can promote. On the other hand, nonconvex regularizers are more flexible in the forms of structure they can promote and they have showcased strong empirical performance in some applications, but they come with the computational challenge of solving the associated optimization problems. In this paper, we seek a systematic understanding of the power and the limitations of convex regularization by investigating the following questions: Given a distribution, what is the optimal regularizer for data drawn from the distribution? What properties of a data source govern whether the optimal regularizer is convex? We address these questions for the class of regularizers specified by functionals that are continuous, positively homogeneous, and positive away from the origin. We say that a regularizer is optimal for a data distribution if the Gibbs density with energy given by the regularizer maximizes the population likelihood (or equivalently, minimizes cross-entropy loss) over all regularizer-induced Gibbs densities. As the regularizers we consider are in one-to-one correspondence with star bodies, we leverage dual Brunn-Minkowski theory to show that a radial function derived from a data distribution is akin to a ``computational sufficient statistic'' as it is the key quantity for identifying optimal regularizers and for assessing the amenability of a data source to convex regularization.

math.OC↗

Multiplicative updates for symmetric-cone factorizations

Given a matrix $X\in \mathbb{R}^{m\times n}_+$ with non-negative entries, the cone factorization problem over a cone $\mathcal{K}\subseteq \mathbb{R}^k$ concerns computing $\{ a_1,\ldots, a_{m} \} \subseteq \mathcal{K}$ and $\{ b_1,\ldots, b_{n} \} \subseteq~\mathcal{K}^*$ belonging to its dual so that $X_{ij} = \langle a_i, b_j \rangle$ for all $i\in [m], j\in [n]$. Cone factorizations are fundamental to mathematical optimization as they allow us to express convex bodies as feasible regions of linear conic programs. In this paper, we introduce and analyze the symmetric-cone multiplicative update (SCMU) algorithm for computing cone factorizations when $\mathcal{K}$ is symmetric; i.e., it is self-dual and homogeneous. Symmetric cones are of central interest in mathematical optimization as they provide a common language for studying linear optimization over the nonnegative orthant (linear programs), over the second-order cone (second order cone programs), and over the cone of positive semidefinite matrices (semidefinite programs). The SCMU algorithm is multiplicative in the sense that the iterates are updated by applying a meticulously chosen automorphism of the cone computed using a generalization of the geometric mean to symmetric cones. Using an extension of Lieb's concavity theorem and von Neumann's trace inequality to symmetric cones, we show that the squared loss objective is non-decreasing along the trajectories of the SCMU algorithm. Specialized to the nonnegative orthant, the SCMU algorithm corresponds to the seminal algorithm by Lee and Seung for computing Nonnegative Matrix Factorizations.

math.OC↗

Learning Semidefinite Regularizers

Regularization techniques are widely employed in optimization-based approaches for solving ill-posed inverse problems in data analysis and scientific computing. These methods are based on augmenting the objective with a penalty function, which is specified based on prior domain-specific expertise to induce a desired structure in the solution. We consider the problem of learning suitable regularization functions from data in settings in which precise domain knowledge is not directly available. Previous work under the title of `dictionary learning' or `sparse coding' may be viewed as learning a regularization function that can be computed via linear programming. We describe generalizations of these methods to learn regularizers that can be computed and optimized via semidefinite programming. Our framework for learning such semidefinite regularizers is based on obtaining structured factorizations of data matrices, and our algorithmic approach for computing these factorizations combines recent techniques for rank minimization problems along with an operator analog of Sinkhorn scaling. Under suitable conditions on the input data, our algorithm provides a locally linearly convergent method for identifying the correct regularizer that promotes the type of structure contained in the data. Our analysis is based on the stability properties of Operator Sinkhorn scaling and their relation to geometric aspects of determinantal varieties (in particular tangent spaces with respect to these varieties). The regularizers obtained using our framework can be employed effectively in semidefinite programming relaxations for solving inverse problems.

math.OC↗

Group Invariant Dictionary Learning

The dictionary learning problem concerns the task of representing data as sparse linear sums drawn from a smaller collection of basic building blocks. In application domains where such techniques are deployed, we frequently encounter datasets where some form of symmetry or invariance is present. Motivated by this observation, we develop a framework for learning dictionaries for data under the constraint that the collection of basic building blocks remains invariant under such symmetries. Our procedure for learning such dictionaries relies on representing the symmetry as the action of a matrix group acting on the data, and subsequently introducing a convex penalty function so as to induce sparsity with respect to the collection of matrix group elements. Our framework specializes to the convolutional dictionary learning problem when we consider integer shifts. Using properties of positive semidefinite Hermitian Toeplitz matrices, we develop an extension that learns dictionaries that are invariant under continuous shifts. Our numerical experiments on synthetic data and ECG data show that the incorporation of such symmetries as priors are most valuable when the dataset has few data-points, or when the full range of symmetries is inadequately expressed in the dataset.

eess.SP↗

A Non-commutative Extension of Lee-Seung's Algorithm for Positive Semidefinite Factorizations

Given a matrix $X\in \mathbb{R}_+^{m\times n}$ with nonnegative entries, a Positive Semidefinite (PSD) factorization of $X$ is a collection of $r \times r$-dimensional PSD matrices $\{A_i\}$ and $\{B_j\}$ satisfying $X_{ij}= \mathrm{tr}(A_i B_j)$ for all $\ i\in [m],\ j\in [n]$. PSD factorizations are fundamentally linked to understanding the expressiveness of semidefinite programs as well as the power and limitations of quantum resources in information theory. The PSD factorization task generalizes the Non-negative Matrix Factorization (NMF) problem where we seek a collection of $r$-dimensional nonnegative vectors $\{a_i\}$ and $\{b_j\}$ satisfying $X_{ij}= a_i^\top b_j$, for all $i\in [m],\ j\in [n]$ -- one can recover the latter problem by choosing matrices in the PSD factorization to be diagonal. The most widely used algorithm for computing NMFs of a matrix is the Multiplicative Update algorithm developed by Lee and Seung, in which nonnegativity of the updates is preserved by scaling with positive diagonal matrices. In this paper, we describe a non-commutative extension of Lee-Seung's algorithm, which we call the Matrix Multiplicative Update (MMU) algorithm, for computing PSD factorizations. The MMU algorithm ensures that updates remain PSD by congruence scaling with the matrix geometric mean of appropriate PSD matrices, and it retains the simplicity of implementation that Lee-Seung's algorithm enjoys. Building on the Majorization-Minimization framework, we show that under our update scheme the squared loss objective is non-increasing and fixed points correspond to critical points. The analysis relies on Lieb's Concavity Theorem. Beyond PSD factorizations, we use the MMU algorithm as a primitive to calculate block-diagonal PSD factorizations and tensor PSD factorizations. We demonstrate the utility of our method with experiments on real and synthetic data.

math.OC↗