arXiv subjects

Shuyang Ling

Publications and source records attributed to Shuyang Ling.

At least 19 recordsLinked to original sources

arXiv2026

Improved Global Landscape Guarantees for Low-rank Factorization in Synchronization

The orthogonal group synchronization problem, which aims to recover a set of $d \times d$ orthogonal matrices from their pairwise noisy products, plays a fundamental role in signal processing, computer vision, and network analysis. In recent years, numerous optimization techniques, such as semidefinite relaxation (SDR) and low-rank (Burer-Monteiro) factorization, have been proposed to address this problem and their theoretical guarantees have been extensively studied. While SDR is provably powerful and exact in recovering the least-squares estimator under certain mild conditions, it is not scalable. In contrast, the low-rank factorization is highly efficient but nonconvex, meaning its iterates may get trapped in local minima. To close the gap, we analyze the low-rank approach and focus on understanding when the associated nonconvex optimization landscape is benign, i.e., free of spurious local minima. Recent works suggest that the benignness depends on the condition number of the Hessian at the global minimizer, but it remains unclear whether sharp guarantees can be achieved. In this work, we consider the low-rank approach which corresponds to an optimization problem over the Stiefel manifold ${\rm St}(p,d)^{\otimes n}$. By formulating the landscape analysis into another convex optimization problem, we provide a unified characterization of the optimization landscape for all parameter pairs $(p,d)$ with $p \geq d+2$ for $d\geq 1$ and $p = d+1$ for $1\leq d\leq 3$ which gives a much improved dependence on the condition number of the Hessian. Our results recover the known sharp state-of-the-art bound for $d=1$ which is extremely useful for characterizing the Kuramoto synchronization, and significantly improved the guarantees for the general case $d \geq 2$ with $p \geq d+2$ over the existing results. The theoretical results are versatile and applicable to a wide range of examples.

math.OC↗

arXiv2024

Beyond Unconstrained Features: Neural Collapse for Shallow Neural Networks with General Data

Neural collapse (NC) is a phenomenon that emerges at the terminal phase of the training (TPT) of deep neural networks (DNNs). The features of the data in the same class collapse to their respective sample means and the sample means exhibit a simplex equiangular tight frame (ETF). In the past few years, there has been a surge of works that focus on explaining why the NC occurs and how it affects generalization. Since the DNNs are notoriously difficult to analyze, most works mainly focus on the unconstrained feature model (UFM). While the UFM explains the NC to some extent, it fails to provide a complete picture of how the network architecture and the dataset affect NC. In this work, we focus on shallow ReLU neural networks and try to understand how the width, depth, data dimension, and statistical property of the training dataset influence the neural collapse. We provide a complete characterization of when the NC occurs for two or three-layer neural networks. For two-layer ReLU neural networks, a sufficient condition on when the global minimizer of the regularized empirical risk function exhibits the NC configuration depends on the data dimension, sample size, and the signal-to-noise ratio in the data instead of the network width. For three-layer neural networks, we show that the NC occurs as long as the first layer is sufficiently wide. Regarding the connection between NC and generalization, we show the generalization heavily depends on the SNR (signal-to-noise ratio) in the data: even if the NC occurs, the generalization can still be bad provided that the SNR in the data is too low. Our results significantly extend the state-of-the-art theoretical analysis of the N C under the UFM by characterizing the emergence of the N C under shallow nonlinear networks and showing how it depends on data properties and network architecture.

stat.ML↗

arXiv2024

On the Exactness of SDP Relaxation for Quadratic Assignment Problem

Quadratic assignment problem (QAP) is a fundamental problem in combinatorial optimization and finds numerous applications in operation research, computer vision, and pattern recognition. However, it is a very well-known NP-hard problem to find the global minimizer to the QAP. In this work, we study the semidefinite relaxation (SDR) of the QAP and investigate when the SDR recovers the global minimizer. In particular, we consider the two input matrices satisfy a simple signal-plus-noise model, and show that when the noise is sufficiently smaller than the signal, then the SDR is exact, i.e., it recovers the global minimizer to the QAP. It is worth noting that this sufficient condition is purely algebraic and does not depend on any statistical assumption of the input data. We apply our bound to several statistical models such as correlated Gaussian Wigner model. Despite the sub-optimality in theory under those models, empirical studies show the remarkable performance of the SDR. Our work could be the first step towards a deeper understanding of the SDR exactness for the QAP.

math.OC↗

arXiv2024

Uncertainty Quantification of Spectral Estimator and MLE for Orthogonal Group Synchronization

Orthogonal group synchronization aims to recover orthogonal group elements from their noisy pairwise measurements. It has found numerous applications including computer vision, imaging science, and community detection. Due to the orthogonal constraints, it is often challenging to find the least squares estimator in presence of noise. In the recent years, semidefinite relaxation (SDR) and spectral methods have proven to be powerful tools in recovering the group elements. In particular, under additive Gaussian noise, the SDR exactly produces the maximum likelihood estimator (MLE), and both MLE and spectral methods are able to achieve near-optimal statistical error. In this work, we take one step further to quantify the uncertainty of the MLE and spectral estimators by considering their distributions. By leveraging the orthogonality constraints in the likelihood function, we obtain a second-order expansion of the MLE and spectral estimator with the leading terms as an anti-symmetric Gaussian random matrix that is on the tangent space of the orthogonal matrix. This also implies state-of-the-art min-max risk bounds and a confidence region of each group element as a by-product. Our works provide a general theoretical framework that is potentially useful to find an approximate distribution of the estimators arising from many statistical inference problems with manifold constraints. The numerical experiments confirm our theoretical contribution.

math.ST↗

arXiv2024

Cross Entropy versus Label Smoothing: A Neural Collapse Perspective

Label smoothing loss is a widely adopted technique to mitigate overfitting in deep neural networks. This paper studies label smoothing from the perspective of Neural Collapse (NC), a powerful empirical and theoretical framework which characterizes model behavior during the terminal phase of training. We first show empirically that models trained with label smoothing converge faster to neural collapse solutions and attain a stronger level of neural collapse. Additionally, we show that at the same level of NC1, models under label smoothing loss exhibit intensified NC2. These findings provide valuable insights into the performance benefits and enhanced model calibration under label smoothing loss. We then leverage the unconstrained feature model to derive closed-form solutions for the global minimizers for both loss functions and further demonstrate that models under label smoothing have a lower conditioning number and, therefore, theoretically converge faster. Our study, combining empirical evidence and theoretical results, not only provides nuanced insights into the differences between label smoothing and cross-entropy losses, but also serves as an example of how the powerful neural collapse framework can be used to improve our understanding of DNNs.

cs.LG↗

arXiv2023

Local Geometry Determines Global Landscape in Low-rank Factorization for Synchronization

The orthogonal group synchronization problem, which focuses on recovering orthogonal group elements from their corrupted pairwise measurements, encompasses examples such as high-dimensional Kuramoto model on general signed networks, $\mathbb{Z}_2$-synchronization, community detection under stochastic block models, and orthogonal Procrustes problem. The semidefinite relaxation (SDR) has proven its power in solving this problem; however, its expensive computational costs impede its widespread practical applications. We consider the Burer-Monteiro factorization approach to the orthogonal group synchronization, an effective and scalable low-rank factorization to solve large scale SDPs. Despite the significant empirical successes of this factorization approach, it is still a challenging task to understand when the nonconvex optimization landscape is benign, i.e., the optimization landscape possesses only one local minimizer, which is also global. In this work, we demonstrate that if the degree of freedom within the factorization exceeds twice the condition number of the ``Laplacian" (certificate matrix) at the global minimizer, the optimization landscape is absent of spurious local minima. Our main theorem is purely algebraic and versatile, and it seamlessly applies to all the aforementioned examples: the nonconvex landscape remains benign under almost identical condition that enables the success of the SDR. Additionally, we illustrate that the Burer-Monteiro factorization is robust to ``monotone adversaries", mirroring the resilience of the SDR. In other words, introducing ``favorable" adversaries into the data will not result in the emergence of new spurious local minimizers.

cs.IT↗

arXiv2023

Neural Collapse for Unconstrained Feature Model under Cross-entropy Loss with Imbalanced Data

Recent years have witnessed the huge success of deep neural networks (DNNs) in various tasks of computer vision and text processing. Interestingly, these DNNs with massive number of parameters share similar structural properties on their feature representation and last-layer classifier at terminal phase of training (TPT). Specifically, if the training data are balanced (each class shares the same number of samples), it is observed that the feature vectors of samples from the same class converge to their corresponding in-class mean features and their pairwise angles are the same. This fascinating phenomenon is known as Neural Collapse (N C), first termed by Papyan, Han, and Donoho in 2019. Many recent works manage to theoretically explain this phenomenon by adopting so-called unconstrained feature model (UFM). In this paper, we study the extension of N C phenomenon to the imbalanced data under cross-entropy loss function in the context of unconstrained feature model. Our contribution is multi-fold compared with the state-of-the-art results: (a) we show that the feature vectors exhibit collapse phenomenon, i.e., the features within the same class collapse to the same mean vector; (b) the mean feature vectors no longer form an equiangular tight frame. Instead, their pairwise angles depend on the sample size; (c) we also precisely characterize the sharp threshold on which the minority collapse (the feature vectors of the minority groups collapse to one single vector) will take place; (d) finally, we argue that the effect of the imbalance in datasize diminishes as the sample size grows. Our results provide a complete picture of the N C under the cross-entropy loss for the imbalanced data. Numerical experiments confirm our theoretical analysis.

stat.ML↗

arXiv2023

Improved theoretical guarantee for rank aggregation via spectral method

Given pairwise comparisons between multiple items, how to rank them so that the ranking matches the observations? This problem, known as rank aggregation, has found many applications in sports, recommendation systems, and other web applications. As it is generally NP-hard to find a global ranking that minimizes the mismatch (known as the Kemeny optimization), we focus on the Erdös-Rényi outliers (ERO) model for this ranking problem. Here, each pairwise comparison is a corrupted copy of the true score difference. We investigate spectral ranking algorithms that are based on unnormalized and normalized data matrices. The key is to understand their performance in recovering the underlying scores of each item from the observed data. This reduces to deriving an entry-wise perturbation error bound between the top eigenvectors of the unnormalized/normalized data matrix and its population counterpart. By using the leave-one-out technique, we provide a sharper $\ell_{\infty}$-norm perturbation bound of the eigenvectors and also derive an error bound on the maximum displacement for each item, with only $Ω(n\log n)$ samples. Our theoretical analysis improves upon the state-of-the-art results in terms of sample complexity, and our numerical experiments confirm these theoretical findings.

stat.ML↗

arXiv2022

Near-Optimal Performance Bounds for Orthogonal and Permutation Group Synchronization via Spectral Methods

Group synchronization asks to recover group elements from their pairwise measurements. It has found numerous applications across various scientific disciplines. In this work, we focus on orthogonal and permutation group synchronization which are widely used in computer vision such as object matching and structure from motion. Among many available approaches, the spectral methods have enjoyed great popularity due to their efficiency and convenience. We will study the performance guarantees of the spectral methods in solving these two synchronization problems by investigating how well the computed eigenvectors approximate each group element individually. We establish our theory by applying the recent popular~\emph{leave-one-out} technique and derive a~\emph{block-wise} performance bound for the recovery of each group element via eigenvectors. In particular, for orthogonal group synchronization, we obtain a near-optimal performance bound for the group recovery in presence of additive Gaussian noise. For permutation group synchronization under random corruption, we show that the widely-used two-step procedure (spectral method plus rounding) can recover all the group elements exactly if the SNR (signal-to-noise ratio) is close to the information theoretical limit. Our numerical experiments confirm our theory and indicate a sharp phase transition for the exact group recovery.

cs.IT↗

arXiv2021

Improved Performance Guarantees for Orthogonal Group Synchronization via Generalized Power Method

Given the noisy pairwise measurements among a set of unknown group elements, how to recover them efficiently and robustly? This problem, known as group synchronization, has drawn tremendous attention in the scientific community. In this work, we focus on orthogonal group synchronization that has found many applications, including computer vision, robotics, and cryo-electron microscopy. One commonly used approach is the least squares estimation that requires solving a highly nonconvex optimization program. The past few years have witnessed considerable advances in tackling this challenging problem by convex relaxation and efficient first-order methods. However, one fundamental theoretical question remains to be answered: how does the recovery performance depend on the noise strength? To answer this question, we study a benchmark model: recovering orthogonal group elements from their pairwise measurements corrupted by Gaussian noise. We investigate the performance of convex relaxation and the generalized power method (GPM). By applying the novel~\emph{leave-one-out} technique, we prove that the GPM with spectral initialization enjoys linear convergence to the global optima to the convex relaxation that also matches the maximum likelihood estimator. Our result achieves a near-optimal performance bound on the convergence of the GPM and improves the state-of-the-art theoretical guarantees on the tightness of convex relaxation by a large margin.

cs.IT↗

arXiv2021

Near-Optimal Bounds for Generalized Orthogonal Procrustes Problem via Generalized Power Method

Given multiple point clouds, how to find the rigid transform (rotation, reflection, and shifting) such that these point clouds are well aligned? This problem, known as the generalized orthogonal Procrustes problem (GOPP), has found numerous applications in statistics, computer vision, and imaging science. While one commonly-used method is finding the least squares estimator, it is generally an NP-hard problem to obtain the least squares estimator exactly due to the notorious nonconvexity. In this work, we apply the semidefinite programming (SDP) relaxation and the generalized power method to solve this generalized orthogonal Procrustes problem. In particular, we assume the data are generated from a signal-plus-noise model: each observed point cloud is a noisy copy of the same unknown point cloud transformed by an unknown orthogonal matrix and also corrupted by additive Gaussian noise. We show that the generalized power method (equivalently alternating minimization algorithm) with spectral initialization converges to the unique global optimum to the SDP relaxation, provided that the signal-to-noise ratio is high. Moreover, this limiting point is exactly the least squares estimator and also the maximum likelihood estimator. In addition, we derive a block-wise estimation error for each orthogonal matrix and the underlying point cloud. Our theoretical bound is near-optimal in terms of the information-theoretic limit (only loose by a factor of the dimension and a log factor). Our results significantly improve the state-of-the-art results on the tightness of the SDP relaxation for the generalized orthogonal Procrustes problem, an open problem posed by Bandeira, Khoo, and Singer in 2014.

cs.IT↗

arXiv2021

Generalized Orthogonal Procrustes Problem under Arbitrary Adversaries

The generalized orthogonal Procrustes problem (GOPP) plays a fundamental role in several scientific disciplines including statistics, imaging science and computer vision. Despite its tremendous practical importance, it is generally an NP-hard problem to find the least squares estimator. We study the semidefinite relaxation (SDR) and an iterative method named generalized power method (GPM) to find the least squares estimator, and investigate the performance under a signal-plus-noise model. We show that the SDR recovers the least squares estimator exactly and moreover the generalized power method with a proper initialization converges linearly to the global minimizer to the SDR, provided that the signal-to-noise ratio is large. The main technique follows from showing the nonlinear mapping involved in the GPM is essentially a local contraction mapping and then applying the well-known Banach fixed-point theorem finishes the proof. In addition, we analyze the low-rank factorization algorithm and show the corresponding optimization landscape is free of spurious local minimizers under nearly identical conditions that enables the success of SDR approach. The highlight of our work is that the theoretical guarantees are purely algebraic and do not assume any statistical priors of the additive adversaries, and thus it applies to various interesting settings.

cs.IT↗

arXiv2020

Solving Orthogonal Group Synchronization via Convex and Low-Rank Optimization: Tightness and Landscape Analysis

Group synchronization aims to recover the group elements from their noisy pairwise measurements. It has found many applications in community detection, clock synchronization, and joint alignment problem. This paper focuses on the orthogonal group synchronization which is often used in cryo-EM and computer vision. However, it is generally NP-hard to retrieve the group elements by finding the least squares estimator. In this work, we first study the semidefinite programming (SDP) relaxation of the orthogonal group synchronization and its tightness, i.e., the SDP estimator is exactly equal to the least squares estimator. Moreover, we investigate the performance of the Burer-Monteiro factorization in solving the SDP relaxation by analyzing its corresponding optimization landscape. We provide deterministic sufficient conditions which guarantee: (i) the tightness of SDP relaxation; (ii) optimization landscape arising from the Burer-Monteiro approach is benign, i.e., the global optimum is exactly the least squares estimator and no other spurious local optima exist. Our result provides a solid theoretical justification of why the Burer-Monteiro approach is remarkably efficient and effective in solving the large-scale SDPs arising from orthogonal group synchronization. We perform numerical experiments to complement our theoretical analysis, which gives insights into future research directions.

math.OC↗

arXiv2020

Strong Consistency, Graph Laplacians, and the Stochastic Block Model

Spectral clustering has become one of the most popular algorithms in data clustering and community detection. We study the performance of classical two-step spectral clustering via the graph Laplacian to learn the stochastic block model. Our aim is to answer the following question: when is spectral clustering via the graph Laplacian able to achieve strong consistency, i.e., the exact recovery of the underlying hidden communities? Our work provides an entrywise analysis (an $\ell_{\infty}$-norm perturbation bound) of the Fielder eigenvector of both the unnormalized and the normalized Laplacian associated with the adjacency matrix sampled from the stochastic block model. We prove that spectral clustering is able to achieve exact recovery of the planted community structure under conditions that match the information-theoretic limits.

stat.ML↗

arXiv2020

On the Critical Coupling of the Finite Kuramoto Model on Dense Networks

Kuramoto model is one of the most prominent models for the synchronization of coupled oscillators. It has long been a research hotspot to understand how natural frequencies, the interaction between oscillators, and network topology determine the onset of synchronization. In this paper, we investigate the critical coupling of Kuramoto oscillators on deterministic dense networks, viewed as a natural generalization from all-to-all networks of identical oscillators. We provide a sufficient condition under which the Kuramoto model with non-identical oscillators has one unique and stable equilibrium. Moreover, this equilibrium is phase cohesive and enjoys local exponential synchronization. We perform numerical simulations of the Kuramoto model on random networks and circulant networks to complement our theoretical analysis and provide insights for future research.

nlin.AO↗

arXiv2019

On the Landscape of Synchronization Networks: A Perspective from Nonconvex Optimization

Studying the landscape of nonconvex cost function is key towards a better understanding of optimization algorithms widely used in signal processing, statistics, and machine learning. Meanwhile, the famous Kuramoto model has been an important mathematical model to study the synchronization phenomena of coupled oscillators over various network topologies. In this paper, we bring together these two seemingly unrelated objects by investigating the optimization landscape of a nonlinear function $E(\boldsymbolθ) = \frac{1}{2}\sum_{1\leq i,j\leq n} a_{ij}(1-\cos(θ_i - θ_j))$ associated to an underlying network and exploring the relationship between the existence of local minima and network topology. This function arises naturally in Burer-Monteiro method applied to $\mathbb{Z}_2$ synchronization as well as matrix completion on the torus. Moreover, it corresponds to the energy function of the homogeneous Kuramoto model on complex networks for coupled oscillators. We prove the minimizer of the energy function is unique up to a global translation under deterministic dense graphs and Erdős-Rényi random graphs with tools from optimization and random matrix theory. Consequently, the stable equilibrium of the corresponding homogeneous Kuramoto model is unique and the basin of attraction for the synchronous state of these coupled oscillators is the whole phase space minus a set of measure zero. In addition, our results address when the Burer-Monteiro method recovers the ground truth exactly from highly incomplete observations in $\mathbb{Z}_2$ synchronization and shed light on the robustness of nonconvex optimization algorithms against certain types of so-called monotone adversaries. Numerical simulations are performed to illustrate our results.

math.OC↗

arXiv2019

Certifying Global Optimality of Graph Cuts via Semidefinite Relaxation: A Performance Guarantee for Spectral Clustering

Spectral clustering has become one of the most widely used clustering techniques when the structure of the individual clusters is non-convex or highly anisotropic. Yet, despite its immense popularity, there exists fairly little theory about performance guarantees for spectral clustering. This issue is partly due to the fact that spectral clustering typically involves two steps which complicated its theoretical analysis: first, the eigenvectors of the associated graph Laplacian are used to embed the dataset, and second, k-means clustering algorithm is applied to the embedded dataset to get the labels. This paper is devoted to the theoretical foundations of spectral clustering and graph cuts. We consider a convex relaxation of graph cuts, namely ratio cuts and normalized cuts, that makes the usual two-step approach of spectral clustering obsolete and at the same time gives rise to a rigorous theoretical analysis of graph cuts and spectral clustering. We derive deterministic bounds for successful spectral clustering via a spectral proximity condition that naturally depends on the algebraic connectivity of each cluster and the inter-cluster connectivity. Moreover, we demonstrate by means of some popular examples that our bounds can achieve near-optimality. Our findings are also fundamental for the theoretical understanding of kernel k-means. Numerical simulations confirm and complement our analysis.

stat.ML↗

arXiv2018

When Do Birds of a Feather Flock Together? k-Means, Proximity, and Conic Programming

Given a set of data, one central goal is to group them into clusters based on some notion of similarity between the individual objects. One of the most popular and widely-used approaches is k-means despite the computational hardness to find its global minimum. We study and compare the properties of different convex relaxations by relating them to corresponding proximity conditions, an idea originally introduced by Kumar and Kannan. Using conic duality theory, we present an improved proximity condition under which the Peng-Wei relaxation of k-means recovers the underlying clusters exactly. Our proximity condition improves upon Kumar and Kannan, and is comparable to that of Awashti and Sheffet where proximity conditions are established for projective k-means. In addition, we provide a necessary proximity condition for the exactness of the Peng-Wei relaxation. For the special case of equal cluster sizes, we establish a different and completely localized proximity condition under which the Amini-Levina relaxation yields exact clustering, thereby having addressed an open problem by Awasthi and Sheffet in the balanced case. Our framework is not only deterministic and model-free but also comes with a clear geometric meaning which allows for further analysis and generalization. Moreover, it can be conveniently applied to analyzing various data generative models such as the stochastic ball models and Gaussian mixture models. With this method, we improve the current minimum separation bound for the stochastic ball models and achieve the state-of-the-art results of learning Gaussian mixture models.

math.OC↗