arXiv subjects

Haoyang Liu

Publications and source records attributed to Haoyang Liu.

82 records · Page 5Linked to original sources

arXiv2020

An equivalence between critical points for rank constraints versus low-rank factorizations

Two common approaches in low-rank optimization problems are either working directly with a rank constraint on the matrix variable, or optimizing over a low-rank factorization so that the rank constraint is implicitly ensured. In this paper, we study the natural connection between the rank-constrained and factorized approaches. We show that all second-order stationary points of the factorized objective function correspond to fixed points of projected gradient descent run on the original problem (where the projection step enforces the rank constraint). This result allows us to unify many existing optimization guarantees that have been proved specifically in either the rank-constrained or the factorized setting, and leads to new results for certain settings of the problem. We demonstrate application of our results to several concrete low-rank optimization problems arising in matrix inverse problems.

math.OC↗

arXiv2020

Minimax rates in sparse, high-dimensional changepoint detection

We study the detection of a sparse change in a high-dimensional mean vector as a minimax testing problem. Our first main contribution is to derive the exact minimax testing rate across all parameter regimes for $n$ independent, $p$-variate Gaussian observations. This rate exhibits a phase transition when the sparsity level is of order $\sqrt{p \log \log (8n)}$ and has a very delicate dependence on the sample size: in a certain sparsity regime it involves a triple iterated logarithmic factor in~$n$. Further, in a dense asymptotic regime, we identify the sharp leading constant, while in the corresponding sparse asymptotic regime, this constant is determined to within a factor of $\sqrt{2}$. Extensions that cover spatial and temporal dependence, primarily in the dense case, are also provided.

math.ST↗

arXiv2019

Exact high-dimensional asymptotics for Support Vector Machine

The Support Vector Machine (SVM) is one of the most widely used classification methods. In this paper, we consider the soft-margin SVM used on data points with independent features, where the sample size $n$ and the feature dimension $p$ grows to $\infty$ in a fixed ratio $p/n\rightarrow δ$. We propose a set of equations that exactly characterizes the asymptotic behavior of support vector machine. In particular, we give exact formulas for (1) the variability of the optimal coefficients, (2) the proportion of data points lying on the margin boundary (i.e. number of support vectors), (3) the final objective function value, and (4) the expected misclassification error on new data points, which in particular implies the exact formula for the optimal tuning parameter given a data generating mechanism. We first establish these formulas in the case where the label $y\in\{+1,-1\}$ is independent of the feature $x$. Then the results are generalized to the case where the label $y\in\{+1,-1\}$ is allowed to have a general dependence on the feature $x$ through a linear combination $a_0^Tx$. These formulas for the non-smooth hinge loss are analogous to the recent results in \citep{sur2018modern} for smooth logistic loss. Our approach is based on heuristic leave-one-out calculations.

stat.ML↗

arXiv2019

Between hard and soft thresholding: optimal iterative thresholding algorithms

Iterative thresholding algorithms seek to optimize a differentiable objective function over a sparsity or rank constraint by alternating between gradient steps that reduce the objective, and thresholding steps that enforce the constraint. This work examines the choice of the thresholding operator, and asks whether it is possible to achieve stronger guarantees than what is possible with hard thresholding. We develop the notion of relative concavity of a thresholding operator, a quantity that characterizes the worst-case convergence performance of any thresholding operator on the target optimization problem. Surprisingly, we find that commonly used thresholding operators, such as hard thresholding and soft thresholding, are suboptimal in terms of worst-case convergence guarantees. Instead, a general class of thresholding operators, lying between hard thresholding and soft thresholding, is shown to be optimal with the strongest possible convergence guarantee among all thresholding operators. Examples of this general class includes $\ell_q$ thresholding with appropriate choices of $q$, and a newly defined {\em reciprocal thresholding} operator. We also investigate the implications of the improved optimization guarantee in the statistical setting of sparse linear regression, and show that this new class of thresholding operators attain the optimal rate for computationally efficient estimators, matching the Lasso.

stat.ME↗

arXiv2019

Low-rank Matrix Optimization Using Polynomial-filtered Subspace Extraction

In this paper, we study first-order methods on a large variety of low-rank matrix optimization problems, whose solutions only live in a low dimensional eigenspace. Traditional first-order methods depend on the eigenvalue decomposition at each iteration which takes most of the computation time. In order to reduce the cost, we propose an inexact algorithm framework based on a polynomial subspace extraction. The idea is to use an additional polynomial-filtered iteration to extract an approximated eigenspace, and project the iteration matrix on this subspace, followed by an optimization update. The accuracy of the extracted subspace can be controlled by the degree of the polynomial filters. This kind of subspace extraction also enjoys the warm start property: the subspace of the current iteration is refined from the previous one. Then this framework is instantiated into two algorithms: the polynomial-filtered proximal gradient method and the polynomial-filtered alternating direction method of multipliers. We give a theoretical guarantee to the two algorithms that the polynomial degree is not necessarily very large. They share the same convergence speed as the corresponding original methods if the polynomial degree grows with an order $Ω(\log k)$ at the $k$-th iteration. If the warm-start property is considered, the degree can be reduced to a constant, independent of the iteration $k$. Preliminary numerical experiments on several low-rank matrix optimization problems show that the polynomial filtered algorithms usually provide multi-fold speedups.

math.OC↗

arXiv2018

Density Estimation with Contaminated Data: Minimax Rates and Theory of Adaptation

This paper studies density estimation under pointwise loss in the setting of contamination model. The goal is to estimate $f(x_0)$ at some $x_0\in\mathbb{R}$ with i.i.d. observations, $$ X_1,\dots,X_n\sim (1-ε)f+εg, $$ where $g$ stands for a contamination distribution. In the context of multiple testing, this can be interpreted as estimating the null density at a point. We carefully study the effect of contamination on estimation through the following model indices: contamination proportion $ε$, smoothness of target density $β_0$, smoothness of contamination density $β_1$, and level of contamination $m$ at the point to be estimated, i.e. $g(x_0)\leq m$. It is shown that the minimax rate with respect to the squared error loss is of order $$ [n^{-\frac{2β_0}{2β_0+1}}]\vee[ε^2(1\wedge m)^2]\vee[n^{-\frac{2β_1}{2β_1+1}}ε^{\frac{2}{2β_1+1}}], $$ which characterizes the exact influence of contamination on the difficulty of the problem. We then establish the minimal cost of adaptation to contamination proportion, to smoothness and to both of the numbers. It is shown that some small price needs to be paid for adaptation in any of the three cases. Variations of Lepski's method are considered to achieve optimal adaptation. The problem is also studied when there is no smoothness assumption on the contamination distribution. This setting that allows for an arbitrary contamination distribution is recognized as Huber's $ε$-contamination model. The minimax rate is shown to be $$ [n^{-\frac{2β_0}{2β_0+1}}]\vee [ε^{\frac{2β_0}{β_0+1}}]. $$ The adaptation theory is also different from the smooth contamination case. While adaptation to either contamination proportion or smoothness only costs a logarithmic factor, adaptation to both numbers is proved to be impossible.

math.ST↗

arXiv2017

A Sparse Completely Positive Relaxation of the Modularity Maximization for Community Detection

In this paper, we consider the community detection problem under either the stochastic block model (SBM) assumption or the degree-correlated stochastic block model (DCSBM) assumption. The modularity maximization formulation for the community detection problem is NP-hard in general. In this paper, we propose a sparse and low-rank completely positive relaxation for the modularity maximization problem, we then develop an efficient row-by-row (RBR) type block coordinate descent (BCD) algorithm to solve the relaxation and prove an $\mathcal{O}(1/\sqrt{N})$ convergence rate to a stationary point where $N$ is the number of iterations. A fast rounding scheme is constructed to retrieve the community structure from the solution. Non-asymptotic high probability bounds on the misclassification rate are established to justify our approach. We further develop an asynchronous parallel RBR algorithm to speed up the convergence. Extensive numerical experiments on both synthetic and real world networks show that the proposed approach enjoys advantages in both clustering accuracy and numerical efficiency. Our numerical results indicate that the newly proposed method is a quite competitive alternative for community detection on sparse networks with over 50 million nodes.

math.OC↗

arXiv2015

On the Marčenko-Pastur law for linear time series

This paper is concerned with extensions of the classical Marčenko-Pastur law to time series. Specifically, $p$-dimensional linear processes are considered which are built from innovation vectors with independent, identically distributed (real- or complex-valued) entries possessing zero mean, unit variance and finite fourth moments. The coefficient matrices of the linear process are assumed to be simultaneously diagonalizable. In this setting, the limiting behavior of the empirical spectral distribution of both sample covariance and symmetrized sample autocovariance matrices is determined in the high-dimensional setting $p/n\to c\in (0,\infty)$ for which dimension $p$ and sample size $n$ diverge to infinity at the same rate. The results extend existing contributions available in the literature for the covariance case and are one of the first of their kind for the autocovariance case.

math.ST↗

arXiv2011

Learning in A Changing World: Restless Multi-Armed Bandit with Unknown Dynamics

We consider the restless multi-armed bandit (RMAB) problem with unknown dynamics in which a player chooses M out of N arms to play at each time. The reward state of each arm transits according to an unknown Markovian rule when it is played and evolves according to an arbitrary unknown random process when it is passive. The performance of an arm selection policy is measured by regret, defined as the reward loss with respect to the case where the player knows which M arms are the most rewarding and always plays the M best arms. We construct a policy with an interleaving exploration and exploitation epoch structure that achieves a regret with logarithmic order when arbitrary (but nontrivial) bounds on certain system parameters are known. When no knowledge about the system is available, we show that the proposed policy achieves a regret arbitrarily close to the logarithmic order. We further extend the problem to a decentralized setting where multiple distributed players share the arms without information exchange. Under both an exogenous restless model and an endogenous restless model, we show that a decentralized extension of the proposed policy preserves the logarithmic regret order as in the centralized setting. The results apply to adaptive learning in various dynamic systems and communication networks, as well as financial investment.

math.OC↗

arXiv2011

Decentralized Restless Bandit with Multiple Players and Unknown Dynamics

We consider decentralized restless multi-armed bandit problems with unknown dynamics and multiple players. The reward state of each arm transits according to an unknown Markovian rule when it is played and evolves according to an arbitrary unknown random process when it is passive. Players activating the same arm at the same time collide and suffer from reward loss. The objective is to maximize the long-term reward by designing a decentralized arm selection policy to address unknown reward models and collisions among players. A decentralized policy is constructed that achieves a regret with logarithmic order when an arbitrary nontrivial bound on certain system parameters is known. When no knowledge about the system is available, we extend the policy to achieve a regret arbitrarily close to the logarithmic order. The result finds applications in communication networks, financial investment, and industrial engineering.

math.OC↗