Searcharxiv⌕ Search

arXiv subjects

Santosh S. Vempala

Publications and source records attributed to Santosh S. Vempala.

At least 55 records · Page 3Linked to original sources
arXiv2021

Faster $p$-Norm Regression Using Sparsity

For a matrix $A\in \mathbb{R}^{n\times d}$ with $n\geq d$, we consider the dual problems of $\min \|Ax-b\|_p^p, \, b\in \mathbb{R}^n$ and $\min_{A^\top x=b} \|x\|_p^p,\, b\in \mathbb{R}^d$. We improve the runtimes for solving these problems to high accuracy for every $p>1$ for sufficiently sparse matrices. We show that recent progress on fast sparse linear solvers can be leveraged to obtain faster than matrix-multiplication algorithms for any $p > 1$, i.e., in time $\tilde{O}(pn^θ)$ for some $θ< ω$, the matrix multiplication constant. We give the first high-accuracy input sparsity $p$-norm regression algorithm for solving $\min \|Ax-b\|_p^p$ with $1 < p \leq 2$, via a new row sampling theorem for the smoothed $p$-norm function. This algorithm runs in time $\tilde{O}(\text{nnz}(A) + d^4)$ for any $1<p\leq 2$, and in time $\tilde{O}(\text{nnz}(A) + d^θ)$ for $p$ close to $2$, improving on the previous best bound where the exponent of $d$ grows with $\max\{p, p/(p-1)\}$.

cs.DS↗
arXiv2021

The Mirror Langevin Algorithm Converges with Vanishing Bias

The technique of modifying the geometry of a problem from Euclidean to Hessian metric has proved to be quite effective in optimization, and has been the subject of study for sampling. The Mirror Langevin Diffusion (MLD) is a sampling analogue of mirror flow in continuous time, and it has nice convergence properties under log-Sobolev or Poincare inequalities relative to the Hessian metric, as shown by Chewi et al. (2020). In discrete time, a simple discretization of MLD is the Mirror Langevin Algorithm (MLA) studied by Zhang et al. (2020), who showed a biased convergence bound with a non-vanishing bias term (does not go to zero as step size goes to zero). This raised the question of whether we need a better analysis or a better discretization to achieve a vanishing bias. Here we study the basic Mirror Langevin Algorithm and show it indeed has a vanishing bias. We apply mean-square analysis based on Li et al. (2019) and Li et al. (2021) to show the mixing time bound for MLA under the modified self-concordance condition introduced by Zhang et al. (2020).

cs.DS↗
arXiv2021

Robustly Learning Mixtures of $k$ Arbitrary Gaussians

We give a polynomial-time algorithm for the problem of robustly estimating a mixture of $k$ arbitrary Gaussians in $\mathbb{R}^d$, for any fixed $k$, in the presence of a constant fraction of arbitrary corruptions. This resolves the main open problem in several previous works on algorithmic robust statistics, which addressed the special cases of robustly estimating (a) a single Gaussian, (b) a mixture of TV-distance separated Gaussians, and (c) a uniform mixture of two Gaussians. Our main tools are an efficient \emph{partial clustering} algorithm that relies on the sum-of-squares method, and a novel \emph{tensor decomposition} algorithm that allows errors in both Frobenius norm and low-rank terms.

cs.DS↗
arXiv2019

The Communication Complexity of Optimization

We consider the communication complexity of a number of distributed optimization problems. We start with the problem of solving a linear system. Suppose there is a coordinator together with $s$ servers $P_1, \ldots, P_s$, the $i$-th of which holds a subset $A^{(i)} x = b^{(i)}$ of $n_i$ constraints of a linear system in $d$ variables, and the coordinator would like to output $x \in \mathbb{R}^d$ for which $A^{(i)} x = b^{(i)}$ for $i = 1, \ldots, s$. We assume each coefficient of each constraint is specified using $L$ bits. We first resolve the randomized and deterministic communication complexity in the point-to-point model of communication, showing it is $\tildeΘ(d^2L + sd)$ and $\tildeΘ(sd^2L)$, respectively. We obtain similar results for the blackboard model. When there is no solution to the linear system, a natural alternative is to find the solution minimizing the $\ell_p$ loss. While this problem has been studied, we give improved upper or lower bounds for every value of $p \ge 1$. One takeaway message is that sampling and sketching techniques, which are commonly used in earlier work on distributed optimization, are neither optimal in the dependence on $d$ nor on the dependence on the approximation $ε$, thus motivating new techniques from optimization to solve these problems. Towards this end, we consider the communication complexity of optimization tasks which generalize linear systems. For linear programming, we first resolve the communication complexity when $d$ is constant, showing it is $\tildeΘ(sL)$ in the point-to-point model. For general $d$ and in the point-to-point model, we show an $\tilde{O}(sd^3 L)$ upper bound and an $\tildeΩ(d^2 L + sd)$ lower bound. We also show if one perturbs the coefficients randomly by numbers as small as $2^{-Θ(L)}$, then the upper bound is $\tilde{O}(sd^2 L) + \textrm{poly}(dL)$.

cs.DS↗
arXiv2019

A Generalized Central Limit Conjecture for Convex Bodies

The central limit theorem for convex bodies says that with high probability the marginal of an isotropic log-concave distribution along a random direction is close to a Gaussian, with the quantitative difference determined asymptotically by the Cheeger/Poincare/KLS constant. Here we propose a generalized CLT for marginals along random directions drawn from any isotropic log-concave distribution; namely, for $x,y$ drawn independently from isotropic log-concave densities $p,q$, the random variable $\langle x,y\rangle$ is close to Gaussian. Our main result is that this generalized CLT is quantitatively equivalent (up to a small factor) to the KLS conjecture. Any polynomial improvement in the current KLS bound of $n^{1/4}$ in $\mathbb{R}^n$ implies the generalized CLT, and vice versa. This tight connection suggests that the generalized CLT might provide insight into basic open questions in asymptotic convex geometry.

math.FA↗
arXiv2019

Optimal Convergence Rate of Hamiltonian Monte Carlo for Strongly Logconcave Distributions

We study Hamiltonian Monte Carlo (HMC) for sampling from a strongly logconcave density proportional to $e^{-f}$ where $f:\mathbb{R}^d \to \mathbb{R}$ is $μ$-strongly convex and $L$-smooth (the condition number is $κ= L/μ$). We show that the relaxation time (inverse of the spectral gap) of ideal HMC is $O(κ)$, improving on the previous best bound of $O(κ^{1.5})$; we complement this with an example where the relaxation time is $Ω(κ)$. When implemented using a nearly optimal ODE solver, HMC returns an $\varepsilon$-approximate point in $2$-Wasserstein distance using $\widetilde{O}((κd)^{0.5} \varepsilon^{-1})$ gradient evaluations per step and $\widetilde{O}((κd)^{1.5}\varepsilon^{-1})$ total time.

cs.DS↗
arXiv2019

Eldan's Stochastic Localization and the KLS Conjecture: Isoperimetry, Concentration and Mixing

We show that the Cheeger constant for $n$-dimensional isotropic logconcave measures is $O(n^{1/4})$, improving on the previous best bound of $O(n^{1/3}\sqrt{\log n}).$ As corollaries we obtain the same improved bound on the thin-shell estimate, Poincaré constant and Lipschitz concentration constant and an alternative proof of this bound for the isotropic (slicing) constant; it also follows that the ball walk for sampling from an isotropic logconcave density in ${\bf R}^{n}$ converges in $O^{*}(n^{2.5})$ steps from a warm start. The proof is based on gradually transforming any logconcave density to one that has a significant Gaussian factor via a Martingale process. Extending this proof technique, we prove that the log-Sobolev constant of any isotropic logconcave density in ${\bf R}^{n}$ with support of diameter $D$ is $Ω(1/D)$, resolving a question posed by Frieze and Kannan in 1997. This is asymptotically the best possible estimate and improves on the previous bound of $Ω(1/D^{2})$ by Kannan-Lovász-Montenegro. It follows that for any isotropic logconcave density, the ball walk with step size $δ=Θ(1/\sqrt{n})$ mixes in $O\left(n^{2}D\right)$ proper steps from \emph{any }starting point. This improves on the previous best bound of $O(n^{2}D^{2})$ and is also asymptotically tight. The new bound leads to the following large deviation inequality for an $L$-Lipschitz function $g$ over an isotropic logconcave density $p$: for any $t>0$, \[ Pr_{x\sim p}\left(\left|g(x)-\bar{g}\right|\geq L\cdot t\right)\leq\exp(-\frac{c\cdot t^{2}}{t+\sqrt{n}}) \] where $\bar{g}$ is the median or mean of $g$ for $x\sim p$; this generalizes and improves on previous bounds by Paouris and by Guedon-Milman. The technique also bounds the ``small ball'' probability in terms of the Cheeger constant, and recovers the current best bound.

math.FA↗
arXiv2018

Algorithmic Theory of ODEs and Sampling from Well-conditioned Logconcave Densities

Sampling logconcave functions arising in statistics and machine learning has been a subject of intensive study. Recent developments include analyses for Langevin dynamics and Hamiltonian Monte Carlo (HMC). While both approaches have dimension-independent bounds for the underlying $\mathit{continuous}$ processes under sufficiently strong smoothness conditions, the resulting discrete algorithms have complexity and number of function evaluations growing with the dimension. Motivated by this problem, in this paper, we give a general algorithm for solving multivariate ordinary differential equations whose solution is close to the span of a known basis of functions (e.g., polynomials or piecewise polynomials). The resulting algorithm has polylogarithmic depth and essentially tight runtime - it is nearly linear in the size of the representation of the solution. We apply this to the sampling problem to obtain a nearly linear implementation of HMC for a broad class of smooth, strongly logconcave densities, with the number of iterations (parallel depth) and gradient evaluations being $\mathit{polylogarithmic}$ in the dimension (rather than polynomial as in previous work). This class includes the widely-used loss function for logistic regression with incoherent weight matrices and has been subject of much study recently. We also give a faster algorithm with $ \mathit{polylogarithmic~depth}$ for the more general and standard class of strongly convex functions with Lipschitz gradient. These results are based on (1) an improved contraction bound for the exact HMC process and (2) logarithmic bounds on the degree of polynomials that approximate solutions of the differential equations arising in implementing HMC.

cs.DS↗
arXiv2018

The Kannan-Lovász-Simonovits Conjecture

The Kannan-Lovász-Simonovits conjecture says that the Cheeger constant of any logconcave density is achieved to within a universal, dimension-independent constant factor by a hyperplane-induced subset. Here we survey the origin and consequences of the conjecture (in geometry, probability, information theory and algorithms) as well as recent progress resulting in the current best bounds. The conjecture has lead to several techniques of general interest.

math.PR↗
arXiv2017

Stochastic Localization + Stieltjes Barrier = Tight Bound for Log-Sobolev

Logarithmic Sobolev inequalities are a powerful way to estimate the rate of convergence of Markov chains and to derive concentration inequalities on distributions. We prove that the log-Sobolev constant of any isotropic logconcave density in $R^n$ with support of diameter $D$ is $1/D$, resolving a question posed by Frieze and Kannan in 1997. This is asymptotically the best possible estimate and improves on the previous bound of $1/D^2$ by Kannan-Lovász-Montenegro. It follows that for any isotropic logconcave density, the ball walk with step size $δ= (1/\sqrt{n})$ mixes in $O(n^2D)$ proper steps from any starting point. This improves on the previous best bound of $O(n^2D^2)$ and is also asymptotically tight. The new bound leads to the following refined large deviation inequality for any L-Lipschitz function g over an isotropic logconcave density p: for any t > 0, $$P(|g(x)- \bar{g}(x)| \ge c . L. t) \le \exp(-\frac{t^2}{t+\sqrt{n}})$$ where $\bar{g}$ is the median or mean of $g$ for $x \sim p$; this generalizes/improves on previous bounds by Paouris and by Guedon-Milman. The technique also bounds the "small ball" probability in terms of the Cheeger constant, and recovers the current best bound. Our main proof is based on stochastic localization together with a Stieltjes-type barrier function.

math.PR↗
arXiv2017

Convergence Rate of Riemannian Hamiltonian Monte Carlo and Faster Polytope Volume Computation

We give the first rigorous proof of the convergence of Riemannian Hamiltonian Monte Carlo, a general (and practical) method for sampling Gibbs distributions. Our analysis shows that the rate of convergence is bounded in terms of natural smoothness parameters of an associated Riemannian manifold. We then apply the method with the manifold defined by the log barrier function to the problems of (1) uniformly sampling a polytope and (2) computing its volume, the latter by extending Gaussian cooling to the manifold setting. In both cases, the total number of steps needed is O^{*}(mn^{\frac{2}{3}}), improving the state of the art. A key ingredient of our analysis is a proof of an analog of the KLS conjecture for Gibbs distributions over manifolds.

cs.DS↗
arXiv2017

Efficient Convex Optimization with Membership Oracles

We consider the problem of minimizing a convex function over a convex set given access only to an evaluation oracle for the function and a membership oracle for the set. We give a simple algorithm which solves this problem with $\tilde{O}(n^2)$ oracle calls and $\tilde{O}(n^3)$ additional arithmetic operations. Using this result, we obtain more efficient reductions among the five basic oracles for convex sets and functions defined by Grötschel, Lovasz and Schrijver.

cs.DS↗
arXiv2017

Geodesic Walks in Polytopes

We introduce the geodesic walk for sampling Riemannian manifolds and apply it to the problem of generating uniform random points from polytopes in R^n specified by m inequalities. The walk is a discrete-time simulation of a stochastic differential equation (SDE) on the Riemannian manifold equipped with the metric induced by the Hessian of a convex function; each step is the solution of an ordinary differential equation (ODE). The resulting sampling algorithm for polytopes mixes in O*(mn^{3/4}) steps. This is the first walk that breaks the quadratic barrier for mixing in high dimension, improving on the previous best bound of O*(mn) by Kannan and Narayanan for the Dikin walk. We also show that each step of the geodesic walk (solving an ODE) can be implemented efficiently, thus improving the time complexity for sampling polytopes. Our analysis of the geodesic walk for general Hessian manifolds does not assume positive curvature and might be of independent interest.

cs.DS↗
arXiv2016

Accelerated Newton Iteration: Roots of Black Box Polynomials and Matrix Eigenvalues

We study the problem of computing the largest root of a real rooted polynomial $p(x)$ to within error $\varepsilon $ given only black box access to it, i.e., for any $x \in {\mathbb R}$, the algorithm can query an oracle for the value of $p(x)$, but the algorithm is not allowed access to the coefficients of $p(x)$. A folklore result for this problem is that the largest root of a polynomial can be computed in $O(n \log (1/\varepsilon ))$ polynomial queries using the Newton iteration. We give a simple algorithm that queries the oracle at only $O(\log n \log(1/\varepsilon ))$ points, where $n$ is the degree of the polynomial. Our algorithm is based on a novel approach for accelerating the Newton method by using higher derivatives. As a special case, we consider the problem of computing the top eigenvalue of a symmetric matrix in ${\mathbb Q}^{n \times n}$ to within error $\varepsilon $ in time polynomial in the input description, i.e., the number of bits to describe the matrix and $\log(1/\varepsilon )$. Well-known methods such as the power iteration and Lanczos iteration incur running time polynomial in $1/\varepsilon $, while Gaussian elimination takes $Ω(n^4)$ bit operations. As a corollary of our main result, we obtain a $\tilde{O}(n^ω \log^2 ( ||A||_F/\varepsilon ))$ bit complexity algorithm to compute the top eigenvalue of the matrix $A$ or to check if it is approximately PSD ($A \succeq -\varepsilon I$).

cs.DS↗
arXiv2015

Max vs Min: Tensor Decomposition and ICA with nearly Linear Sample Complexity

We present a simple, general technique for reducing the sample complexity of matrix and tensor decomposition algorithms applied to distributions. We use the technique to give a polynomial-time algorithm for standard ICA with sample complexity nearly linear in the dimension, thereby improving substantially on previous bounds. The analysis is based on properties of random polynomials, namely the spacings of an ensemble of polynomials. Our technique also applies to other applications of tensor decompositions, including spherical Gaussian mixture models.

cs.DS↗
arXiv2014

Unsupervised Learning through Prediction in a Model of Cortex

We propose a primitive called PJOIN, for "predictive join," which combines and extends the operations JOIN and LINK, which Valiant proposed as the basis of a computational theory of cortex. We show that PJOIN can be implemented in Valiant's model. We also show that, using PJOIN, certain reasonably complex learning and pattern matching tasks can be performed, in a way that involves phenomena which have been observed in cognition and the brain, namely memory-based prediction and downward traffic in the cortical hierarchy.

cs.NE↗
arXiv2012

Structure from Local Optima: Learning Subspace Juntas via Higher Order PCA

We present a generalization of the well-known problem of learning k-juntas in R^n, and a novel tensor algorithm for unraveling the structure of high-dimensional distributions. Our algorithm can be viewed as a higher-order extension of Principal Component Analysis (PCA). Our motivating problem is learning a labeling function in R^n, which is determined by an unknown k-dimensional subspace. This problem of learning a k-subspace junta is a common generalization of learning a k-junta (a function of k coordinates in R^n) and learning intersections of k halfspaces. In this context, we introduce an irrelevant noisy attributes model where the distribution over the "relevant" k-dimensional subspace is independent of the distribution over the (n-k)-dimensional "irrelevant" subspace orthogonal to it. We give a spectral tensor algorithm which identifies the relevant subspace, and thereby learns k-subspace juntas under some additional assumptions. We do this by exploiting the structure of local optima of higher moment tensors over the unit sphere; PCA finds the global optima of the second moment tensor (covariance matrix). Our main result is that when the distribution in the irrelevant (n-k)-dimensional subspace is any Gaussian, the complexity of our algorithm is T(k,ε) + \poly(n), where T is the complexity of learning the concept in k dimensions, and the polynomial is a function of the k-dimensional concept class being learned. This substantially generalizes existing results on learning low-dimensional concepts.

cs.CC↗