arXiv subjects

Richard Peng

Publications and source records attributed to Richard Peng.

At least 55 records · Page 3Linked to original sources

arXiv2020

Fast, Provably convergent IRLS Algorithm for p-norm Linear Regression

Linear regression in $\ell_p$-norm is a canonical optimization problem that arises in several applications, including sparse recovery, semi-supervised learning, and signal processing. Generic convex optimization algorithms for solving $\ell_p$-regression are slow in practice. Iteratively Reweighted Least Squares (IRLS) is an easy to implement family of algorithms for solving these problems that has been studied for over 50 years. However, these algorithms often diverge for p > 3, and since the work of Osborne (1985), it has been an open problem whether there is an IRLS algorithm that is guaranteed to converge rapidly for p > 3. We propose p-IRLS, the first IRLS algorithm that provably converges geometrically for any $p \in [2,\infty).$ Our algorithm is simple to implement and is guaranteed to find a $(1+\varepsilon)$-approximate solution in $O(p^{3.5} m^{\frac{p-2}{2(p-1)}} \log \frac{m}{\varepsilon}) \le O_p(\sqrt{m} \log \frac{m}{\varepsilon} )$ iterations. Our experiments demonstrate that it performs even better than our theoretical bounds, beats the standard Matlab/CVX implementation for solving these problems by 10--50x, and is the fastest among available implementations in the high-accuracy regime.

cs.DS↗

arXiv2019

Vertex Sparsifiers for c-Edge Connectivity

We show the existence of O(f(c)k) sized vertex sparsifiers that preserve all edge-connectivity values up to c between a set of k terminal vertices, where f(c) is a function that only depends on c, the edge-connectivity value. This construction is algorithmic: we also provide an algorithm whose running time depends linearly on k, but exponentially in c. It implies that for constant values of c, an offline sequence of edge insertions/deletions and c-edge-connectivity queries can be answered in polylog time per operation. These results are obtained by combining structural results about minimum terminal separating cuts in undirected graphs with recent developments in expander decomposition based methods for finding small vertex/edge cuts in graphs.

cs.DS↗

arXiv2019

Deterministic Graph Cuts in Subquadratic Time: Sparse, Balanced, and k-Vertex

We study deterministic algorithms for computing graph cuts, with focus on two fundamental problems: balanced sparse cut and $k$-vertex connectivity for small $k$ ($k=O(\polylog n)$). Both problems can be solved in near-linear time with randomized algorithms, but their previous deterministic counterparts take at least quadratic time. In this paper, we break this bound for both problems. Interestingly, achieving this for one problem crucially relies on doing so for the other. In particular, via a divide-and-conquer argument, a variant of the cut-matching game by [Khandekar et al.`07], and the local vertex connectivity algorithm of [Nanongkai et al. STOC'19], we give a subquadratic time algorithm for $k$-vertex connectivity using a subquadratic time algorithm for computing balanced sparse cuts on sparse graphs. To achieve the latter, we improve the previously best $mn$ bound for approximating balanced sparse cut for the whole range of $m$. This starts from (1) breaking the $n^3$ barrier on dense graphs to $n^{ω+ o(1)}$ (where $ω< 2.372$) using the the PageRank matrix, but without explicitly sweeping to find sparse cuts; to (2) getting the $\tilde O(m^{1.58})$ bound by combining the $J$-trees by [Madry FOCS `10] with the $n^{ω+ o(1)}$ bound above, and finally; to (3) getting the $m^{1.5 + o(1)}$ bound by recursively invoking the second bound in conjunction with expander-based graph sparsification. Interestingly, our final $m^{1.5 + o(1)}$ bound lands at a natural stopping point in the sense that polynomially breaking it would lead to a breakthrough for the dynamic connectivity problem.

cs.DS↗

arXiv2019

Flowless: Extracting Densest Subgraphs Without Flow Computations

We propose a simple and computationally efficient method for dense subgraph discovery, which is a classic problem both in theory and in practice. It is well known that dense subgraphs can have strong correlation with structures of interest in real-world networks across various domains ranging from social networks to biological systems [Gionis and Tsourakakis `15]. For the densest subgraph problem, Charikar's greedy algorithm [Asashiro `00, Charikar `00] is very simple, and can typically find result of quality much better than the provable factor 2-approximation, which makes it very popular in practice. However, it is also known to give suboptimal output in many real-world examples. On the other hand, finding the exact optimal solution requires the computation of maximum flow. Despite the existence of highly optimized maximum flow solvers, such computation still incurs prohibitive computational costs for the massive graphs arising in modern data science applications. We devise an iterative algorithm which naturally generalizes the greedy algorithm of Charikar. Our algorithm draws insights from the iterative approaches from convex optimization, and also exploits the dual interpretation of the densest subgraph problem. We have empirical evidence that our algorithm is much more robust against the structural heterogeneities in real-world datasets, and converges to the optimal subgraph density even when the simple greedy algorithm fails. On the other hand, in instances where Charikar's algorithm performs well, our algorithm is able to quickly verify its optimality. Furthermore, we demonstrate that our method is significantly faster than the maximum flow based exact optimal algorithm. We conduct experiments on datasets from broad domains, and our algorithm achieves $\sim$145$\times$ speedup on average to find subgraphs whose density is at least 90\% of the optimal value.

cs.DS↗

arXiv2019

Parallel Batch-Dynamic Graphs: Algorithms and Lower Bounds

In this paper we study the problem of dynamically maintaining graph properties under batches of edge insertions and deletions in the massively parallel model of computation. In this setting, the graph is stored on a number of machines, each having space strongly sublinear with respect to the number of vertices, that is, $n^ε$ for some constant $0 < ε< 1$. Our goal is to handle batches of updates and queries where the data for each batch fits onto one machine in constant rounds of parallel computation, as well as to reduce the total communication between the machines. This objective corresponds to the gradual buildup of databases over time, while the goal of obtaining constant rounds of communication for problems in the static setting has been elusive for problems as simple as undirected graph connectivity. We give an algorithm for dynamic graph connectivity in this setting with constant communication rounds and communication cost almost linear in terms of the batch size. Our techniques combine a new graph contraction technique, an independent random sample extractor from correlated samples, as well as distributed data structures supporting parallel updates and queries in batches. We also illustrate the power of dynamic algorithms in the MPC model by showing that the batched version of the adaptive connectivity problem is $\mathsf{P}$-complete in the centralized setting, but sub-linear sized batches can be handled in a constant number of rounds. Due to the wide applicability of our approaches, we believe it represents a practically-motivated workaround to the current difficulties in designing more efficient massively parallel static graph algorithms.

cs.DS↗

arXiv2019

Dynamic Optimality Refuted -- For Tournament Heaps

We prove a separation between offline and online algorithms for finger-based tournament heaps undergoing key modifications. These heaps are implemented by binary trees with keys stored on leaves, and intermediate nodes tracking the min of their respective subtrees. They represent a natural starting point for studying self-adjusting heaps due to the need to access the root-to-leaf path upon modifications. We combine previous studies on the competitive ratios of unordered binary search trees by [Fredman WADS2011] and on order-by-next request by [Martínez-Roura TCS2000] and [Munro ESA2000] to show that for any number of fingers, tournament heaps cannot handle a sequence of modify-key operations with competitive ratio in $o(\sqrt{\log{n}})$. Critical to this analysis is the characterization of the modifications that a heap can undergo upon an access. There are $\exp(Θ(n \log{n}))$ valid heaps on $n$ keys, but only $\exp(Θ(n))$ binary search trees. We parameterize the modification power through the well-studied concept of fingers: additional pointers the data structure can manipulate arbitrarily. Here we demonstrate that fingers can be significantly more powerful than servers moving on a static tree by showing that access to $k$ fingers allow an offline algorithm to handle any access sequence with amortized cost $O(\log_{k}(n) + 2^{\lg^{*}n})$.

cs.DS↗

arXiv2019

Flows in Almost Linear Time via Adaptive Preconditioning

We present algorithms for solving a large class of flow and regression problems on unit weighted graphs to $(1 + 1 / poly(n))$ accuracy in almost-linear time. These problems include $\ell_p$-norm minimizing flow for $p$ large ($p \in [ω(1), o(\log^{2/3} n) ]$), and their duals, $\ell_p$-norm semi-supervised learning for $p$ close to $1$. As $p$ tends to infinity, $\ell_p$-norm flow and its dual tend to max-flow and min-cut respectively. Using this connection and our algorithms, we give an alternate approach for approximating undirected max-flow, and the first almost-linear time approximations of discretizations of total variation minimization objectives. This algorithm demonstrates that many tools previous viewed as limited to linear systems are in fact applicable to a much wider range of convex objectives. It is based on the the routing-based solver for Laplacian linear systems by Spielman and Teng (STOC '04, SIMAX '14), but require several new tools: adaptive non-linear preconditioning, tree-routing based ultra-sparsification for mixed $\ell_2$ and $\ell_p$ norm objectives, and decomposing graphs into uniform expanders.

cs.DS↗

arXiv2019

Fully Dynamic Spectral Vertex Sparsifiers and Applications

We study \emph{dynamic} algorithms for maintaining spectral vertex sparsifiers of graphs with respect to a set of terminals $T$ of our choice. Such objects preserve pairwise resistances, solutions to systems of linear equations, and energy of electrical flows between the terminals in $T$. We give a data structure that supports insertions and deletions of edges, and terminal additions, all in sublinear time. Our result is then applied to the following problems. (1) A data structure for maintaining solutions to Laplacian systems $\mathbf{L} \mathbf{x} = \mathbf{b}$, where $\mathbf{L}$ is the Laplacian matrix and $\mathbf{b}$ is a demand vector. For a bounded degree, unweighted graph, we support modifications to both $\mathbf{L}$ and $\mathbf{b}$ while providing access to $ε$-approximations to the energy of routing an electrical flow with demand $\mathbf{b}$, as well as query access to entries of a vector $\tilde{\mathbf{x}}$ such that $\left\lVert \tilde{\mathbf{x}}-\mathbf{L}^{\dagger} \mathbf{b} \right\rVert_{\mathbf{L}} \leq ε\left\lVert \mathbf{L}^{\dagger} \mathbf{b} \right\rVert_{\mathbf{L}}$ in $\tilde{O}(n^{11/12}ε^{-5})$ expected amortized update and query time. (2) A data structure for maintaining All-Pairs Effective Resistance. For an intermixed sequence of edge insertions, deletions, and resistance queries, our data structure returns $(1 \pm ε)$-approximation to all the resistance queries against an oblivious adversary with high probability. Its expected amortized update and query times are $\tilde{O}(\min(m^{3/4},n^{5/6} ε^{-2}) ε^{-4})$ on an unweighted graph, and $\tilde{O}(n^{5/6}ε^{-6})$ on weighted graphs. These results represent the first data structures for maintaining key primitives from the Laplacian paradigm for graph algorithms in sublinear time without assumptions on the underlying graph topologies.

cs.DS↗

arXiv2019

Higher-Order Accelerated Methods for Faster Non-Smooth Optimization

We provide improved convergence rates for various \emph{non-smooth} optimization problems via higher-order accelerated methods. In the case of $\ell_\infty$ regression, we achieves an $O(ε^{-4/5})$ iteration complexity, breaking the $O(ε^{-1})$ barrier so far present for previous methods. We arrive at a similar rate for the problem of $\ell_1$-SVM, going beyond what is attainable by first-order methods with prox-oracle access for non-smooth non-strongly convex problems. We further show how to achieve even faster rates by introducing higher-order regularization. Our results rely on recent advances in near-optimal accelerated methods for higher-order smooth convex optimization. In particular, we extend Nesterov's smoothing technique to show that the standard softmax approximation is not only smooth in the usual sense, but also \emph{higher-order} smooth. With this observation in hand, we provide the first example of higher-order acceleration techniques yielding faster rates for \emph{non-smooth} optimization, to the best of our knowledge.

math.OC↗

arXiv2019

Optimal Offline Dynamic $2,3$-Edge/Vertex Connectivity

We give offline algorithms for processing a sequence of $2$ and $3$ edge and vertex connectivity queries in a fully-dynamic undirected graph. While the current best fully-dynamic online data structures for $3$-edge and $3$-vertex connectivity require $O(n^{2/3})$ and $O(n)$ time per update, respectively, our per-operation cost is only $O(\log n)$, optimal due to the dynamic connectivity lower bound of Patrascu and Demaine. Our approach utilizes a divide and conquer scheme that transforms a graph into smaller equivalents that preserve connectivity information. This construction of equivalents is closely-related to the development of vertex sparsifiers, and shares important connections to several upcoming results in dynamic graph data structures, outside of just the offline model.

cs.DS↗

arXiv2019

Iterative Refinement for $\ell_p$-norm Regression

We give improved algorithms for the $\ell_{p}$-regression problem, $\min_{x} \|x\|_{p}$ such that $A x=b,$ for all $p \in (1,2) \cup (2,\infty).$ Our algorithms obtain a high accuracy solution in $\tilde{O}_{p}(m^{\frac{|p-2|}{2p + |p-2|}}) \le \tilde{O}_{p}(m^{\frac{1}{3}})$ iterations, where each iteration requires solving an $m \times m$ linear system, $m$ being the dimension of the ambient space. By maintaining an approximate inverse of the linear systems that we solve in each iteration, we give algorithms for solving $\ell_{p}$-regression to $1 / \text{poly}(n)$ accuracy that run in time $\tilde{O}_p(m^{\max\{ω, 7/3\}}),$ where $ω$ is the matrix multiplication constant. For the current best value of $ω> 2.37$, we can thus solve $\ell_{p}$ regression as fast as $\ell_{2}$ regression, for all constant $p$ bounded away from $1.$ Our algorithms can be combined with fast graph Laplacian linear equation solvers to give minimum $\ell_{p}$-norm flow / voltage solutions to $1 / \text{poly}(n)$ accuracy on an undirected graph with $m$ edges in $\tilde{O}_{p}(m^{1 + \frac{|p-2|}{2p + |p-2|}}) \le \tilde{O}_{p}(m^{\frac{4}{3}})$ time. For sparse graphs and for matrices with similar dimensions, our iteration counts and running times improve on the $p$-norm regression algorithm by [Bubeck-Cohen-Lee-Li STOC`18] and general-purpose convex optimization algorithms. At the core of our algorithms is an iterative refinement scheme for $\ell_{p}$-norms, using the smoothed $\ell_{p}$-norms introduced in the work of Bubeck et al. Given an initial solution, we construct a problem that seeks to minimize a quadratically-smoothed $\ell_{p}$ norm over a subspace, such that a crude solution to this problem allows us to improve the initial solution by a constant factor, leading to algorithms with fast convergence.

cs.DS↗

arXiv2018

Solving Directed Laplacian Systems in Nearly-Linear Time through Sparse LU Factorizations

We show how to solve directed Laplacian systems in nearly-linear time. Given a linear system in an $n \times n$ Eulerian directed Laplacian with $m$ nonzero entries, we show how to compute an $ε$-approximate solution in time $O(m \log^{O(1)} (n) \log (1/ε))$. Through reductions from [Cohen et al. FOCS'16] , this gives the first nearly-linear time algorithms for computing $ε$-approximate solutions to row or column diagonally dominant linear systems (including arbitrary directed Laplacians) and computing $ε$-approximations to various properties of random walks on directed graphs, including stationary distributions, personalized PageRank vectors, hitting times, and escape probabilities. These bounds improve upon the recent almost-linear algorithms of [Cohen et al. STOC'17], which gave an algorithm to solve Eulerian Laplacian systems in time $O((m+n2^{O(\sqrt{\log n \log \log n})})\log^{O(1)}(n ε^{-1}))$. To achieve our results, we provide a structural result that we believe is of independent interest. We show that Laplacians of all strongly connected directed graphs have sparse approximate LU-factorizations. That is, for every such directed Laplacian $ {\mathbf{L}}$, there is a lower triangular matrix $\boldsymbol{\mathit{\mathfrak{L}}}$ and an upper triangular matrix $\boldsymbol{\mathit{\mathfrak{U}}}$, each with at most $\tilde{O}(n)$ nonzero entries, such that their product $\boldsymbol{\mathit{\mathfrak{L}}} \boldsymbol{\mathit{\mathfrak{U}}}$ spectrally approximates $ {\mathbf{L}}$ in an appropriate norm. This claim can be viewed as an analogue of recent work on sparse Cholesky factorizations of Laplacians of undirected graphs. We show how to construct such factorizations in nearly-linear time and prove that, once constructed, they yield nearly-linear time algorithms for solving directed Laplacian systems.

cs.DS↗

arXiv2018

Constant Arboricity Spectral Sparsifiers

We show that every graph is spectrally similar to the union of a constant number of forests. Moreover, we show that Spielman-Srivastava sparsifiers are the union of O(logn) forests. This result can be used to estimate boundaries of small subsets of vertices in nearly optimal query time.

cs.DS↗

arXiv2018

Graph Sparsification, Spectral Sketches, and Faster Resistance Computation, via Short Cycle Decompositions

We develop a framework for graph sparsification and sketching, based on a new tool, short cycle decomposition -- a decomposition of an unweighted graph into an edge-disjoint collection of short cycles, plus few extra edges. A simple observation gives that every graph G on n vertices with m edges can be decomposed in $O(mn)$ time into cycles of length at most $2\log n$, and at most $2n$ extra edges. We give an $m^{1+o(1)}$ time algorithm for constructing a short cycle decomposition, with cycles of length $n^{o(1)}$, and $n^{1+o(1)}$ extra edges. These decompositions enable us to make progress on several open questions: * We give an algorithm to find $(1\pmε)$-approximations to effective resistances of all edges in time $m^{1+o(1)}ε^{-1.5}$, improving over the previous best of $\tilde{O}(\min\{mε^{-2},n^2 ε^{-1}\})$. This gives an algorithm to approximate the determinant of a Laplacian up to $(1\pmε)$ in $m^{1 + o(1)} + n^{15/8+o(1)}ε^{-7/4}$ time. * We show existence and efficient algorithms for constructing graphical spectral sketches -- a distribution over sparse graphs H such that for a fixed vector $x$, we have w.h.p. $x'L_Hx=(1\pmε)x'L_Gx$ and $x'L_H^+x=(1\pmε)x'L_G^+x$. This implies the existence of resistance-sparsifiers with about $nε^{-1}$ edges that preserve the effective resistances between every pair of vertices up to $(1\pmε).$ * By combining short cycle decompositions with known tools in graph sparsification, we show the existence of nearly-linear sized degree-preserving spectral sparsifiers, as well as significantly sparser approximations of directed graphs. The latter is critical to recent breakthroughs on faster algorithms for solving linear systems in directed Laplacians. Improved algorithms for constructing short cycle decompositions will lead to improvements for each of the above results.

cs.DS↗

arXiv2018

Incomplete Nested Dissection

We present an asymptotically faster algorithm for solving linear systems in well-structured 3-dimensional truss stiffness matrices. These linear systems arise from linear elasticity problems, and can be viewed as extensions of graph Laplacians into higher dimensions. Faster solvers for the 2-D variants of such systems have been studied using generalizations of tools for solving graph Laplacians [Daitch-Spielman CSC'07, Shklarski-Toledo SIMAX'08]. Given a 3-dimensional truss over $n$ vertices which is formed from a union of $k$ convex structures (tetrahedral meshes) with bounded aspect ratios, whose individual tetrahedrons are also in some sense well-conditioned, our algorithm solves a linear system in the associated stiffness matrix up to accuracy $ε$ in time $O(k^{1/3} n^{5/3} \log (1 / ε))$. This asymptotically improves the running time $O(n^2)$ by Nested Dissection for all $k \ll n$. We also give a result that improves on Nested Dissection even when we allow any aspect ratio for each of the $k$ convex structures (but we still require well-conditioned individual tetrahedrons). In this regime, we improve on Nested Dissection for $k \ll n^{1/44}$. The key idea of our algorithm is to combine nested dissection and support theory. Both of these techniques for solving linear systems are well studied, but usually separately. Our algorithm decomposes a 3-dimensional truss into separate and balanced regions with small boundaries. We then bound the spectrum of each such region separately, and utilize such bounds to obtain improved algorithms by preconditioning with partial states of separator-based Gaussian elimination.

cs.DS↗

arXiv2018

Fully Dynamic Effective Resistances

In this paper we consider the \emph{fully-dynamic} All-Pairs Effective Resistance problem, where the goal is to maintain effective resistances on a graph $G$ among any pair of query vertices under an intermixed sequence of edge insertions and deletions in $G$. The effective resistance between a pair of vertices is a physics-motivated quantity that encapsulates both the congestion and the dilation of a flow. It is directly related to random walks, and it has been instrumental in the recent works for designing fast algorithms for combinatorial optimization problems, graph sparsification, and network science. We give a data-structure that maintains $(1+ε)$-approximations to all-pair effective resistances of a fully-dynamic unweighted, undirected multi-graph $G$ with $\tilde{O}(m^{4/5}ε^{-4})$ expected amortized update and query time, against an oblivious adversary. Key to our result is the maintenance of a dynamic \emph{Schur complement}~(also known as vertex resistance sparsifier) onto a set of terminal vertices of our choice. This maintenance is obtained (1) by interpreting the Schur complement as a sum of random walks and (2) by randomly picking the vertex subset into which the sparsifier is constructed. We can then show that each update in the graph affects a small number of such walks, which in turn leads to our sub-linear update time. We believe that this local representation of vertex sparsifiers may be of independent interest.

cs.DS↗

arXiv2018

Graph Sketching Against Adaptive Adversaries Applied to the Minimum Degree Algorithm

Motivated by the study of matrix elimination orderings in combinatorial scientific computing, we utilize graph sketching and local sampling to give a data structure that provides access to approximate fill degrees of a matrix undergoing elimination in $O(\text{polylog}(n))$ time per elimination and query. We then study the problem of using this data structure in the minimum degree algorithm, which is a widely-used heuristic for producing elimination orderings for sparse matrices by repeatedly eliminating the vertex with (approximate) minimum fill degree. This leads to a nearly-linear time algorithm for generating approximate greedy minimum degree orderings. Despite extensive studies of algorithms for elimination orderings in combinatorial scientific computing, our result is the first rigorous incorporation of randomized tools in this setting, as well as the first nearly-linear time algorithm for producing elimination orderings with provable approximation guarantees. While our sketching data structure readily works in the oblivious adversary model, by repeatedly querying and greedily updating itself, it enters the adaptive adversarial model where the underlying sketches become prone to failure due to dependency issues with their internal randomness. We show how to use an additional sampling procedure to circumvent this problem and to create an independent access sequence. Our technique for decorrelating the interleaved queries and updates to this randomized data structure may be of independent interest.

cs.DS↗

arXiv2018

Current Flow Group Closeness Centrality for Complex Networks

Current flow closeness centrality (CFCC) has a better discriminating ability than the ordinary closeness centrality based on shortest paths. In this paper, we extend this notion to a group of vertices in a weighted graph, and then study the problem of finding a subset $S$ of $k$ vertices to maximize its CFCC $C(S)$, both theoretically and experimentally. We show that the problem is NP-hard, but propose two greedy algorithms for minimizing the reciprocal of $C(S)$ with provable guarantees using the monotoncity and supermodularity. The first is a deterministic algorithm with an approximation factor $(1-\frac{k}{k-1}\cdot\frac{1}{e})$ and cubic running time; while the second is a randomized algorithm with a $(1-\frac{k}{k-1}\cdot\frac{1}{e}-ε)$-approximation and nearly-linear running time for any $ε> 0$. Extensive experiments on model and real networks demonstrate that our algorithms are effective and efficient, with the second algorithm being scalable to massive networks with more than a million vertices.

cs.DS↗