arXiv subjects

Ming-Jun Lai

Publications and source records attributed to Ming-Jun Lai.

34 records · Page 2Linked to original sources

arXiv2021

Graph Sparsification by Universal Greedy Algorithms

Graph sparsification is to approximate an arbitrary graph by a sparse graph and is useful in many applications, such as simplification of social networks, least squares problems, numerical solution of symmetric positive definite linear systems and etc. In this paper, inspired by the well-known sparse signal recovery algorithm called orthogonal matching pursuit (OMP), we introduce a deterministic, greedy edge selection algorithm called universal greedy approach (UGA) for graph sparsification. For a general spectral sparsification problem, e.g., positive subset selection problem from a set of $m$ vectors from $\mathbb{R}^n$, we propose a nonnegative UGA algorithm which needs $O(mn^2+ n^3/ε^2)$ time to find a $\frac{1+ε/β}{1-ε/β}$-spectral sparsifier with positive coefficients with sparsity $\le\lceil\frac{n}{ε^2}\rceil$, where $β$ is the ratio between the smallest length and largest length of the vectors. The convergence of the nonnegative UGA algorithm will be established. For the graph sparsification problem, another UGA algorithm will be proposed which can output a $\frac{1+O(ε)}{1-O(ε)}$-spectral sparsifier with $\lceil\frac{n}{ε^2}\rceil$ edges in $O(m+n^2/ε^2)$ time from a graph with $m$ edges and $n$ vertices under some mild assumptions. This is a linear time algorithm in terms of the number of edges that the community of graph sparsification is looking for. The best result in the literature to the knowledge of the authors is the existence of a deterministic algorithm which is almost linear, i.e. $O(m^{1+o(1)})$ for some $o(1)=O(\frac{(\log\log(m))^{2/3}}{\log^{1/3}(m)})$. Finally, extensive experimental results, including applications to graph clustering and least squares regression, show the effectiveness of proposed approaches.

cs.DS↗

arXiv2020

A Quasi-Orthogonal Matching Pursuit Algorithm for Compressive Sensing

In this paper, we propose a new orthogonal matching pursuit algorithm called quasi-OMP algorithm which greatly enhances the performance of classical orthogonal matching pursuit (OMP) algorithm, at some cost of computational complexity. We are able to show that under some sufficient conditions of mutual coherence of the sensing matrix, the QOMP Algorithm succeeds in recovering the s-sparse signal vector x within s iterations where a total number of 2s columns are selected under the both noiseless and noisy settings. In addition, we show that for Gaussian sensing matrix, the norm of the residual of each iteration will go to zero linearly depends on the size of the matrix with high probability. The numerical experiments are demonstrated to show the effectiveness of QOMP algorithm in recovering sparse solutions which outperforms the classic OMP and GOMP algorithm.

math.NA↗

arXiv2020

Compressive Sensing for cut improvement and local clustering

We show how one can phrase the cut improvement problem for graphs as a sparse recovery problem, whence one can use algorithms originally developed for use in compressive sensing (such as SubspacePursuit or CoSaMP) to solve it. We show that this approach to cut improvement is fast, both in theory and practice and moreover enjoys statistical guarantees of success when applied to graphs drawn from probabilistic models such as the Stochastic Block Model. Using this new cut improvement approach, which we call ClusterPursuit, as an algorithmic primitive we then propose new methods for local clustering and semi-supervised clustering, which enjoy similar guarantees of success and speed. Finally, we verify the promise of our approach with extensive numerical benchmarking.

cs.IT↗

arXiv2018

On DC based Methods for Phase Retrieval

In this paper, we develop a new computational approach which is based on minimizing the difference of two convex functionals (DC) to solve a broader class of phase retrieval problems. The approach splits a standard nonlinear least squares minimizing function associated with the phase retrieval problem into the difference of two convex functions and then solves a sequence of convex minimization sub-problems. For each subproblem, the Nesterov's accelerated gradient descent algorithm or the Barzilai-Borwein (BB) algorithm is used. In the setting of sparse phase retrieval, a standard $\ell_1$ norm term is added into the minimization mentioned above. The subproblem is approximated by a proximal gradient method which is solved by the shrinkage-threshold technique directly without iterations. In addition, a modified Attouch-Peypouquet technique is used to accelerate the iterative computation. These lead to more effective algorithms than the Wirtinger flow (WF) algorithm and the Gauss-Newton (GN) algorithm and etc.. A convergence analysis of both DC based algorithms shows that the iterative solutions is convergent linearly to a critical point and will be closer to a global minimizer than the given initial starting point. Our study is a deterministic analysis while the study for the Wirtinger flow (WF) algorithm and its variants, the Gauss-Newton (GN) algorithm, the trust region algorithm is based on the probability analysis. In particular, the DC based algorithms are able to retrieve solutions using a number $m$ of measurements which is about twice of the number $n$ of entries in the solution with high frequency of successes. When $m\approx n$, the $\ell_1$ DC based algorithm is able to retrieve sparse signals.

cs.IT↗

arXiv2018

A Compressive Sensing Approach to Community Detection with Applications

The community detection problem for graphs asks one to partition the n vertices V of a graph G into k communities, or clusters, such that there are many intracluster edges and few intercluster edges. Of course this is equivalent to finding a permutation matrix P such that, if A denotes the adjacency matrix of G, then PAP^T is approximately block diagonal. As there are k^n possible partitions of n vertices into k subsets, directly determining the optimal clustering is clearly infeasible. Instead one seeks to solve a more tractable approximation to the clustering problem. In this paper we reformulate the community detection problem via sparse solution of a linear system associated with the Laplacian of a graph G and then develop a two-stage approach based on a thresholding technique and a compressive sensing algorithm to find a sparse solution which corresponds to the community containing a vertex of interest in G. Crucially, our approach results in an algorithm which is able to find a single cluster of size n_0 in O(nlog(n)n_0) operations and all k clusters in fewer than O(n^2ln(n)) operations. This is a marked improvement over the classic spectral clustering algorithm, which is unable to find a single cluster at a time and takes approximately O(n^3) operations to find all k clusters. Moreover, we are able to provide robust guarantees of success for the case where G is drawn at random from the Stochastic Block Model, a popular model for graphs with clusters. Extensive numerical results are also provided, showing the efficacy of our algorithm on both synthetic and real-world data sets.

cs.IT↗

arXiv2018

Sharp Sufficient Conditions for Stable Recovery of Block Sparse Signals by Block Orthogonal Matching Pursuit

In this paper, we use the block orthogonal matching pursuit (BOMP) algorithm to recover block sparse signals $\x$ from measurements $\y=\A\x+\v$, where $\v$ is an $\ell_2$-bounded noise vector (i.e., $\|\v\|_2\leq ε$ for some constant $ε$). We investigate some sufficient conditions based on the block restricted isometry property (block-RIP) for exact (when $\v=\0$) and stable (when $\v\neq\0$) recovery of block sparse signals $\x$. First, on the one hand, we show that if $\A$ satisfies the block-RIP with $δ_{K+1}<1/\sqrt{K+1}$, then every block $K$-sparse signal $\x$ can be exactly or stably recovered by BOMP in $K$ iterations. On the other hand, we show that, for any $K\geq 1$ and $1/\sqrt{K+1}\leq δ<1$, there exists a matrix $\A$ satisfying the block-RIP with $δ_{K+1}=δ$ and a block $K$-sparse signal $\x$ such that BOMP may fail to recover $\x$ in $K$ iterations. Then, we study some sufficient conditions for recovering block $α$-strongly-decaying $K$-sparse signals. We show that if $\A$ satisfies the block-RIP with $δ_{K+1}<\sqrt{2}/2$, then every $α$-strongly-decaying block $K$-sparse signal can be exactly or stably recovered by BOMP in $K$ iterations under some conditions on $α$. Our newly found sufficient condition on the block-RIP of $\A$ is less restrictive than that for $\ell_1$ minimization for this special class of sparse signals. Furthermore, for any $K\geq 1$, $α>1$ and $\sqrt{2}/2\leq δ<1$, the recovery of $\x$ may fail in $K$ iterations for a sensing matrix $\A$ which satisfies the block-RIP with $δ_{K+1}=δ$. Finally, we study some sufficient conditions for partial recovery of block sparse signals. Specifically, if $\A$ satisfies the block-RIP with $δ_{K+1}<\sqrt{2}/2$, then BOMP is guaranteed to recover some blocks of $\x$ if these blocks satisfy a sufficient condition.

cs.IT↗

arXiv2017

A Bivariate Spline Method for Second Order Elliptic Equations in Non-Divergence Form

A bivariate spline method is developed to numerically solve second order elliptic partial differential equations (PDE) in non-divergence form. The existence, uniqueness, stability as well as approximation properties of the discretized solution will be established by using the well-known Ladyzhenskaya-Babuska-Brezzi (LBB) condition. Bivariate splines, discontinuous splines with smoothness constraints are used to implement the method. A plenty of computational results based on splines of various degrees are presented to demonstrate the effectiveness and efficiency of our method.

math.NA↗

arXiv2015

Efficient spatial modelling using the SPDE approach with bivariate splines

Gaussian fields (GFs) are frequently used in spatial statistics for their versatility. The associated computational cost can be a bottleneck, especially in realistic applications. It has been shown that computational efficiency can be gained by doing the computations using Gaussian Markov random fields (GMRFs) as the GFs can be seen as weak solutions to corresponding stochastic partial differential equations (SPDEs) using piecewise linear finite elements. We introduce a new class of representations of GFs with bivariate splines instead of finite elements. This allows an easier implementation of piecewise polynomial representations of various degrees. It leads to GMRFs that can be inferred efficiently and can be easily extended to non-stationary fields. The solutions approximated with higher order bivariate splines converge faster, hence the computational cost can be alleviated. Numerical simulations using both real and simulated data also demonstrate that our framework increases the flexibility and efficiency.

stat.CO↗

arXiv2014

Stochastic Coordinate Coding and Its Application for Drosophila Gene Expression Pattern Annotation

\textit{Drosophila melanogaster} has been established as a model organism for investigating the fundamental principles of developmental gene interactions. The gene expression patterns of \textit{Drosophila melanogaster} can be documented as digital images, which are annotated with anatomical ontology terms to facilitate pattern discovery and comparison. The automated annotation of gene expression pattern images has received increasing attention due to the recent expansion of the image database. The effectiveness of gene expression pattern annotation relies on the quality of feature representation. Previous studies have demonstrated that sparse coding is effective for extracting features from gene expression images. However, solving sparse coding remains a computationally challenging problem, especially when dealing with large-scale data sets and learning large size dictionaries. In this paper, we propose a novel algorithm to solve the sparse coding problem, called Stochastic Coordinate Coding (SCC). The proposed algorithm alternatively updates the sparse codes via just a few steps of coordinate descent and updates the dictionary via second order stochastic gradient descent. The computational cost is further reduced by focusing on the non-zero components of the sparse codes and the corresponding columns of the dictionary only in the updating procedure. Thus, the proposed algorithm significantly improves the efficiency and the scalability, making sparse coding applicable for large-scale data sets and large dictionary sizes. Our experiments on Drosophila gene expression data sets demonstrate the efficiency and the effectiveness of the proposed algorithm.

cs.LG↗

arXiv2014

Orthogonal Rank-One Matrix Pursuit for Low Rank Matrix Completion

In this paper, we propose an efficient and scalable low rank matrix completion algorithm. The key idea is to extend orthogonal matching pursuit method from the vector case to the matrix case. We further propose an economic version of our algorithm by introducing a novel weight updating rule to reduce the time and storage complexity. Both versions are computationally inexpensive for each matrix pursuit iteration, and find satisfactory results in a few iterations. Another advantage of our proposed algorithm is that it has only one tunable parameter, which is the rank. It is easy to understand and to use by the user. This becomes especially important in large-scale learning problems. In addition, we rigorously show that both versions achieve a linear convergence rate, which is significantly better than the previous known results. We also empirically compare the proposed algorithms with several state-of-the-art matrix completion algorithms on many real-world datasets, including the large-scale recommendation dataset Netflix as well as the MovieLens datasets. Numerical results show that our proposed algorithm is more efficient than competing algorithms while achieving similar or better prediction performance.

cs.LG↗

arXiv2014

Parallel Multi-Block ADMM with o(1/k) Convergence

This paper introduces a parallel and distributed extension to the alternating direction method of multipliers (ADMM) for solving convex problem: minimize $\sum_{i=1}^N f_i(x_i)$ subject to $\sum_{i=1}^N A_i x_i=c, x_i\in \mathcal{X}_i$. The algorithm decomposes the original problem into N smaller subproblems and solves them in parallel at each iteration. This Jacobian-type algorithm is well suited for distributed computing and is particularly attractive for solving certain large-scale problems. This paper introduces a few novel results. Firstly, it shows that extending ADMM straightforwardly from the classic Gauss-Seidel setting to the Jacobian setting, from 2 blocks to N blocks, will preserve convergence if matrices $A_i$ are mutually near-orthogonal and have full column-rank. Secondly, for general matrices $A_i$, this paper proposes to add proximal terms of different kinds to the N subproblems so that the subproblems can be solved in flexible and efficient ways and the algorithm converges globally at a rate of o(1/k). Thirdly, a simple technique is introduced to improve some existing convergence rates from O(1/k) to o(1/k). In practice, some conditions in our convergence theorems are conservative. Therefore, we introduce a strategy for dynamically tuning the parameters in the algorithm, leading to substantial acceleration of the convergence in practice. Numerical results are presented to demonstrate the efficiency of the proposed method in comparison with several existing parallel algorithms. We implemented our algorithm on Amazon EC2, an on-demand public computing cloud, and report its performance on very large-scale basis pursuit problems with distributed data.

math.OC↗

arXiv2013

Augmented L1 and Nuclear-Norm Models with a Globally Linearly Convergent Algorithm

This paper studies the long-existing idea of adding a nice smooth function to "smooth" a non-differentiable objective function in the context of sparse optimization, in particular, the minimization of $||x||_1+1/(2α)||x||_2^2$, where $x$ is a vector, as well as the minimization of $||X||_*+1/(2α)||X||_F^2$, where $X$ is a matrix and $||X||_*$ and $||X||_F$ are the nuclear and Frobenius norms of $X$, respectively. We show that they can efficiently recover sparse vectors and low-rank matrices. In particular, they enjoy exact and stable recovery guarantees similar to those known for minimizing $||x||_1$ and $||X||_*$ under the conditions on the sensing operator such as its null-space property, restricted isometry property, spherical section property, or RIPless property. To recover a (nearly) sparse vector $x^0$, minimizing $||x||_1+1/(2α)||x||_2^2$ returns (nearly) the same solution as minimizing $||x||_1$ almost whenever $α\ge 10||x^0||_\infty$. The same relation also holds between minimizing $||X||_*+1/(2α)||X||_F^2$ and minimizing $||X||_*$ for recovering a (nearly) low-rank matrix $X^0$, if $α\ge 10||X^0||_2$. Furthermore, we show that the linearized Bregman algorithm for minimizing $||x||_1+1/(2α)||x||_2^2$ subject to $Ax=b$ enjoys global linear convergence as long as a nonzero solution exists, and we give an explicit rate of convergence. The convergence property does not require a solution solution or any properties on $A$. To our knowledge, this is the best known global convergence result for first-order sparse optimization algorithms.

cs.IT↗

arXiv2013

Convergence Analysis of a Finite Difference Scheme for the Gradient Flow associated with the ROF Model

We present a convergence analysis of a finite difference scheme for the time dependent partial different equation called gradient flow associated with the Rudin-Osher-Fatemi model. We devise an iterative algorithm to compute the solution of the finite difference scheme and prove the convergence of the iterative algorithm. Finally computational experiments are shown to demonstrate the convergence of the finite difference scheme. An application for image denoising is given.

math.NA↗

arXiv2011

Construction of bivariate symmetric orthonormal wavelets with short support

In this paper, we give a parameterization of the class of bivariate symmetric orthonormal scaling functions with filter size $6\times 6$ using the standard dilation matrix 2I. In addition, we give two families of refinable functions which are not orthonormal but have associated tight frames. Finally, we show that the class of bivariate symmetric scaling functions with filter size $8\times 8$ can not have two or more vanishing moments.

math.NA↗

arXiv2007

Prewavelet Solution to Poisson Equations

Finite element method is one of powerful numerical methods to solve PDE. Usually, if a finite element solution to a Poisson equation based on a triangulation of the underlying domain is not accurate enough, one will discard the solution and then refine the triangulation uniformly and compute a new finite element solution over the refined triangulation. It is wasteful to discard the original finite element solution. We propose a prewavelet method to save the original solution by adding a prewavelet subsolution to obtain the refined level finite element solution. To increase the accuracy of numerical solution to Poisson equations, we can keep adding prewavelet subsolutions. Our prewavelets are orthogonal in the $H^1$ norm and they are compactly supported except for one globally supported basis function in a rectangular domain. We have implemented these prewavelet basis functions in MATLAB and used them for numerical solution of Poisson equation with Dirichlet boundary conditions. Numerical simulation demonstrates that our prewavelet solution is much more efficient than the standard finite element method.

math.NA↗

arXiv2005

Factorization of Multivariate Positive Laurent Polynomials

Recently Dritschel proves that any positive multivariate Laurent polynomial can be factorized into a sum of square magnitudes of polynomials. We first give another proof of the Dritschel theorem. Our proof is based on the univariate matrix Fejer-Riesz theorem. Then we discuss a computational method to find approximates of polynomial matrix factorization. Some numerical examples will be shown. Finally we discuss how to compute nonnegative Laurent polynomial factorizations in the multivariate setting.

math.CA↗