arXiv subjects

Wenjing Liao

Publications and source records attributed to Wenjing Liao.

52 records · Page 3Linked to original sources

arXiv2022

Deep Nonparametric Estimation of Operators between Infinite Dimensional Spaces

Learning operators between infinitely dimensional spaces is an important learning task arising in wide applications in machine learning, imaging science, mathematical modeling and simulations, etc. This paper studies the nonparametric estimation of Lipschitz operators using deep neural networks. Non-asymptotic upper bounds are derived for the generalization error of the empirical risk minimizer over a properly chosen network class. Under the assumption that the target operator exhibits a low dimensional structure, our error bounds decay as the training sample size increases, with an attractive fast rate depending on the intrinsic dimension in our estimation. Our assumptions cover most scenarios in real applications and our results give rise to fast rates by exploiting low dimensional structures of data in operator estimation. We also investigate the influence of network structures (e.g., network width, depth, and sparsity) on the generalization error of the neural network estimator and propose a general suggestion on the choice of network structures to maximize the learning efficiency quantitatively.

stat.ML↗

arXiv2021

Learning functions varying along a central subspace

Many functions of interest are in a high-dimensional space but exhibit low-dimensional structures. This paper studies regression of a $s$-Hölder function $f$ in $\mathbb{R}^D$ which varies along a central subspace of dimension $d$ while $d\ll D$. A direct approximation of $f$ in $\mathbb{R}^D$ with an $\varepsilon$ accuracy requires the number of samples $n$ in the order of $\varepsilon^{-(2s+D)/s}$. In this paper, we analyze the Generalized Contour Regression (GCR) algorithm for the estimation of the central subspace and use piecewise polynomials for function approximation. GCR is among the best estimators for the central subspace, but its sample complexity is an open question. We prove that GCR leads to a mean squared estimation error of $O(n^{-1})$ for the central subspace, if a variance quantity is exactly known. The estimation error of this variance quantity is also given in this paper. The mean squared regression error of $f$ is proved to be in the order of $\left(n/\log n\right)^{-\frac{2s}{2s+d}}$ where the exponent depends on the dimension of the central subspace $d$ instead of the ambient space $D$. This result demonstrates that GCR is effective in learning the low-dimensional central subspace. We also propose a modified GCR with improved efficiency. The convergence rate is validated through several numerical experiments.

math.ST↗

arXiv2021

Besov Function Approximation and Binary Classification on Low-Dimensional Manifolds Using Convolutional Residual Networks

Most of existing statistical theories on deep neural networks have sample complexities cursed by the data dimension and therefore cannot well explain the empirical success of deep learning on high-dimensional data. To bridge this gap, we propose to exploit low-dimensional geometric structures of the real world data sets. We establish theoretical guarantees of convolutional residual networks (ConvResNet) in terms of function approximation and statistical estimation for binary classification. Specifically, given the data lying on a $d$-dimensional manifold isometrically embedded in $\mathbb{R}^D$, we prove that if the network architecture is properly chosen, ConvResNets can (1) approximate Besov functions on manifolds with arbitrary accuracy, and (2) learn a classifier by minimizing the empirical logistic risk, which gives an excess risk in the order of $n^{-\frac{s}{2s+2(s\vee d)}}$, where $s$ is a smoothness parameter. This implies that the sample complexity depends on the intrinsic dimension $d$, instead of the data dimension $D$. Our results demonstrate that ConvResNets are adaptive to low-dimensional structures of data sets.

stat.ML↗

arXiv2021

Multiscale regression on unknown manifolds

We consider the regression problem of estimating functions on $\mathbb{R}^D$ but supported on a $d$-dimensional manifold $ \mathcal{M} \subset \mathbb{R}^D $ with $ d \ll D $. Drawing ideas from multi-resolution analysis and nonlinear approximation, we construct low-dimensional coordinates on $\mathcal{M}$ at multiple scales, and perform multiscale regression by local polynomial fitting. We propose a data-driven wavelet thresholding scheme that automatically adapts to the unknown regularity of the function, allowing for efficient estimation of functions exhibiting nonuniform regularity at different locations and scales. We analyze the generalization error of our method by proving finite sample bounds in high probability on rich classes of priors. Our estimator attains optimal learning rates (up to logarithmic factors) as if the function was defined on a known Euclidean domain of dimension $d$, instead of an unknown manifold embedded in $\mathbb{R}^D$. The implemented algorithm has quasilinear complexity in the sample size, with constants linear in $D$ and exponential in $d$. Our work therefore establishes a new framework for regression on low-dimensional sets embedded in high dimensions, with fast implementation and strong theoretical guarantees.

stat.ML↗

arXiv2020

An auto-parameter denoising method for nuclear magnetic resonance spectroscopy based on low-rank Hankel matrix

Nuclear Magnetic Resonance (NMR) spectroscopy, which is modeled as the sum of damped exponential signals, has become an indispensable tool in various scenarios, such as the structure and function determination, chemical analysis, and disease diagnosis. NMR spectroscopy signals, however, are usually corrupted by Gaussian noise in practice, raising difficulties in sequential analysis and quantification of the signals. The low-rank Hankel property plays an important role in the denoising issue, but selecting an appropriate parameter still remains a problem. In this work, we explore the effect of the regularization parameter of a convex optimization denoising method based on low-rank Hankel matrices for exponential signals corrupted by Gaussian noise. An accurate estimate on the spectral norm of weighted Hankel matrices is provided as a guidance to set the regularization parameter. The bound can be efficiently calculated since it only depends on the standard deviation of the noise and a constant. Aided by the bound, one can easily obtain an auto-setting regularization parameter to produce promising denoised results. Our experiments on synthetic and realistic NMR spectroscopy data demonstrate a superior denoising performance of our proposed approach in comparison with the typical Cadzow and the state-of-the-art QR decomposition methods, especially in the low signal-to-noise ratio regime.

physics.med-ph↗

arXiv2020

Doubly Robust Off-Policy Learning on Low-Dimensional Manifolds by Deep Neural Networks

Causal inference explores the causation between actions and the consequent rewards on a covariate set. Recently deep learning has achieved a remarkable performance in causal inference, but existing statistical theories cannot well explain such an empirical success, especially when the covariates are high-dimensional. Most theoretical results in causal inference are asymptotic, suffer from the curse of dimensionality, and only work for the finite-action scenario. To bridge such a gap between theory and practice, this paper studies doubly robust off-policy learning by deep neural networks. When the covariates lie on a low-dimensional manifold, we prove nonasymptotic regret bounds, which converge at a fast rate depending on the intrinsic dimension of the manifold. Our results cover both the finite- and continuous-action scenarios. Our theory shows that deep neural networks are adaptive to the low-dimensional geometric structures of the covariates, and partially explains the success of deep learning for causal inference.

cs.LG↗

arXiv2019

Super-resolution limit of the ESPRIT algorithm

The problem of imaging point objects can be formulated as estimation of an unknown atomic measure from its $M+1$ consecutive noisy Fourier coefficients. The standard resolution of this inverse problem is $1/M$ and super-resolution refers to the capability of resolving atoms at a higher resolution. When any two atoms are less than $1/M$ apart, this recovery problem is highly challenging and many existing algorithms either cannot deal with this situation or require restrictive assumptions on the sign of the measure. ESPRIT is an efficient method that does not depend on the sign of the measure. This paper provides an explicit error bound on the support matching distance of ESPRIT in terms of the minimum singular value of Vandermonde matrices. When the support consists of multiple well-separated clumps and noise is sufficiently small, the support error by ESPRIT scales like ${\rm SRF}^{2λ+2} \times {\rm Noise}$, where the Super-Resolution Factor (${\rm SRF}$) governs the difficulty of the problem and $λ$ is the cardinality of the largest clump. {If the support contains one clump of closely spaced atoms, the min-max error is ${\rm SRF}^{2λ+2} \times {\rm Noise}/M$. Our error bound matches the min-max rate up to a factor of $M$ in the small noise regime. Our results therefore establishes the near-optimality of ESPRIT,} and our theory is validated by numerical experiments.

cs.IT↗

arXiv2019

Conditioning of restricted Fourier matrices and super-resolution of MUSIC

This paper studies stable recovery of a collection of point sources from its noisy $M+1$ low-frequency Fourier coefficients. We focus on the super-resolution regime where the minimum separation of the point sources is below $1/M$. We propose a separated clumps model where point sources are clustered in far apart sets, and prove an accurate lower bound of the Fourier matrix with nodes restricted to the source locations. This estimate gives rise to a theoretical analysis on the super-resolution limit of the MUSIC algorithm.

cs.IT↗

arXiv2019

IDENT: Identifying Differential Equations with Numerical Time evolution

Identifying unknown differential equations from a given set of discrete time dependent data is a challenging problem. A small amount of noise can make the recovery unstable, and nonlinearity and differential equations with varying coefficients add complexity to the problem. We assume that the governing partial differential equation (PDE) is a linear combination of a subset of a prescribed dictionary containing different differential terms, and the objective of this paper is to find the correct coefficients. We propose a new direction based on the fundamental idea of convergence analysis of numerical PDE schemes. We utilize Lasso for efficiency, and a performance guarantee is established based on an incoherence property. The main contribution is to validate and correct the results by Time Evolution Error (TEE). The new algorithm, called Identifying Differential Equations with Numerical Time evolution (IDENT), is explored for data with non-periodic boundary conditions, noisy data and PDEs with varying coefficients. From the recovery analysis of Lasso, we propose a new definition of Noise-to-Signal ratio, which better represents the level of noise in the case of PDE identification. We systematically analyze the effects of data generations and downsampling, and propose an order preserving denoising method called Least-Squares Moving Average (LSMA), to preprocess the given data. For the identification of PDEs with varying coefficients, we propose to add Base Element Expansion (BEE) to aide the computation. Various numerical experiments from basic tests to noisy data, downsampling effects and varying coefficients are presented.

math.NA↗

arXiv2019

Sensor Calibration for Off-the-Grid Spectral Estimation

This paper studies sensor calibration in spectral estimation where the true frequencies are located on a continuous domain. We consider a uniform array of sensors that collects measurements whose spectrum is composed of a finite number of frequencies, where each sensor has an unknown calibration parameter. Our goal is to recover the spectrum and the calibration parameters simultaneously from multiple snapshots of the measurements. In the noiseless case with an infinite number of snapshots, we prove uniqueness of this problem up to certain trivial, inevitable ambiguities based on an algebraic method, as long as there are more sensors than frequencies. We then analyze the sensitivity of this algebraic technique with respect to the number of snapshots and noise. We next propose an optimization approach that makes full use of the measurements by minimizing a non-convex objective which is non-negative and continuously differentiable over all calibration parameters and Toeplitz matrices. We prove that, in the case of infinite snapshots and noiseless measurements, the objective vanishes only at equivalent solutions to the true calibration parameters and the measurement covariance matrix. The objective is minimized using Wirtinger gradient descent which is proven to converge to a critical point. We show empirically that this critical point provides a good approximation of the true calibration parameters and the underlying frequencies.

cs.IT↗

arXiv2017

Adaptive Geometric Multiscale Approximations for Intrinsically Low-dimensional Data

We consider the problem of efficiently approximating and encoding high-dimensional data sampled from a probability distribution $ρ$ in $\mathbb{R}^D$, that is nearly supported on a $d$-dimensional set $\mathcal{M}$ - for example supported on a $d$-dimensional Riemannian manifold. Geometric Multi-Resolution Analysis (GMRA) provides a robust and computationally efficient procedure to construct low-dimensional geometric approximations of $\mathcal{M}$ at varying resolutions. We introduce a thresholding algorithm on the geometric wavelet coefficients, leading to what we call adaptive GMRA approximations. We show that these data-driven, empirical approximations perform well, when the threshold is chosen as a suitable universal function of the number of samples $n$, on a wide variety of measures $ρ$, that are allowed to exhibit different regularity at different scales and locations, thereby efficiently encoding data from more complex measures than those supported on manifolds. These approximations yield a data-driven dictionary, together with a fast transform mapping data to coefficients, and an inverse of such a map. The algorithms for both the dictionary construction and the transforms have complexity $C n \log n$ with the constant linear in $D$ and exponential in $d$. Our work therefore establishes adaptive GMRA as a fast dictionary learning algorithm with approximation guarantees. We include several numerical experiments on both synthetic and real data, confirming our theoretical results and demonstrating the effectiveness of adaptive GMRA.

stat.ML↗

arXiv2015

MUSIC for multidimensional spectral estimation: stability and super-resolution

This paper presents a performance analysis of the MUltiple SIgnal Classification (MUSIC) algorithm applied on $D$ dimensional single-snapshot spectral estimation while $s$ true frequencies are located on the continuum of a bounded domain. Inspired by the matrix pencil form, we construct a D-fold Hankel matrix from the measurements and exploit its Vandermonde decomposition in the noiseless case. MUSIC amounts to identifying a noise subspace, evaluating a noise-space correlation function, and localizing frequencies by searching the $s$ smallest local minima of the noise-space correlation function. In the noiseless case, $(2s)^D$ measurements guarantee an exact reconstruction by MUSIC as the noise-space correlation function vanishes exactly at true frequencies. When noise exists, we provide an explicit estimate on the perturbation of the noise-space correlation function in terms of noise level, dimension $D$, the minimum separation among frequencies, the maximum and minimum amplitudes while frequencies are separated by two Rayleigh Length (RL) at each direction. As a by-product the maximum and minimum non-zero singular values of the multidimensional Vandermonde matrix whose nodes are on the unit sphere are estimated under a gap condition of the nodes. Under the 2-RL separation condition, if noise is i.i.d. gaussian, we show that perturbation of the noise-space correlation function decays like $\sqrt{\log(\#(\mathbf{N}))/\#(\mathbf{N})}$ as the sample size $\#(\mathbf{N})$ increases. When the separation among frequencies drops below 2 RL, our numerical experiments show that the noise tolerance of MUSIC obeys a power law with the minimum separation of frequencies.

cs.IT↗

arXiv2014

MUSIC for Single-Snapshot Spectral Estimation: Stability and Super-resolution

This paper studies the problem of line spectral estimation in the continuum of a bounded interval with one snapshot of array measurement. The single-snapshot measurement data is turned into a Hankel data matrix which admits the Vandermonde decomposition and is suitable for the MUSIC algorithm. The MUSIC algorithm amounts to finding the null space (the noise space) of the Hankel matrix, forming the noise-space correlation function and identifying the s smallest local minima of the noise-space correlation as the frequency set. In the noise-free case exact reconstruction is guaranteed for any arbitrary set of frequencies as long as the number of measurements is at least twice the number of distinct frequencies to be recovered. In the presence of noise the stability analysis shows that the perturbation of the noise-space correlation is proportional to the spectral norm of the noise matrix as long as the latter is smaller than the smallest (nonzero) singular value of the noiseless Hankel data matrix. Under the assumption that frequencies are separated by at least twice the Rayleigh Length (RL), the stability of the noise-space correlation is proved by means of novel discrete Ingham inequalities which provide bounds on nonzero singular values of the noiseless Hankel data matrix. The numerical performance of MUSIC is tested in comparison with other algorithms such as BLO-OMP and SDP (TV-min). While BLO-OMP is the stablest algorithm for frequencies separated above 4 RL, MUSIC becomes the best performing one for frequencies separated between 2 RL and 3 RL. Also, MUSIC is more efficient than other methods. MUSIC truly shines when the frequency separation drops to 1 RL or below when all other methods fail. Indeed, the resolution length of MUSIC decreases to zero as noise decreases to zero as a power law with an exponent much smaller than an upper bound established by Donoho.

cs.IT↗

arXiv2013

Fourier phasing with phase-uncertain mask

Fourier phasing is the problem of retrieving Fourier phase information from Fourier intensity data. The standard Fourier phase retrieval (without a mask) is known to have many solutions which cause the standard phasing algorithms to stagnate and produce wrong or inaccurate solutions. In this paper Fourier phase retrieval is carried out with the introduction of a randomly fabricated mask in measurement and reconstruction. Highly probable uniqueness of solution, up to a global phase, was previously proved with exact knowledge of the mask. Here the uniqueness result is extended to the case where only rough information about the mask's phases is assumed. The exponential probability bound for uniqueness is given in terms of the uncertainty-to-diversity ratio (UDR) of the unknown mask. New phasing algorithms alternating between the object update and the mask update are systematically tested and demonstrated to have the capability of recovering both the object and the mask (within the object support) simultaneously, consistent with the uniqueness result. Phasing with a phase-uncertain mask is shown to be robust with respect to the correlation in the mask as well as the Gaussian and Poisson noises.

physics.optics↗

arXiv2012

Phase Retrieval with Random Phase Illumination

This paper presents a detailed, numerical study on the performance of the standard phasing algorithms with random phase illumination (RPI). Phasing with high resolution RPI and the oversampling ratio $σ=4$ determines a unique phasing solution up to a global phase factor. Under this condition, the standard phasing algorithms converge rapidly to the true solution without stagnation. Excellent approximation is achieved after a small number of iterations, not just with high resolution but also low resolution RPI in the presence of additive as well multiplicative noises. It is shown that RPI with $σ=2$ is sufficient for phasing complex-valued images under a sector condition and $σ=1$ for phasing nonnegative images. The Error Reduction algorithm with RPI is proved to converge to the true solution under proper conditions.

physics.optics↗

arXiv2011

Mismatch and resolution in compressive imaging

Highly coherent sensing matrices arise in discretization of continuum problems such as radar and medical imaging when the grid spacing is below the Rayleigh threshold as well as in using highly coherent, redundant dictionaries as sparsifying operators. Algorithms (BOMP, BLOOMP) based on techniques of band exclusion and local optimization are proposed to enhance Orthogonal Matching Pursuit (OMP) and deal with such coherent sensing matrices. BOMP and BLOOMP have provably performance guarantee of reconstructing sparse, widely separated objects {\em independent} of the redundancy and have a sparsity constraint and computational cost similar to OMP's. Numerical study demonstrates the effectiveness of BLOOMP for compressed sensing with highly coherent, redundant sensing matrices.

cs.IT↗