arXiv subjects

Justin Romberg

Publications and source records attributed to Justin Romberg.

At least 73 records · Page 4Linked to original sources

arXiv2018

Distributed Stochastic Approximation for Solving Network Optimization Problems Under Random Quantization

We study distributed optimization problems over a network when the communication between the nodes is constrained, and so information that is exchanged between the nodes must be quantized. This imperfect communication poses a fundamental challenge, and this imperfect communication, if not properly accounted for, prevents the convergence of these algorithms. Our first contribution in this paper is to propose a modified consensus-based gradient method for solving such problems using random (dithered) quantization. This algorithm can be interpreted as a distributed variant of a well-known two-time-scale stochastic algorithm. We then study the convergence and derive upper bounds on the rates of convergence of the proposed method as a function of the bandwidths available between the nodes and the underlying network topology, for both convex and strongly convex objective functions. Our results complement for existing literature where such convergence and explicit formulas of the convergence rates are missing. Finally, we provide numerical simulations to compare the convergence properties of the distributed gradient methods with and without quantization for solving the well-known regression problems over networks, for both quadratic and absolute loss functions.

math.OC↗

arXiv2018

A Hardware Realization of Superresolution Combining Random Coding and Blurring

Resolution enhancements are often desired in imaging applications where high-resolution sensor arrays are difficult to obtain. Many computational imaging methods have been proposed to encode high-resolution scene information on low-resolution sensors by cleverly modulating light from the scene before it hits the sensor. These methods often require movement of some portion of the imaging apparatus or only acquire images up to the resolution of a modulating element. Here a technique is presented for resolving beyond the resolutions of both a pointwise-modulating mask element and a sensor array through the introduction of a controlled blur into the optical pathway. The analysis contains an intuitive and exact expression for the overall superresolvability of the system, and arguments are presented to explain how the combination of random coding and blurring makes the superresolution problem well-posed. Experimental results demonstrate that a resolution enhancement of approximately $4\times$ is possible in practice using standard optical components, without mechanical motion of the imaging apparatus, and without any a priori assumptions on scene structure.

eess.IV↗

arXiv2018

ROAST: Rapid Orthogonal Approximate Slepian Transform

In this paper, we provide a Rapid Orthogonal Approximate Slepian Transform (ROAST) for the discrete vector that one obtains when collecting a finite set of uniform samples from a baseband analog signal. The ROAST offers an orthogonal projection which is an approximation to the orthogonal projection onto the leading discrete prolate spheroidal sequence (DPSS) vectors (also known as Slepian basis vectors). As such, the ROAST is guaranteed to accurately and compactly represent not only oversampled bandlimited signals but also the leading DPSS vectors themselves. Moreover, the subspace angle between the ROAST subspace and the corresponding DPSS subspace can be made arbitrarily small. The complexity of computing the representation of a signal using the ROAST is comparable to the FFT, which is much less than the complexity of using the DPSS basis vectors. We also give non-asymptotic results to guarantee that the proposed basis not only provides a very high degree of approximation accuracy in a mean squared error sense for bandlimited sample vectors, but also that it can provide high-quality approximations of all sampled sinusoids within the band of interest.

cs.IT↗

arXiv2018

Solving Equations of Random Convex Functions via Anchored Regression

We consider the question of estimating a solution to a system of equations that involve convex nonlinearities, a problem that is common in machine learning and signal processing. Because of these nonlinearities, conventional estimators based on empirical risk minimization generally involve solving a non-convex optimization program. We propose anchored regression, a new approach based on convex programming that amounts to maximizing a linear functional (perhaps augmented by a regularizer) over a convex set. The proposed convex program is formulated in the natural space of the problem, and avoids the introduction of auxiliary variables, making it computationally favorable. Working in the native space also provides great flexibility as structural priors (e.g., sparsity) can be seamlessly incorporated. For our analysis, we model the equations as being drawn from a fixed set according to a probability law. Our main results provide guarantees on the accuracy of the estimator in terms of the number of equations we are solving, the amount of noise present, a measure of statistical complexity of the random equations, and the geometry of the regularizer at the true solution. We also provide recipes for constructing the anchor vector (that determines the linear functional to maximize) directly from the observed data.

cs.LG↗

arXiv2018

Blind Deconvolution using Convex Programming

We consider the problem of recovering two unknown vectors, $\boldsymbol{w}$ and $\boldsymbol{x}$, of length $L$ from their circular convolution. We make the structural assumption that the two vectors are members of known subspaces, one with dimension $N$ and the other with dimension $K$. Although the observed convolution is nonlinear in both $\boldsymbol{w}$ and $\boldsymbol{x}$, it is linear in the rank-1 matrix formed by their outer product $\boldsymbol{w}\boldsymbol{x}^*$. This observation allows us to recast the deconvolution problem as low-rank matrix recovery problem from linear measurements, whose natural convex relaxation is a nuclear norm minimization program. We prove the effectiveness of this relaxation by showing that for "generic" signals, the program can deconvolve $\boldsymbol{w}$ and $\boldsymbol{x}$ exactly when the maximum of $N$ and $K$ is almost on the order of $L$. That is, we show that if $\boldsymbol{x}$ is drawn from a random subspace of dimension $N$, and $\boldsymbol{w}$ is a vector in a subspace of dimension $K$ whose basis vectors are "spread out" in the frequency domain, then nuclear norm minimization recovers $\boldsymbol{w}\boldsymbol{x}^*$ without error. We discuss this result in the context of blind channel estimation in communications. If we have a message of length $N$ which we code using a random $L\times N$ coding matrix, and the encoded message travels through an unknown linear time-invariant channel of maximum length $K$, then the receiver can recover both the channel response and the message when $L\gtrsim N+K$, to within constant and log factors.

cs.IT↗

arXiv2018

Compressive Multiplexing of Correlated Signals

We present a general architecture for the acquisition of ensembles of correlated signals. The signals are multiplexed onto a single line by mixing each one against a different code and then adding them together, and the resulting signal is sampled at a high rate. We show that if the $M$ signals, each bandlimited to $W/2$ Hz, can be approximated by a superposition of $R < M$ underlying signals, then the ensemble can be recovered by sampling at a rate within a logarithmic factor of $RW$ (as compared to the Nyquist rate of $MW$). This sampling theorem shows that the correlation structure of the signal ensemble can be exploited in the acquisition process even though it is unknown a priori. The reconstruction of the ensemble is recast as a low-rank matrix recovery problem from linear measurements. The architectures we are considering impose a certain type of structure on the linear operators. Although our results depend on the mixing forms being random, this imposed structure results in a very different type of random projection than those analyzed in the low-rank recovery literature to date.

cs.IT↗

arXiv2018

Compressive Sampling of Ensembles of Correlated Signals

We propose several sampling architectures for the efficient acquisition of an ensemble of correlated signals. We show that without prior knowledge of the correlation structure, each of our architectures (under different sets of assumptions) can acquire the ensemble at a sub-Nyquist rate. Prior to sampling, the analog signals are diversified using simple, implementable components. The diversification is achieved by injecting types of "structured randomness" into the ensemble, the result of which is subsampled. For reconstruction, the ensemble is modeled as a low-rank matrix that we have observed through an (undetermined) set of linear equations. Our main results show that this matrix can be recovered using standard convex programming techniques when the total number of samples is on the order of the intrinsic degree of freedom of the ensemble --- the more heavily correlated the ensemble, the fewer samples are needed. To motivate this study, we discuss how such ensembles arise in the context of array processing.

cs.IT↗

arXiv2017

Net-Trim: Convex Pruning of Deep Neural Networks with Performance Guarantee

We introduce and analyze a new technique for model reduction for deep neural networks. While large networks are theoretically capable of learning arbitrarily complex models, overfitting and model redundancy negatively affects the prediction accuracy and model variance. Our Net-Trim algorithm prunes (sparsifies) a trained network layer-wise, removing connections at each layer by solving a convex optimization program. This program seeks a sparse set of weights at each layer that keeps the layer inputs and outputs consistent with the originally trained model. The algorithms and associated analysis are applicable to neural networks operating with the rectified linear unit (ReLU) as the nonlinear activation. We present both parallel and cascade versions of the algorithm. While the latter can achieve slightly simpler models with the same generalization performance, the former can be computed in a distributed manner. In both cases, Net-Trim significantly reduces the number of connections in the network, while also providing enough regularization to slightly reduce the generalization error. We also provide a mathematical analysis of the consistency between the initial network and the retrained model. To analyze the model sample complexity, we derive the general sufficient conditions for the recovery of a sparse transform matrix. For a single layer taking independent Gaussian random vectors of length $N$ as inputs, we show that if the network response can be described using a maximum number of $s$ non-zero weights per node, these weights can be learned from $\mathcal{O}(s\log N)$ samples.

cs.LG↗

arXiv2017

Fast and guaranteed blind multichannel deconvolution under a bilinear system model

We consider the multichannel blind deconvolution problem where we observe the output of multiple channels that are all excited with the same unknown input. From these observations, we wish to estimate the impulse responses of each of the channels. We show that this problem is well-posed if the channels follow a bilinear model where the ensemble of channel responses is modeled as lying in a low-dimensional subspace but with each channel modulated by an independent gain. Under this model, we show how the channel estimates can be found by minimizing a quadratic functional over a non-convex set. We analyze two methods for solving this non-convex program, and provide performance guarantees for each. The first is a method of alternating eigenvectors that breaks the program down into a series of eigenvalue problems. The second is a truncated power iteration, which can roughly be interpreted as a method for finding the largest eigenvector of a symmetric matrix with the additional constraint that it adheres to our bilinear model. As with most non-convex optimization algorithms, the performance of both of these algorithms is highly dependent on having a good starting point. We show how such a starting point can be constructed from the channel measurements. Our performance guarantees are non-asymptotic, and provide a sufficient condition on the number of samples observed per channel in order to guarantee channel estimates of a certain accuracy. Our analysis uses a model with a "generic" subspace that is drawn at random, and we show the performance bounds hold with high probability. Mathematically, the key estimates are derived by quantifying how well the eigenvectors of certain random matrices approximate the eigenvectors of their mean. We also present a series of numerical results demonstrating that the empirical performance is consistent with the presented theory.

math.NA↗

arXiv2017

Spectral Methods for Passive Imaging: Non-asymptotic Performance and Robustness

We study the problem of passive imaging through convolutive channels. A scene is illuminated with an unknown, unstructured source, and the measured response is the convolution of this source with multiple channel responses, each of which is time-limited. Spectral methods based on the commutativity of convolution, first proposed and analyzed in the 1990s, provide an elegant mathematical framework for attacking this problem. However, these now classical methods are very sensitive to noise, especially when working from relatively small sample sizes. In this paper, we show that a linear subspace model on the coefficients of the impulse responses of the channels can make this problem well-posed. We derive non-asymptotic error bounds for the generic subspace model by analyzing the spectral gap of the cross-correlation matrix of the channels relative to the perturbation introduced by noise. Numerical results show that this modified spectral method offers significant improvements over the classical method and outperforms other competing methods for multichannel blind deconvolution.

cs.IT↗

arXiv2017

The Fast Slepian Transform

The discrete prolate spheroidal sequences (DPSS's) provide an efficient representation for discrete signals that are perfectly timelimited and nearly bandlimited. Due to the high computational complexity of projecting onto the DPSS basis - also known as the Slepian basis - this representation is often overlooked in favor of the fast Fourier transform (FFT). We show that there exist fast constructions for computing approximate projections onto the leading Slepian basis elements. The complexity of the resulting algorithms is comparable to the FFT, and scales favorably as the quality of the desired approximation is increased. In the process of bounding the complexity of these algorithms, we also establish new nonasymptotic results on the eigenvalue distribution of discrete time-frequency localization operators. We then demonstrate how these algorithms allow us to efficiently compute the solution to certain least-squares problems that arise in signal processing. We also provide simulations comparing these fast, approximate Slepian methods to exact Slepian methods as well as the traditional FFT based methods.

math.NA↗

arXiv2017

The Eigenvalue Distribution of Discrete Periodic Time-Frequency Limiting Operators

Bandlimiting and timelimiting operators play a fundamental role in analyzing bandlimited signals that are approximately timelimited (or vice versa). In this paper, we consider a time-frequency (in the discrete Fourier transform (DFT) domain) limiting operator whose eigenvectors are known as the periodic discrete prolate spheroidal sequences (PDPSSs). We establish new nonasymptotic results on the eigenvalue distribution of this operator. As a byproduct, we also characterize the eigenvalue distribution of a set of submatrices of the DFT matrix, which is of independent interest.

cs.IT↗

arXiv2017

Phase Retrieval Meets Statistical Learning Theory: A Flexible Convex Relaxation

We propose a flexible convex relaxation for the phase retrieval problem that operates in the natural domain of the signal. Therefore, we avoid the prohibitive computational cost associated with "lifting" and semidefinite programming (SDP) in methods such as PhaseLift and compete with recently developed non-convex techniques for phase retrieval. We relax the quadratic equations for phaseless measurements to inequality constraints each of which representing a symmetric "slab". Through a simple convex program, our proposed estimator finds an extreme point of the intersection of these slabs that is best aligned with a given anchor vector. We characterize geometric conditions that certify success of the proposed estimator. Furthermore, using classic results in statistical learning theory, we show that for random measurements the geometric certificates hold with high probability at an optimal sample complexity. Phase transition of our estimator is evaluated through simulations. Our numerical experiments also suggest that the proposed method can solve phase retrieval problems with coded diffraction measurements as well.

cs.IT↗

arXiv2017

Algebraic Connectivity Under Site Percolation in Finite Weighted Graphs

We study the behavior of algebraic connectivity in a weighted graph that is subject to site percolation, random deletion of the vertices. Using a refined concentration inequality for random matrices we show in our main theorem that the (augmented) Laplacian of the percolated graph concentrates around its expectation. This concentration bound then provides a lower bound on the algebraic connectivity of the percolated graph. As a special case for $(n,d,λ)$-graphs (i.e., $d$-regular graphs on $n$ vertices with non-trivial eigenvalues less than $λ$ in magnitude) our result shows that, with high probability, the graph remains connected under a homogeneous site percolation with survival probability $p\ge 1-C_{1}n^{-C_{2}/d}$ with $C_{1}$ and $C_{2}$ depending only on $λ/d$.

math.PR↗

arXiv2016

Empirical Chaos Processes and Blind Deconvolution

This paper investigates conditions under which certain kinds of systems of bilinear equations have a unique structured solution. In particular, we look at when we can recover vectors $\boldsymbol{w},\boldsymbol{q}$ from observations of the form \[ y_{\ell} = <\boldsymbol{w},\boldsymbol{b}_{\ell}><\boldsymbol{c}_{\ell},\boldsymbol{q}>, \quad \ell = 1,\ldots,L, \] where $\boldsymbol{b}_\ell,\boldsymbol{c}_\ell$ are known. We show that if $\boldsymbol{w}\in\mathbb{C}^{M_1}$ and $\boldsymbol{q}\in\mathbb{C}^{M_2}$ are sparse, with no more than $K$ and $N$ nonzero entries, respectively, and the $\boldsymbol{b}_\ell,\boldsymbol{c}_\ell$ are generic, selected as independent Gaussian random vectors, then $\boldsymbol{w},\boldsymbol{q}$ are uniquely determined from \[ L \geq \mathrm{Const}\cdot (K+N)\log^5(M_1M_2) \] such equations with high probability. The key ingredient in our analysis is a uniform probabilistic bound on how far a random process of the form \[Z(\boldsymbol{X}) = \sum_{\ell=1}^L|\boldsymbol{b}_\ell^*\boldsymbol{X}\boldsymbol{c}_\ell|^2 \] deviates from its mean over a set of structured matrices $\boldsymbol{X}\in\mathcal{X}$. As both $\boldsymbol{b}_\ell$ and $\boldsymbol{c}_\ell$ are random, this is a specialized type of $4$th order chaos; we refer to $Z(\boldsymbol{X})$ as an {\em empirical chaos process}. Bounding this process yields a set of general conditions for when the map $\boldsymbol{X}\rightarrow \{\boldsymbol{b}_\ell^*\boldsymbol{X}\boldsymbol{c}_\ell\}_{\ell=1}^L$ is a restricted isometry over the set of matrices $\mathcal{X}$. The conditions are stated in terms of general geometric properties of the set $\mathcal{X}$, and are explicitly computed for the case where $\mathcal{X}$ is the set of matrices that are simultaneously sparse and low rank.

cs.IT↗

arXiv2016

A Light-powered, Always-On, Smart Camera with Compressed Domain Gesture Detection

In this paper we propose an energy-efficient camera-based gesture recognition system powered by light energy for "always on" applications. Low energy consumption is achieved by directly extracting gesture features from the compressed measurements, which are the block averages and the linear combinations of the image sensor's pixel values. The gestures are recognized using a nearest-neighbour (NN) classifier followed by Dynamic Time Warping (DTW). The system has been implemented on an Analog Devices Black Fin ULP vision processor and powered by PV cells whose output is regulated by TI's DC-DC buck converter with Maximum Power Point Tracking (MPPT). Measured data reveals that with only 400 compressed measurements (768x compression ratio) per frame, the system is able to recognize key wake-up gestures with greater than 80% accuracy and only 95mJ of energy per frame. Owing to its fully self-powered operation, the proposed system can find wide applications in "always-on" vision systems such as in surveillance, robotics and consumer electronics with touch-less operation.

cs.CV↗

arXiv2016

Learning Shapes by Convex Composition

We present a mathematical and algorithmic scheme for learning the principal geometric elements in an image or 3D object. We build on recent work that convexifies the basic problem of finding a combination of a small number shapes that overlap and occlude one another in such a way that they "match" a given scene as closely as possible. This paper derives general sufficient conditions under which this convex shape composition identifies a target composition. From a computational standpoint, we present two different methods for solving the associated optimization programs. The first method simply recasts the problem as a linear program, while the second uses the alternating direction method of multipliers with a series of easily computed proximal operators.

cs.CV↗

arXiv2016

Sweep Distortion Removal from THz Images via Blind Demodulation

Heavy sweep distortion induced by alignments and inter-reflections of layers of a sample is a major burden in recovering 2D and 3D information in time resolved spectral imaging. This problem cannot be addressed by conventional denoising and signal processing techniques as it heavily depends on the physics of the acquisition. Here we propose and implement an algorithmic framework based on low-rank matrix recovery and alternating minimization that exploits the forward model for THz acquisition. The method allows recovering the original signal in spite of the presence of temporal-spatial distortions. We address a blind-demodulation problem, where based on several observations of the sample texture modulated by an undesired sweep pattern, the two classes of signals are separated. The performance of the method is examined in both synthetic and experimental data, and the successful reconstructions are demonstrated. The proposed general scheme can be implemented to advance inspection and imaging applications in THz and other time-resolved sensing modalities.

cs.CV↗