SearcharxivSearch

arXiv subjects

Mark Iwen

Publications and source records attributed to Mark Iwen.

At least 19 recordsLinked to original sources

Fast Dimensionality Reduction from $\ell_2$ to $\ell_p$

The Johnson-Lindenstrauss (JL) lemma is a fundamental result in dimensionality reduction, ensuring that any finite set $X \subseteq \mathbb{R}^d$ can be embedded into a lower-dimensional space $\mathbb{R}^k$ while approximately preserving all pairwise Euclidean distances. In recent years, embeddings that preserve Euclidean distances when measured via the $\ell_1$ norm in the target space have received increasing attention due to their relevance in applications such as nearest neighbor search in high dimensions. A recent breakthrough by Dirksen, Mendelson, and Stollenwerk established an optimal $\ell_2 \to \ell_1$ embedding with computational complexity $O(d \log d)$. In this work, we generalize this direction and propose a simple linear embedding from $\ell_2$ to $\ell_p$ for any $p \in [1,2]$ based on a construction of Ailon and Liberty. Our method achieves a reduced runtime of $O(d \log k)$ when $k \leq d^{1/4}$, improving upon prior runtime results when the target dimension is small. Additionally, we show that for \emph{any norm} $\|\cdot\|$ in the target space, any embedding of $(\mathbb{R}^d, \|\cdot\|_2)$ into $(\mathbb{R}^k, \|\cdot\|)$ with distortion $\varepsilon$ generally requires $k = \Omega\big(\varepsilon^{-2} \log(\varepsilon^2 n)/\log(1/\varepsilon)\big)$, matching the optimal bound for the $\ell_2$ case up to a logarithmic factor.

math.PR

Fast One-Pass Sparse Approximation of the Top Eigenvectors of Huge Approximately Low-Rank Matrices? Yes, $MAM^*$!

Motivated by applications such as sparse PCA, in this paper we present provably-accurate one-pass algorithms for the sparse approximation of the top eigenvectors of extremely massive matrices based on a single compact linear sketch. The resulting compressive-sensing-based approaches can approximate the leading eigenvectors of huge approximately low-rank matrices that are too large to store in memory based on a single pass over its entries while utilizing a total memory footprint on the order of the much smaller desired sparse eigenvector approximations. Finally, the compressive sensing recovery algorithm itself (which takes the gathered compressive matrix measurements as input, and then outputs sparse approximations of its top eigenvectors) can also be formulated to run in a time which principally depends on the size of the sought sparse approximations, making its runtime sublinear in the size of the large matrix whose eigenvectors one aims to approximate. Preliminary experiments on huge matrices having $\sim 10^{16}$ entries illustrate the developed theory and demonstrate the practical potential of the proposed approach.

cs.IT

Phasebook: A Survey of Selected Open Problems in Phase Retrieval

Phase retrieval is an inverse problem that, on one hand, is crucial in many applications across imaging and physics, and, on the other hand, leads to deep research questions in theoretical signal processing and applied harmonic analysis. This survey paper is an outcome of the recent workshop Phase Retrieval in Mathematics and Applications (PRiMA) (held on August 5--9 2024 at the Lorentz Center in Leiden, The Netherlands) that brought together experts working on theoretical and practical aspects of the phase retrieval problem with the purpose to formulate and explore essential open problems in the field.

cs.IT

On Extended Concentration Inequalities for Fast JL Embeddings of Infinite Sets

The Johnson-Lindenstrauss (JL) lemma allows subsets of a high-dimensional space to be embedded into a lower-dimensional space while approximately preserving all pairwise Euclidean distances. This important result has inspired an extensive literature, with a significant portion dedicated to constructing structured random matrices with fast matrix-vector multiplication algorithms that generate such embeddings for finite point sets. In this paper, we briefly consider fast JL embedding matrices for {\it infinite} subsets of $\mathbb{R}^d$. Prior work in this direction such as \cite{oymak2018isometric, mendelson2023column} has focused on constructing fast JL matrices $HD \in \mathbb{R}^{k \times d}$ by multiplying structured matrices with RIP(-like) properties $H \in \mathbb{R}^{k \times d}$ against a random diagonal matrix $D \in \mathbb{R}^{d \times d}$. However, utilizing RIP(-like) matrices $H$ in this fashion necessarily has the unfortunate side effect that the resulting embedding dimension $k$ must depend on the ambient dimension $d$ no matter how simple the infinite set is that one aims to embed. Motivated by this, we explore an alternate strategy for removing this $d$-dependence from $k$ herein: Extending a concentration inequality proven by Ailon and Liberty \cite{Ailon2008fast} in the hope of later utilizing it in a chaining argument to obtain a near-optimal result for infinite sets. %, and $(ii)$ utilizing a simple secondary Gaussian embedding of an initial fast JL embedding of a given infinite set. Though this strategy ultimately fails to provide the near-optimal embedding dimension we seek, along the way we obtain a stronger-than-sub-exponential extension of the concentration inequality in \cite{Ailon2008fast} which may be of independent interest.

cs.DS

Implicit Regularization for Tubal Tensor Factorizations via Gradient Descent

We provide a rigorous analysis of implicit regularization in an overparametrized tensor factorization problem beyond the lazy training regime. For matrix factorization problems, this phenomenon has been studied in a number of works. A particular challenge has been to design universal initialization strategies which provably lead to implicit regularization in gradient-descent methods. At the same time, it has been argued by Cohen et. al. 2016 that more general classes of neural networks can be captured by considering tensor factorizations. However, in the tensor case, implicit regularization has only been rigorously established for gradient flow or in the lazy training regime. In this paper, we prove the first tensor result of its kind for gradient descent rather than gradient flow. We focus on the tubal tensor product and the associated notion of low tubal rank, encouraged by the relevance of this model for image data. We establish that gradient descent in an overparametrized tensor factorization model with a small random initialization exhibits an implicit bias towards solutions of low tubal rank. Our theoretical findings are illustrated in an extensive set of numerical simulations show-casing the dynamics predicted by our theory as well as the crucial role of using a small random initialization.

cs.LG

On Continuous Terminal Embeddings of Sets of Positive Reach

In this paper we prove the existence of H\"{o}lder continuous terminal embeddings of any desired $X \subseteq \mathbb{R}^d$ into $\mathbb{R}^{m}$ with $m=\mathcal{O}(\varepsilon^{-2}\omega(S_X)^2)$, for arbitrarily small distortion $\varepsilon$, where $\omega(S_X)$ denotes the Gaussian width of the unit secants of $X$. More specifically, when $X$ is a finite set we provide terminal embeddings that are locally $\frac{1}{2}$-H\"{o}lder almost everywhere, and when $X$ is infinite with positive reach we give terminal embeddings that are locally $\frac{1}{4}$-H\"{o}lder everywhere sufficiently close to $X$ (i.e., within all tubes around $X$ of radius less than $X$'s reach). When $X$ is a compact $d$-dimensional submanifold of $\mathbb{R}^N$, an application of our main results provides terminal embeddings into $\tilde{\mathcal{O}}(d)$-dimensional space that are locally H\"{o}lder everywhere sufficiently close to the manifold.

math.OC

Tensor Deli: Tensor Completion for Low CP-Rank Tensors via Random Sampling

We propose two provably accurate methods for low CP-rank tensor completion - one using adaptive sampling and one using nonadaptive sampling. Both of our algorithms combine matrix completion techniques for a small number of slices along with Jennrich's algorithm to learn the factors corresponding to the first two modes, and then solve systems of linear equations to learn the factors corresponding to the remaining modes. For order-$3$ tensors, our algorithms follow a "sandwich" sampling strategy that more densely samples a few outer slices (the bread), and then more sparsely samples additional inner slices (the bbq-braised tofu) for the final completion. For an order-$d$, CP-rank $r$ tensor of size $n \times \cdots \times n$ that satisfies mild assumptions, our adaptive sampling algorithm recovers the CP-decomposition with high probability while using at most $O(nr\log r + dnr)$ samples and $O(n^2r^2+dnr^2)$ operations. Our nonadaptive sampling algorithm recovers the CP-decomposition with high probability while using at most $O(dnr^2\log n + nr\log^2 n)$ samples and runs in polynomial time. Numerical experiments demonstrate that both of our methods work well on noisy synthetic data as well as on real world data.

math.NA

Tensor Sandwich: Tensor Completion for Low CP-Rank Tensors via Adaptive Random Sampling

We propose an adaptive and provably accurate tensor completion approach based on combining matrix completion techniques (see, e.g., arXiv:0805.4471, arXiv:1407.3619, arXiv:1306.2979) for a small number of slices with a modified noise robust version of Jennrich's algorithm. In the simplest case, this leads to a sampling strategy that more densely samples two outer slices (the bread), and then more sparsely samples additional inner slices (the bbq-braised tofu) for the final completion. Under mild assumptions on the factor matrices, the proposed algorithm completes an $n \times n \times n$ tensor with CP-rank $r$ with high probability while using at most $\mathcal{O}(nr\log^2 r)$ adaptively chosen samples. Empirical experiments further verify that the proposed approach works well in practice, including as a low-rank approximation method in the presence of additive noise.

math.NA

Sparse Spectral Methods for Solving High-Dimensional and Multiscale Elliptic PDEs

In his monograph Chebyshev and Fourier Spectral Methods, John Boyd claimed that, regarding Fourier spectral methods for solving differential equations, ``[t]he virtues of the Fast Fourier Transform will continue to improve as the relentless march to larger and larger [bandwidths] continues''. This paper attempts to further the virtue of the Fast Fourier Transform (FFT) as not only bandwidth is pushed to its limits, but also the dimension of the problem. Instead of using the traditional FFT however, we make a key substitution: a high-dimensional, sparse Fourier transform (SFT) paired with randomized rank-1 lattice methods. The resulting sparse spectral method rapidly and automatically determines a set of Fourier basis functions whose span is guaranteed to contain an accurate approximation of the solution of a given elliptic PDE. This much smaller, near-optimal Fourier basis is then used to efficiently solve the given PDE in a runtime which only depends on the PDE's data compressibility and ellipticity properties, while breaking the curse of dimensionality and relieving linear dependence on any multiscale structure in the original problem. Theoretical performance of the method is established herein with convergence analysis in the Sobolev norm for a general class of non-constant diffusion equations, as well as pointers to technical extensions of the convergence analysis to more general advection-diffusion-reaction equations. Numerical experiments demonstrate good empirical performance on several multiscale and high-dimensional example problems, further showcasing the promise of the proposed methods in practice.

math.NA

Toward Fast and Provably Accurate Near-field Ptychographic Phase Retrieval

Ptychography is an imaging technique which involves a sample being illuminated by a coherent, localized probe of illumination. When the probe interacts with the sample, the light is diffracted and a diffraction pattern is detected. Then the sample (or probe) is shifted laterally in space to illuminate a new area of the sample whilst ensuring sufficient overlap. Near-field Ptychography (NFP) occurs when the sample is placed at a short defocus distance having a large Fresnel number. In this paper, we prove that certain NFP measurements are robustly invertible (up to an unavoidable global phase ambiguity) by constructing a point spread function and physical mask which leads to a well-conditioned lifted linear system. We then apply a block phase retrieval algorithm using weighted angular synchronization and prove that the proposed approach accurately recovers the measured sample. Finally, we also propose using a Wirtinger Flow for NFP problems and numerically evaluate that alternate approach both against our main proposed approach, as well as with NFP measurements for which our main approach does not apply.

math.NA

Neural Network Approximation of Continuous Functions in High Dimensions with Applications to Inverse Problems

The remarkable successes of neural networks in a huge variety of inverse problems have fueled their adoption in disciplines ranging from medical imaging to seismic analysis over the past decade. However, the high dimensionality of such inverse problems has simultaneously left current theory, which predicts that networks should scale exponentially in the dimension of the problem, unable to explain why the seemingly small networks used in these settings work as well as they do in practice. To reduce this gap between theory and practice, we provide a general method for bounding the complexity required for a neural network to approximate a H\"older (or uniformly) continuous function defined on a high-dimensional set with a low-complexity structure. The approach is based on the observation that the existence of a Johnson-Lindenstrauss embedding $A\in\mathbb{R}^{d\times D}$ of a given high-dimensional set $S\subset\mathbb{R}^D$ into a low dimensional cube $[-M,M]^d$ implies that for any H\"older (or uniformly) continuous function $f:S\to\mathbb{R}^p$, there exists a H\"older (or uniformly) continuous function $g:[-M,M]^d\to\mathbb{R}^p$ such that $g(Ax)=f(x)$ for all $x\in S$. Hence, if one has a neural network which approximates $g:[-M,M]^d\to\mathbb{R}^p$, then a layer can be added that implements the JL embedding $A$ to obtain a neural network that approximates $f:S\to\mathbb{R}^p$. By pairing JL embedding results along with results on approximation of H\"older (or uniformly) continuous functions by neural networks, one then obtains results which bound the complexity required for a neural network to approximate H\"older (or uniformly) continuous functions on high dimensional sets. The end result is a general theoretical framework which can then be used to better explain the observed empirical successes of smaller networks in a wider variety of inverse problems than current theory allows.

stat.ML

Characterizing unit spheres in Euclidean spaces via reach and volume

Let $M$ be a smooth, connected, compact submanifold of $\mathbb{R}^n$ without boundary and of dimension $k\geq 2$. Let $\mathbb{S}^k \subset \mathbb{R}^{k+1}\subset \mathbb{R}^n$ denote the $k$-dimesnional unit sphere. We show if $M$ has reach equal to one, then its volume satisfies $\text{vol}(M)\geq \text{vol}(\mathbb{S}^k)$ with equality holding only if $M$ is congruent to $\mathbb{S}^k$.

math.DG

A Hybrid Scattering Transform for Signals with Isolated Singularities

The scattering transform is a wavelet-based model of Convolutional Neural Networks originally introduced by S. Mallat. Mallat's analysis shows that this network has desirable stability and invariance guarantees and therefore helps explain the observation that the filters learned by early layers of a Convolutional Neural Network typically resemble wavelets. Our aim is to understand what sort of filters should be used in the later layers of the network. Towards this end, we propose a two-layer hybrid scattering transform. In our first layer, we convolve the input signal with a wavelet filter transform to promote sparsity, and, in the second layer, we convolve with a Gabor filter to leverage the sparsity created by the first layer. We show that these measurements characterize information about signals with isolated singularities. We also show that the Gabor measurements used in the second layer can be used to synthesize sparse signals such as those produced by the first layer.

eess.SP

Phase Retrieval for $L^2([-π,π])$ via the Provably Accurate and Noise Robust Numerical Inversion of Spectrogram Measurements

In this paper, we focus on the approximation of smooth functions $f: [-π, π] \rightarrow \mathbb{C}$, up to an unresolvable global phase ambiguity, from a finite set of Short Time Fourier Transform (STFT) magnitude (i.e., spectrogram) measurements. Two algorithms are developed for approximately inverting such measurements, each with theoretical error guarantees establishing their correctness. A detailed numerical study also demonstrates that both algorithms work well in practice and have good numerical convergence behavior.

math.NA

Lower Bounds on the Low-Distortion Embedding Dimension of Submanifolds of $\mathbb{R}^n$

Let $\mathcal{M}$ be a smooth submanifold of $\mathbb{R}^n$ equipped with the Euclidean (chordal) metric. This note considers the smallest dimension $m$ for which there exists a bi-Lipschitz function $f: \mathcal{M} \mapsto \mathbb{R}^m$ with bi-Lipschitz constants close to one. The main result bounds the embedding dimension $m$ below in terms of the bi-Lipschitz constants of $f$ and the reach, volume, diameter, and dimension of $\mathcal{M}$. This new lower bound is applied to show that prior upper bounds by Eftekhari and Wakin (arXiv:1306.4748) on the minimal low-distortion embedding dimension of such manifolds using random matrices achieve near-optimal dependence on both reach and volume. This supports random linear maps as being nearly as efficient as the best possible nonlinear maps at reducing the ambient dimension for manifold data. In the process of proving our main result, we also prove similar results concerning the impossibility of achieving better nonlinear measurement maps with the Restricted Isometry Property (RIP) in compressive sensing applications.

math.NA

On the $\ell^\infty$-norms of the Singular Vectors of Arbitrary Powers of a Difference Matrix with Applications to Sigma-Delta Quantization

Let $\| A \|_{\max} := \max_{i,j} |A_{i,j}|$ denote the maximum magnitude of entries of a given matrix $A$. In this paper we show that $$\max \left\{ \|U_r \|_{\max},\|V_r\|_{\max} \right\} \le \frac{(Cr)^{6r}}{\sqrt{N}},$$ where $U_r$ and $V_r$ are the matrices whose columns are, respectively, the left and right singular vectors of the $r$-th order finite difference matrix $D^{r}$ with $r \geq 2$, and where $D$ is the $N\times N$ finite difference matrix with $1$ on the diagonal, $-1$ on the sub-diagonal, and $0$ elsewhere. Here $C$ is a universal constant that is independent of both $N$ and $r$. Among other things, this establishes that both the right and left singular vectors of such finite difference matrices are Bounded Orthonormal Systems (BOSs) with known upper bounds on their BOS constants, objects of general interest in classical compressive sensing theory. Such finite difference matrices are also fundamental to standard $r^{\rm th}$ order Sigma-Delta quantization schemes more specifically, and as a result the new bounds provided herein on the maximum $\ell^{\infty}$-norms of their $\ell^2$-normalized singular vectors allow for several previous Sigma-Delta quantization results to be generalized and improved.

math.NA

Sparse Fourier Transforms on Rank-1 Lattices for the Rapid and Low-Memory Approximation of Functions of Many Variables

We consider fast, provably accurate algorithms for approximating functions on the $d$-dimensional torus, $f: \mathbb{ T }^d \rightarrow \mathbb{C}$, that are sparse (or compressible) in the Fourier basis. In particular, suppose that the Fourier coefficients of $f$, $\{c_{\bf k} (f) \}_{{\bf k} \in \mathbb{Z}^d}$, are concentrated in a finite set $I \subset \mathbb{Z}^d$ so that $$\min_{Ω\subset I s.t. |Ω| =s } \left\| f - \sum_{{\bf k} \in Ω} c_{\bf k} (f) e^{ -2 πi {\bf k} \cdot \circ} \right\|_2 < ε\|f \|_2$$ holds for $s \ll |I|$ and $ε\in (0,1)$. We aim to identify a near-minimizing subset $Ω\subset I$ and accurately approximate the associated Fourier coefficients $\{ c_{\bf k} (f) \}_{{\bf k} \in Ω}$ as rapidly as possible. We present both deterministic as well as randomized algorithms using $O(s^2 d \log^c (|I|))$-time/memory and $O(s d \log^c (|I|))$-time/memory, respectively. Most crucially, all of the methods proposed herein achieve these runtimes while satisfying theoretical best $s$-term approximation guarantees which guarantee their numerical accuracy and robustness to noise for general functions. These are achieved by modifying several one-dimensional Sparse Fourier Transform (SFT) methods to subsample a function along a reconstructing rank-1 lattice for the given frequency set $I$ to rapidly identify a near-minimizing subset $Ω\subset I$ without using anything about the lattice beyond its generating vector. This requires new fast and low-memory frequency identification techniques capable of rapidly recovering vector-valued frequencies in $\mathbb{Z}^d$ as opposed to simple integer frequencies in the univariate setting. Two different strategies are proposed and analyzed, each with different accuracy versus computational speed and memory tradeoffs.

math.NA

Sparse Harmonic Transforms: A New Class of Sublinear-time Algorithms for Learning Functions of Many Variables

We develop fast and memory efficient numerical methods for learning functions of many variables that admit sparse representations in terms of general bounded orthonormal tensor product bases. Such functions appear in many applications including, e.g., various Uncertainty Quantification(UQ) problems involving the solution of parametric PDE that are approximately sparse in Chebyshev or Legendre product bases. We expect that our results provide a starting point for a new line of research on sublinear-time solution techniques for UQ applications of the type above which will eventually be able to scale to significantly higher-dimensional problems than what are currently computationally feasible. More concretely, let $B$ be a finite Bounded Orthonormal Product Basis (BOPB) of cardinality $|B|=N$. We will develop methods that approximate any function $f$ that is sparse in the BOPB, that is, $f:\mathcal{D}\subset R^D\rightarrow C$ of the form $f(\mathbf{x})=\sum_{b\in S}c_b\cdot b(\mathbf{x})$ with $S\subset B$ of cardinality $|S| =s\ll N$. Our method has a runtime of just $(s\log N)^{O(1)}$, uses only $(s\log N)^{O(1)}$ function evaluations on a fixed and nonadaptive grid, and not more than $(s\log N)^{O(1)}$ bits of memory. For $s\ll N$, the runtime $(s\log N)^{O(1)}$ will be less than what is required to simply enumerate the elements of the basis $B$; thus our method is the first approach applicable in a general BOPB framework that falls into the class referred to as "sublinear-time". This and the similarly reduced sample and memory requirements set our algorithm apart from previous works based on standard compressive sensing algorithms such as basis pursuit which typically store and utilize full intermediate basis representations of size $Ω(N)$.

math.NA