arXiv subjects

John Wright

Publications and source records attributed to John Wright.

At least 55 records · Page 3Linked to original sources

arXiv2021

Deep Networks Provably Classify Data on Curves

Data with low-dimensional nonlinear structure are ubiquitous in engineering and scientific problems. We study a model problem with such structure -- a binary classification task that uses a deep fully-connected neural network to classify data drawn from two disjoint smooth curves on the unit sphere. Aside from mild regularity conditions, we place no restrictions on the configuration of the curves. We prove that when (i) the network depth is large relative to certain geometric properties that set the difficulty of the problem and (ii) the network width and number of samples is polynomial in the depth, randomly-initialized gradient descent quickly learns to correctly classify all points on the two curves with high probability. To our knowledge, this is the first generalization guarantee for deep networks with nonlinear data that depends only on intrinsic data properties. Our analysis proceeds by a reduction to dynamics in the neural tangent kernel (NTK) regime, where the network depth plays the role of a fitting resource in solving the classification problem. In particular, via fine-grained control of the decay properties of the NTK, we demonstrate that when the network is sufficiently deep, the NTK can be locally approximated by a translationally invariant operator on the manifolds and stably inverted over smooth functions, which guarantees convergence and generalization.

stat.ML↗

arXiv2021

Principal Component Pursuit for Pattern Identification in Environmental Mixtures

Environmental health researchers often aim to identify sources/behaviors that give rise to potentially harmful exposures. We adapted principal component pursuit (PCP)-a robust technique for dimensionality reduction in computer vision and signal processing-to identify patterns in environmental mixtures. PCP decomposes the exposure mixture into a low-rank matrix containing consistent exposure patterns across pollutants and a sparse matrix isolating unique exposure events. We adapted PCP to accommodate non-negative and missing data, and values below a given limit of detection (LOD). We simulated data to represent environmental mixtures of two sizes with increasing proportions <LOD and three noise structures. We compared PCP-LOD to principal component analysis (PCA) to evaluate performance. We next applied PCP-LOD to a mixture of 21 persistent organic pollutants (POPs) measured in 1,000 U.S. adults from the 2001-2002 National Health and Nutrition Examination Survey. We applied singular value decomposition to the estimated low-rank matrix to characterize the patterns. PCP-LOD recovered the true number of patterns through cross-validation for all simulations; based on an a priori specified criterion, PCA recovered the true number of patterns in 32% of simulations. PCP-LOD achieved lower relative predictive error than PCA for all simulated datasets with up to 50% of the data <LOD. When 75% of values were <LOD, PCP-LOD outperformed PCA only when noise was low. In the POP mixture, PCP-LOD identified a rank-three underlying structure and separated 6% of values as unique events. One pattern represented comprehensive exposure to all POPs. The other patterns grouped chemicals based on known structure and toxicity. PCP-LOD serves as a useful tool to express multi-dimensional exposures as consistent patterns that, if found to be related to adverse health, are amenable to targeted interventions.

stat.AP↗

arXiv2021

Square Root Principal Component Pursuit: Tuning-Free Noisy Robust Matrix Recovery

We propose a new framework -- Square Root Principal Component Pursuit -- for low-rank matrix recovery from observations corrupted with noise and outliers. Inspired by the square root Lasso, this new formulation does not require prior knowledge of the noise level. We show that a single, universal choice of the regularization parameter suffices to achieve reconstruction error proportional to the (a priori unknown) noise level. In comparison, previous formulations such as stable PCP rely on noise-dependent parameters to achieve similar performance, and are therefore challenging to deploy in applications where the noise level is unknown. We validate the effectiveness of our new method through experiments on simulated and real datasets. Our simulations corroborate the claim that a universal choice of the regularization parameter yields near optimal performance across a range of noise levels, indicating that the proposed method outperforms the (somewhat loose) bound proved here.

cs.LG↗

arXiv2021

Gemmini: Enabling Systematic Deep-Learning Architecture Evaluation via Full-Stack Integration

DNN accelerators are often developed and evaluated in isolation without considering the cross-stack, system-level effects in real-world environments. This makes it difficult to appreciate the impact of System-on-Chip (SoC) resource contention, OS overheads, and programming-stack inefficiencies on overall performance/energy-efficiency. To address this challenge, we present Gemmini, an open-source*, full-stack DNN accelerator generator. Gemmini generates a wide design-space of efficient ASIC accelerators from a flexible architectural template, together with flexible programming stacks and full SoCs with shared resources that capture system-level effects. Gemmini-generated accelerators have also been fabricated, delivering up to three orders-of-magnitude speedups over high-performance CPUs on various DNN benchmarks. * https://github.com/ucb-bar/gemmini

cs.DC↗

arXiv2021

Ferroelectricity in Polar ScAlN/GaN Epitaxial Semiconductor Heterostructures

Room temperature ferroelectricity is observed in lattice-matched ~18% ScAlN/GaN heterostructures grown by molecular beam epitaxy on single-crystal GaN substrates. The epitaxial films have smooth surface morphologies and high crystallinity. Pulsed current-voltage measurements confirm stable and repeatable polarization switching in such ferroelectric/semiconductor structures at several measurement conditions, and in multiple samples. The measured coercive field values are Ec~0.7 MV/cm at room temperature, with remnant polarization Pr~10 μC/cm2 for ~100 nm thick ScAlN layers. These values are substantially lower than comparable ScAlN control layers deposited by sputtering. Importantly, the coercive field of MBE ScAlN is smaller than the critical breakdown field of GaN, offering the potential for low voltage ferroelectric switching. The low coercive field ferroelectricity of ScAlN on GaN heralds the possibility of new forms of electronic and photonic devices with epitaxially integrated ferroelectric/semiconductor heterostructures that take advantage of the GaN electronic and photonic semiconductor platform, where the underlying semiconductors themselves exhibit spontaneous and piezoelectric polarization.

cond-mat.mtrl-sci↗

arXiv2021

Deep Networks and the Multiple Manifold Problem

We study the multiple manifold problem, a binary classification task modeled on applications in machine vision, in which a deep fully-connected neural network is trained to separate two low-dimensional submanifolds of the unit sphere. We provide an analysis of the one-dimensional case, proving for a simple manifold configuration that when the network depth $L$ is large relative to certain geometric and statistical properties of the data, the network width $n$ grows as a sufficiently large polynomial in $L$, and the number of i.i.d. samples from the manifolds is polynomial in $L$, randomly-initialized gradient descent rapidly learns to classify the two manifolds perfectly with high probability. Our analysis demonstrates concrete benefits of depth and width in the context of a practically-motivated model problem: the depth acts as a fitting resource, with larger depths corresponding to smoother networks that can more readily separate the class manifolds, and the width acts as a statistical resource, enabling concentration of the randomly-initialized network and its gradients. The argument centers around the neural tangent kernel and its role in the nonasymptotic analysis of training overparameterized neural networks; to this literature, we contribute essentially optimal rates of concentration for the neural tangent kernel of deep fully-connected networks, requiring width $n \gtrsim L\,\mathrm{poly}(d_0)$ to achieve uniform concentration of the initial kernel over a $d_0$-dimensional submanifold of the unit sphere $\mathbb{S}^{n_0-1}$, and a nonasymptotic framework for establishing generalization of networks trained in the NTK regime with structured data. The proof makes heavy use of martingale concentration to optimally treat statistical dependencies across layers of the initial random network. This approach should be of use in establishing similar results for other network architectures.

stat.ML↗

arXiv2021

Complete Dictionary Learning via $\ell^4$-Norm Maximization over the Orthogonal Group

This paper considers the fundamental problem of learning a complete (orthogonal) dictionary from samples of sparsely generated signals. Most existing methods solve the dictionary (and sparse representations) based on heuristic algorithms, usually without theoretical guarantees for either optimality or complexity. The recent $\ell^1$-minimization based methods do provide such guarantees but the associated algorithms recover the dictionary one column at a time. In this work, we propose a new formulation that maximizes the $\ell^4$-norm over the orthogonal group, to learn the entire dictionary. We prove that under a random data model, with nearly minimum sample complexity, the global optima of the $\ell^4$ norm are very close to signed permutations of the ground truth. Inspired by this observation, we give a conceptually simple and yet effective algorithm based on "matching, stretching, and projection" (MSP). The algorithm provably converges locally at a superlinear (cubic) rate and cost per iteration is merely an SVD. In addition to strong theoretical guarantees, experiments show that the new algorithm is significantly more efficient and effective than existing methods, including KSVD and $\ell^1$-based methods. Preliminary experimental results on mixed real imagery data clearly demonstrate advantages of so learned dictionary over classic PCA bases.

cs.LG↗

arXiv2021

Momentum-resolved electronic band structure and offsets in an epitaxial NbN/GaN superconductor/semiconductor heterojunction

The electronic structure of heterointerfaces play a pivotal role in their device functionality. Recently, highly crystalline ultrathin films of superconducting NbN have been integrated by molecular beam epitaxy with the semiconducting GaN. We use soft X-ray angle-resolved photoelectron spectroscopy to directly measure the momentum-resolved electronic band structures for both NbN and GaN constituents of this Schottky heterointerface, and determine their momentum-dependent interfacial band offset as well as the band-bending profile into GaN. We find, in particular, that the Fermi states in NbN are aligned against the band gap in GaN, which excludes any significant electronic cross-talk of the superconducting states in NbN through the interface to GaN. We support the experimental findings with first-principles calculations for bulk NbN and GaN. The Schottky barrier height obtained from photoemission is corroborated by electronic transport and optical measurements. The momentum-resolved understanding of electronic properties elucidated by the combined materials advances and experimental methods in our work opens up new possibilities in systems where interfacial states play a defining role.

cond-mat.mtrl-sci↗

arXiv2020

An unexplored MBE growth mode reveals new properties of superconducting NbN

Accessing unexplored conditions in crystal growth often reveals remarkable surprises and new regimes of physical behavior. In this work, performing molecular beam epitaxy of the technologically important superconductor NbN at temperatures greater than 1000$^\circ$C, higher than in the past, is found to reveal persistent RHEED oscillations throughout the growth, atomically smooth surfaces, normal metal resistivities as low as 37$μΩ$-cm and superconducting critical temperatures in excess of 15 K. Most remarkably, a reversal of the sign of the Hall coefficient is observed as the NbN films are cooled, and the high material quality allows the first imaging of Abrikosov vortex lattices in this superconductor.

cond-mat.mtrl-sci↗

arXiv2020

Deep Networks from the Principle of Rate Reduction

This work attempts to interpret modern deep (convolutional) networks from the principles of rate reduction and (shift) invariant classification. We show that the basic iterative gradient ascent scheme for optimizing the rate reduction of learned features naturally leads to a multi-layer deep network, one iteration per layer. The layered architectures, linear and nonlinear operators, and even parameters of the network are all explicitly constructed layer-by-layer in a forward propagation fashion by emulating the gradient scheme. All components of this "white box" network have precise optimization, statistical, and geometric interpretation. This principled framework also reveals and justifies the role of multi-channel lifting and sparse coding in early stage of deep networks. Moreover, all linear operators of the so-derived network naturally become multi-channel convolutions when we enforce classification to be rigorously shift-invariant. The derivation also indicates that such a convolutional network is significantly more efficient to construct and learn in the spectral domain. Our preliminary simulations and experiments indicate that so constructed deep network can already learn a good discriminative representation even without any back propagation training.

cs.LG↗

arXiv2020

Quantum soundness of the classical low individual degree test

Low degree tests play an important role in classical complexity theory, serving as basic ingredients in foundational results such as $\mathsf{MIP} = \mathsf{NEXP}$ [BFL91] and the PCP theorem [AS98,ALM+98]. Over the last ten years, versions of these tests which are sound against quantum provers have found increasing applications to the study of nonlocal games and the complexity class~$\mathsf{MIP}^*$. The culmination of this line of work is the result $\mathsf{MIP}^* = \mathsf{RE}$ [arXiv:2001.04383]. One of the key ingredients in the first reported proof of $\mathsf{MIP}^* = \mathsf{RE}$ is a two-prover variant of the low degree test, initially shown to be sound against multiple quantum provers in [arXiv:1302.1242]. Unfortunately a mistake was recently discovered in the latter result, invalidating the main result of [arXiv:1302.1242] as well as its use in subsequent works, including [arXiv:2001.04383]. We analyze a variant of the low degree test called the low individual degree test. Our main result is that the two-player version of this test is sound against quantum provers. This soundness result is sufficient to re-derive several bounds on~$\mathsf{MIP}^*$ that relied on [arXiv:1302.1242], including $\mathsf{MIP}^* = \mathsf{RE}$.

quant-ph↗

arXiv2020

Epitaxial niobium nitride superconducting nanowire single-photon detectors

Superconducting nanowires used in single-photon detectors have been realized on amorphous or poly-crystalline films. Here, we report the use of single-crystalline NbN thin films for superconducting nanowire single-photon detectors (SNSPDs). Grown by molecular beam epitaxy (MBE) at high temperature on nearly lattice-matched AlN-on-sapphire substrates, the NbN films exhibit high degree of uniformity and homogeneity. Even with relatively thick films, the fabricated nanowire detectors show saturated internal efficiency at near-IR wavelengths, demonstrating the potential of MBE-grown NbN for realizing large arrays of on-chip SNSPDs and their integration with AlN-based $χ^{(2)}$ quantum photonic circuits.

physics.app-ph↗

arXiv2020

Finding the Sparsest Vectors in a Subspace: Theory, Algorithms, and Applications

The problem of finding the sparsest vector (direction) in a low dimensional subspace can be considered as a homogeneous variant of the sparse recovery problem, which finds applications in robust subspace recovery, dictionary learning, sparse blind deconvolution, and many other problems in signal processing and machine learning. However, in contrast to the classical sparse recovery problem, the most natural formulation for finding the sparsest vector in a subspace is usually nonconvex. In this paper, we overview recent advances on global nonconvex optimization theory for solving this problem, ranging from geometric analysis of its optimization landscapes, to efficient optimization algorithms for solving the associated nonconvex optimization problem, to applications in machine intelligence, representation learning, and imaging sciences. Finally, we conclude this review by pointing out several interesting open problems for future research.

cs.LG↗

arXiv2019

Convolutional Phase Retrieval via Gradient Descent

We study the convolutional phase retrieval problem, of recovering an unknown signal $\mathbf x \in \mathbb C^n $ from $m$ measurements consisting of the magnitude of its cyclic convolution with a given kernel $\mathbf a \in \mathbb C^m $. This model is motivated by applications such as channel estimation, optics, and underwater acoustic communication, where the signal of interest is acted on by a given channel/filter, and phase information is difficult or impossible to acquire. We show that when $\mathbf a$ is random and the number of observations $m$ is sufficiently large, with high probability $\mathbf x$ can be efficiently recovered up to a global phase shift using a combination of spectral initialization and generalized gradient descent. The main challenge is coping with dependencies in the measurement operator. We overcome this challenge by using ideas from decoupling theory, suprema of chaos processes and the restricted isometry property of random circulant matrices, and recent analysis of alternating minimization methods.

stat.CO↗

arXiv2019

Short-and-Sparse Deconvolution -- A Geometric Approach

Short-and-sparse deconvolution (SaSD) is the problem of extracting localized, recurring motifs in signals with spatial or temporal structure. Variants of this problem arise in applications such as image deblurring, microscopy, neural spike sorting, and more. The problem is challenging in both theory and practice, as natural optimization formulations are nonconvex. Moreover, practical deconvolution problems involve smooth motifs (kernels) whose spectra decay rapidly, resulting in poor conditioning and numerical challenges. This paper is motivated by recent theoretical advances, which characterize the optimization landscape of a particular nonconvex formulation of SaSD. This is used to derive a $provable$ algorithm which exactly solves certain non-practical instances of the SaSD problem. We leverage the key ideas from this theory (sphere constraints, data-driven initialization) to develop a $practical$ algorithm, which performs well on data arising from a range of application areas. We highlight key additional challenges posed by the ill-conditioning of real SaSD problems, and suggest heuristics (acceleration, continuation, reweighting) to mitigate them. Experiments demonstrate both the performance and generality of the proposed method.

eess.SP↗

arXiv2019

NEEXP in MIP*

We study multiprover interactive proof systems. The power of classical multiprover interactive proof systems, in which the provers do not share entanglement, was characterized in a famous work by Babai, Fortnow, and Lund (Computational Complexity 1991), whose main result was the equality MIP = NEXP. The power of quantum multiprover interactive proof systems, in which the provers are allowed to share entanglement, has proven to be much more difficult to characterize. The best known lower-bound on MIP* is NEXP, due to Ito and Vidick (FOCS 2012). As for upper bounds, MIP* could be as large as RE, the class of recursively enumerable languages. The main result of this work is the inclusion of NEEXP (nondeterministic doubly exponential time) in MIP*. This is an exponential improvement over the prior lower bound and shows that proof systems with entangled provers are at least exponentially more powerful than classical provers. In our protocol the verifier delegates a classical, exponentially large MIP protocol for NEEXP to two entangled provers: the provers obtain their exponentially large questions by measuring their shared state, and use a classical PCP to certify the correctness of their exponentially-long answers. For the soundness of our protocol, it is crucial that each player should not only sample its own question correctly but also avoid performing measurements that would reveal the other player's sampled question. We ensure this by commanding the players to perform a complementary measurement, relying on the Heisenberg uncertainty principle to prevent the forbidden measurements from being performed.

quant-ph↗

arXiv2019

Molecular Beam Epitaxy Growth of Scandium Nitride on Hexagonal SiC, GaN, and AlN

RF plasma assisted MBE growth of Scandium Nitride (ScN) thin films on GaN (0001)/SiC, AlN (0001)/Al2O3 and on 6H-SiC (0001) hexagonal substrates is found to lead to a face centered cubic (rock-salt) crystal structure with (111) out-of-plane orientation instead of hexagonal orientation. For the first time, cubic (111) twinned patterns in ScN are observed by in-situ electron diffraction during epitaxy, and the twin domains in ScN are detected by electron backscattered diffraction, and further corroborated with X-ray diffraction. The epitaxial ScN films display very smooth, sub nanometer surface roughness at a growth temperature of 750C. Temperature-dependent Hall-effect measurements indicate a constant high n-type carrier concentration of ~1x1020/cm3 and electron mobilities of ~ 20 cm2/Vs.

cond-mat.mtrl-sci↗

arXiv2019

Structured Local Optima in Sparse Blind Deconvolution

Blind deconvolution is a ubiquitous problem of recovering two unknown signals from their convolution. Unfortunately, this is an ill-posed problem in general. This paper focuses on the {\em short and sparse} blind deconvolution problem, where the one unknown signal is short and the other one is sparsely and randomly supported. This variant captures the structure of the unknown signals in several important applications. We assume the short signal to have unit $\ell^2$ norm and cast the blind deconvolution problem as a nonconvex optimization problem over the sphere. We demonstrate that (i) in a certain region of the sphere, every local optimum is close to some shift truncation of the ground truth, and (ii) for a generic short signal of length $k$, when the sparsity of activation signal $θ\lesssim k^{-2/3}$ and number of measurements $m\gtrsim poly(k)$, a simple initialization method together with a descent algorithm which escapes strict saddle points recovers a near shift truncation of the ground truth kernel.

eess.SP↗