arXiv subjects

Yuan Cao

Publications and source records attributed to Yuan Cao.

At least 163 records · Page 9Linked to original sources

arXiv2021

Echo State Speech Recognition

We propose automatic speech recognition (ASR) models inspired by echo state network (ESN), in which a subset of recurrent neural networks (RNN) layers in the models are randomly initialized and untrained. Our study focuses on RNN-T and Conformer models, and we show that model quality does not drop even when the decoder is fully randomized. Furthermore, such models can be trained more efficiently as the decoders do not require to be updated. By contrast, randomizing encoders hurts model quality, indicating that optimizing encoders and learn proper representations for acoustic inputs are more vital for speech recognition. Overall, we challenge the common practice of training ASR models for all components, and demonstrate that ESN-based models can perform equally well but enable more efficient training and storage than fully-trainable counterparts.

cs.CL↗

arXiv2021

Provable Generalization of SGD-trained Neural Networks of Any Width in the Presence of Adversarial Label Noise

We consider a one-hidden-layer leaky ReLU network of arbitrary width trained by stochastic gradient descent (SGD) following an arbitrary initialization. We prove that SGD produces neural networks that have classification accuracy competitive with that of the best halfspace over the distribution for a broad class of distributions that includes log-concave isotropic and hard margin distributions. Equivalently, such networks can generalize when the data distribution is linearly separable but corrupted with adversarial label noise, despite the capacity to overfit. To the best of our knowledge, this is the first work to show that overparameterized neural networks trained by SGD can generalize when the data is corrupted with adversarial label noise.

cs.LG↗

arXiv2021

Agnostic Learning of Halfspaces with Gradient Descent via Soft Margins

We analyze the properties of gradient descent on convex surrogates for the zero-one loss for the agnostic learning of linear halfspaces. If $\mathsf{OPT}$ is the best classification error achieved by a halfspace, by appealing to the notion of soft margins we are able to show that gradient descent finds halfspaces with classification error $\tilde O(\mathsf{OPT}^{1/2}) + \varepsilon$ in $\mathrm{poly}(d,1/\varepsilon)$ time and sample complexity for a broad class of distributions that includes log-concave isotropic distributions as a subclass. Along the way we answer a question recently posed by Ji et al. (2020) on how the tail behavior of a loss function can affect sample complexity and runtime guarantees for gradient descent.

cs.LG↗

arXiv2021

High-Temperature Structure Detection in Ferromagnets

This paper studies structure detection problems in high temperature ferromagnetic (positive interaction only) Ising models. The goal is to distinguish whether the underlying graph is empty, i.e., the model consists of independent Rademacher variables, versus the alternative that the underlying graph contains a subgraph of a certain structure. We give matching upper and lower minimax bounds under which testing this problem is possible/impossible respectively. Our results reveal that a key quantity called graph arboricity drives the testability of the problem. On the computational front, under a conjecture of the computational hardness of sparse principal component analysis, we prove that, unless the signal is strong enough, there are no polynomial time tests which are capable of testing this problem. In order to prove this result we exhibit a way to give sharp inequalities for the even moments of sums of i.i.d. Rademacher random variables which may be of independent interest.

math.ST↗

arXiv2021

Unconventional sequence of correlated Chern insulators in magic-angle twisted bilayer graphene

The interplay between strong electron-electron interactions and band topology can lead to novel electronic states that spontaneously break symmetries. The discovery of flat bands in magic-angle twisted bilayer graphene (MATBG) with nontrivial topology has provided a unique platform in which to search for new symmetry-broken phases. Recent scanning tunneling microscopy and transport experiments have revealed a sequence of topological insulating phases in MATBG with Chern numbers $C=\pm 3, \, \pm 2, \, \pm 1$ near moiré band filling factors $ν= \pm 1, \, \pm 2, \, \pm 3$, corresponding to a simple pattern of flavor-symmetry-breaking Chern insulators. Here, we report high-resolution local compressibility measurements of MATBG with a scanning single electron transistor that reveal a new sequence of incompressible states with unexpected Chern numbers observed down to zero magnetic field. We find that the Chern numbers for eight of the observed incompressible states are incompatible with the simple picture in which the $C= \pm 1$ bands are sequentially filled. We show that the emergence of these unusual incompressible phases can be understood as a consequence of broken translation symmetry that doubles the moiré unit cell and splits each $C=\pm 1$ band into a $C=\pm 1$ band and a $C=0$ band. Our findings significantly expand the known phase diagram of MATBG, and shed light onto the origin of the close competition between different correlated phases in the system.

cond-mat.mes-hall↗

arXiv2021

Quantum Random Number Generation with Uncharacterized Laser and Sunlight

The entropy or randomness source is an essential ingredient in random number generation. Quantum random number generators generally require well modeled and calibrated light sources, such as a laser, to generate randomness. With uncharacterized light sources, such as sunlight or an uncharacterized laser, genuine randomness is practically hard to be quantified or extracted owing to its unknown or complicated structure. By exploiting a recently proposed source-independent randomness generation protocol, we theoretically modify it by considering practical issues and experimentally realize the modified scheme with an uncharacterized laser and a sunlight source. The extracted randomness is guaranteed to be secure independent of its source and the randomness generation speed reaches 1 Mbps, three orders of magnitude higher than the original realization. Our result signifies the power of quantum technology in randomness generation and paves the way to high-speed semi-self-testing quantum random number generators with practical light sources.

quant-ph↗

arXiv2021

Distortionless pulse transmission in valley photonic crystal slab waveguide

Valley photonic crystal is one type of photonic topological insulator, whose realization only needs P-symmetry breaking. The domain wall between two valley-contrasting photonic crystals support robust edge states which can wrap around sharp corners without backscattering. Using the robust edge states, one can achieve the pulse transmission. Here, using time-domain measurement in the microwave regime, we show distortionless pulse transmission in a sharply bended waveguide. An Omega-shaped waveguide with four 120-degree bends is constructed with the domain wall between two valley photonic crystal slabs. Experimental results show the progress of Gaussian pulse transmission without distortion, and the full width at half maximum of the output signal was changed slightly in the Omega-shaped waveguide. By measuring steady state electric field distribution, we also confirmed the confined edge states without out-of-plane radiation which benefits from the dispersion below the light line. Our work provides a way for high-fidelity optical pulse signal transmission and develop high-performance optical elements such as photonic circuits or optical delay lines.

physics.app-ph↗

arXiv2020

Single-mode fiber coupling with a M-SPGD algorithm for long-range quantum communications

Satellite-based quantum communication is a promising approach for realizing global-scale quantum networks. For free-space quantum channel, single-mode fiber coupling is particularly important for improving signal-to-noise ratio of daylight quantum key distribution (QKD) and compatibility with standard fiber-based QKD. However, achieving a highly efficient and stable single-mode coupling efficiency under strong atmospheric turbulence remains experimentally challenging. Here, we develop a single-mode receiver with an adaptive optics (AO) system based on a modal version of the stochastic parallel gradient descent (M-SPGD) algorithm and test its performance over an 8 km urban terrestrial free-space channel. Under strong atmospheric turbulence, the M-SPGD AO system obtains an improvement of about 3.7 dB in the single-mode fiber coupling efficiency and a significant suppression of fluctuation, which can find its applications in free-space long-range quantum communications.

quant-ph↗

arXiv2020

Tunable Phase Boundaries and Ultra-Strong Coupling Superconductivity in Mirror Symmetric Magic-Angle Trilayer Graphene

Moiré superlattices have recently emerged as a novel platform where correlated physics and superconductivity can be studied with unprecedented tunability. Although correlated effects have been observed in several other moiré systems, magic-angle twisted bilayer graphene (MATBG) remains the only one where robust superconductivity has been reproducibly measured. Here we realize a new moiré superconductor, mirror symmetric magic-angle twisted trilayer graphene (MATTG) with dramatically richer tunability in electronic structure and superconducting properties. Hall effect and quantum oscillations measurements as a function of density and electric field allow us to determine the system's tunable phase boundaries in the normal state. Zero magnetic field resistivity measurements then reveal that the existence of superconductivity is intimately connected to the broken symmetry phase emerging from two carriers per moiré unit cell. Strikingly, we find that the superconducting phase gets suppressed and bounded at the van Hove singularities (vHs) partially surrounding the broken-symmetry phase, which is difficult to reconcile with weak-coupling BCS theory. Moreover, the extensive in situ tunability of our system allows us to achieve the ultra-strong coupling regime, characterized by a Ginzburg-Landau coherence length reaching the average inter-particle distance and very large $T_\mathrm{BKT}/T_{F}$ ratios in excess of 0.1, where $T_\mathrm{BKT}$ and $T_F$ are the Berezinskii-Kosterlitz-Thouless transition and Fermi temperatures, respectively. These observations suggest that MATTG can be electrically tuned close to the two-dimensional BCS-BEC crossover. Our results establish a new generation of tunable moiré superconductors with the potential to revolutionize our fundamental understanding and the applications of strong coupling superconductivity.

cond-mat.supr-con↗

arXiv2020

Towards NNGP-guided Neural Architecture Search

The predictions of wide Bayesian neural networks are described by a Gaussian process, known as the Neural Network Gaussian Process (NNGP). Analytic forms for NNGP kernels are known for many models, but computing the exact kernel for convolutional architectures is prohibitively expensive. One can obtain effective approximations of these kernels through Monte-Carlo estimation using finite networks at initialization. Monte-Carlo NNGP inference is orders-of-magnitude cheaper in FLOPs compared to gradient descent training when the dataset size is small. Since NNGP inference provides a cheap measure of performance of a network architecture, we investigate its potential as a signal for neural architecture search (NAS). We compute the NNGP performance of approximately 423k networks in the NAS-bench 101 dataset on CIFAR-10 and compare its utility against conventional performance measures obtained by shortened gradient-based training. We carry out a similar analysis on 10k randomly sampled networks in the mobile neural architecture search (MNAS) space for ImageNet. We discover comparative advantages of NNGP-based metrics, and discuss potential applications. In particular, we propose that NNGP performance is an inexpensive signal independent of metrics obtained from training that can either be used for reducing big search spaces, or improving training-based performance measures.

cs.LG↗

arXiv2020

Highly Tunable Junctions and Nonlocal Josephson Effect in Magic Angle Graphene Tunneling Devices

Magic-angle twisted bilayer graphene (MATBG) has recently emerged as a highly tunable two-dimensional (2D) material platform exhibiting a wide range of phases, such as metal, insulator, and superconductor states. Local electrostatic control over these phases may enable the creation of versatile quantum devices that were previously not achievable in other single material platforms. Here, we exploit the electrical tunability of MATBG to engineer Josephson junctions and tunneling transistors all within one material, defined solely by electrostatic gates. Our multi-gated device geometry offers complete control over the Josephson junction, with the ability to independently tune the weak link, barriers, and tunneling electrodes. We show that these purely 2D MATBG Josephson junctions exhibit nonlocal electrodynamics in a magnetic field, in agreement with the Pearl theory for ultrathin superconductors. Utilizing the intrinsic bandgaps of MATBG, we also demonstrate monolithic edge tunneling spectroscopy within the same MATBG devices and measure the energy spectrum of MATBG in the superconducting phase. Furthermore, by inducing a double barrier geometry, the devices can be operated as a single-electron transistor, exhibiting Coulomb blockade. These MATBG tunneling devices, with versatile functionality encompassed within a single material, may find applications in graphene-based tunable superconducting qubits, on-chip superconducting circuits, and electromagnetic sensing in next-generation quantum nanoelectronics.

cond-mat.supr-con↗

arXiv2020

Rapid Domain Adaptation for Machine Translation with Monolingual Data

One challenge of machine translation is how to quickly adapt to unseen domains in face of surging events like COVID-19, in which case timely and accurate translation of in-domain information into multiple languages is critical but little parallel data is available yet. In this paper, we propose an approach that enables rapid domain adaptation from the perspective of unsupervised translation. Our proposed approach only requires in-domain monolingual data and can be quickly applied to a preexisting translation system trained on general domain, reaching significant gains on in-domain translation quality with little or no drop on general-domain. We also propose an effective procedure of simultaneous adaptation for multiple domains and languages. To the best of our knowledge, this is the first attempt that aims to address unsupervised multilingual domain adaptation.

cs.CL↗

arXiv2020

Deciphering Undersegmented Ancient Scripts Using Phonetic Prior

Most undeciphered lost languages exhibit two characteristics that pose significant decipherment challenges: (1) the scripts are not fully segmented into words; (2) the closest known language is not determined. We propose a decipherment model that handles both of these challenges by building on rich linguistic constraints reflecting consistent patterns in historical sound change. We capture the natural phonological geometry by learning character embeddings based on the International Phonetic Alphabet (IPA). The resulting generative framework jointly models word segmentation and cognate alignment, informed by phonological constraints. We evaluate the model on both deciphered languages (Gothic, Ugaritic) and an undeciphered one (Iberian). The experiments show that incorporating phonetic geometry leads to clear and consistent gains. Additionally, we propose a measure for language closeness which correctly identifies related languages for Gothic and Ugaritic. For Iberian, the method does not show strong evidence supporting Basque as a related language, concurring with the favored position by the current scholarship.

cs.CL↗

arXiv2020

Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Models

Massively multilingual models subsuming tens or even hundreds of languages pose great challenges to multi-task optimization. While it is a common practice to apply a language-agnostic procedure optimizing a joint multilingual task objective, how to properly characterize and take advantage of its underlying problem structure for improving optimization efficiency remains under-explored. In this paper, we attempt to peek into the black-box of multilingual optimization through the lens of loss function geometry. We find that gradient similarity measured along the optimization trajectory is an important signal, which correlates well with not only language proximity but also the overall model performance. Such observation helps us to identify a critical limitation of existing gradient-based multi-task learning methods, and thus we derive a simple and scalable optimization procedure, named Gradient Vaccine, which encourages more geometrically aligned parameter updates for close tasks. Empirically, our method obtains significant model performance gains on multilingual machine translation and XTREME benchmark tasks for multilingual language models. Our work reveals the importance of properly measuring and utilizing language proximity in multilingual optimization, and has broader implications for multi-task learning beyond multilingual modeling.

cs.CL↗

arXiv2020

A Generalized Neural Tangent Kernel Analysis for Two-layer Neural Networks

A recent breakthrough in deep learning theory shows that the training of over-parameterized deep neural networks can be characterized by a kernel function called \textit{neural tangent kernel} (NTK). However, it is known that this type of results does not perfectly match the practice, as NTK-based analysis requires the network weights to stay very close to their initialization throughout training, and cannot handle regularizers or gradient noises. In this paper, we provide a generalized neural tangent kernel analysis and show that noisy gradient descent with weight decay can still exhibit a "kernel-like" behavior. This implies that the training loss converges linearly up to a certain accuracy. We also establish a novel generalization error bound for two-layer neural networks trained by noisy gradient descent with weight decay.

cs.LG↗

arXiv2020

Towards Understanding the Spectral Bias of Deep Learning

An intriguing phenomenon observed during training neural networks is the spectral bias, which states that neural networks are biased towards learning less complex functions. The priority of learning functions with low complexity might be at the core of explaining generalization ability of neural network, and certain efforts have been made to provide theoretical explanation for spectral bias. However, there is still no satisfying theoretical result justifying the underlying mechanism of spectral bias. In this paper, we give a comprehensive and rigorous explanation for spectral bias and relate it with the neural tangent kernel function proposed in recent work. We prove that the training process of neural networks can be decomposed along different directions defined by the eigenfunctions of the neural tangent kernel, where each direction has its own convergence rate and the rate is determined by the corresponding eigenvalue. We then provide a case study when the input data is uniformly distributed over the unit sphere, and show that lower degree spherical harmonics are easier to be learned by over-parameterized neural networks. Finally, we provide numerical experiments to demonstrate the correctness of our theory. Our experimental results also show that our theory can tolerate certain model misspecification in terms of the input data distribution.

cs.LG↗

arXiv2020

Entropic evidence for a Pomeranchuk effect in magic angle graphene

In the 1950's, Pomeranchuk predicted that, counterintuitively, liquid 3He may solidify upon heating, due to a high excess spin entropy in the solid phase. Here, using both local and global electronic entropy and compressibility measurements, we show that an analogous effect occurs in magic angle twisted bilayer graphene. Near a filling of one electron per moir'e unit cell, we observe a dramatic increase in the electronic entropy to about 1kB per unit cell. This large excess entropy is quenched by an in-plane magnetic field, pointing to its magnetic origin. A sharp drop in the compressibility as a function of the electron density, associated with a reset of the Fermi level back to the vicinity of the Dirac point, marks a clear boundary between two phases. We map this jump as a function of electron density, temperature, and magnetic field. This reveals a phase diagram that is consistent with a Pomeranchuk-like temperature- and field-driven transition from a low-entropy electronic liquid to a high-entropy correlated state with nearly-free magnetic moments. The correlated state features an unusual combination of seemingly contradictory properties, some associated with itinerant electrons, such as the absence of a thermodynamic gap, metallicity, and a Dirac-like compressibility, and others associated with localized moments, such as a large entropy and its disappearance with magnetic field. Moreover, the energy scales characterizing these two sets of properties are very different: whereas the compressibility jump onsets at T~30K, the bandwidth of magnetic excitations is ~3K or smaller. The hybrid nature of the new correlated state and the large separation of energy scales have key implications for the physics of correlated states in twisted bilayer graphene.

cond-mat.mes-hall↗

arXiv2020

Agnostic Learning of a Single Neuron with Gradient Descent

We consider the problem of learning the best-fitting single neuron as measured by the expected square loss $\mathbb{E}_{(x,y)\sim \mathcal{D}}[(σ(w^\top x)-y)^2]$ over some unknown joint distribution $\mathcal{D}$ by using gradient descent to minimize the empirical risk induced by a set of i.i.d. samples $S\sim \mathcal{D}^n$. The activation function $σ$ is an arbitrary Lipschitz and non-decreasing function, making the optimization problem nonconvex and nonsmooth in general, and covers typical neural network activation functions and inverse link functions in the generalized linear model setting. In the agnostic PAC learning setting, where no assumption on the relationship between the labels $y$ and the input $x$ is made, if the optimal population risk is $\mathsf{OPT}$, we show that gradient descent achieves population risk $O(\mathsf{OPT})+ε$ in polynomial time and sample complexity when $σ$ is strictly increasing. For the ReLU activation, our population risk guarantee is $O(\mathsf{OPT}^{1/2})+ε$. When labels take the form $y = σ(v^\top x) + ξ$ for zero-mean sub-Gaussian noise $ξ$, we show that the population risk guarantees for gradient descent improve to $\mathsf{OPT} + ε$. Our sample complexity and runtime guarantees are (almost) dimension independent, and when $σ$ is strictly increasing, require no distributional assumptions beyond boundedness. For ReLU, we show the same results under a nondegeneracy assumption for the marginal distribution of the input.

cs.LG↗