arXiv subjects

Yuan Cao

Publications and source records attributed to Yuan Cao.

At least 145 records · Page 8Linked to original sources

arXiv2022

Multilingual Mix: Example Interpolation Improves Multilingual Neural Machine Translation

Multilingual neural machine translation models are trained to maximize the likelihood of a mix of examples drawn from multiple language pairs. The dominant inductive bias applied to these models is a shared vocabulary and a shared set of parameters across languages; the inputs and labels corresponding to examples drawn from different language pairs might still reside in distinct sub-spaces. In this paper, we introduce multilingual crossover encoder-decoder (mXEncDec) to fuse language pairs at an instance level. Our approach interpolates instances from different language pairs into joint `crossover examples' in order to encourage sharing input and output spaces across languages. To ensure better fusion of examples in multilingual settings, we propose several techniques to improve example interpolation across dissimilar languages under heavy data imbalance. Experiments on a large-scale WMT multilingual dataset demonstrate that our approach significantly improves quality on English-to-Many, Many-to-English and zero-shot translation tasks (from +0.5 BLEU up to +5.5 BLEU points). Results on code-switching sets demonstrate the capability of our approach to improve model generalization to out-of-distribution multilingual examples. We also conduct qualitative and quantitative representation comparisons to analyze the advantages of our approach at the representation level.

cs.CL↗

arXiv2022

Description-Driven Task-Oriented Dialog Modeling

Task-oriented dialogue (TOD) systems are required to identify key information from conversations for the completion of given tasks. Such information is conventionally specified in terms of intents and slots contained in task-specific ontology or schemata. Since these schemata are designed by system developers, the naming convention for slots and intents is not uniform across tasks, and may not convey their semantics effectively. This can lead to models memorizing arbitrary patterns in data, resulting in suboptimal performance and generalization. In this paper, we propose that schemata should be modified by replacing names or notations entirely with natural language descriptions. We show that a language description-driven system exhibits better understanding of task specifications, higher performance on state tracking, improved data efficiency, and effective zero-shot transfer to unseen tasks. Following this paradigm, we present a simple yet effective Description-Driven Dialog State Tracking (D3ST) model, which relies purely on schema descriptions and an "index-picking" mechanism. We demonstrate the superiority in quality, data efficiency and robustness of our approach as measured on the MultiWOZ (Budzianowski et al.,2018), SGD (Rastogi et al., 2020), and the recent SGD-X (Lee et al., 2021) benchmarks.

cs.CL↗

arXiv2022

Towards the Next 1000 Languages in Multilingual Machine Translation: Exploring the Synergy Between Supervised and Self-Supervised Learning

Achieving universal translation between all human language pairs is the holy-grail of machine translation (MT) research. While recent progress in massively multilingual MT is one step closer to reaching this goal, it is becoming evident that extending a multilingual MT system simply by training on more parallel data is unscalable, since the availability of labeled data for low-resource and non-English-centric language pairs is forbiddingly limited. To this end, we present a pragmatic approach towards building a multilingual MT model that covers hundreds of languages, using a mixture of supervised and self-supervised objectives, depending on the data availability for different language pairs. We demonstrate that the synergy between these two training paradigms enables the model to produce high-quality translations in the zero-resource setting, even surpassing supervised translation quality for low- and mid-resource languages. We conduct a wide array of experiments to understand the effect of the degree of multilingual supervision, domain mismatches and amounts of parallel and monolingual data on the quality of our self-supervised multilingual models. To demonstrate the scalability of the approach, we train models with over 200 languages and demonstrate high performance on zero-resource translation on several previously under-studied languages. We hope our findings will serve as a stepping stone towards enabling translation for the next thousand languages.

cs.CL↗

arXiv2022

Risk Bounds for Over-parameterized Maximum Margin Classification on Sub-Gaussian Mixtures

Modern machine learning systems such as deep neural networks are often highly over-parameterized so that they can fit the noisy training data exactly, yet they can still achieve small test errors in practice. In this paper, we study this "benign overfitting" phenomenon of the maximum margin classifier for linear classification problems. Specifically, we consider data generated from sub-Gaussian mixtures, and provide a tight risk bound for the maximum margin linear classifier in the over-parameterized setting. Our results precisely characterize the condition under which benign overfitting can occur in linear classification problems, and improve on previous work. They also have direct implications for over-parameterized logistic regression.

cs.LG↗

arXiv2021

Benign Overfitting in Adversarially Robust Linear Classification

"Benign overfitting", where classifiers memorize noisy training data yet still achieve a good generalization performance, has drawn great attention in the machine learning community. To explain this surprising phenomenon, a series of works have provided theoretical justification in over-parameterized linear regression, classification, and kernel methods. However, it is not clear if benign overfitting still occurs in the presence of adversarial examples, i.e., examples with tiny and intentional perturbations to fool the classifiers. In this paper, we show that benign overfitting indeed occurs in adversarial training, a principled approach to defend against adversarial examples. In detail, we prove the risk bounds of the adversarially trained linear classifier on the mixture of sub-Gaussian data under $\ell_p$ adversarial perturbations. Our result suggests that under moderate perturbations, adversarially trained linear classifiers can achieve the near-optimal standard and adversarial risks, despite overfitting the noisy training data. Numerical experiments validate our theoretical findings.

cs.LG↗

arXiv2021

How Much Over-parameterization Is Sufficient to Learn Deep ReLU Networks?

A recent line of research on deep learning focuses on the extremely over-parameterized setting, and shows that when the network width is larger than a high degree polynomial of the training sample size $n$ and the inverse of the target error $ε^{-1}$, deep neural networks learned by (stochastic) gradient descent enjoy nice optimization and generalization guarantees. Very recently, it is shown that under certain margin assumptions on the training data, a polylogarithmic width condition suffices for two-layer ReLU networks to converge and generalize (Ji and Telgarsky, 2019). However, whether deep neural networks can be learned with such a mild over-parameterization is still an open question. In this work, we answer this question affirmatively and establish sharper learning guarantees for deep ReLU networks trained by (stochastic) gradient descent. In specific, under certain assumptions made in previous work, our optimization and generalization guarantees hold with network width polylogarithmic in $n$ and $ε^{-1}$. Our results push the study of over-parameterized deep neural networks towards more practical settings.

cs.LG↗

arXiv2021

Magic-Angle Multilayer Graphene: A Robust Family of Moiré Superconductors

The discovery of correlated states and superconductivity in magic-angle twisted bilayer graphene (MATBG) has established moiré quantum matter as a new platform to explore interaction-driven and topological quantum phenomena. Multitudes of phases have been realized in moiré systems, but surprisingly, robust superconductivity has been one of the least common of all, initially found in MATBG and only more recently also in magic-angle twisted trilayer graphene (MATTG). While MATBG and MATTG share some similar characteristics, they also exhibit substantial differences, such as in their response to external electric and magnetic fields. This raises the question of whether they are simply two separate unique systems, or whether they form part of a broader family of superconducting materials. Here, we report the experimental realization of magic-angle twisted 4-layer and 5-layer graphene (MAT4G and MAT5G, respectively), which turn out to be superconductors, hence establishing alternating-twist magic-angle multilayer graphene as a robust family of moiré superconductors. The members of this family have flat bands in their electronic structure as a common feature, suggesting their central role in the observed robust superconductivity. On the other hand, there are also important variations across the family, such as different symmetries for members with even and odd number of layers. However, our measurements in parallel magnetic fields, in particular the investigation of Pauli limit violation and spontaneous rotational symmetry breaking, reveal that the most pronounced distinction is between the N=2 and N>2-layer structures. Our results expand the emergent family of moiré superconductors, providing new insight with potential implications for the design of novel superconducting materials platforms.

cond-mat.supr-con↗

arXiv2021

Efficient and Private Federated Learning with Partially Trainable Networks

Federated learning is used for decentralized training of machine learning models on a large number (millions) of edge mobile devices. It is challenging because mobile devices often have limited communication bandwidth and local computation resources. Therefore, improving the efficiency of federated learning is critical for scalability and usability. In this paper, we propose to leverage partially trainable neural networks, which freeze a portion of the model parameters during the entire training process, to reduce the communication cost with little implications on model performance. Through extensive experiments, we empirically show that Federated learning of Partially Trainable neural networks (FedPT) can result in superior communication-accuracy trade-offs, with up to $46\times$ reduction in communication cost, at a small accuracy cost. Our approach also enables faster training, with a smaller memory footprint, and better utility for strong differential privacy guarantees. The proposed FedPT method can be particularly interesting for pushing the limitations of over-parameterization in on-device learning.

cs.LG↗

arXiv2021

Understanding How Encoder-Decoder Architectures Attend

Encoder-decoder networks with attention have proven to be a powerful way to solve many sequence-to-sequence tasks. In these networks, attention aligns encoder and decoder states and is often used for visualizing network behavior. However, the mechanisms used by networks to generate appropriate attention matrices are still mysterious. Moreover, how these mechanisms vary depending on the particular architecture used for the encoder and decoder (recurrent, feed-forward, etc.) are also not well understood. In this work, we investigate how encoder-decoder networks solve different sequence-to-sequence tasks. We introduce a way of decomposing hidden states over a sequence into temporal (independent of input) and input-driven (independent of sequence position) components. This reveals how attention matrices are formed: depending on the task requirements, networks rely more heavily on either the temporal or input-driven components. These findings hold across both recurrent and feed-forward architectures despite their differences in forming the temporal components. Overall, our results provide new insight into the inner workings of attention-based encoder-decoder networks.

cs.LG↗

arXiv2021

Towards Zero-Label Language Learning

This paper explores zero-label learning in Natural Language Processing (NLP), whereby no human-annotated data is used anywhere during training and models are trained purely on synthetic data. At the core of our framework is a novel approach for better leveraging the powerful pretrained language models. Specifically, inspired by the recent success of few-shot inference on GPT-3, we present a training data creation procedure named Unsupervised Data Generation (UDG), which leverages few-shot prompts to synthesize high-quality training data without real human annotations. Our method enables zero-label learning as we train task-specific models solely on the synthetic data, yet we achieve better or comparable results from strong baseline models trained on human-labeled data. Furthermore, when mixed with labeled data, our approach serves as a highly effective data augmentation procedure, achieving new state-of-the-art results on the SuperGLUE benchmark.

cs.CL↗

arXiv2021

Effective Sequence-to-Sequence Dialogue State Tracking

Sequence-to-sequence models have been applied to a wide variety of NLP tasks, but how to properly use them for dialogue state tracking has not been systematically investigated. In this paper, we study this problem from the perspectives of pre-training objectives as well as the formats of context representations. We demonstrate that the choice of pre-training objective makes a significant difference to the state tracking quality. In particular, we find that masked span prediction is more effective than auto-regressive language modeling. We also explore using Pegasus, a span prediction-based pre-training objective for text summarization, for the state tracking model. We found that pre-training for the seemingly distant summarization task works surprisingly well for dialogue state tracking. In addition, we found that while recurrent state context representation works also reasonably well, the model may have a hard time recovering from earlier mistakes. We conducted experiments on the MultiWOZ 2.1-2.4, WOZ 2.0, and DSTC2 datasets with consistent observations.

cs.CL↗

arXiv2021

Understanding the Generalization of Adam in Learning Neural Networks with Proper Regularization

Adaptive gradient methods such as Adam have gained increasing popularity in deep learning optimization. However, it has been observed that compared with (stochastic) gradient descent, Adam can converge to a different solution with a significantly worse test error in many deep learning applications such as image classification, even with a fine-tuned regularization. In this paper, we provide a theoretical explanation for this phenomenon: we show that in the nonconvex setting of learning over-parameterized two-layer convolutional neural networks starting from the same random initialization, for a class of data distributions (inspired from image data), Adam and gradient descent (GD) can converge to different global solutions of the training objective with provably different generalization errors, even with weight decay regularization. In contrast, we show that if the training objective is convex, and the weight decay regularization is employed, any optimization algorithms including Adam and GD will converge to the same solution if the training is successful. This suggests that the inferior generalization performance of Adam is fundamentally tied to the nonconvex landscape of deep learning optimization.

cs.LG↗

arXiv2021

Fractional Chern insulators in magic-angle twisted bilayer graphene

Fractional Chern insulators (FCIs) are lattice analogues of fractional quantum Hall states that may provide a new avenue toward manipulating non-abelian excitations. Early theoretical studies have predicted their existence in systems with energetically flat Chern bands and highlighted the critical role of a particular quantum band geometry. Thus far, however, FCI states have only been observed in Bernal-stacked bilayer graphene aligned with hexagonal boron nitride (BLG/hBN), in which a very large magnetic field is responsible for the existence of the Chern bands, precluding the realization of FCIs at zero field and limiting its potential for applications. By contrast, magic angle twisted bilayer graphene (MATBG) supports flat Chern bands at zero magnetic field, and therefore offers a promising route toward stabilizing zero-field FCIs. Here we report the observation of eight FCI states at low magnetic field in MATBG enabled by high-resolution local compressibility measurements. The first of these states emerge at 5 T, and their appearance is accompanied by the simultaneous disappearance of nearby topologically-trivial charge density wave states. Unlike the BLG/hBN platform, we demonstrate that the principal role of the weak magnetic field here is merely to redistribute the Berry curvature of the native Chern bands and thereby realize a quantum band geometry favorable for the emergence of FCIs. Our findings strongly suggest that FCIs may be realized at zero magnetic field and pave the way for the exploration and manipulation of anyonic excitations in moiré systems with native flat Chern bands.

cond-mat.mes-hall↗

arXiv2021

Your GAN is Secretly an Energy-based Model and You Should use Discriminator Driven Latent Sampling

We show that the sum of the implicit generator log-density $\log p_g$ of a GAN with the logit score of the discriminator defines an energy function which yields the true data density when the generator is imperfect but the discriminator is optimal, thus making it possible to improve on the typical generator (with implicit density $p_g$). To make that practical, we show that sampling from this modified density can be achieved by sampling in latent space according to an energy-based model induced by the sum of the latent prior log-density and the discriminator output score. This can be achieved by running a Langevin MCMC in latent space and then applying the generator function, which we call Discriminator Driven Latent Sampling~(DDLS). We show that DDLS is highly efficient compared to previous methods which work in the high-dimensional pixel space and can be applied to improve on previously trained GANs of many types. We evaluate DDLS on both synthetic and real-world datasets qualitatively and quantitatively. On CIFAR-10, DDLS substantially improves the Inception Score of an off-the-shelf pre-trained SN-GAN~\citep{sngan} from $8.22$ to $9.09$ which is even comparable to the class-conditional BigGAN~\citep{biggan} model. This achieves a new state-of-the-art in unconditional image synthesis setting without introducing extra parameters or additional training.

cs.LG↗

arXiv2021

Construction and enumeration of left dihedral codes satisfying certain duality properties

Let $\mathbb{F}_{q}$ be the finite field of $q$ elements and let $D_{2n}=\langle x,y\mid x^n=1, y^2=1, yxy=x^{n-1}\rangle$ be the dihedral group of order $n$. Left ideals of the group algebra $\mathbb{F}_{q}[D_{2n}]$ are known as left dihedral codes over $\mathbb{F}_{q}$ of length $2n$, and abbreviated as left $D_{2n}$-codes. Let ${\rm gcd}(n,q)=1$. In this paper, we give an explicit representation for the Euclidean hull of every left $D_{2n}$-code over $\mathbb{F}_{q}$. On this basis, we determine all distinct Euclidean LCD codes and Euclidean self-orthogonal codes which are left $D_{2n}$-codes over $\mathbb{F}_{q}$. In particular, we provide an explicit representation and a precise enumeration for these two subclasses of left $D_{2n}$-codes and self-dual left $D_{2n}$-codes, respectively. Moreover, we give a direct and simple method for determining the encoder (generator matrix) of any left $D_{2n}$-code over $\mathbb{F}_{q}$, and present several numerical examples to illustrative our applications.

cs.IT↗

arXiv2021

Improving Longer-range Dialogue State Tracking

Dialogue state tracking (DST) is a pivotal component in task-oriented dialogue systems. While it is relatively easy for a DST model to capture belief states in short conversations, the task of DST becomes more challenging as the length of a dialogue increases due to the injection of more distracting contexts. In this paper, we aim to improve the overall performance of DST with a special focus on handling longer dialogues. We tackle this problem from three perspectives: 1) A model designed to enable hierarchical slot status prediction; 2) Balanced training procedure for generic and task-specific language understanding; 3) Data perturbation which enhances the model's ability in handling longer conversations. We conduct experiments on the MultiWOZ benchmark, and demonstrate the effectiveness of each component via a set of ablation tests, especially on longer conversations.

cs.CL↗

arXiv2021

Large Pauli Limit Violation and Reentrant Superconductivity in Magic-Angle Twisted Trilayer Graphene

Moiré quantum matter has emerged as a novel materials platform where correlated and topological phases can be explored with unprecedented control. Among them, magic-angle systems constructed from two or three layers of graphene have shown robust superconducting phases with unconventional characteristics. However, direct evidence for unconventional pairing remains to be experimentally demonstrated. Here, we show that magic-angle twisted trilayer graphene (MATTG) exhibits superconductivity up to in-plane magnetic fields in excess of 10 T, which represents a large ($2\sim3$ times) violation of the Pauli limit for conventional spin-singlet superconductors. This observation is surprising for a system which is not expected to have strong spin-orbit coupling. Furthermore, the Pauli limit violation is observed over the entire superconducting phase, indicating that it is not related to a possible pseudogap phase with large superconducting amplitude pairing. More strikingly, we observe reentrant superconductivity at large magnetic fields, which is present in a narrower range of carrier density and displacement field. These findings suggest that the superconductivity in MATTG is likely driven by a mechanism resulting in non-spin-singlet Cooper pairs, where the external magnetic field can cause transitions between phases with potentially different order parameters. Our results showcase the richness of moiré superconductivity and may pave a new route towards designing next-generation exotic quantum matter.

cond-mat.mes-hall↗

arXiv2021

Single-photon imaging over 200 km

Long-range active imaging has widespread applications in remote sensing and target recognition. Single-photon light detection and ranging (lidar) has been shown to have high sensitivity and temporal resolution. On the application front, however, the operating range of practical single-photon lidar systems is limited to about tens of kilometers over the Earth's atmosphere, mainly due to the weak echo signal mixed with high background noise. Here, we present a compact coaxial single-photon lidar system capable of realizing 3D imaging at up to 201.5 km. It is achieved by using high-efficiency optical devices for collection and detection, and what we believe is a new noise-suppression technique that is efficient for long-range applications. We show that photon-efficient computational algorithms enable accurate 3D imaging over hundreds of kilometers with as few as 0.44 signal photons per pixel. The results represent a significant step toward practical, low-power lidar over extra-long ranges.

eess.IV↗