arXiv subjects

Ekin D. Cubuk

Publications and source records attributed to Ekin D. Cubuk.

At least 37 records · Page 2Linked to original sources

arXiv2020

Affinity and Diversity: Quantifying Mechanisms of Data Augmentation

Though data augmentation has become a standard component of deep neural network training, the underlying mechanism behind the effectiveness of these techniques remains poorly understood. In practice, augmentation policies are often chosen using heuristics of either distribution shift or augmentation diversity. Inspired by these, we seek to quantify how data augmentation improves model generalization. To this end, we introduce interpretable and easy-to-compute measures: Affinity and Diversity. We find that augmentation performance is predicted not by either of these alone but by jointly optimizing the two.

cs.LG↗

arXiv2020

AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty

Modern deep neural networks can achieve high accuracy when the training distribution and test distribution are identically distributed, but this assumption is frequently violated in practice. When the train and test distributions are mismatched, accuracy can plummet. Currently there are few techniques that improve robustness to unforeseen data shifts encountered during deployment. In this work, we propose a technique to improve the robustness and uncertainty estimates of image classifiers. We propose AugMix, a data processing technique that is simple to implement, adds limited computational overhead, and helps models withstand unforeseen corruptions. AugMix significantly improves robustness and uncertainty measures on challenging image classification benchmarks, closing the gap between previous methods and the best possible performance in some cases by more than half.

stat.ML↗

arXiv2020

ReMixMatch: Semi-Supervised Learning with Distribution Alignment and Augmentation Anchoring

We improve the recently-proposed "MixMatch" semi-supervised learning algorithm by introducing two new techniques: distribution alignment and augmentation anchoring. Distribution alignment encourages the marginal distribution of predictions on unlabeled data to be close to the marginal distribution of ground-truth labels. Augmentation anchoring feeds multiple strongly augmented versions of an input into the model and encourages each output to be close to the prediction for a weakly-augmented version of the same input. To produce strong augmentations, we propose a variant of AutoAugment which learns the augmentation policy while the model is being trained. Our new algorithm, dubbed ReMixMatch, is significantly more data-efficient than prior work, requiring between $5\times$ and $16\times$ less data to reach the same accuracy. For example, on CIFAR-10 with 250 labeled examples we reach $93.73\%$ accuracy (compared to MixMatch's accuracy of $93.58\%$ with $4{,}000$ examples) and a median accuracy of $84.92\%$ with just four labels per class. We make our code and data open-source at https://github.com/google-research/remixmatch.

cs.LG↗

arXiv2019

SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition

We present SpecAugment, a simple data augmentation method for speech recognition. SpecAugment is applied directly to the feature inputs of a neural network (i.e., filter bank coefficients). The augmentation policy consists of warping the features, masking blocks of frequency channels, and masking blocks of time steps. We apply SpecAugment on Listen, Attend and Spell networks for end-to-end speech recognition tasks. We achieve state-of-the-art performance on the LibriSpeech 960h and Swichboard 300h tasks, outperforming all prior work. On LibriSpeech, we achieve 6.8% WER on test-other without the use of a language model, and 5.8% WER with shallow fusion with a language model. This compares to the previous state-of-the-art hybrid system of 7.5% WER. For Switchboard, we achieve 7.2%/14.6% on the Switchboard/CallHome portion of the Hub5'00 test set without the use of a language model, and 6.8%/14.1% with shallow fusion, which compares to the previous state-of-the-art hybrid system at 8.3%/17.3% WER.

eess.AS↗

arXiv2019

RandAugment: Practical automated data augmentation with a reduced search space

Recent work has shown that data augmentation has the potential to significantly improve the generalization of deep learning models. Recently, automated augmentation strategies have led to state-of-the-art results in image classification and object detection. While these strategies were optimized for improving validation accuracy, they also led to state-of-the-art results in semi-supervised learning and improved robustness to common corruptions of images. An obstacle to a large-scale adoption of these methods is a separate search phase which increases the training complexity and may substantially increase the computational cost. Additionally, due to the separate search phase, these approaches are unable to adjust the regularization strength based on model or dataset size. Automated augmentation policies are often found by training small models on small datasets and subsequently applied to train larger models. In this work, we remove both of these obstacles. RandAugment has a significantly reduced search space which allows it to be trained on the target task with no need for a separate proxy task. Furthermore, due to the parameterization, the regularization strength may be tailored to different model and dataset sizes. RandAugment can be used uniformly across different tasks and datasets and works out of the box, matching or surpassing all previous automated augmentation approaches on CIFAR-10/100, SVHN, and ImageNet. On the ImageNet dataset we achieve 85.0% accuracy, a 0.6% increase over the previous state-of-the-art and 1.0% increase over baseline augmentation. On object detection, RandAugment leads to 1.0-1.3% improvement over baseline augmentation, and is within 0.3% mAP of AutoAugment on COCO. Finally, due to its interpretable hyperparameter, RandAugment may be used to investigate the role of data augmentation with varying model and dataset size. Code is available online.

cs.CV↗

arXiv2019

Learning Data Augmentation Strategies for Object Detection

Data augmentation is a critical component of training deep learning models. Although data augmentation has been shown to significantly improve image classification, its potential has not been thoroughly investigated for object detection. Given the additional cost for annotating images for object detection, data augmentation may be of even greater importance for this computer vision task. In this work, we study the impact of data augmentation on object detection. We first demonstrate that data augmentation operations borrowed from image classification may be helpful for training detection models, but the improvement is limited. Thus, we investigate how learned, specialized data augmentation policies improve generalization performance for detection models. Importantly, these augmentation policies only affect training and leave a trained model unchanged during evaluation. Experiments on the COCO dataset indicate that an optimized data augmentation policy improves detection accuracy by more than +2.3 mAP, and allow a single inference model to achieve a state-of-the-art accuracy of 50.7 mAP. Importantly, the best policy found on COCO may be transferred unchanged to other detection datasets and models to improve predictive accuracy. For example, the best augmentation policy identified with COCO improves a strong baseline on PASCAL-VOC by +2.7 mAP. Our results also reveal that a learned augmentation policy is superior to state-of-the-art architecture regularization methods for object detection, even when considering strong baselines. Code for training with the learned policy is available online at https://github.com/tensorflow/tpu/tree/master/models/official/detection

cs.CV↗

arXiv2019

Realistic Evaluation of Deep Semi-Supervised Learning Algorithms

Semi-supervised learning (SSL) provides a powerful framework for leveraging unlabeled data when labels are limited or expensive to obtain. SSL algorithms based on deep neural networks have recently proven successful on standard benchmark tasks. However, we argue that these benchmarks fail to address many issues that these algorithms would face in real-world applications. After creating a unified reimplementation of various widely-used SSL techniques, we test them in a suite of experiments designed to address these issues. We find that the performance of simple baselines which do not use unlabeled data is often underreported, that SSL methods differ in sensitivity to the amount of labeled and unlabeled data, and that performance can degrade substantially when the unlabeled dataset contains out-of-class examples. To help guide SSL research towards real-world applicability, we make our unified reimplemention and evaluation platform publicly available.

cs.LG↗

arXiv2019

Using learned optimizers to make models robust to input noise

State-of-the art vision models can achieve superhuman performance on image classification tasks when testing and training data come from the same distribution. However, when models are tested on corrupted images (e.g. due to scale changes, translations, or shifts in brightness or contrast), performance degrades significantly. Here, we explore the possibility of meta-training a learned optimizer that can train image classification models such that they are robust to common image corruptions. Specifically, we are interested training models that are more robust to noise distributions not present in the training data. We find that a learned optimizer meta-trained to produce models which are robust to Gaussian noise trains models that are more robust to Gaussian noise at other scales compared to traditional optimizers like Adam. The effect of meta-training is more complicated when targeting a more general set of noise distributions, but led to improved performance on half of held-out corruption tasks. Our results suggest that meta-learning provides a novel approach for studying and improving the robustness of deep learning models.

cs.LG↗

arXiv2019

Improving Robustness Without Sacrificing Accuracy with Patch Gaussian Augmentation

Deploying machine learning systems in the real world requires both high accuracy on clean data and robustness to naturally occurring corruptions. While architectural advances have led to improved accuracy, building robust models remains challenging. Prior work has argued that there is an inherent trade-off between robustness and accuracy, which is exemplified by standard data augment techniques such as Cutout, which improves clean accuracy but not robustness, and additive Gaussian noise, which improves robustness but hurts accuracy. To overcome this trade-off, we introduce Patch Gaussian, a simple augmentation scheme that adds noise to randomly selected patches in an input image. Models trained with Patch Gaussian achieve state of the art on the CIFAR-10 and ImageNetCommon Corruptions benchmarks while also improving accuracy on clean data. We find that this augmentation leads to reduced sensitivity to high frequency noise(similar to Gaussian) while retaining the ability to take advantage of relevant high frequency information in the image (similar to Cutout). Finally, we show that Patch Gaussian can be used in conjunction with other regularization methods and data augmentation policies such as AutoAugment, and improves performance on the COCO object detection benchmark.

cs.LG↗

arXiv2019

AutoAugment: Learning Augmentation Policies from Data

Data augmentation is an effective technique for improving the accuracy of modern image classifiers. However, current data augmentation implementations are manually designed. In this paper, we describe a simple procedure called AutoAugment to automatically search for improved data augmentation policies. In our implementation, we have designed a search space where a policy consists of many sub-policies, one of which is randomly chosen for each image in each mini-batch. A sub-policy consists of two operations, each operation being an image processing function such as translation, rotation, or shearing, and the probabilities and magnitudes with which the functions are applied. We use a search algorithm to find the best policy such that the neural network yields the highest validation accuracy on a target dataset. Our method achieves state-of-the-art accuracy on CIFAR-10, CIFAR-100, SVHN, and ImageNet (without additional data). On ImageNet, we attain a Top-1 accuracy of 83.5% which is 0.4% better than the previous record of 83.1%. On CIFAR-10, we achieve an error rate of 1.5%, which is 0.6% better than the previous state-of-the-art. Augmentation policies we find are transferable between datasets. The policy learned on ImageNet transfers well to achieve significant improvements on other datasets, such as Oxford Flowers, Caltech-101, Oxford-IIT Pets, FGVC Aircraft, and Stanford Cars.

cs.CV↗

arXiv2018

Machine learning determination of atomic dynamics at grain boundaries

In polycrystalline materials, grain boundaries are sites of enhanced atomic motion, but the complexity of the atomic structures within a grain boundary network makes it difficult to link the structure and atomic dynamics. Here we use a machine learning technique to establish a connection between local structure and dynamics of these materials. Following previous work on bulk glassy materials, we define a purely structural quantity, softness, that captures the propensity of an atom to rearrange. This approach correctly identifies crystalline regions, stacking faults, and twin boundaries as having low likelihood of atomic rearrangements, while finding a large variability within high-energy grain boundaries. As has been found in glasses [9,19,26], the probability that atoms of a given softness will rearrange is nearly Arrhenius. This indicates a well-defined energy barrier as well as a well-defined prefactor for the Arrhenius form for atoms of a given softness. The decrease in the prefactor for low-softness atoms indicates that variations in entropy exhibit a dominant influence on the atomic dynamics in grain boundaries.

cond-mat.mtrl-sci↗

arXiv2018

Accelerated search and design of stretchable graphene kirigami using machine learning

Making kirigami-inspired cuts into a sheet has been shown to be an effective way of designing stretchable materials with metamorphic properties where the 2D shape can transform into complex 3D shapes. However, finding the optimal solutions is not straightforward as the number of possible cutting patterns grows exponentially with system size. Here, we report on how machine learning (ML) can be used to approximate the target properties, such as yield stress and yield strain, as a function of cutting pattern. Our approach enables the rapid discovery of kirigami designs that yield extreme stretchability as verified by molecular dynamics (MD) simulations. We find that convolutional neural networks (CNN), commonly used for classification in vision tasks, can be applied for regression to achieve an accuracy close to the precision of the MD simulations. This approach can then be used to search for optimal designs that maximize elastic stretchability with only 1000 training samples in a large design space of $\sim 4\times10^6$ candidate designs. This example demonstrates the power and potential of ML in finding optimal kirigami designs at a fraction of iterations that would be required of a purely MD or experiment-based approach, where no prior knowledge of the governing physics is known or available.

physics.comp-ph↗

arXiv2018

Machine learning-assisted discovery of many new solid Li-ion conducting materials

We discover many new crystalline solid materials with fast single crystal Li ion conductivity at room temperature, discovered through density functional theory simulations guided by machine learning-based methods. The discovery of new solid Li superionic conductors is of critical importance to the development of safe all-solid-state Li-ion batteries. With a predictive universal structure-property relationship for fast ion conduction not well understood, the search for new solid Li ion conductors has relied largely on trial-and-error computational and experimental searches over the last several decades. In this work, we perform a guided search of materials space with a machine learning (ML)-based prediction model for material selection and density functional theory molecular dynamics (DFT-MD) simulations for calculating ionic conductivity. These materials are screened from over 12,000 experimentally synthesized and characterized candidates with very diverse structures and compositions. When compared to a random search of materials space, we find that the ML-guided search is 2.7 times more likely to identify fast Li ion conductors, with at least a 45x improvement in the log-average of room temperature Li ion conductivity. The F1 score of the ML-based model is 0.50, 3.5 times better than the F1 score expected from completely random guesswork. In a head-to-head competition against six Ph.D. students working in the field, we find that the ML-based model doubles the F1 score of human experts in its ability to identify fast Li-ion conductors from atomistic structure with a thousand- fold increase in speed, clearly demonstrating the utility of this model for the research community. All conducting materials reported here lack transition metals and are predicted to exhibit low electronic conduction, high stability against oxidation, and high thermodynamic stability.

cond-mat.mtrl-sci↗

arXiv2017

Intriguing Properties of Adversarial Examples

It is becoming increasingly clear that many machine learning classifiers are vulnerable to adversarial examples. In attempting to explain the origin of adversarial examples, previous studies have typically focused on the fact that neural networks operate on high dimensional data, they overfit, or they are too linear. Here we argue that the origin of adversarial examples is primarily due to an inherent uncertainty that neural networks have about their predictions. We show that the functional form of this uncertainty is independent of architecture, dataset, and training protocol; and depends only on the statistics of the logit differences of the network, which do not change significantly during training. This leads to adversarial error having a universal scaling, as a power-law, with respect to the size of the adversarial perturbation. We show that this universality holds for a broad range of datasets (MNIST, CIFAR10, ImageNet, and random data), models (including state-of-the-art deep networks, linear models, adversarially trained networks, and networks trained on randomly shuffled labels), and attacks (FGSM, step l.l., PGD). Motivated by these results, we study the effects of reducing prediction entropy on adversarial robustness. Finally, we study the effect of network architectures on adversarial sensitivity. To do this, we use neural architecture search with reinforcement learning to find adversarially robust architectures on CIFAR10. Our resulting architecture is more robust to white \emph{and} black box attacks compared to previous attempts.

stat.ML↗

arXiv2016

Disconnecting structure and dynamics in glassy thin films

Nanometrically thin glassy films depart strikingly from the behavior of their bulk counterparts. We investigate whether the dynamical differences between bulk and thin film glasses can be understood by differences in local microscopic structure. We employ machine-learning methods that have previously identified strong correlations between local structure and particle rearrangement dynamics in bulk systems. We show that these methods completely fail to detect key aspects of thin-film glassy dynamics. Furthermore, we show that no combination of local structural features drawn from a very general set of two- and multi-point functions is able to distinguish between particles at the center of film and those in intermediate layers where the dynamics are strongly perturbed.

cond-mat.soft↗

arXiv2016

The Relationship Between Local Structure and Relaxation in Out-of-Equilibrium Glassy Systems

The dynamical glass transition is typically taken to be the temperature at which a glassy liquid is no longer able to equilibrate on experimental timescales. Consequently, the physical properties of these systems just above or below the dynamical glass transition, such as viscosity, can change by many orders of magnitude over long periods of time following external perturbation. During this progress towards equilibrium, glassy systems exhibit a history dependence that has complicated their study. In previous work, we bridged the gap between structure and dynamics in glassy liquids above their dynamical glass transition temperatures by introducing a scalar field called "softness", a quantity obtained using machine learning methods. Softness is designed to capture the hidden patterns in relative particle positions that correlate strongly with dynamical rearrangements of particle positions. Here we show that the out-of-equilibrium behavior of a model glassforming system can be understood in terms of softness. To do this we first demonstrate that the evolution of behavior following a temperature quench is a primarily structural phenomenon: the structure changes considerably, but the relationship between structure and dynamics remains invariant. We then show that the history-dependent relaxation time can be robustly computed from structure as quantified by softness. Together, these results motivate the use of softness to characterize the history dependence of glasses.

cond-mat.soft↗

arXiv2015

A structural approach to relaxation in glassy liquids

When a liquid freezes, a change in the local atomic structure marks the transition to the crystal. When a liquid is cooled to form a glass, however, no noticeable structural change marks the glass transition. Indeed, characteristic features of glassy dynamics that appear below an onset temperature, T_0, are qualitatively captured by mean field theory, which assumes uniform local structure at all temperatures. Even studies of more realistic systems have found only weak correlations between structure and dynamics. This raises the question: is structure important to glassy dynamics in three dimensions? Here, we answer this question affirmatively by using machine learning methods to identify a new field, that we call softness, which characterizes local structure and is strongly correlated with rearrangement dynamics. We find that the onset of glassy dynamics at T_0 is marked by the onset of correlations between softness (i.e. structure) and dynamics. Moreover, we use softness to construct a simple model of slow glassy relaxation that is in excellent agreement with our simulation results, showing that a theory of the evolution of softness in time would constitute a theory of glassy dynamics.

cond-mat.soft↗

arXiv2015

High-Temperature Quantum Anomalous Hall Effect in n-p Codoped Topological Insulators

The quantum anomalous Hall effect (QAHE) is a fundamental quantum transport phenomenon that manifests as a quantized transverse conductance in response to a longitudinally applied electric field in the absence of an external magnetic field, and promises to have immense application potentials in future dissipation-less quantum electronics. Here we present a novel kinetic pathway to realize the QAHE at high temperatures by $n$-$p$ codoping of three-dimensional topological insulators. We provide proof-of-principle numerical demonstration of this approach using vanadium-iodine (V-I) codoped Sb$_2$Te$_3$ and demonstrate that, strikingly, even at low concentrations of $\sim$2\% V and $\sim$1\% I, the system exhibits a quantized Hall conductance, the tell-tale hallmark of QAHE, at temperatures of at least $\sim$ 50 Kelvin, which is three orders of magnitude higher than the typical temperatures at which it has been realized so far. The proposed approach is conceptually general and may shed new light in experimental realization of high-temperature QAHE.

cond-mat.mtrl-sci↗