Searcharxiv⌕ Search

arXiv subjects

Jaehoon Lee

Publications and source records attributed to Jaehoon Lee.

At least 91 records · Page 5Linked to original sources
arXiv2020

On Empirical Comparisons of Optimizers for Deep Learning

Selecting an optimizer is a central step in the contemporary deep learning pipeline. In this paper, we demonstrate the sensitivity of optimizer comparisons to the hyperparameter tuning protocol. Our findings suggest that the hyperparameter search space may be the single most important factor explaining the rankings obtained by recent empirical comparisons in the literature. In fact, we show that these results can be contradicted when hyperparameter search spaces are changed. As tuning effort grows without bound, more general optimizers should never underperform the ones they can approximate (i.e., Adam should never perform worse than momentum), but recent attempts to compare optimizers either assume these inclusion relationships are not practically relevant or restrict the hyperparameters in ways that break the inclusions. In our experiments, we find that inclusion relationships between optimizers matter in practice and always predict optimizer comparisons. In particular, we find that the popular adaptive gradient methods never underperform momentum or gradient descent. We also report practical tips around tuning often ignored hyperparameters of adaptive gradient methods and raise concerns about fairly benchmarking optimizers for neural network training.

cs.LG↗
arXiv2020

NTIRE 2020 Challenge on Video Quality Mapping: Methods and Results

This paper reviews the NTIRE 2020 challenge on video quality mapping (VQM), which addresses the issues of quality mapping from source video domain to target video domain. The challenge includes both a supervised track (track 1) and a weakly-supervised track (track 2) for two benchmark datasets. In particular, track 1 offers a new Internet video benchmark, requiring algorithms to learn the map from more compressed videos to less compressed videos in a supervised training manner. In track 2, algorithms are required to learn the quality mapping from one device to another when their quality varies substantially and weakly-aligned video pairs are available. For track 1, in total 7 teams competed in the final test phase, demonstrating novel and effective solutions to the problem. For track 2, some existing methods are evaluated, showing promising solutions to the weakly-supervised video quality mapping problem.

eess.IV↗
arXiv2020

On the infinite width limit of neural networks with a standard parameterization

There are currently two parameterizations used to derive fixed kernels corresponding to infinite width neural networks, the NTK (Neural Tangent Kernel) parameterization and the naive standard parameterization. However, the extrapolation of both of these parameterizations to infinite width is problematic. The standard parameterization leads to a divergent neural tangent kernel while the NTK parameterization fails to capture crucial aspects of finite width networks such as: the dependence of training dynamics on relative layer widths, the relative training dynamics of weights and biases, and overall learning rate scale. Here we propose an improved extrapolation of the standard parameterization that preserves all of these properties as width is taken to infinity and yields a well-defined neural tangent kernel. We show experimentally that the resulting kernels typically achieve similar accuracy to those resulting from an NTK parameterization, but with better correspondence to the parameterization of typical finite width networks. Additionally, with careful tuning of width parameters, the improved standard parameterization kernels can outperform those stemming from an NTK parameterization. We release code implementing this improved standard parameterization as part of the Neural Tangents library at https://github.com/google/neural-tangents.

cs.LG↗
arXiv2019

Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent

A longstanding goal in deep learning research has been to precisely characterize training and generalization. However, the often complex loss landscapes of neural networks have made a theory of learning dynamics elusive. In this work, we show that for wide neural networks the learning dynamics simplify considerably and that, in the infinite width limit, they are governed by a linear model obtained from the first-order Taylor expansion of the network around its initial parameters. Furthermore, mirroring the correspondence between wide Bayesian neural networks and Gaussian processes, gradient-based training of wide neural networks with a squared loss produces test set predictions drawn from a Gaussian process with a particular compositional kernel. While these theoretical results are only exact in the infinite width limit, we nevertheless find excellent empirical agreement between the predictions of the original network and those of the linearized version even for finite practically-sized networks. This agreement is robust across different architectures, optimization methods, and loss functions.

stat.ML↗
arXiv2019

Neural Tangents: Fast and Easy Infinite Neural Networks in Python

Neural Tangents is a library designed to enable research into infinite-width neural networks. It provides a high-level API for specifying complex and hierarchical neural network architectures. These networks can then be trained and evaluated either at finite-width as usual or in their infinite-width limit. Infinite-width networks can be trained analytically using exact Bayesian inference or using gradient descent via the Neural Tangent Kernel. Additionally, Neural Tangents provides tools to study gradient descent training dynamics of wide but finite networks in either function space or weight space. The entire library runs out-of-the-box on CPU, GPU, or TPU. All computations can be automatically distributed over multiple accelerators with near-linear scaling in the number of devices. Neural Tangents is available at www.github.com/google/neural-tangents. We also provide an accompanying interactive Colab notebook.

stat.ML↗
arXiv2019

Measuring the Effects of Data Parallelism on Neural Network Training

Recent hardware developments have dramatically increased the scale of data parallelism available for neural network training. Among the simplest ways to harness next-generation hardware is to increase the batch size in standard mini-batch neural network training algorithms. In this work, we aim to experimentally characterize the effects of increasing the batch size on training time, as measured by the number of steps necessary to reach a goal out-of-sample error. We study how this relationship varies with the training algorithm, model, and data set, and find extremely large variation between workloads. Along the way, we show that disagreements in the literature on how batch size affects model quality can largely be explained by differences in metaparameter tuning and compute budgets at different batch sizes. We find no evidence that larger batch sizes degrade out-of-sample performance. Finally, we discuss the implications of our results on efforts to train neural networks much faster in the future. Our experimental data is publicly available as a database of 71,638,836 loss measurements taken over the course of training for 168,160 individual models across 35 workloads.

cs.LG↗
arXiv2019

Physics Enhanced Artificial Intelligence

We propose that intelligently combining models from the domains of Artificial Intelligence or Machine Learning with Physical and Expert models will yield a more "trustworthy" model than any one model from a single domain, given a complex and narrow enough problem. Based on mean-variance portfolio theory and bias-variance trade-off analysis, we prove combining models from various domains produces a model that has lower risk, increasing user trust. We call such combined models - physics enhanced artificial intelligence (PEAI), and suggest use cases for PEAI.

cs.AI↗
arXiv2018

Structure of ordinary $Λ$-adic arithmetic cohomology groups

We study the $Λ$-module structure of the ordinary parts of the arithmetic cohomology groups of modular Jacobians made out of various towers of modular curves. We prove that the ordinary parts of $Λ$-adic Selmer groups coming from two different towers have "almost same" $Λ$-module structures. We also prove the cotorsionness of $Λ$-adic Tate-Shafarevich group under mild assumptions.

math.NT↗
arXiv2018

Deep Neural Networks as Gaussian Processes

It has long been known that a single-layer fully-connected neural network with an i.i.d. prior over its parameters is equivalent to a Gaussian process (GP), in the limit of infinite network width. This correspondence enables exact Bayesian inference for infinite width neural networks on regression tasks by means of evaluating the corresponding GP. Recently, kernel functions which mimic multi-layer random neural networks have been developed, but only outside of a Bayesian framework. As such, previous work has not identified that these kernels can be used as covariance functions for GPs and allow fully Bayesian prediction with a deep neural network. In this work, we derive the exact equivalence between infinitely wide deep networks and GPs. We further develop a computationally efficient pipeline to compute the covariance function for these GPs. We then use the resulting GPs to perform Bayesian inference for wide deep neural networks on MNIST and CIFAR-10. We observe that trained neural network accuracy approaches that of the corresponding GP with increasing layer width, and that the GP uncertainty is strongly correlated with trained network prediction error. We further find that test performance increases as finite-width trained networks are made wider and more similar to a GP, and thus that GP predictions typically outperform those of finite-width networks. Finally we connect the performance of these GPs to the recent theory of signal propagation in random neural networks.

stat.ML↗
arXiv2016

3d $\mathcal{N}=2$ minimal SCFTs from Wrapped M5-branes

We study CFT data of 3-dimensional superconformal field theories (SCFTs) arising from wrapped two M5-branes on closed hyperbolic 3-manifolds. Via so-called 3d/3d correspondence, central charges of these SCFTs are related to a $SL(2)$ Chern-Simons (CS) invariant on the 3-manifolds. We give a rigorous definition of the invariant in terms of resurgence theory and a state-integral model for the complex CS theory. We numerically evaluate the central charges for several closed 3-manifolds with small hyperbolic volume. The computation suggests that the wrapped M5-brane systems give infinitely many discrete SCFTs with small central charges. We also analyze these `minimal' SCFTs in the eye of 3d $\mathcal{N}=2$ superconformal bootstrap.

hep-th↗
arXiv2016

Linking dynamical heterogeneity to static amorphous order

Glass-forming liquids grow dramatically sluggish upon cooling. This slowdown has long been thought to be accompanied by a growing correlation length. Characteristic dynamical and static length scales, however, have been observed to grow at different rates, which perplexes the relationship between the two and with the slowdown. Here, we show the existence of a direct link between dynamical sluggishness and static point-to-set correlations, holding at the local level as we probe different environments within a liquid. This link, which is stronger and more general than that observed with locally preferred structures, suggests the existence of an intimate relationship between structure and dynamics in a broader range of glass-forming liquids than previously thought.

cond-mat.dis-nn↗
arXiv2016

Entanglement entropy from one-point functions in holographic states

For holographic CFT states near the vacuum, entanglement entropies for spatial subsystems can be expressed perturbatively as an expansion in the one-point functions of local operators dual to light bulk fields. Using the connection between quantum Fisher information for CFT states and canonical energy for the dual spacetimes, we describe a general formula for this expansion up to second-order in the one-point functions, for an arbitrary ball-shaped region, extending the first-order result given by the entanglement first law. For two-dimensional CFTs, we use this to derive a completely explicit formula for the second-order contribution to the entanglement entropy from the stress tensor. We show that this stress tensor formula can be reproduced by a direct CFT calculation for states related to the vacuum by a local conformal transformation. This result can also be reproduced via the perturbative solution to a non-linear scalar wave equation on an auxiliary de Sitter spacetime, extending the first-order result in arXiv/1509.00113.

hep-th↗
arXiv2015

Glassy slowdown and replica-symmetry-breaking instantons

Glass-forming liquids exhibit a dramatic dynamical slowdown as the temperature is lowered. This can be attributed to relaxation proceeding via large structural rearrangements whose characteristic size increases as the system cools. These cooperative rearrangements are well modeled by instantons in a replica effective field theory, with the size of the dominant instanton encoding the liquid's cavity point-to-set correlation length. Varying the parameters of the effective theory corresponds to varying the statistics of the underlying free-energy landscape. We demonstrate that, for a wide range of parameters, replica-symmetry-breaking instantons dominate. The detailed structure of the dominant instanton provides a rich window into point-to-set correlations and glassy dynamics.

cond-mat.stat-mech↗
arXiv2014

Exact Correlators of BPS Operators from the 3d Superconformal Bootstrap

We use the superconformal bootstrap to derive exact relations between OPE coefficients in three-dimensional superconformal field theories with ${\cal N} \geq 4$ supersymmetry. These relations follow from a consistent truncation of the crossing symmetry equations that is associated with the cohomology of a certain supercharge. In ${\cal N} = 4$ SCFTs, the non-trivial cohomology classes are in one-to-one correspondence with certain half-BPS operators, provided that these operators are restricted to lie on a line. The relations we find are powerful enough to allow us to determine an infinite number of OPE coefficients in the interacting SCFT ($U(2)_2 \times U(1)_{-2}$ ABJ theory) that constitutes the IR limit of $O(3)$ ${\cal N} = 8$ super-Yang-Mills theory. More generally, in ${\cal N} = 8$ SCFTs with a unique stress tensor, we are led to conjecture that many superconformal multiplets allowed by group theory must actually be absent from the spectrum, and we test this conjecture in known ${\cal N} = 8$ SCFTs using the superconformal index. For generic ${\cal N} = 8$ SCFTs, we also improve on numerical bootstrap bounds on OPE coefficients of short and semi-short multiplets and discuss their relation to the exact relations between OPE coefficients we derived. In particular, we show that the kink previously observed in these bounds arises from the disappearance of a certain quarter-BPS multiplet, and that the location of the kink is likely tied to the existence of the $U(2)_2 \times U(1)_{-2}$ ABJ theory, which can be argued to not possess this multiplet.

hep-th↗
arXiv2014

The ${\cal N} = 8$ Superconformal Bootstrap in Three Dimensions

We analyze the constraints imposed by unitarity and crossing symmetry on the four-point function of the stress-tensor multiplet of ${\cal N}=8$ superconformal field theories in three dimensions. We first derive the superconformal blocks by analyzing the superconformal Ward identity. Our results imply that the OPE of the primary operator of the stress-tensor multiplet with itself must have parity symmetry. We then analyze the relations between the crossing equations, and we find that these equations are mostly redundant. We implement the independent crossing constraints numerically and find bounds on OPE coefficients and operator dimensions as a function of the stress-tensor central charge. To make contact with known ${\cal N}=8$ superconformal field theories, we compute this central charge in a few particular cases using supersymmetric localization. For limiting values of the central charge, our numerical bounds are nearly saturated by the large $N$ limit of ABJM theory and also by the free $U(1)\times U(1)$ ABJM theory.

hep-th↗
arXiv2014

Critical Exponents for Supercooled Liquids

We compute critical exponents governing universal features of supercooled liquids through the effective theory of an overlap field. The correlation length diverges with the Ising exponent; the size of dynamically heterogeneous patches grows more rapidly; and the relaxation time obeys a generalized Vogel-Fulcher-Tammann relation.

cond-mat.dis-nn↗
arXiv2014

Frobenius numbers of Pythagorean triples

Given relatively prime integers $a_1, \dotsc, a_n$, the Frobenius number $g(a_1, \dotsc, a_n)$ is defined as the largest integer which cannot be expressed as $x_1 a_1 + \dotsb + x_n a_n$ with $x_i$ nonnegative integers. In this article, we give the Frobenius number of primitive Pythagorean triples. That is, \[ g(m^2-n^2, 2mn, m^2+n^2) = (m-1)(m^2-n^2) + (m-1)(2mn) - (m^2 + n^2). \]

math.NT↗