arXiv subjects

Ben Adlam

Publications and source records attributed to Ben Adlam.

25 records · Page 2Linked to original sources

arXiv2020

The Surprising Simplicity of the Early-Time Learning Dynamics of Neural Networks

Modern neural networks are often regarded as complex black-box functions whose behavior is difficult to understand owing to their nonlinear dependence on the data and the nonconvexity in their loss landscapes. In this work, we show that these common perceptions can be completely false in the early phase of learning. In particular, we formally prove that, for a class of well-behaved input distributions, the early-time learning dynamics of a two-layer fully-connected neural network can be mimicked by training a simple linear model on the inputs. We additionally argue that this surprising simplicity can persist in networks with more layers and with convolutional architecture, which we verify empirically. Key to our analysis is to bound the spectral norm of the difference between the Neural Tangent Kernel (NTK) at initialization and an affine transform of the data kernel; however, unlike many previous results utilizing the NTK, we do not require the network to have disproportionately large width, and the network is allowed to escape the kernel regime later in training.

cs.LG↗

arXiv2019

Learning GANs and Ensembles Using Discrepancy

Generative adversarial networks (GANs) generate data based on minimizing a divergence between two distributions. The choice of that divergence is therefore critical. We argue that the divergence must take into account the hypothesis set and the loss function used in a subsequent learning task, where the data generated by a GAN serves for training. Taking that structural information into account is also important to derive generalization guarantees. Thus, we propose to use the discrepancy measure, which was originally introduced for the closely related problem of domain adaptation and which precisely takes into account the hypothesis set and the loss function. We show that discrepancy admits favorable properties for training GANs and prove explicit generalization guarantees. We present efficient algorithms using discrepancy for two tasks: training a GAN directly, namely DGAN, and mixing previously trained generative models, namely EDGAN. Our experiments on toy examples and several benchmark datasets show that DGAN is competitive with other GANs and that EDGAN outperforms existing GAN ensembles, such as AdaGAN.

cs.LG↗

arXiv2019

Investigating Under and Overfitting in Wasserstein Generative Adversarial Networks

We investigate under and overfitting in Generative Adversarial Networks (GANs), using discriminators unseen by the generator to measure generalization. We find that the model capacity of the discriminator has a significant effect on the generator's model quality, and that the generator's poor performance coincides with the discriminator underfitting. Contrary to our expectations, we find that generators with large model capacities relative to the discriminator do not show evidence of overfitting on CIFAR10, CIFAR100, and CelebA.

stat.ML↗

arXiv2019

AdaNet: A Scalable and Flexible Framework for Automatically Learning Ensembles

AdaNet is a lightweight TensorFlow-based (Abadi et al., 2015) framework for automatically learning high-quality ensembles with minimal expert intervention. Our framework is inspired by the AdaNet algorithm (Cortes et al., 2017) which learns the structure of a neural network as an ensemble of subnetworks. We designed it to: (1) integrate with the existing TensorFlow ecosystem, (2) offer sensible default search spaces to perform well on novel datasets, (3) present a flexible API to utilize expert information when available, and (4) efficiently accelerate training with distributed CPU, GPU, and TPU hardware. The code is open-source and available at: https://github.com/tensorflow/adanet.

cs.LG↗

arXiv2018

Stationary frequencies and mixing times for neutral drift processes with spatial structure

We study a general setting of neutral evolution in which the population is of finite, constant size and can have spatial structure. Mutation leads to different genetic types ("traits"), which can be discrete or continuous. Under minimal assumptions, we show that the marginal trait distributions of the evolutionary process, which specify the probability that any given individual has a certain trait, all converge to the stationary distribution of the mutation process. In particular, the stationary frequencies of traits in the population are independent of its size, spatial structure, and evolutionary update rule, and these frequencies can be calculated by evaluating a simple stochastic process describing a population of size one (i.e. the mutation process itself). We conclude by analyzing mixing times, which characterize rates of convergence of the mutation process along the lineages, in terms of demographic variables of the evolutionary process.

q-bio.PE↗

arXiv2015

Spectral Statistics of Sparse Random Graphs with a General Degree Distribution

We consider the adjacency matrices of sparse random graphs from the Chung-Lu model, where edges are added independently between the $N$ vertices with varying probabilities $p_{ij}$. The rank of the matrix $(p_{ij})$ is some fixed positive integer. We prove that the distribution of eigenvalues is given by the solution of a functional self-consistent equation. We prove a local law down to the optimal scale and prove bulk universality. The results are parallel to \cite{Erdos2013b} and \cite{Landon2015}.

math.PR↗

arXiv2014

Universality of fixation probabilities in randomly structured populations

The stage of evolution is the population of reproducing individuals. The structure of the population is know to affect the dynamics and outcome of evolutionary processes, but analytical results for generic random structures have been lacking. The most general result so far, the isothermal theorem, assumes the propensity for change in each position is exactly the same, but realistic biological structures are always subject to variation and noise. We consider a population of finite size $n$ under constant selection whose structure is given by a wide variety of weighted, directed, random graphs; vertices represent individuals and edges interactions between individuals. By establishing a robustness result for the isothermal theorem and using large deviation estimates to understand the typical structure of random graphs, we prove that for a generalization of the Erdős-Rényi model the fixation probability of an invading mutant is approximately the same as that of a mutant of equal fitness in a well-mixed population with high probability. Simulations of perturbed lattices, small-world networks, and scale-free networks behave similarly. We conjecture that the fixation probability in a well-mixed population, $(1-r^{-1})/(1-r^{-n})$, is universal: for many random graph models, the fixation probability approaches the above function uniformly as the graphs become large.

q-bio.PE↗