arXiv subjects

Tom Henighan

Publications and source records attributed to Tom Henighan.

23 records · Page 2Linked to original sources

arXiv2020

Scaling Laws for Autoregressive Generative Modeling

We identify empirical scaling laws for the cross-entropy loss in four domains: generative image modeling, video modeling, multimodal image$\leftrightarrow$text models, and mathematical problem solving. In all cases autoregressive Transformers smoothly improve in performance as model size and compute budgets increase, following a power-law plus constant scaling law. The optimal model size also depends on the compute budget through a power-law, with exponents that are nearly universal across all data domains. The cross-entropy loss has an information theoretic interpretation as $S($True$) + D_{\mathrm{KL}}($True$||$Model$)$, and the empirical scaling laws suggest a prediction for both the true data distribution's entropy and the KL divergence between the true and model distributions. With this interpretation, billion-parameter Transformers are nearly perfect models of the YFCC100M image distribution downsampled to an $8\times 8$ resolution, and we can forecast the model size needed to achieve any given reducible loss (ie $D_{\mathrm{KL}}$) in nats/image for other resolutions. We find a number of additional scaling laws in specific domains: (a) we identify a scaling relation for the mutual information between captions and images in multimodal models, and show how to answer the question "Is a picture worth a thousand words?"; (b) in the case of mathematical problem solving, we identify scaling laws for model performance when extrapolating beyond the training distribution; (c) we finetune generative image models for ImageNet classification and find smooth scaling of the classification loss and error rate, even as the generative loss levels off. Taken together, these results strengthen the case that scaling laws have important implications for neural network performance, including on downstream tasks.

cs.LG↗

arXiv2020

Language Models are Few-Shot Learners

Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task. While typically task-agnostic in architecture, this method still requires task-specific fine-tuning datasets of thousands or tens of thousands of examples. By contrast, humans can generally perform a new language task from only a few examples or from simple instructions - something which current NLP systems still largely struggle to do. Here we show that scaling up language models greatly improves task-agnostic, few-shot performance, sometimes even reaching competitiveness with prior state-of-the-art fine-tuning approaches. Specifically, we train GPT-3, an autoregressive language model with 175 billion parameters, 10x more than any previous non-sparse language model, and test its performance in the few-shot setting. For all tasks, GPT-3 is applied without any gradient updates or fine-tuning, with tasks and few-shot demonstrations specified purely via text interaction with the model. GPT-3 achieves strong performance on many NLP datasets, including translation, question-answering, and cloze tasks, as well as several tasks that require on-the-fly reasoning or domain adaptation, such as unscrambling words, using a novel word in a sentence, or performing 3-digit arithmetic. At the same time, we also identify some datasets where GPT-3's few-shot learning still struggles, as well as some datasets where GPT-3 faces methodological issues related to training on large web corpora. Finally, we find that GPT-3 can generate samples of news articles which human evaluators have difficulty distinguishing from articles written by humans. We discuss broader societal impacts of this finding and of GPT-3 in general.

cs.CL↗

arXiv2020

Scaling Laws for Neural Language Models

We study empirical scaling laws for language model performance on the cross-entropy loss. The loss scales as a power-law with model size, dataset size, and the amount of compute used for training, with some trends spanning more than seven orders of magnitude. Other architectural details such as network width or depth have minimal effects within a wide range. Simple equations govern the dependence of overfitting on model/dataset size and the dependence of training speed on model size. These relationships allow us to determine the optimal allocation of a fixed compute budget. Larger models are significantly more sample-efficient, such that optimally compute-efficient training involves training very large models on a relatively modest amount of data and stopping significantly before convergence.

cs.LG↗

arXiv2017

Direct Measurement of Anharmonic Decay Channels of a Coherent Phonon

We observe anharmonic decay of the photoexcited coherent A1g phonon in bismuth to points in the Brillouin zone where conservation of momentum and energy are satisfied for three-phonon scattering. The decay of a coherent phonon can be understood as a parametric resonance process whereby the atomic displacement periodically modulates the frequency of a broad continuum of modes. This results in energy transfer through resonant squeezing of the target modes. Using ultrafast diffuse x-ray scattering, we observe build up of coherent oscillations in the target modes driven by this parametric resonance over a wide range of the Brillouin zone. We compare the extracted anharmonic coupling constant to first principles calculations for a representative decay channel.

cond-mat.mtrl-sci↗

arXiv2015

Phonon Spectroscopy with Sub-meV Resolution by Femtosecond X-ray Diffuse Scattering

We present a reconstruction of the transverse acoustic phonon dispersion of germanium from femtosecond time-resolved x-ray diffuse scattering measurements at the Linac Coherent Light Source. We demonstrate an energy resolution of 0.3 meV with momentum resolution of 0.01 nm^-1 using 10 keV x-rays with a bandwidth of ~ 1 eV. This high resolution was achieved simultaneously for a large section of reciprocal space including regions closely following three of the principle symmetry directions. The phonon dispersion was reconstructed with less than three hours of measurement time, during which neither the x-ray energy, the sample orientation, nor the detector position were scanned. These results demonstrate how time-domain measurements can complement conventional frequency domain inelastic scattering techniques.

cond-mat.mtrl-sci↗