arXiv subjects

Deep Ganguli

Publications and source records attributed to Deep Ganguli.

25 records · Page 2Linked to original sources

arXiv2022

In-context Learning and Induction Heads

"Induction heads" are attention heads that implement a simple algorithm to complete token sequences like [A][B] ... [A] -> [B]. In this work, we present preliminary and indirect evidence for a hypothesis that induction heads might constitute the mechanism for the majority of all "in-context learning" in large transformer models (i.e. decreasing loss at increasing token indices). We find that induction heads develop at precisely the same point as a sudden sharp increase in in-context learning ability, visible as a bump in the training loss. We present six complementary lines of evidence, arguing that induction heads may be the mechanistic source of general in-context learning in transformer models of any size. For small attention-only models, we present strong, causal evidence; for larger models with MLPs, we present correlational evidence.

cs.LG↗

arXiv2022

Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback

We apply preference modeling and reinforcement learning from human feedback (RLHF) to finetune language models to act as helpful and harmless assistants. We find this alignment training improves performance on almost all NLP evaluations, and is fully compatible with training for specialized skills such as python coding and summarization. We explore an iterated online mode of training, where preference models and RL policies are updated on a weekly cadence with fresh human feedback data, efficiently improving our datasets and models. Finally, we investigate the robustness of RLHF training, and identify a roughly linear relation between the RL reward and the square root of the KL divergence between the policy and its initialization. Alongside our main results, we perform peripheral analyses on calibration, competing objectives, and the use of OOD detection, compare our models with human writers, and provide samples from our models using prompts appearing in recent related work.

cs.CL↗

arXiv2021

A General Language Assistant as a Laboratory for Alignment

Given the broad capabilities of large language models, it should be possible to work towards a general-purpose, text-based assistant that is aligned with human values, meaning that it is helpful, honest, and harmless. As an initial foray in this direction we study simple baseline techniques and evaluations, such as prompting. We find that the benefits from modest interventions increase with model size, generalize to a variety of alignment evaluations, and do not compromise the performance of large models. Next we investigate scaling trends for several training objectives relevant to alignment, comparing imitation learning, binary discrimination, and ranked preference modeling. We find that ranked preference modeling performs much better than imitation learning, and often scales more favorably with model size. In contrast, binary discrimination typically performs and scales very similarly to imitation learning. Finally we study a `preference model pre-training' stage of training, with the goal of improving sample efficiency when finetuning on human preferences.

cs.CL↗

arXiv2021

The AI Index 2021 Annual Report

Welcome to the fourth edition of the AI Index Report. This year we significantly expanded the amount of data available in the report, worked with a broader set of external organizations to calibrate our data, and deepened our connections with the Stanford Institute for Human-Centered Artificial Intelligence (HAI). The AI Index Report tracks, collates, distills, and visualizes data related to artificial intelligence. Its mission is to provide unbiased, rigorously vetted, and globally sourced data for policymakers, researchers, executives, journalists, and the general public to develop intuitions about the complex field of AI. The report aims to be the most credible and authoritative source for data and insights about AI in the world.

cs.AI↗

arXiv2021

Understanding the Capabilities, Limitations, and Societal Impact of Large Language Models

On October 14th, 2020, researchers from OpenAI, the Stanford Institute for Human-Centered Artificial Intelligence, and other universities convened to discuss open research questions surrounding GPT-3, the largest publicly-disclosed dense language model at the time. The meeting took place under Chatham House Rules. Discussants came from a variety of research backgrounds including computer science, linguistics, philosophy, political science, communications, cyber policy, and more. Broadly, the discussion centered around two main questions: 1) What are the technical capabilities and limitations of large language models? 2) What are the societal effects of widespread use of large language models? Here, we provide a detailed summary of the discussion organized by the two themes above.

cs.CL↗

arXiv2016

Neural and perceptual signatures of efficient sensory coding

The mammalian brain is a metabolically expensive device, and evolutionary pressures have presumably driven it to make productive use of its resources. For sensory areas, this concept has been expressed more formally as an optimality principle: the brain maximizes the information that is encoded about relevant sensory variables, given available resources. Here, we develop this efficiency principle for encoding a sensory variable with a heterogeneous population of noisy neurons, each responding to a particular range of values. The accuracy with which the population represents any particular value depends on the number of cells that respond to that value, their selectivity, and their response levels. We derive the optimal solution for these parameters in closed form, as a function of the probability of stimulus values encountered in the environment. This optimal neural population also imposes limitations on the ability of the organism to discriminate different values of the encoded variable. As a result, we predict an explicit relationship between the statistical properties of the environment, the allocation and selectivity of neurons within populations, and perceptual discriminability. We test this relationship for three visual and two auditory attributes, and find that it is remarkably consistent with existing data.

q-bio.NC↗

arXiv2012

Implicit embedding of prior probabilities in optimally efficient neural populations

We examine how the prior probability distribution of a sensory variable in the environment influences the optimal allocation of neurons and spikes in a population that represents that variable. We start with a conventional response model, in which the spikes of each neuron are drawn from a Poisson distribution with a mean rate governed by an associated tuning curve. For this model, we approximate the Fisher information in terms of the density and amplitude of the tuning curves, under the assumption that tuning width varies inversely with cell density. We consider a family of objective functions based on the expected value, over the sensory prior, of a functional of the Fisher information. This family includes lower bounds on mutual information and perceptual discriminability as special cases. For all cases, we obtain a closed form expression for the optimum, in which the density and gain of the cells in the population are power law functions of the stimulus prior. Thus, the allocation of these resources is uniquely specified by the prior. Since perceptual discriminability may be expressed directly in terms of the Fisher information, it too will be a power law function of the prior. We show that these results hold for tuning curves of arbitrary shape and correlated neuronal variability. This framework thus provides direct and experimentally testable predictions regarding the relationship between sensory priors, tuning properties of neural representations, and perceptual discriminability.

q-bio.NC↗