arXiv subjects

William Bialek

Publications and source records attributed to William Bialek.

At least 19 recordsLinked to original sources

arXiv2025

Large language models and the entropy of English

We use large language models (LLMs) to uncover long-ranged structure in English texts from a variety of sources. The conditional entropy or code length in many cases continues to decrease with context length at least to $N\sim 10^4$ characters, implying that there are direct dependencies or interactions across these distances. A corollary is that there are small but significant correlations between characters at these separations, as we show from the data independent of models. The distribution of code lengths reveals an emergent certainty about an increasing fraction of characters at large $N$. Over the course of model training, we observe different dynamics at long and short context lengths, suggesting that long-ranged structure is learned only gradually. Our results constrain efforts to build statistical physics models of LLMs or language itself.

cond-mat.stat-mech↗

arXiv2025

Ambiguous signals and efficient codes

In many biological networks the responses of individual elements are ambiguous. We consider a scenario in which many sensors respond to a shared signal, each with limited information capacity, and ask that the outputs together convey as much information as possible about an underlying relevant variable. In a low noise limit where we can make analytic progress, we show that individually ambiguous responses optimize overall information transmission.

physics.bio-ph↗

arXiv2025

When many noisy genes optimize information flow

It often is emphasized that gene expression is noisy. A seemingly contradictory view is that control mechanisms have been optimized to squeeze as much information as possible out of a limited number of molecules. Here we revisit these issues in a simple model where a single transcription factor (TF) controls a large number of target genes. We include only the physically required noise sources: random arrival of TFs at their targets and counting noise in the synthesis and degradation of mRNA. If the cell has a limited total number of mRNA molecules, then the capacity to transmit information about TF concentration is maximized when these resources are distributed across the largest possible number of target genes. To realize this capacity the distribution of TF concentrations must be biased toward smaller values. Thus, in some limits, information transmission is optimized when individual expression levels are noisy. In addition, the dependence of information transmission on the parameters of this multi-gene system has a "sloppy" spectrum, so that optimal performance can co-exist with substantial variability.

physics.bio-ph↗

arXiv2025

Context dependent adaptation in a neural computation

Brains adapt to the statistical structure of their input. In the visual system, local light intensities change rapidly, the variance of the intensity changes more slowly, and the dynamic range of contrast itself changes more slowly still. We use a motion-sensitive neuron in the fly visual system to probe this hierarchy of adaptation phenomena, delivering naturalistic stimuli that have been simplified to have a clear separation of time scales. We show that the neural response to visual motion depends on contrast, and this dependence itself varies with context. Using the spike-triggered average velocity trajectory as a response measure, we find that context dependence is confined to a low-dimensional space, with a single dominant dimension. Across a wide range of conditions this adaptation serves to match the integration time to the mean interval between spikes, reducing redundancy.

q-bio.NC↗

arXiv2025

Neural subspaces, minimax entropy, and mean-field theory for networks of neurons

Recent advances in experimental techniques enable the simultaneous recording of activity from thousands of neurons in the brain, presenting both an opportunity and a challenge: to build meaningful, scalable models of large neural populations. Correlations in the brain are typically weak but widespread, suggesting that a mean-field approach might be effective in describing real neural populations, and we explore a hierarchy of maximum entropy models guided by this idea. We begin with models that match only the mean and variance of the total population activity, and extend to models that match the experimentally observed mean and variance of activity along multiple projections of the neural state. Confronted by data from several different brain regions, these models are driven toward a first-order phase transition, characterized by the presence of two nearly degenerate minima in the energy landscape, and this leads to predictions in qualitative disagreement with other features of the data. To resolve this problem we introduce a novel class of models that constrain the full probability distribution of activity along selected projections. We develop the mean-field theory for this class of models and apply it to recordings from 1000+ neurons in the mouse hippocampus. This 'distributional mean--field' model provides an accurate and consistent description of the data, offering a scalable and principled approach to modeling complex neural population dynamics.

physics.bio-ph↗

arXiv2025

Optimization and variability can coexist

Many biological systems perform close to their physical limits, but promoting this optimality to a general principle seems to require implausibly fine tuning of parameters. Using examples from a wide range of systems, we show that this intuition is wrong. Near an optimum, functional performance depends on parameters in a "sloppy'' way, with some combinations of parameters being only weakly constrained. Absent any other constraints, this predicts that we should observe widely varying parameters, and we make this precise: the entropy in parameter space can be extensive even if performance on average is very close to optimal. This removes a major objection to optimization as a general principle, and rationalizes the observed variability.

q-bio.QM↗

arXiv2025

Extended mean-field theories for networks of real neurons

If the behavior of a system with many degrees of freedom can be captured by a small number of collective variables, then plausibly there is an underlying mean-field theory. We show that simple versions of this idea fail to describe the patterns of activity in networks of real neurons. An extended mean-field theory that matches the distribution of collective variables is at least consistent, though shows signs that these networks are poised near a critical point, in agreement with other observations. These results suggest a path to analysis of emerging data on ever larger numbers of neurons.

physics.bio-ph↗

arXiv2024

The FlEye camera: Sampling the joint distribution of natural scenes and motion

To make efficient use of limited physical resources, the brain must match its coding and computational strategies to the statistical structure of input signals. An attractive testing ground for these principles is the problem of motion estimation in the fly visual system: we understand the optics of the compound eye, have a quantitative description of input signals and noise from the retina, and can record from output neurons that encode estimates of different velocity components. Furthermore, recent work provides a nearly complete wiring diagram of the intervening circuitry. What is missing is a characterization of the visual signals and motions that flies encounter in a natural context. We attack this directly with the development of a specialized camera that matches the high temporal resolution, optical properties, and spectral sensitivity of the fly's eye; inertial motion sensors provide ground truth about rotations and translations through the world. We describe the design, construction, and performance characteristics of this FlEye camera. To illustrate the opportunities created by this instrument we use data on movies and motion to construct optimal local motion estimators that can be compared with the responses of the fly's motion sensitive neurons.

q-bio.NC↗

arXiv2024

Direct estimates of irreversibility from time series

The arrow of time can be quantified through the Kullback-Leibler divergence ($D_{KL}$) between the distributions of forward and reverse trajectories in a system. Many approaches to estimate this rely on specific models, but the use of incorrect models can introduce uncontrolled errors. Here, we describe a model-free method that uses trajectory data directly to estimate the evidence for irreversibility over finite windows of time. To do this we build on previous work to identify and correct for errors that arise from limited sample size. Importantly, our approach accurately recovers $D_{KL} = 0$ in systems that adhere to detailed balance, and the correct nonzero $D_{KL}$ for data generated by well understood models of nonequilibrium systems. We apply our method to trajectories of neural activity in the retina as it responds to naturalistic inputs, and find evidence of irreversibility in single neurons, emphasizing the non-Markovian character of these data. These results open new avenues for investigating how the brain represents the arrow of time.

cond-mat.stat-mech↗

arXiv2024

Moving boundaries: An appreciation of John Hopfield

The 2024 Nobel Prize in Physics was awarded to John Hopfield and Geoffrey Hinton, "for foundational discoveries and inventions that enable machine learning with artificial neural networks." As noted by the Nobel committee, their work moved the boundaries of physics. This is a brief reflection on Hopfield's work, its implications for the emergence of biological physics as a part of physics, the path from his early papers to the modern revolution in artificial intelligence, and prospects for the future.

physics.hist-ph↗

arXiv2024

Statistical mechanics for networks of real neurons

Perceptions and actions, thoughts and memories result from coordinated activity in hundreds or even thousands of neurons in the brain. It is an old dream of the physics community to provide a statistical mechanics description for these and other emergent phenomena of life. These aspirations appear in a new light because of developments in our ability to measure the electrical activity of the brain, sampling thousands of individual neurons simultaneously over hours or days. We review the progress that has been made in bringing theory and experiment together, focusing on maximum entropy methods and a phenomenological renormalization group. These approaches have uncovered new, quantitatively reproducible collective behaviors in networks of real neurons, and provide examples of rich parameter--free predictions that agree in detail with experiment.

cond-mat.dis-nn↗

arXiv2024

Searching for long time scales without fine tuning

Most of animal and human behavior occurs on time scales much longer than the response times of individual neurons. In many cases, it is plausible that these long time scales emerge from the recurrent dynamics of electrical activity in networks of neurons. In linear models, time scales are set by the eigenvalues of a dynamical matrix whose elements measure the strengths of synaptic connections between neurons. It is not clear to what extent these matrix elements need to be tuned in order to generate long time scales; in some cases, one needs not just a single long time scale but a whole range. Starting from the simplest case of random symmetric connections, we combine maximum entropy and random matrix theory methods to construct ensembles of networks, exploring the constraints required for long time scales to become generic. We argue that a single long time scale can emerge generically from realistic constraints, but a full spectrum of slow modes requires more tuning. Langevin dynamics that generates patterns of synaptic connections drawn from these ensembles involves a combination of Hebbian learning and activity-dependent synaptic scaling.

physics.bio-ph↗

arXiv2024

Maximum entropy models for patterns of gene expression

New experimental methods make it possible to measure the expression levels of many genes, simultaneously, in snapshots from thousands or even millions of individual cells. Current approaches to analyze these experiments involve clustering or low-dimensional projections. Here we use the principle of maximum entropy to obtain a probabilistic description that captures the observed presence or absence of mRNAs from hundreds of genes in cells from the mammalian brain. We construct the Ising model compatible with experimental means and pairwise correlations, and validate it by showing that it gives good predictions for higher-order statistics. We notice that the probability distribution of cell states has many local maxima. By labeling cell states according to the associated maximum, we obtain a cell classification that agrees well with previous results that use traditional clustering techniques. Our results provide quantitative descriptions of gene expression statistics and interpretable criteria for defining cell classes, supporting the hypothesis that cell classes emerge from the collective interaction of gene expression levels.

physics.bio-ph↗

arXiv2024

Ambitions for theory in the physics of life

Theoretical physicists have been fascinated by the phenomena of life for more than a century. As we engage with more realistic descriptions of living systems, however, things get complicated. After reviewing different reactions to this complexity, I explore the optimization of information flow as a potentially general theoretical principle. The primary example is a genetic network guiding development of the fly embryo, but each idea also is illustrated by examples from neural systems. In each case, optimization makes detailed, largely parameter-free predictions that connect quantitatively with experiment

physics.bio-ph↗

arXiv2024

A brief tutorial on information theory

At the 2023 Les Houches Summer School on Theoretical Biological Physics, several students asked for some background on information theory, and so we added a tutorial to the scheduled lectures. This is largely a transcript of that tutorial, lightly edited. It covers basic definitions and context rather than detailed calculations. We hope to have maintained the informality of the presentation, including exchanges with the students, while still being useful.

physics.bio-ph↗

arXiv2023

Scale invariance in early embryonic development

The body plan of the fruit fly is determined by the expression of just a handful of genes. We show that the spatial patterns of expression for several of these genes scale precisely with the size of the embryo. Concretely, discrete positional markers such as the peaks in striped patterns have absolute positions along the anterior-posterior axis that are proportional to embryo length, with better than 1% accuracy. Further, the information (in bits) that graded patterns of expression provide about position can be decomposed into information about fractional or scaled position and information about absolute position or embryo length; all of the available information is about scaled position, again with ~1% accuracy. These observations suggest that the underlying genetic network exhibits scale invariance in a deeper mathematical sense. Taking this mathematical statement seriously requires that the network dynamics have a zero mode, which connects to many other observations on this system.

q-bio.MN↗

arXiv2023

Exact minimax entropy models of large-scale neuronal activity

In the brain, fine-scale correlations combine to produce macroscopic patterns of activity. However, as experiments record from larger and larger populations, we approach a fundamental bottleneck: the number of correlations one would like to include in a model grows larger than the available data. In this undersampled regime, one must focus on a sparse subset of correlations; the optimal choice contains the maximum information about patterns of activity or, equivalently, minimizes the entropy of the inferred maximum entropy model. Applying this ``minimax entropy" principle is generally intractable, but here we present an exact and scalable solution for pairwise correlations that combine to form a tree (a network without loops). Applying our method to over one thousand neurons in the mouse hippocampus, we find that the optimal tree of correlations reduces our uncertainty about the population activity by 14% (over 50 times more than a random tree). Despite containing only 0.1% of all pairwise correlations, this minimax entropy model accurately predicts the observed large-scale synchrony in neural activity and becomes even more accurate as the population grows. The inferred Ising model is almost entirely ferromagnetic (with positive interactions) and exhibits signatures of thermodynamic criticality. These results suggest that a sparse backbone of excitatory interactions may play an important role in driving collective neuronal activity.

physics.bio-ph↗

arXiv2023

Finding the last bits of positional information

In a developing embryo, information about the position of cells is encoded in the concentrations of "morphogen" molecules. In the fruit fly, the local concentrations of just a handful of proteins encoded by the gap genes are sufficient to specify position with a precision comparable to the spacing between cells along the anterior--posterior axis. This matches the precision of downstream events such as the striped patterns of expression in the pair-rule genes, but is not quite sufficient to define unique identities for individual cells. We demonstrate theoretically that this information gap can be bridged if positional errors are spatially correlated, with relatively long correlation lengths. We then show experimentally that these correlations are present, with the required strength, in the fluctuating positions of the pair-rule stripes, and this can be traced back to the gap genes. Taking account of these correlations, the available information matches the information needed for unique cellular specification, within error bars of ~2%. These observation support a precisionist view of information flow through the underlying genetic networks, in which accurate signals are available from the start and preserved as they are transformed into the final spatial patterns.

q-bio.MN↗