arXiv subjects

William Bialek

Publications and source records attributed to William Bialek.

121 records · Page 7Linked to original sources

arXiv2002

Spike timing and the coding of naturalistic sounds in a central auditory area of songbirds

In nature, animals encounter high dimensional sensory stimuli that have complex statistical and dynamical structure. Attempts to study the neural coding of these natural signals face challenges both in the selection of the signal ensemble and in the analysis of the resulting neural responses. For zebra finches, naturalistic stimuli can be defined as sounds that they encounter in a colony of conspecific birds. We assembled an ensemble of these sounds by recording groups of 10-40 zebra finches, and then analyzed the response of single neurons in the songbird central auditory area (field L) to continuous playback of long segments from this ensemble. Following methods developed in the fly visual system, we measured the information that spike trains provide about the acoustic stimulus without any assumptions about which features of the stimulus are relevant. Preliminary results indicate that large amounts of information are carried by spike timing, with roughly half of the information accessible only at time resolutions better than 10 ms; additional information is still being revealed as time resolution is improved to 2 ms. Information can be decomposed into that carried by the locking of individual spikes to the stimulus (or modulations of spike rate) vs. that carried by timing in spike patterns. Initial results show that in field L, temporal patterns give at least ~20% extra information. Thus, single central auditory neurons can provide an informative representation of naturalistic sounds, in which spike timing may play a significant role.

physics.bio-ph↗

arXiv2002

Entropy and inference, revisited

We study properties of popular near-uniform (Dirichlet) priors for learning undersampled probability distributions on discrete nonmetric spaces and show that they lead to disastrous results. However, an Occam-style phase space argument expands the priors into their infinite mixture and resolves most of the observed problems. This leads to a surprisingly good estimator of entropies of discrete distributions.

physics.data-an↗

arXiv2001

Complexity Through Nonextensivity

The problem of defining and studying complexity of a time series has interested people for years. In the context of dynamical systems, Grassberger has suggested that a slow approach of the entropy to its extensive asymptotic limit is a sign of complexity. We investigate this idea further by information theoretic and statistical mechanics techniques and show that these arguments can be made precise, and that they generalize many previous approaches to complexity, in particular unifying ideas from the physics literature with ideas from learning and coding theory; there are even connections of this statistical approach to algorithmic or Kolmogorov complexity. Moreover, a set of simple axioms similar to those used by Shannon in his development of information theory allows us to prove that the divergent part of the subextensive component of the entropy is a unique complexity measure. We classify time series by their complexities and demonstrate that beyond the `logarithmic' complexity classes widely anticipated in the literature there are qualitatively more complex, `power--law' classes which deserve more attention.

physics.data-an↗

arXiv2001

Predictability, complexity and learning

We define {\em predictive information} $I_{\rm pred} (T)$ as the mutual information between the past and the future of a time series. Three qualitatively different behaviors are found in the limit of large observation times $T$: $I_{\rm pred} (T)$ can remain finite, grow logarithmically, or grow as a fractional power law. If the time series allows us to learn a model with a finite number of parameters, then $I_{\rm pred} (T)$ grows logarithmically with a coefficient that counts the dimensionality of the model space. In contrast, power--law growth is associated, for example, with the learning of infinite parameter (or nonparametric) models such as continuous functions with smoothness constraints. There are connections between the predictive information and measures of complexity that have been defined both in learning theory and in the analysis of physical systems through statistical mechanics and dynamical systems theory. Further, in the same way that entropy provides the unique measure of available information consistent with some simple and plausible conditions, we argue that the divergent part of $I_{\rm pred} (T)$ provides the unique measure for the complexity of dynamics underlying a time series. Finally, we discuss how these ideas may be useful in different problems in physics, statistics, and biology.

physics.data-an↗

arXiv2000

Universality and individuality in a neural code

The problem of neural coding is to understand how sequences of action potentials (spikes) are related to sensory stimuli, motor outputs, or (ultimately) thoughts and intentions. One clear question is whether the same coding rules are used by different neurons, or by corresponding neurons in different individuals. We present a quantitative formulation of this problem using ideas from information theory, and apply this approach to the analysis of experiments in the fly visual system. We find significant individual differences in the structure of the code, particularly in the way that temporal patterns of spikes are used to convey information beyond that available from variations in spike rate. On the other hand, all the flies in our ensemble exhibit a high coding efficiency, so that every spike carries the same amount of information in all the individuals. Thus the neural code has a quantifiable mixture of individuality and universality.

physics.bio-ph↗

arXiv2000

Stability and noise in biochemical switches

Many processes in biology, from the regulation of gene expression in bacteria to memory in the brain, involve switches constructed from networks of biochemical reactions. Crucial molecules are present in small numbers, raising questions about noise and stability. Analysis of noise in simple reaction schemes indicates that switches stable for years and switchable in milliseconds can be built from fewer than one hundred molecules. Prospects for direct tests of this prediction, as well as implications, are discussed.

cond-mat.soft↗

arXiv2000

Real time encoding of motion: Answerable questions and questionable answers from the fly's visual system

In the past decade, a small corner of the fly's visual system has become an important testing ground for ideas about coding and computation in the nervous system. A number of results demonstrate that this system operates with a precision and efficiency near the limits imposed by physics, and more generally these results point to the reliability and efficiency of the strategies that nature has selected for representing and processing visual signals. A recent series of papers by Egelhaaf and coworkers, however, suggests that almost all these conclusions are incorrect. In this contribution we place these controversies in a larger context, emphasizing that the arguments are not just about flies, but rather about how we should quantify the neural response to complex, naturalistic inputs. As an example, Egelhaaf et al. (and many others) compute certain correlation functions and use the apparent correlation times as a measure of temporal precision in the neural response. This analysis neglects the structure of the correlation function at short times, and we show how to analyze this structure to reveal a temporal precision 30 times better than suggested by the correlation time; this precision is confirmed by a much more detailed information theoretic analysis. In reviewing other aspects of the controversy, we find that the analysis methods used by Egelhaaf et al. suffer from some mathematical inconsistencies, and that in some cases we are unable to reproduce their experimental results. Finally, we present results from new experiments that probe the neural response to inputs that approach more closely the natural context for freely flying flies. These new experiments demonstrate that the fly's visual system is even more precise and efficient under natural conditions than had been inferred from our earlier work.

physics.bio-ph↗

arXiv2000

The information bottleneck method

We define the relevant information in a signal $x\in X$ as being the information that this signal provides about another signal $y\in \Y$. Examples include the information that face images provide about the names of the people portrayed, or the information that speech sounds provide about the words spoken. Understanding the signal $x$ requires more than just predicting $y$, it also requires specifying which features of $\X$ play a role in the prediction. We formalize this problem as that of finding a short code for $\X$ that preserves the maximum information about $\Y$. That is, we squeeze the information that $\X$ provides about $\Y$ through a `bottleneck' formed by a limited set of codewords $\tX$. This constrained optimization problem can be seen as a generalization of rate distortion theory in which the distortion measure $d(x,\x)$ emerges from the joint statistics of $\X$ and $\Y$. This approach yields an exact set of self consistent equations for the coding rules $X \to \tX$ and $\tX \to \Y$. Solutions to these equations can be found by a convergent re-estimation method that generalizes the Blahut-Arimoto algorithm. Our variational principle provides a surprisingly rich framework for discussing a variety of problems in signal processing and learning, as will be described in detail elsewhere.

physics.data-an↗

arXiv1999

Predictive Information

Observations on the past provide some hints about what will happen in the future, and this can be quantified using information theory. The ``predictive information'' defined in this way has connections to measures of complexity that have been proposed both in the study of dynamical systems and in mathematical statistics. In particular, the predictive information diverges when the observed data stream allows us to learn an increasingly precise model for the dynamics that generate the data, and the structure of this divergence measures the complexity of the model. We argue that divergent contributions to the predictive information provide the only measure of complexity or richness that is consistent with certain plausible requirements.

cond-mat.stat-mech↗

arXiv1997

Entropy and Information in Neural Spike Trains

The nervous system represents time-dependent signals in sequences of discrete action potentials or spikes, all spikes are identical so that information is carried only in the spike arrival times. We show how to quantify this information, in bits, free from any assumptions about which features of the spike train or input waveform are most important. We apply this approach to the analysis of experiments on a motion-sensitive neuron in the fly visual system.

cond-mat↗

arXiv1997

Adaptation and Optimal Chemotactic Strategy for E. Coli

Extending the classic works of Berg and Purcell on the biophysics of bacterial chemotaxis, we find the optimal chemotactic strategy for the peritrichous bacterium E. Coli in the high and low signal to noise ratio limits. The optimal strategy depends on properties of the environment and properties of the individual bacterium and is therefore highly adaptive. We review experiments relevant to testing both the form of the proposed strategy and its adaptability and propose extensions of them which could test the limits of the adaptability in this simplest sensory processing system.

adap-org↗

arXiv1996

Field Theories for Learning Probability Distributions

Imagine being shown $N$ samples of random variables drawn independently from the same distribution. What can you say about the distribution? In general, of course, the answer is nothing, unless we have some prior notions about what to expect. From a Bayesian point of view we need an {\it a priori} distribution on the space of possible probability distributions, which defines a scalar field theory. In one dimension, free field theory with a constraint provides a tractable formulation of the problem, and we also discus generalizations to higher dimensions.

cond-mat↗

arXiv1994

Statistical Mechanics and Visual Signal Processing

The nervous system solves a wide variety of problems in signal processing. In many cases the performance of the nervous system is so good that it apporaches fundamental physical limits, such as the limits imposed by diffraction and photon shot noise in vision. In this paper we show how to use the language of statistical field theory to address and solve problems in signal processing, that is problems in which one must estimate some aspect of the environment from the data in an array of sensors. In the field theory formulation the optimal estimator can be written as an expectation value in an ensemble where the input data act as external field. Problems at low signal-to-noise ratio can be solved in perturbation theory, while high signal-to-noise ratios are treated with a saddle-point approximation. These ideas are illustrated in detail by an example of visual motion estimation which is chosen to model a problem solved by the fly's brain. In this problem the optimal estimator has a rich structure, adapting to various parameters of the environment such as the mean-square contrast and the correlation time of contrast fluctuations. This structure is in qualitative accord with existing measurements on motion sensitive neurons in the fly's brain, and we argue that the adaptive properties of the optimal estimator may help resolve conlficts among different interpretations of these data. Finally we propose some crucial direct tests of the adaptive behavior.

cond-mat↗