arXiv subjects

A. C. C. Coolen

Publications and source records attributed to A. C. C. Coolen.

At least 55 records · Page 3Linked to original sources

arXiv2001

Supervised Learning with Restricted Training Sets: a Generating Functional Analysis

We study the dynamics of supervised on-line learning of realizable tasks in feed-forward neural networks. We focus on the regime where the number of examples used for training is proportional to the number of input channels N. Using generating function techniques from spin glass theory, we are able to average over the composition of the training set and transform the problem for N to infinity to an effective single pattern system, described completely by the student autocovariance, the student-teacher overlap and the student response function, with exact closed equations. Our method applies to arbitrary learning rules, i.e. not necessarily of a gradient-descent type. The resulting exact macroscopic dynamical equations can be integrated without finite-size effects up to any degree of accuracy, but their main value is in providing an exact and simple starting point for analytical approximation schemes. Finally, we show how, in the region of absent anomalous response and using the hypothesis that (as in detailed balance systems) the short-time part of the various operators can be transformed away, one can describe the stationary state of the network succesfully by a set of coupled equations involving only four scalar order parameters.

cond-mat.dis-nn↗

arXiv2001

A Solvable Model of Secondary Structure Formation in Random Hetero-Polymers

We propose and solve a simple model describing secondary structure formation in random hetero-polymers. It describes monomers with a combination of one-dimensional short-range interactions (representing steric forces and hydrogen bonds) and infinite range interactions (representing polarity forces). We solve our model using a combination of mean field and random field techniques, leading to phase diagrams exhibiting second-order transitions between folded, partially folded and unfolded states, including regions where folding depends on initial conditions. Our theoretical results, which are in excellent agreement with numerical simulations, lead to an appealing physical picture of the folding process: the polarity forces drive the transition to a collapsed state, the steric forces introduce monomer specificity, and the hydrogen bonds stabilise the conformation by damping the frustration-induced multiplicity of states.

cond-mat.dis-nn↗

arXiv2000

Generating Functional Analysis of the Dynamics of the Batch Minority Game with Random External Information

We study the dynamics of the batch minority game, with random external information, using generating functional techniques a la De Dominicis. The relevant control parameter in this model is the ratio $α=p/N$ of the number $p$ of possible values for the external information over the number $N$ of trading agents. In the limit $N\to\infty$ we calculate the location $α_c$ of the phase transition (signaling the onset of anomalous response), and solve the statics for $α>α_c$ exactly. The temporal correlations in global market fluctuations turn out not to decay to zero for infinitely widely separated times. For $α<α_c$ the stationary state is shown to be non-unique. For $α\to 0$ we analyse our equations in leading order in $α$, and find asymptotic solutions with diverging volatility $σ=\order(α^{-{1/2}})$ (as regularly observed in simulations), but also asymptotic solutions with vanishing volatility $σ=\order(α^{1/2})$. The former, however, are shown to emerge only if the agents' initial strategy valuations are below a specific critical value.

cond-mat.dis-nn↗

arXiv2000

Cluster derivation of Parisi's RSB solution for disordered systems

We propose a general scheme in which disordered systems are allowed to sacrifice energy equi-partitioning and separate into a hierarchy of ergodic sub-systems (clusters) with different characteristic time-scales and temperatures. The details of the break-up follow from the requirement of stationarity of the entropy of the slower cluster, at every level in the hierarchy. We apply our ideas to the Sherrington-Kirkpatrick model, and show how the Parisi solution can be {\it derived} quantitatively from plausible physical principles. Our approach gives new insight into the physics behind Parisi's solution and its relations with other theories, numerical experiments, and short range models.

cond-mat.dis-nn↗

arXiv2000

Attractor Modulation and Proliferation in 1+$\infty$ Dimensional Neural Networks

We extend a recently introduced class of exactly solvable models for recurrent neural networks with competition between 1D nearest neighbour and infinite range information processing. We increase the potential for further frustration and competition in these models, as well as their biological relevance, by adding next-nearest neighbour couplings, and we allow for modulation of the attractors so that we can interpolate continuously between situations with different numbers of stored patterns. Our models are solved by combining mean field and random field techniques. They exhibit increasingly complex phase diagrams with novel phases, separated by multiple first- and second order transitions (dynamical and thermodynamic ones), and, upon modulating the attractor strengths, non-trivial scenarios of phase diagram deformation. Our predictions are in excellent agreement with numerical simulations.

cond-mat.dis-nn↗

arXiv2000

Cluster Derivation of the Parisi Scheme for Disordered Systems

We propose a general quantitative scheme in which systems are given the freedom to sacrifice energy equi-partitioning on the relevant time-scales of observation, and have phase transitions by separating autonomously into ergodic sub-systems (clusters) with different characteristic time-scales and temperatures. The details of the break-up follow uniquely from the requirement of zero entropy for the slower cluster. Complex systems, such as the Sherrington-Kirkpatrick model, are found to minimise their free energy by spontaneously decomposing into a hierarchy of ergodically equilibrating degrees of freedom at different (effective) temperatures. This leads exactly and uniquely to Parisi's replica symmetry breaking scheme. Our approach, which is somewhat akin to an earlier one by Sompolinsky, gives new insight into the physical interpretation of the Parisi scheme and its relations with other approaches, numerical experiments, and short range models. Furthermore, our approach shows that the Parisi scheme can be derived quantitatively and uniquely from plausible physical principles.

cond-mat.stat-mech↗

arXiv2000

Non-Deterministic Learning Dynamics in Large Neural Networks due to Structural Data Bias

We study the dynamics of on-line learning in large perceptrons, for the case of training sets with a structural bias of the input vectors, by deriving exact and closed macroscopic dynamical laws using non-equilibrium statistical mechanical tools. In sharp contrast to the more conventional theories developed for homogeneously distributed or only weakly biased data, these laws are found to describe a non-trivial and persistently non-deterministic macroscopic evolution, and a generalisation error which retains both stochastic and sample-to-sample fluctuations, even for infinitely large networks. Furthermore, for the standard error-correcting microscopic algorithms (such as the perceptron learning rule) one obtains learning curves with distinct bias-induced phases. Our theoretical predictions find excellent confirmation in numerical simulations.

cond-mat.dis-nn↗

arXiv2000

Statistical Mechanics of Recurrent Neural Networks I. Statics

A lecture notes style review of the equilibrium statistical mechanics of recurrent neural networks with discrete and continuous neurons (e.g. Ising, coupled-oscillators). To be published in the Handbook of Biological Physics (North-Holland). Accompanied by a similar review (part II) dealing with the dynamics.

cond-mat.dis-nn↗

arXiv2000

Statistical Mechanics of Recurrent Neural Networks II. Dynamics

A lecture notes style review of the non-equilibrium statistical mechanics of recurrent neural networks with discrete and continuous neurons (e.g. Ising, graded-response, coupled-oscillators). To be published in the Handbook of Biological Physics (North-Holland). Accompanied by a similar review (part I) dealing with the statics.

cond-mat.dis-nn↗

arXiv2000

Exact Solution of the Infinite-Range Quantum Mattis Model

We have solved the quantum version of the Mattis model with infinite-range interactions. A variational approach gives the exact solution for the infinite-range system, in spite of the non-commutative nature of the quantum spin components; this implies that quantum effects are not predominant in determining the macroscopic properties of the system. Nevertheless, the model has a surprisingly rich phase behaviour, exhibiting phase diagrams with tricritical, three-phase and critical end points.

cond-mat.dis-nn↗

arXiv2000

One+Infinite Dimensional Attractor Neural Networks

We solve a class of attractor neural network models with a mixture of 1D nearest-neighbour and infinite-range interactions, which are of a Hebbian-type form. Our solution is based on a combination of mean-field methods, transfer matrices and 1D random-field techniques, and is obtained for Boltzmann-type equilibrium (following sequential Glauber dynamics) and Peretto-type equilibrium (following parallel dynamics). Competition between the alignment forces mediated via short-range interactions, and those mediated via infinite-range ones, is found to generate novel phenomena, such as multiple locally stable `pure' states, first-order transitions between recall states, 2-cycles and non-recall states, and domain formation leading to extremely long relaxation times. We test our results against numerical simulations and simple benchmark cases and find excellent agreement.

cond-mat.dis-nn↗

arXiv1999

Dynamics of Learning with Restricted Training Sets II: Tests and Applications

We apply a general theory describing the dynamics of supervised learning in layered neural networks in the regime where the size p of the training set is proportional to the number of inputs N, as developed in a previous paper, to several choices of learning rules. In the case of (on-line and batch) Hebbian learning, where a direct exact solution is possible, we show that our theory provides exact results at any time in many different verifiable cases. For non-Hebbian learning rules, such as Perceptron and AdaTron, we find very good agreement between the predictions of our theory and numerical simulations. Finally, we derive three approximation schemes aimed at eliminating the need to solve a functional saddle-point equation at each time step, and assess their performance. The simplest of these schemes leads to a fully explicit and relatively simple non-linear diffusion equation for the joint field distribution, which already describes the learning dynamics surprisingly well over a wide range of parameters.

cond-mat.dis-nn↗

arXiv1999

Dynamics of Learning with Restricted Training Sets I: General Theory

We study the dynamics of supervised learning in layered neural networks, in the regime where the size $p$ of the training set is proportional to the number $N$ of inputs. Here the local fields are no longer described by Gaussian probability distributions and the learning dynamics is of a spin-glass nature, with the composition of the training set playing the role of quenched disorder. We show how dynamical replica theory can be used to predict the evolution of macroscopic observables, including the two relevant performance measures (training error and generalization error), incorporating the old formalism developed for complete training sets in the limit $α=p/N\to\infty$ as a special case. For simplicity we restrict ourselves in this paper to single-layer networks and realizable tasks.

cond-mat.dis-nn↗

arXiv1998

On-Line Learning with Restricted Training Sets: An Exactly Solvable Case

We solve the dynamics of on-line Hebbian learning in large perceptrons exactly, for the regime where the size of the training set scales linearly with the number of inputs. We consider both noiseless and noisy teachers. Our calculation cannot be extended to non-Hebbian rules, but the solution provides a convenient and welcome benchmark with which to test more general and advanced theories for solving the dynamics of learning with restricted training sets.

cond-mat.dis-nn↗

arXiv1998

The XY Spin-Glass with Slow Dynamic Couplings

We investigate an XY spin-glass model in which both spins and couplings evolve in time: the spins change rapidly according to Glauber-type rules, whereas the couplings evolve slowly with a dynamics involving spin correlations and Gaussian disorder. For large times the model can be solved using replica theory. In contrast to the XY-model with static disordered couplings, solving the present model requires two levels of replicas, one for the spins and one for the couplings. Relevant order parameters are defined and a phase diagram is obtained upon making the replica-symmetric Ansatz. The system exhibits two different spin-glass phases, with distinct de Almeida-Thouless lines, marking continuous replica-symmetry breaking: one describing freezing of the spins only, and one describing freezing of both spins and couplings.

cond-mat.dis-nn↗

arXiv1998

Phase Diagram and Storage Capacity of Sequence Processing Neural Networks

We solve the dynamics of Hopfield-type neural networks which store sequences of patterns, close to saturation. The asymmetry of the interaction matrix in such models leads to violation of detailed balance, ruling out an equilibrium statistical mechanical analysis. Using generating functional methods we derive exact closed equations for dynamical order parameters, viz. the sequence overlap and correlation- and response functions, in the thermodynamic limit. We calculate the time translation invariant solutions of these equations, describing stationary limit-cycles, which leads to a phase diagram. The effective retarded self-interaction usually appearing in symmetric models is here found to vanish, which causes a significantly enlarged storage capacity of $α_c\sim 0.269$, compared to $α_\c\sim 0.139$ for Hopfield networks storing static patterns. Our results are tested against extensive computer simulations and excellent agreement is found.

cond-mat.dis-nn↗

arXiv1998

Finite Size Effects in Separable Recurrent Neural Networks

We perform a systematic analytical study of finite size effects in separable recurrent neural network models with sequential dynamics, away from saturation. We find two types of finite size effects: thermal fluctuations, and disorder-induced `frozen' corrections to the mean-field laws. The finite size effects are described by equations that correspond to a time-dependent Ornstein-Uhlenbeck process. We show how the theory can be used to understand and quantify various finite size phenomena in recurrent neural networks, with and without detailed balance.

cond-mat.dis-nn↗

arXiv1998

Dynamics of Supervised Learning with Restricted Training Sets

We study the dynamics of supervised learning in layered neural networks, in the regime where the size $p$ of the training set is proportional to the number $N$ of inputs. Here the local fields are no longer described by Gaussian probability distributions. We show how dynamical replica theory can be used to predict the evolution of macroscopic observables, including the relevant performance measures, incorporating the old formalism in the limit $α=p/N\to\infty$ as a special case. For simplicity we restrict ourselves to single-layer networks and realizable tasks.

cond-mat.dis-nn↗