arXiv subjects

Masato Okada

Publications and source records attributed to Masato Okada.

At least 91 records · Page 5Linked to original sources

arXiv2004

The path-integral analysis of an associative memory model storing an infinite number of finite limit cycles

It is shown that an exact solution of the transient dynamics of an associative memory model storing an infinite number of limit cycles with l finite steps by means of the path-integral analysis. Assuming the Maxwell construction ansatz, we have succeeded in deriving the stationary state equations of the order parameters from the macroscopic recursive equations with respect to the finite-step sequence processing model which has retarded self-interactions. We have also derived the stationary state equations by means of the signal-to-noise analysis (SCSNA). The signal-to-noise analysis must assume that crosstalk noise of an input to spins obeys a Gaussian distribution. On the other hand, the path-integral method does not require such a Gaussian approximation of crosstalk noise. We have found that both the signal-to-noise analysis and the path-integral analysis give the completely same result with respect to the stationary state in the case where the dynamics is deterministic, when we assume the Maxwell construction ansatz. We have shown the dependence of storage capacity (alpha_c) on the number of patterns per one limit cycle (l). Storage capacity monotonously increases with the number of steps, and converges to alpha_c=0.269 at l ~= 10. The original properties of the finite-step sequence processing model appear as long as the number of steps of the limit cycle has order l=O(1).

cond-mat.dis-nn↗

arXiv2004

Pulse-coupled resonate-and-fire models

We analyze two pulse-coupled resonate-and-fire neurons. Numerical simulation reveals that an anti-phase state is an attractor of this model. We can analytically explain the stability of anti-phase states by means of a return map of firing times, which we propose in this paper. The resultant stability condition turns out to be quite simple. The phase diagram based on our theory shows that there are two types of anti-phase states. One of these cannot be seen in coupled integrate-and-fire models and is peculiar to resonate-and-fire models. The results of our theory coincide with those of numerical simulations.

q-bio.NC↗

arXiv2004

Impact of deviation from precise balance of spike-timing-dependent plasticity

Recent biological experimental findings have shown that synaptic plasticity depends on the relative timing of pre- and post-synaptic spikes and this is called spike-timing-dependent plasticity (STDP). Many authors have claimed that a precise balance between long-term potentiation (LTP) and long-term depression (LTD) of STDP is crucial in the storage of spatio-temporal patterns. Some authors have numerically investigated the impact of an imbalance between LTP and LTD on the network properties. However, the mathematical mechanism remains unknown. We analytically show that an associative memory network has the robust retrieval properties of spatio-temporal patterns, and these properties make the network less vulnerable to any deviation from a precise balance between LTP and LTD when the network contains a finite number of neurons.

cond-mat.dis-nn↗

arXiv2004

Analysis of Bidirectional Associative Memory using SCSNA and Statistical Neurodynamics

Bidirectional associative memory (BAM) is a kind of an artificial neural network used to memorize and retrieve heterogeneous pattern pairs. Many efforts have been made to improve BAM from the the viewpoint of computer application, and few theoretical studies have been done. We investigated the theoretical characteristics of BAM using a framework of statistical-mechanical analysis. To investigate the equilibrium state of BAM, we applied self-consistent signal to noise analysis (SCSNA) and obtained a macroscopic parameter equations and relative capacity. Moreover, to investigate not only the equilibrium state but also the retrieval process of reaching the equilibrium state, we applied statistical neurodynamics to the update rule of BAM and obtained evolution equations for the macroscopic parameters. These evolution equations are consistent with the results of SCSNA in the equilibrium state.

cond-mat.dis-nn↗

arXiv2004

Ensemble learning of linear perceptron; Online learning theory

Within the framework of on-line learning, we study the generalization error of an ensemble learning machine learning from a linear teacher perceptron. The generalization error achieved by an ensemble of linear perceptrons having homogeneous or inhomogeneous initial weight vectors is precisely calculated at the thermodynamic limit of a large number of input elements and shows rich behavior. Our main findings are as follows. For learning with homogeneous initial weight vectors, the generalization error using an infinite number of linear student perceptrons is equal to only half that of a single linear perceptron, and converges with that of the infinite case with O(1/K) for a finite number of K linear perceptrons. For learning with inhomogeneous initial weight vectors, it is advantageous to use an approach of weighted averaging over the output of the linear perceptrons, and we show the conditions under which the optimal weights are constant during the learning process. The optimal weights depend on only correlation of the initial weight vectors.

cond-mat.dis-nn↗

arXiv2003

Robustness of retrieval properties against imbalance between long-term potentiation and depression of spike-timing-dependent plasticity

Spike-timing-dependent plasticity (STDP) has recently been shown in some physiological studies. STDP depends on the precise temporal relationship of pre- and post-synaptic spikes. Many authors have indicated that a precise balance between long-term potentiation (LTP) and long-term depression (LTD) of STDP is significant for a stable learning. However, a situation in which the balance is maintained precisely is inconceivable in the brain. Using a method of the statistical neurodynamics, we show robust retrieval properties of spatio-temporal patterns in an associative memory model against the imbalance between LTP and LTD. When the fluctuation of LTD is assumed to obey a Gaussian distribution with mean 0 and variance $δ^2$, the storage capacity takes a finite value even at large $δ$. This means that the balance between LTP and LTD of STDP need not to be maintained precisely, but must be maintained on average. Furthermore, we found that a basin of attraction becomes smaller as $δ$ increases while an initial critical overlap remains unchanged.

cond-mat.dis-nn↗

arXiv2003

Synapse efficiency diverges due to synaptic pruning following over-growth

In the development of the brain, it is known that synapses are pruned following over-growth. This pruning following over-growth seems to be a universal phenomenon that occurs in almost all areas -- visual cortex, motor area, association area, and so on. It has been shown numerically that the synapse efficiency is increased by systematic deletion. We discuss the synapse efficiency to evaluate the effect of pruning following over-growth, and analytically show that the synapse efficiency diverges as O(log c) at the limit where connecting rate c is extremely small. Under a fixed synapse number criterion, the optimal connecting rate, which maximize memory performance, exists.

cond-mat.dis-nn↗

arXiv2003

One step RSB scheme for the rate distortion function

We apply statistical mechanics to an inverse problem of linear mapping to investigate the physics of the irreversible compression. We use the replica symmetry breaking (RSB) technique with a toy model to demonstrate the Shannon's result. The rate distortion function, which is widely known as the theoretical limit of the compression with a fidelity criterion, is derived using the Parisi one step RSB scheme. The bound can not be achieved in the sparsely-connected systems, where suboptimal solutions dominate the capacity.

cond-mat.dis-nn↗

arXiv2003

On-line learning through simple perceptron with a margin

We analyze a learning method that uses a margin $κ$ {\it a la} Gardner for simple perceptron learning. This method corresponds to the perceptron learning when $κ=0$, and to the Hebbian learning when $κ\to \infty$. Nevertheless, we found that the generalization ability of the method was superior to that of the perceptron and the Hebbian methods at an early stage of learning. We analyzed the asymptotic property of the learning curve of this method through computer simulation and found that it was the same as for perceptron learning. We also investigated an adaptive margin control method.

cond-mat.dis-nn↗

arXiv2003

Naive Mean Field Approximation for the Error Correcting Code

Solving the error correcting code is an important goal with regard to communication theory.To reveal the error correcting code characteristics, several researchers have applied a statistical-mechanical approach to this problem. In our research, we have treated the error correcting code as a Bayes inference framework. Carrying out the inference in practice, we have applied the NMF (naive mean field) approximation to the MPM (maximizer of the posterior marginals) inference, which is a kind of Bayes inference. In the field of artificial neural networks, this approximation is used to reduce computational cost through the substitution of stochastic binary units with the deterministic continuous value units. However, few reports have quantitatively described the performance of this approximation. Therefore, we have analyzed the approximation performance from a theoretical viewpoint, and have compared our results with the computer simulation.

cond-mat.dis-nn↗

arXiv2003

Storage Capacity Diverges with Synaptic Efficiency in an Associative Memory Model with Synaptic Delay and Pruning

It is known that storage capacity per synapse increases by synaptic pruning in the case of a correlation-type associative memory model. However, the storage capacity of the entire network then decreases. To overcome this difficulty, we propose decreasing the connecting rate while keeping the total number of synapses constant by introducing delayed synapses. In this paper, a discrete synchronous-type model with both delayed synapses and their prunings is discussed as a concrete example of the proposal. First, we explain the Yanai-Kim theory by employing the statistical neurodynamics. This theory involves macrodynamical equations for the dynamics of a network with serial delay elements. Next, considering the translational symmetry of the explained equations, we re-derive macroscopic steady state equations of the model by using the discrete Fourier transformation. The storage capacities are analyzed quantitatively. Furthermore, two types of synaptic prunings are treated analytically: random pruning and systematic pruning. As a result, it becomes clear that in both prunings, the storage capacity increases as the length of delay increases and the connecting rate of the synapses decreases when the total number of synapses is constant. Moreover, an interesting fact becomes clear: the storage capacity asymptotically approaches $2/π$ due to random pruning. In contrast, the storage capacity diverges in proportion to the logarithm of the length of delay by systematic pruning and the proportion constant is $4/π$. These results theoretically support the significance of pruning following an overgrowth of synapses in the brain and strongly suggest that the brain prefers to store dynamic attractors such as sequences and limit cycles rather than equilibrium states.

cond-mat.dis-nn↗

arXiv2003

Low-dimensional chaos induced by frustration in a non-monotonic system

We report a novel mechanism for the occurrence of chaos at the macroscopic level induced by the frustration of interaction, namely frustration-induced chaos, in a non-monotonic sequential associative memory model. We succeed in deriving exact macroscopic dynamical equations from the microscopic dynamics in the case of the thermodynamic limit and prove that two order parameters dominate this large-degree-of-freedom system. Two-parameter bifurcation diagrams are obtained from the order-parameter equations. Then we analytically show that the chaos is low-dimensional at the macroscopic level when the system has some degree of frustration, but that the chaos definitely does not occur without the frustration.

cond-mat.dis-nn↗

arXiv2003

Mixed states in a sparsely encoded associative memory model storing ultrametric patterns

When mixed states are composed of s memory patterns, s types of mixed states, which can become equilibrium states of the model, can be generated. We previously reported that storage capacities for most mixed states composed of uncorrelated memory patterns do not diverge at the sparse limit of the firing rate, f -> 0. On the contrary to the uncorrelated case, we show that the storage capacities for all mixed states composed of correlated memory patterns diverge as 1/|f log f| even when the correlation of memory patterns is infinitestimal small. We also show that, when the firing rate is fixed, as the correlation coefficient increases, the storage capacities of the mixed states composed of all correlated memory patterns increase, while those of the mixed states composed of only some of the correlated memory patterns decrease.

cond-mat.dis-nn↗

arXiv2002

On-Line Learning Theory of Soft Committee Machines with Correlated Hidden Units - Steepest Gradient Descent and Natural Gradient Descent -

The permutation symmetry of the hidden units in multilayer perceptrons causes the saddle structure and plateaus of the learning dynamics in gradient learning methods. The correlation of the weight vectors of hidden units in a teacher network is thought to affect this saddle structure, resulting in a prolonged learning time, but this mechanism is still unclear. In this paper, we discuss it with regard to soft committee machines and on-line learning using statistical mechanics. Conventional gradient descent needs more time to break the symmetry as the correlation of the teacher weight vectors rises. On the other hand, no plateaus occur with natural gradient descent regardless of the correlation for the limit of a low learning rate. Analytical results support these dynamics around the saddle point.

cond-mat.dis-nn↗

arXiv2002

Dynamically-Coupled Oscillators -- Cooperative Behavior via Dynamical Interaction --

We propose a theoretical framework to study the cooperative behavior of dynamically coupled oscillators (DCOs) that possess dynamical interactions. Then, to understand synchronization phenomena in networks of interneurons which possess inhibitory interactions, we propose a DCO model with dynamics of interactions that tend to cause 180-degree phase lags. Employing an approach developed here, we demonstrate that although our model displays synchronization at high frequencies, it does not exhibit synchronization at low frequencies because this dynamical interaction does not cause a phase lag sufficiently large to cancel the effect of the inhibition. We interpret the disappearance of synchronization in our model with decreasing frequency as describing the breakdown of synchronization in the interneuron network of the CA1 area below the critical frequency of 20 Hz.

cond-mat.dis-nn↗

arXiv2002

Statistical Mechanics of the Bayesian Image Restoration under Spatially Correlated Noise

We investigated the use of the Bayesian inference to restore noise-degraded images under conditions of spatially correlated noise. The generative statistical models used for the original image and the noise were assumed to obey multi-dimensional Gaussian distributions whose covariance matrices are translational invariant. We derived an exact description to be used as the expectation for the restored image by the Fourier transformation and restored an image distorted by spatially correlated noise by using a spatially uncorrelated noise model. We found that the resulting hyperparameter estimations for the minimum error and maximal posterior marginal criteria did not coincide when the generative probabilistic model and the model used for restoration were in different classes, while they did coincide when they were in the same class.

cond-mat.dis-nn↗

arXiv2002

Associative Memory by Recurrent Neural Networks with Delay Elements

The synapses of real neural systems seem to have delays. Therefore, it is worthwhile to analyze associative memory models with delayed synapses. Thus, a sequential associative memory model with delayed synapses is discussed, where a discrete synchronous updating rule and a correlation learning rule are employed. Its dynamic properties are analyzed by the statistical neurodynamics. In this paper, we first re-derive the Yanai-Kim theory, which involves macrodynamical equations for the dynamics of the network with serial delay elements. Since their theory needs a computational complexity of $O(L^4t)$ to obtain the macroscopic state at time step t where L is the length of delay, it is intractable to discuss the macroscopic properties for a large L limit. Thus, we derive steady state equations using the discrete Fourier transformation, where the computational complexity does not formally depend on L. We show that the storage capacity $α_C$ is in proportion to the delay length L with a large L limit, and the proportion constant is 0.195, i.e., $α_C = 0.195 L$. These results are supported by computer simulations.

cond-mat.dis-nn↗

arXiv2002

Naive mean field approximation for image restoration

We attempt image restoration in the framework of the Baysian inference. Recently, it has been shown that under a certain criterion the MAP (Maximum A Posterior) estimate, which corresponds to the minimization of energy, can be outperformed by the MPM (Maximizer of the Posterior Marginals) estimate, which is equivalent to a finite-temperature decoding method. Since a lot of computational time is needed for the MPM estimate to calculate the thermal averages, the mean field method, which is a deterministic algorithm, is often utilized to avoid this difficulty. We present a statistical-mechanical analysis of naive mean field approximation in the framework of image restoration. We compare our theoretical results with those of computer simulation, and investigate the potential of naive mean field approximation.

cond-mat.dis-nn↗