arXiv subjects

Tomoki Fukai

Publications and source records attributed to Tomoki Fukai.

13 recordsLinked to original sources

arXiv2024

Associative memory inspires improvements for in-context learning using a novel attention residual stream architecture

Large language models (LLMs) demonstrate an impressive ability to utilise information within the context of their input sequences to appropriately respond to data unseen by the LLM during its training procedure. This ability is known as in-context learning (ICL). Humans and non-human animals demonstrate similar abilities, however their neural architectures differ substantially from LLMs. Despite this, a critical component within LLMs, the attention mechanism, resembles modern associative memory models, widely used in and influenced by the computational neuroscience community to model biological memory systems. Using this connection, we introduce an associative memory model capable of performing ICL. We use this as inspiration for a novel residual stream architecture which allows information to directly flow between attention heads. We test this architecture during training within a two-layer Transformer and show its ICL abilities manifest more quickly than without this modification. We then apply our architecture in small language models with 8 million and 1 billion parameters, focusing on attention head values, with results also indicating improved performance at these larger and more naturalistic scales.

cs.NE↗

arXiv2024

convSeq: Fast and Scalable Method for Detecting Patterns in Spike Data

Spontaneous neural activity, crucial in memory, learning, and spatial navigation, often manifests itself as repetitive spatiotemporal patterns. Despite their importance, analyzing these patterns in large neural recordings remains challenging due to a lack of efficient and scalable detection methods. Addressing this gap, we introduce convSeq, an unsupervised method that employs backpropagation for optimizing spatiotemporal filters that effectively identify these neural patterns. Our method's performance is validated on various synthetic data and real neural recordings, revealing spike sequences with unprecedented scalability and efficiency. Significantly surpassing existing methods in speed, convSeq sets a new standard for analyzing spontaneous neural activity, potentially advancing our understanding of information processing in neural circuits.

eess.SP↗

arXiv2023

An Inherent Trade-Off in Noisy Neural Communication with Rank-Order Coding

Rank-order coding, a form of temporal coding, has emerged as a promising scheme to explain the rapid ability of the mammalian brain. Owing to its speed as well as efficiency, rank-order coding is increasingly gaining interest in diverse research areas beyond neuroscience. However, much uncertainty still exists about the performance of rank-order coding under noise. Herein we show what information rates are fundamentally possible and what trade-offs are at stake. An unexpected finding in this paper is the emergence of a special class of errors that, in a regime, increase with less noise.

cs.CV↗

arXiv2023

Simplicial Hopfield networks

Hopfield networks are artificial neural networks which store memory patterns on the states of their neurons by choosing recurrent connection weights and update rules such that the energy landscape of the network forms attractors around the memories. How many stable, sufficiently-attracting memory patterns can we store in such a network using $N$ neurons? The answer depends on the choice of weights and update rule. Inspired by setwise connectivity in biology, we extend Hopfield networks by adding setwise connections and embedding these connections in a simplicial complex. Simplicial complexes are higher dimensional analogues of graphs which naturally represent collections of pairwise and setwise relationships. We show that our simplicial Hopfield networks increase memory storage capacity. Surprisingly, even when connections are limited to a small random subset of equivalent size to an all-pairwise network, our networks still outperform their pairwise counterparts. Such scenarios include non-trivial simplicial topology. We also test analogous modern continuous Hopfield networks, offering a potentially promising avenue for improving the attention mechanism in Transformer models.

cs.NE↗

arXiv2023

Goal-oriented inference of environment from redundant observations

The agent learns to organize decision behavior to achieve a behavioral goal, such as reward maximization, and reinforcement learning is often used for this optimization. Learning an optimal behavioral strategy is difficult under the uncertainty that events necessary for learning are only partially observable, called as Partially Observable Markov Decision Process (POMDP). However, the real-world environment also gives many events irrelevant to reward delivery and an optimal behavioral strategy. The conventional methods in POMDP, which attempt to infer transition rules among the entire observations, including irrelevant states, are ineffective in such an environment. Supposing Redundantly Observable Markov Decision Process (ROMDP), here we propose a method for goal-oriented reinforcement learning to efficiently learn state transition rules among reward-related "core states'' from redundant observations. Starting with a small number of initial core states, our model gradually adds new core states to the transition diagram until it achieves an optimal behavioral strategy consistent with the Bellman equation. We demonstrate that the resultant inference model outperforms the conventional method for POMDP. We emphasize that our model only containing the core states has high explainability. Furthermore, the proposed method suits online learning as it suppresses memory consumption and improves learning speed.

cs.LG↗

arXiv2022

An Extension of Fisher's Criterion: Theoretical Results with a Neural Network Realization

Fisher's criterion is a widely used tool in machine learning for feature selection. For large search spaces, Fisher's criterion can provide a scalable solution to select features. A challenging limitation of Fisher's criterion, however, is that it performs poorly when mean values of class-conditional distributions are close to each other. Motivated by this challenge, we propose an extension of Fisher's criterion to overcome this limitation. The proposed extension utilizes the available heteroscedasticity of class-conditional distributions to distinguish one class from another. Additionally, we describe how our theoretical results can be casted into a neural network framework, and conduct a proof-of-concept experiment to demonstrate the viability of our approach to solve classification problems.

cs.LG↗

arXiv2022

Modeling the Repetition-based Recovering of Acoustic and Visual Sources with Dendritic Neurons

In natural auditory environments, acoustic signals originate from the temporal superimposition of different sound sources. The problem of inferring individual sources from ambiguous mixtures of sounds is known as blind source decomposition. Experiments on humans have demonstrated that the auditory system can identify sound sources as repeating patterns embedded in the acoustic input. Source repetition produces temporal regularities that can be detected and used for segregation. Specifically, listeners can identify sounds occurring more than once across different mixtures, but not sounds heard only in a single mixture. However, whether such a behaviour can be computationally modelled has not yet been explored. Here, we propose a biologically inspired computational model to perform blind source separation on sequences of mixtures of acoustic stimuli. Our method relies on a somatodendritic neuron model trained with a Hebbian-like learning rule which can detect spatio-temporal patterns recurring in synaptic inputs. We show that the segregation capabilities of our model are reminiscent of the features of human performance in a variety of experimental settings involving synthesized sounds with naturalistic properties. Furthermore, we extend the study to investigate the properties of segregation on task settings not yet explored with human subjects, namely natural sounds and images. Overall, our work suggests that somatodendritic neuron models offer a promising neuro-inspired learning strategy to account for the characteristics of the brain segregation capabilities as well as to make predictions on yet untested experimental settings.

cs.SD↗

arXiv2019

Spontaneous and stimulus-induced coherent states of critically balanced neuronal networks

How the information microscopically processed by individual neurons is integrated and used in organizing the behavior of an animal is a central question in neuroscience. The coherence of neuronal dynamics over different scales has been suggested as a clue to the mechanisms underlying this integration. Balanced excitation and inhibition may amplify microscopic fluctuations to a macroscopic level, thus providing a mechanism for generating coherent multiscale dynamics. Previous theories of brain dynamics, however, were restricted to cases in which inhibition dominated excitation and suppressed fluctuations in the macroscopic population activity. In the present study, we investigate the dynamics of neuronal networks at a critical point between excitation-dominant and inhibition-dominant states. In these networks, the microscopic fluctuations are amplified by the strong excitation and inhibition to drive the macroscopic dynamics, while the macroscopic dynamics determine the statistics of the microscopic fluctuations. Developing a novel type of mean-field theory applicable to this class of interscale interactions, we show that the amplification mechanism generates spontaneous, irregular macroscopic rhythms similar to those observed in the brain. Through the same mechanism, microscopic inputs to a small number of neurons effectively entrain the dynamics of the whole network. These network dynamics undergo a probabilistic transition to a coherent state, as the magnitude of either the balanced excitation and inhibition or the external inputs is increased. Our mean-field theory successfully predicts the behavior of this model. Furthermore, we numerically demonstrate that the coherent dynamics can be used for state-dependent read-out of information from the network. These results show a novel form of neuronal information processing that connects neuronal dynamics on different scales.

cond-mat.dis-nn↗

arXiv2018

Extended temporal association memory by inhibitory Hebbian learning

Hebbian learning of excitatory synapses plays a central role in storing activity patterns in associative memory models. Furthermore, interstimulus Hebbian learning associates multiple items in the brain by converting temporal correlation to spatial correlation between attractors. However, growing experimental evidence suggests that learning of inhibitory synapses creates "inhibitory engrams", which presumably balance with the patterns encoded in the excitatory network. Controlling inhibitory engrams may modify the behavior of associative memory in neural networks, but the consequence of such control has not been theoretically understood. Noting that Hebbian learning of inhibitory synapses yields an anti-Hebbian effect, we show that the combination of Hebbian and anti-Hebbian learning can increase the span of temporal association between the correlated attractors. The balance of targetted and global inhibition regulates this span of association in the network. Our results suggest a nontrivial role of anti-Hebbian learning and inhibitory engrams in associative memory.

q-bio.NC↗

arXiv2018

Discrete-attractor-like Tracking in Continuous Attractor Neural Networks

Continuous attractor neural networks generate a set of smoothly connected attractor states. In memory systems of the brain, these attractor states may represent continuous pieces of information such as spatial locations and head directions of animals. However, during the replay of previous experiences, hippocampal neurons show a discontinuous sequence in which discrete transitions of neural state are phase-locked with the slow-gamma (30-40 Hz) oscillation. Here, we explored the underlying mechanisms of the discontinuous sequence generation. We found that a continuous attractor neural network has several phases depending on the interactions between external input and local inhibitory feedback. The discrete-attractor-like behavior naturally emerges in one of these phases without any discreteness assumption. We propose that the dynamics of continuous attractor neural networks is the key to generate discontinuous state changes phase-locked to the brain rhythm.

cond-mat.dis-nn↗

arXiv2009

Temporal precision of spike response to fluctuating input in pulse-coupled networks of oscillating neurons

A single neuron is known to generate almost identical spike trains when the same fluctuating input is repeatedly applied. Here, we study the reliability of spike firing in a pulse-coupled network of oscillator neurons receiving fluctuating inputs. We can study the precise responses of the network as synchronization between uncoupled copies of the network by a common noisy input. To study the noise-induced synchronization between networks, we derive a self-consistent equation for the distribution of spike-time differences between the networks. Solving this equation, we elucidate how the spike precision changes as a function of the coupling strength.

nlin.AO↗

arXiv2007

Reliability of temporal coding on pulse-coupled networks of oscillators

We study the reliability of spike output in a general class of pulse-coupled oscillators receiving a fluctuating input. Showing that this problem is equivalent to noise-induced synchronization between identical networks of oscillators, we employ the phase reduction method to analytically derive the average Lyapunov exponent of the synchronized state. We show that a transition occurs between reliable and unreliable responses at a critical coupling strength, which is determined through the competition between the external input and recurrent input. To our surprise, the critical value does not depend on intrinsic properties of oscillators.

nlin.AO↗

arXiv2007

Synchronization of Excitatory Neurons with Strongly Heterogeneous Phase Responses

In many real-world oscillator systems, the phase response curves are highly heterogeneous. However, dynamics of heterogeneous oscillator networks has not been seriously addressed. We propose a theoretical framework to analyze such a system by dealing explicitly with the heterogeneous phase response curves. We develop a novel method to solve the self-consistent equations for order parameters by using formal complex-valued phase variables, and apply our theory to networks of in vitro cortical neurons. We find a novel state transition that is not observed in previous oscillator network models.

nlin.PS↗