arXiv subjects

Taro Toyoizumi

Publications and source records attributed to Taro Toyoizumi.

At least 19 recordsLinked to original sources

arXiv2026

An anticipated surprise-hazard framework for risky intertemporal choices

People frequently deviate from classical utility theories when making risky and intertemporal decisions. While the effects of risk and temporal delay have been extensively studied in isolation, their interplay and underlying theoretical basis remain debated. In this work, we extend our previously proposed anticipated surprise framework to risky intertemporal choices by modeling delayed rewards as outcomes that fail to materialize at a constant hazard rate. This accounts for key empirical findings: diverse patterns of time inconsistency and aversion to timing risk stem from the avoidance of large negative surprises, while conflicting empirical findings on how risk affects temporal discounting emerge from differences in mental representations of outcome resolution. The extended framework offers a new perspective on how various types of risk may interact and provides a flexible foundation for analyzing complex real-life decision-making scenarios.

econ.TH↗

arXiv2026

Dataset Distillation Efficiently Encodes Low-Dimensional Representations from Gradient-Based Learning of Non-Linear Tasks

Dataset distillation, a training-aware data compression technique, has recently attracted increasing attention as an effective tool for mitigating costs of optimization and data storage. However, progress remains largely empirical. Mechanisms underlying the extraction of task-relevant information from the training process and the efficient encoding of such information into synthetic data points remain elusive. In this paper, we theoretically analyze practical algorithms of dataset distillation applied to the gradient-based training of two-layer neural networks with width $L$. By focusing on a non-linear task structure called multi-index model, we prove that the low-dimensional structure of the problem is efficiently encoded into the resulting distilled data. This dataset reproduces a model with high generalization ability for a required memory complexity of $\tildeΘ$$(r^2d+L)$, where $d$ and $r$ are the input and intrinsic dimensions of the task. To the best of our knowledge, this is one of the first theoretical works that include a specific task structure, leverage its intrinsic dimensionality to quantify the compression rate and study dataset distillation implemented solely via gradient-based algorithms.

cs.LG↗

arXiv2026

Cortex and subcortex play distinct roles over learning when cortical memory is limited

It has been proposed that the brain integrates flexible, computationally expensive cortical processing with simpler, lower-cost subcortical mechanisms to achieve resource-efficient performance greater than that of either system alone. Despite the allure of this perspective, satisfying theoretical frameworks that explore this hypothesis are still limited. We extend existing frameworks in which a model-based module and model-free module learn in tandem by explicitly constraining the memory resources of the model-based module, and investigate the impact of this constraint in a simple decision-making setting. Memory constraints naturally give rise to strategies for allocating memory resources. We evaluate the performance of different strategies in different situations and demonstrate that when the rewarded states change often, it can be advantageous for the model-based module to focus its memory resources not on exploiting the current reward, but on capturing general structure of the environment. This work provides a theoretical foundation for a functional dissociation between cortical and subcortical systems during learning: the cortex supports general structure learning, while subcortical circuits specialize in reward-based learning. We further detail how these hypotheses can be tested on experimental data.

q-bio.NC↗

arXiv2026

Reshaping Neural Representation via Associative, Presynaptic Short-Term Plasticity

Short-term synaptic plasticity (STP) is often regarded as a presynaptic filter of spikes, independent of postsynaptic activity. Recent experiments, however, indicate an associative STP that depends on pre- and postsynaptic coactivation. We develop a normative, information-theoretic theory of associative STP. Extending Fisher-information-based learning to Tsodyks-Markram synapses, we derive learning rules for baseline weight and release probability that maximize stimulus information under resource constraints. The rules split into a postsynaptic term tracking local firing and a presynaptic, phase-advanced term that selectively detects stimulus onset. For slowly varying inputs, this onset sensitivity favors anti-causal connectivity and enhances response offset during drive and reverse replay after drive removal in recurrent circuits. Linear-response analysis shows that STP yields frequency-dependent phase selectivity and that release-probability constraints tune temporal asymmetry. These results identify release-probability plasticity as a principled substrate for rapidly reconfigurable temporal coding.

q-bio.NC↗

arXiv2026

Beyond variance: sensitivity-based dimensions in brain networks underlie individual differences in cognitive ability

Explaining individual differences in cognitive abilities requires both identifying brain parameters that vary across individuals and understanding how brain networks are recruited for specific tasks. Typically, task performance relies on the integration and segregation of functional subnetworks, often captured by parameters like regional excitability and connectivity. Yet, the high dimensionality of these parameters hinders pinpointing their functional relevance. Here, we apply stiff-sloppy analysis to human brain data, revealing that certain subtle parameter combinations ("stiff dimensions") powerfully influence neural activity during task processing, whereas others ("sloppy dimensions") vary more extensively but exert minimal impact. Using a pairwise maximum entropy model of task fMRI, we show that even small deviations in stiff dimensions-derived through Fisher Information Matrix analysis-govern the dynamic interplay of segregation and integration between the default mode network (DMN) and a working memory network (WMN). Crucially, separating a 0-back task (vigilant attention) from a 2-back task (working memory updating) uncovers partially distinct stiff dimensions predicting performance in each condition, along with a global DMN-WMN segregation shared across both tasks. Altogether, stiff-sloppy analysis challenges the conventional focus on large parameter variability by highlighting these subtle yet functionally decisive parameter combinations.

q-bio.NC↗

arXiv2026

Modeling flexible behavior with remapping-based hippocampal sequence learning

Animals flexibly change their behavior depending on context. It is reported that the hippocampus is one of the most prominent regions for contextual behaviors, and its sequential activity shows context dependency. However, how such context-dependent sequential activity is established through reorganization of neuronal activity (remapping) is unclear. To better understand the formation of hippocampal activity and its contribution to context-dependent flexible behavior, we present a novel biologically plausible reinforcement learning model. In this model, Context selector promotes the formation of context-dependent sequential activity and allows for flexible switching of behavior in multiple contexts. This model reproduces a variety of findings from neural activity, optogenetic inactivation, human fMRI, and clinical research. Furthermore, our model predicts that imbalances in the ratio between sensory and contextual representations in Context selector account for schizophrenia (SZ) and autism spectrum disorder (ASD)-like behaviors.

q-bio.NC↗

arXiv2025

Diverse Neural Sequences in QIF Networks: An Analytically Tractable Framework for Synfire Chains and Hippocampal Replay

Sequential neural activity is fundamental to cognition, yet how diverse sequences are recalled under biological constraints remains a key question. Existing models often struggle to balance biophysical realism and analytical tractability. We address this problem by proposing a parsimonious network of Quadratic Integrate-and-Fire (QIF) neurons with sequences embedded via a temporally asymmetric Hebbian (TAH) rule. Our findings demonstrate that this single framework robustly reproduces a spectrum of sequential activities, including persistent synfire-like chains and transient, hippocampal replay-like bursts exhibiting intra-ripple frequency accommodation (IFA), all achieved without requiring specialized delay or adaptation mechanisms. Crucially, we derive exact low-dimensional firing-rate equations (FREs) that provide mechanistic insight, elucidating the bifurcation structure governing these distinct dynamical regimes and explaining their stability. The model also exhibits strong robustness to synaptic heterogeneity and memory pattern overlap. These results establish QIF networks with TAH connectivity as an analytically tractable and biologically plausible platform for investigating the emergence, stability, and diversity of sequential neural activity in the brain.

q-bio.NC↗

arXiv2024

A provable control of sensitivity of neural networks through a direct parameterization of the overall bi-Lipschitzness

While neural networks can enjoy an outstanding flexibility and exhibit unprecedented performance, the mechanism behind their behavior is still not well-understood. To tackle this fundamental challenge, researchers have tried to restrict and manipulate some of their properties in order to gain new insights and better control on them. Especially, throughout the past few years, the concept of \emph{bi-Lipschitzness} has been proved as a beneficial inductive bias in many areas. However, due to its complexity, the design and control of bi-Lipschitz architectures are falling behind, and a model that is precisely designed for bi-Lipschitzness realizing a direct and simple control of the constants along with solid theoretical analysis is lacking. In this work, we investigate and propose a novel framework for bi-Lipschitzness that can achieve such a clear and tight control based on convex neural networks and the Legendre-Fenchel duality. Its desirable properties are illustrated with concrete experiments. We also apply this framework to uncertainty estimation and monotone problem settings to illustrate its broad range of applications.

cs.LG↗

arXiv2023

Causal Graph in Language Model Rediscovers Cortical Hierarchy in Human Narrative Processing

Understanding how humans process natural language has long been a vital research direction. The field of natural language processing (NLP) has recently experienced a surge in the development of powerful language models. These models have proven to be invaluable tools for studying another complex system known to process human language: the brain. Previous studies have demonstrated that the features of language models can be mapped to fMRI brain activity. This raises the question: is there a commonality between information processing in language models and the human brain? To estimate information flow patterns in a language model, we examined the causal relationships between different layers. Drawing inspiration from the workspace framework for consciousness, we hypothesized that features integrating more information would more accurately predict higher hierarchical brain activity. To validate this hypothesis, we classified language model features into two categories based on causal network measures: 'low in-degree' and 'high in-degree'. We subsequently compared the brain prediction accuracy maps for these two groups. Our results reveal that the difference in prediction accuracy follows a hierarchical pattern, consistent with the cortical hierarchy map revealed by activity time constants. This finding suggests a parallel between how language models and the human brain process linguistic information.

cs.CL↗

arXiv2023

Computational role of sleep in memory reorganization

Sleep is considered to play an essential role in memory reorganization. Despite its importance, classical theoretical models did not focus on some sleep characteristics. Here, we review recent theoretical approaches investigating their roles in learning and discuss the possibility that non-rapid eye movement (NREM) sleep selectively consolidates memory, and rapid eye movement (REM) sleep reorganizes the representations of memories. We first review the possibility that slow waves during NREM sleep contribute to memory selection by using sequential firing patterns and the existence of up and down states. Second, we discuss the role of dreaming during REM sleep in developing neuronal representations. We finally discuss how to develop these points further, emphasizing the connections to experimental neuroscience and machine learning.

q-bio.NC↗

arXiv2023

A Hopfield-like model with complementary encodings of memories

We present a Hopfield-like autoassociative network for memories representing examples of concepts. Each memory is encoded by two activity patterns with complementary properties. The first is dense and correlated across examples within concepts, and the second is sparse and exhibits no correlation among examples. The network stores each memory as a linear combination of its encodings. During retrieval, the network recovers sparse or dense patterns with a high or low activity threshold, respectively. As more memories are stored, the dense representation at low threshold shifts from examples to concepts, which are learned from accumulating common example features. Meanwhile, the sparse representation at high threshold maintains distinctions between examples due to the high capacity of sparse, decorrelated patterns. Thus, a single network can retrieve memories at both example and concept scales and perform heteroassociation between them. We obtain our results by deriving macroscopic mean-field equations that yield capacity formulas for sparse examples, dense examples, and dense concepts. We also perform network simulations that verify our theoretical results and explicitly demonstrate the capabilities of the network.

q-bio.NC↗

arXiv2022

Spontaneous Emerging Preference in Two-tower Language Model

The ever-growing size of the foundation language model has brought significant performance gains in various types of downstream tasks. With the existence of side-effects brought about by the large size of the foundation language model such as deployment cost, availability issues, and environmental cost, there is some interest in exploring other possible directions, such as a divide-and-conquer scheme. In this paper, we are asking a basic question: are language processes naturally dividable? We study this problem with a simple two-tower language model setting, where two language models with identical configurations are trained side-by-side cooperatively. With this setting, we discover the spontaneous emerging preference phenomenon, where some of the tokens are consistently better predicted by one tower while others by another tower. This phenomenon is qualitatively stable, regardless of model configuration and type, suggesting this as an intrinsic property of natural language. This study suggests that interesting properties of natural language are still waiting to be discovered, which may aid the future development of natural language processing techniques.

cs.CL↗

arXiv2022

Progressive Interpretation Synthesis: Interpreting Task Solving by Quantifying Previously Used and Unused Information

A deep neural network is a good task solver, but it is difficult to make sense of its operation. People have different ideas about how to form the interpretation about its operation. We look at this problem from a new perspective where the interpretation of task solving is synthesized by quantifying how much and what previously unused information is exploited in addition to the information used to solve previous tasks. First, after learning several tasks, the network acquires several information partitions related to each task. We propose that the network, then, learns the minimal information partition that supplements previously learned information partitions to more accurately represent the input. This extra partition is associated with un-conceptualized information that has not been used in previous tasks. We manage to identify what un-conceptualized information is used and quantify the amount. To interpret how the network solves a new task, we quantify as meta-information how much information from each partition is extracted. We implement this framework with the variational information bottleneck technique. We test the framework with the MNIST and the CLEVR dataset. The framework is shown to be able to compose information partitions and synthesize experience-dependent interpretation in the form of meta-information. This system progressively improves the resolution of interpretation upon new experience by converting a part of the un-conceptualized information partition to a task-related partition. It can also provide a visual interpretation by imaging what is the part of previously un-conceptualized information that is needed to solve a new task.

cs.AI↗

arXiv2022

An economic decision-making model of anticipated surprise with dynamic expectation

When making decisions under risk, people often exhibit behaviors that classical economic theories cannot explain. Newer models that attempt to account for these irrational behaviors often lack neuroscience bases and require the introduction of subjective and problem-specific constructs. Here, we present a decision-making model inspired by the prediction error signals and introspective neuronal replay reported in the brain. In the model, decisions are chosen based on anticipated surprise, defined by a nonlinear average of the differences between individual outcomes and a reference point. The reference point is determined by the expected value of the possible outcomes, which can dynamically change during the mental simulation of decision-making problems involving sequential stages. Our model elucidates the contribution of each stage to the appeal of available options in a decision-making problem. This allows us to explain several economic paradoxes and gambling behaviors. Our work could help bridge the gap between decision-making theories in economics and neurosciences.

econ.TH↗

arXiv2022

Dimensionality reduction to maximize prediction generalization capability

Generalization of time series prediction remains an important open issue in machine learning, wherein earlier methods have either large generalization error or local minima. We develop an analytically solvable, unsupervised learning scheme that extracts the most informative components for predicting future inputs, termed predictive principal component analysis (PredPCA). Our scheme can effectively remove unpredictable noise and minimize test prediction error through convex optimization. Mathematical analyses demonstrate that, provided with sufficient training samples and sufficiently high-dimensional observations, PredPCA can asymptotically identify hidden states, system parameters, and dimensionalities of canonical nonlinear generative processes, with a global convergence guarantee. We demonstrate the performance of PredPCA using sequential visual inputs comprising hand-digits, rotating 3D objects, and natural scenes. It reliably estimates distinct hidden states and predicts future outcomes of previously unseen test input data, based exclusively on noisy observations. The simple architecture and low computational cost of PredPCA are highly desirable for neuromorphic hardware.

stat.ML↗

arXiv2021

Learning poly-synaptic paths with traveling waves

Traveling waves are commonly observed across the brain. While previous studies have suggested the role of traveling waves in learning, the mechanism is still unclear. We adopted a computational approach to investigate the effect of traveling waves on synaptic plasticity. Our results indicate that traveling waves facilitate the learning of poly-synaptic network-paths when combined with a reward-dependent local synaptic plasticity rule. We also demonstrate that traveling waves expedite finding the shortest paths and learning nonlinear input/output-mapping, such as exclusive or (XOR) function.

q-bio.NC↗

arXiv2020

On the achievability of blind source separation for high-dimensional nonlinear source mixtures

For many years, a combination of principal component analysis (PCA) and independent component analysis (ICA) has been used for blind source separation (BSS). However, it remains unclear why these linear methods work well with real-world data that involve nonlinear source mixtures. This work theoretically validates that a cascade of linear PCA and ICA can solve a nonlinear BSS problem accurately -- when the sensory inputs are generated from hidden sources via nonlinear mappings with sufficient dimensionality. Our proposed theorem, termed the asymptotic linearization theorem, theoretically guarantees that applying linear PCA to the inputs can reliably extract a subspace spanned by the linear projections from every hidden source as the major components -- and thus projecting the inputs onto their major eigenspace can effectively recover a linear transformation of the hidden sources. Then, subsequent application of linear ICA can separate all the true independent hidden sources accurately. Zero-element-wise-error nonlinear BSS is asymptotically attained when the source dimensionality is large and the input dimensionality is sufficiently larger than the source dimensionality. Our proposed theorem is validated analytically and numerically. Moreover, the same computation can be performed by using Hebbian-like plasticity rules, implying the biological plausibility of this nonlinear BSS strategy. Our results highlight the utility of linear PCA and ICA for accurately and reliably recovering nonlinearly mixed sources -- and further suggest the importance of employing sensors with sufficient dimensionality to identify true hidden sources of real-world data.

stat.ML↗

arXiv2020

Edge of chaos and avalanches in neural networks with heavy-tailed synaptic weight distribution

We propose an analytically tractable neural connectivity model with power-law distributed synaptic strengths. When threshold neurons with biologically plausible number of incoming connections are considered, our model features a continuous transition to chaos and can reproduce biologically relevant low activity levels and scale-free avalanches, i.e. bursts of activity with power-law distributions of sizes and lifetimes. In contrast, the Gaussian counterpart exhibits a discontinuous transition to chaos and thus cannot be poised near the edge of chaos. We validate our predictions in simulations of networks of binary as well as leaky integrate-and-fire neurons. Our results suggest that heavy-tailed synaptic distribution may form a weakly informative sparse-connectivity prior that can be useful in biological and artificial adaptive systems.

physics.bio-ph↗