arXiv subjects

Toshihiro Ota

Publications and source records attributed to Toshihiro Ota.

11 recordsLinked to original sources

arXiv2026

Phases in a class of associative memories via hidden neurons

Associative memory in the Hopfield network is attractor dynamics in a disordered many-body system, and higher-order and exponential extensions turn its retrieval update into softmax attention. The polynomial and exponential regimes have been analyzed by different methods, with no common architecture in which to ask what fixes the storage scale. In this paper we study the bipartite architecture of Krotov and Hopfield, which we call the class $H$, whose model is fixed by a Lagrangian for each layer, taking the hidden neurons as the order parameter of retrieval. At polynomial load the replica method yields the replica-symmetric phase diagrams and closed-form capacities, and the crosstalk moment is common to Ising and spherical visible neurons, so their differences come from the visible entropy. With a softmax hidden layer the load is exponential, and a copy representation maps the thermodynamics onto random-energy-model counting, with paramagnetic, condensed, and frozen phases. Heating destabilizes retrieval by quantized reassignments of attention, and typical Gaussian patterns remain metastable at every load. The regimes differ in their crosstalk statistics, central-limit at polynomial load and large-deviation at exponential load, and the class $H$ splits retrieval into two roles, the visible Lagrangian fixing stability and the hidden one the storage scale, two axes that may also guide the design of new Lagrangians.

cs.LG↗

arXiv2026

Comments on holographic entanglements in cutoff AdS

We study two-interval holographic entanglement entropy and entanglement wedge cross section in cutoff AdS. In particular, we investigate phase transitions of them. For two-interval entanglement entropy, the transition point monotonically decreases with a deformation parameter, which means that by the TT deformation the degrees of freedom in subsystems are decreasing. This implies that the effect of the TT deformation can be regarded as the rescaling of the energy scale. We also study entanglement wedge cross section in cutoff AdS, and our result implies that for the entanglement of purification in the TT deformed CFTs phase transition could occur even for fixed subsystems.

hep-th↗

arXiv2026

Collective Kernel EFT for Pre-activation ResNets

In finite-width deep neural networks, the empirical kernel $G$ evolves stochastically across layers. We develop a collective kernel effective field theory (EFT) for pre-activation ResNets based on a $G$-only closure hierarchy and diagnose its finite validity window. Exploiting the exact conditional Gaussianity of residual increments, we derive an exact stochastic recursion for $G$. Applying Gaussian approximations systematically yields a continuous-depth ODE system for the mean kernel $K_0$, the kernel covariance $V_4$, and the $1/n$ mean correction $K_{1,\mathrm{EFT}}$, which emerges diagrammatically as a one-loop tadpole correction. Numerically, $K_0$ remains accurate at all depths. However, the $V_4$ equation residual accumulates to an $O(1)$ error at finite time, primarily driven by approximation errors in the $G$-only transport term. Furthermore, $K_{1,\mathrm{EFT}}$ fails due to the breakdown of the source closure, which exhibits a systematic mismatch even at initialization. These findings highlight the limitations of $G$-only state-space reduction and suggest extending the state space to incorporate the sigma-kernel.

cs.LG↗

arXiv2025

Hamiltonian of polymatrix zero-sum games

The understanding of a dynamical system's properties can be significantly advanced by establishing it as a Hamiltonian system and then systematically exploring its inherent symmetries. By formulating agents' strategies and cumulative payoffs as canonically conjugate variables, we identify the Hamiltonian function that generates the dynamics of poly-matrix zero-sum games. We reveal the symmetries of our Hamiltonian and derive the associated conserved quantities, showing how the conservation of probability and the invariance of the Fenchel coupling are intrinsically encoded within the system. Furthermore, we propose the dissipation FTRL (DFTRL) dynamics by introducing a perturbation that dissipates the Fenchel coupling, proving convergence to the Nash equilibrium and linking DFTRL to last-iterate convergent algorithms. Our results highlight the potential of Hamiltonian dynamics in uncovering the structural properties of learning dynamics in games, and pave the way for broader applications of Hamiltonian dynamics in game theory and machine learning.

cs.GT↗

arXiv2024

Hierarchical Associative Memory, Parallelized MLP-Mixer, and Symmetry Breaking

Transformers have established themselves as the leading neural network model in natural language processing and are increasingly foundational in various domains. In vision, the MLP-Mixer model has demonstrated competitive performance, suggesting that attention mechanisms might not be indispensable. Inspired by this, recent research has explored replacing attention modules with other mechanisms, including those described by MetaFormers. However, the theoretical framework for these models remains underdeveloped. This paper proposes a novel perspective by integrating Krotov's hierarchical associative memory with MetaFormers, enabling a comprehensive representation of the entire Transformer block, encompassing token-/channel-mixing modules, layer normalization, and skip connections, as a single Hopfield network. This approach yields a parallelized MLP-Mixer derived from a three-layer Hopfield network, which naturally incorporates symmetric token-/channel-mixing modules and layer normalization. Empirical studies reveal that symmetric interaction matrices in the model hinder performance in image recognition tasks. Introducing symmetry-breaking effects transitions the performance of the symmetric parallelized MLP-Mixer to that of the vanilla MLP-Mixer. This indicates that during standard training, weight matrices of the vanilla MLP-Mixer spontaneously acquire a symmetry-breaking configuration, enhancing their effectiveness. These findings offer insights into the intrinsic properties of Transformers and MLP-Mixers and their theoretical underpinnings, providing a robust framework for future model design and optimization.

cs.LG↗

arXiv2024

iMixer: hierarchical Hopfield network implies an invertible, implicit and iterative MLP-Mixer

In the last few years, the success of Transformers in computer vision has stimulated the discovery of many alternative models that compete with Transformers, such as the MLP-Mixer. Despite their weak inductive bias, these models have achieved performance comparable to well-studied convolutional neural networks. Recent studies on modern Hopfield networks suggest the correspondence between certain energy-based associative memory models and Transformers or MLP-Mixer, and shed some light on the theoretical background of the Transformer-type architectures design. In this paper, we generalize the correspondence to the recently introduced hierarchical Hopfield network, and find iMixer, a novel generalization of MLP-Mixer model. Unlike ordinary feedforward neural networks, iMixer involves MLP layers that propagate forward from the output side to the input side. We characterize the module as an example of invertible, implicit, and iterative mixing module. We evaluate the model performance with various datasets on image classification tasks, and find that iMixer, despite its unique architecture, exhibits stable learning capabilities and achieves performance comparable to or better than the baseline vanilla MLP-Mixer. The results imply that the correspondence between the Hopfield networks and the Mixer models serves as a principle for understanding a broader class of Transformer-like architecture designs.

cs.LG↗

arXiv2024

Decision Mamba: Reinforcement Learning via Sequence Modeling with Selective State Spaces

Decision Transformer, a promising approach that applies Transformer architectures to reinforcement learning, relies on causal self-attention to model sequences of states, actions, and rewards. While this method has shown competitive results, this paper investigates the integration of the Mamba framework, known for its advanced capabilities in efficient and effective sequence modeling, into the Decision Transformer architecture, focusing on the potential performance enhancements in sequential decision-making tasks. Our study systematically evaluates this integration by conducting a series of experiments across various decision-making environments, comparing the modified Decision Transformer, Decision Mamba, with its traditional counterpart. This work contributes to the advancement of sequential decision-making models, suggesting that the architecture and training methodology of neural networks can significantly impact their performance in complex tasks, and highlighting the potential of Mamba as a valuable tool for improving the efficacy of Transformer-based models in reinforcement learning scenarios.

cs.LG↗

arXiv2023

Attention in a family of Boltzmann machines emerging from modern Hopfield networks

Hopfield networks and Boltzmann machines (BMs) are fundamental energy-based neural network models. Recent studies on modern Hopfield networks have broaden the class of energy functions and led to a unified perspective on general Hopfield networks including an attention module. In this letter, we consider the BM counterparts of modern Hopfield networks using the associated energy functions, and study their salient properties from a trainability perspective. In particular, the energy function corresponding to the attention module naturally introduces a novel BM, which we refer to as the attentional BM (AttnBM). We verify that AttnBM has a tractable likelihood function and gradient for certain special cases and is easy to train. Moreover, we reveal the hidden connections between AttnBM and some single-layer models, namely the Gaussian--Bernoulli restricted BM and the denoising autoencoder with softmax units coming from denoising score matching. We also investigate BMs introduced by other energy functions and show that the energy function of dense associative memory models gives BMs belonging to Exponential Family Harmoniums.

cs.LG↗

arXiv2021

Wilson-'t Hooft lines as transfer matrices

We establish a correspondence between a class of Wilson-'t Hooft lines in four-dimensional $\mathcal{N} = 2$ supersymmetric gauge theories described by circular quivers and transfer matrices constructed from dynamical L-operators for trigonometric quantum integrable systems. We compute the vacuum expectation values of the Wilson-'t Hooft lines in a twisted product space $S^1 \times_ε\mathbb{R}^2 \times \mathbb{R}$ by supersymmetric localization and show that they are equal to the Wigner transforms of the transfer matrices. A variant of the AGT correspondence implies an identification of the transfer matrices with Verlinde operators in Toda theory, which we also verify. We explain how these field theory setups are related to four-dimensional Chern-Simons theory via embedding into string theory and dualities.

hep-th↗

arXiv2019

Chaos of QCD string from holography

It is challenging to quantify chaos of QCD, because non-perturbative QCD accompanies non-local observables. By using holography, we find that QCD strings at large $N_c$ and strong coupling limit exhibit chaos, and measure their Lyapunov exponent at zero temperature. A pair of a quark and an antiquark separated by $L_q$ in the large $N_c$ QCD is dual to a Nambu-Goto string hanging from the spatial boundary of the D4-soliton geometry. We numerically solve the motion of the string after putting a pulse force on its boundaries. The chaos is observed for the amplitude of the force larger than a certain lower bound. The bound increases as $L_q$ grows, and its dependence is well approximated by a hypothesis that the chaos originates in the endpoints of the QCD string.

hep-th↗

arXiv2018

Phase diagram of QCD chaos in linear sigma models and holography

Measuring chaos of QCD-like theories is a challenge for formulating a novel characterization of quantum gauge theories. We define a chaos phase diagram of QCD allowing us to locate chaos in the parameter space of energy of homogeneous meson condensates and the QCD parameters such as pion/quark mass. We draw the chaos phase diagrams obtained in two ways: first, by using a linear sigma model, varying parameters of the potential, and second, by using the D4/D6 holographic QCD, varying the number of colors $N_c$ and the 't Hooft coupling constant $λ$. A scaling law drastically simplifies our analyses, and we discovered that the chaos originates in the maximum of the potential, and larger $N_c$ or larger $λ$ diminishes the chaos.

hep-th↗