arXiv subjects

Tankut Can

Publications and source records attributed to Tankut Can.

At least 19 recordsLinked to original sources

arXiv2026

Emergent retokenization symmetry in large language models: phenomenology and applications

Tokenization introduces representational redundancy: under a fixed token vocabulary, every byte string admits many valid token encodings, or segmentations, that decode to the same surface string. However, given a prompt, most language model tokenizers break this representational symmetry by returning a canonical segmentation. Training only on canonical segmentations should influence inference behavior, and there is little reason to expect models to respect segmentation symmetry on downstream tasks. We find that this symmetry partially emerges during training. Here, we probe this emergent symmetry through experiments testing token compositional understanding, representation diversity, and task focused benchmark performance. We primarily use \textbf{retokenization} -- replacing a prompt's canonical tokenization with an alternative segmentation while preserving its bytes exactly. Relative to other prompt perturbations, retokenization is unusually clean because it isolates segmentation effects without changing syntax, semantics or surface form. We use retokenization to study sensitivity and robustness to semantically identical input representations across pretraining and post-training. Moreover, this partial retokenization symmetry suggests a distinct inference-time sampling axis. While temperature sampling generates diverse outputs from the model using its next-token probability distribution, retokenization generates diversity from the model's internal computations through semantically equivalent input representations. We find that while this retokenization sampling strategy can hurt performance on easy problems, it can also recover solutions that conventional sampling does not find. Overall, our work presents retokenization as a simple yet powerful probe of large language models, shedding light on compositional understanding and prompt sensitivity, and offering a novel sampling strategy.

cs.CL↗

arXiv2026

Semantic Chunking and the Entropy of Natural Language

The entropy rate of printed English is famously estimated to be about one bit per character, a benchmark that modern large language models (LLMs) have only recently approached. This entropy rate implies that English contains nearly 80 percent redundancy relative to the five bits per character expected for random text. We introduce a statistical model that attempts to capture the intricate multi-scale structure of natural language, providing a first-principles account of this redundancy level. Our model describes a procedure of self-similarly segmenting text into semantically coherent chunks down to the single-word level. The semantic structure of the text can then be hierarchically decomposed, allowing for analytical treatment. Numerical experiments with modern LLMs and open datasets suggest that our model quantitatively captures the structure of real texts at different levels of the semantic hierarchy. The entropy rate predicted by our model agrees with the estimated entropy rate of printed English. Moreover, our theory further reveals that the entropy rate of natural language is not fixed but should increase systematically with the semantic complexity of corpora, which are captured by the only free parameter in our model.

cs.CL↗

arXiv2025

Statistical Mechanics of Semantic Compression

The basic problem of semantic compression is to minimize the length of a message while preserving its meaning. This differs from classical notions of compression in that the distortion is not measured directly at the level of bits, but rather in an abstract semantic space. In order to make this precise, we take inspiration from cognitive neuroscience and machine learning and model semantic space as a continuous Euclidean vector space. In such a space, stimuli like speech, images, or even ideas, are mapped to high-dimensional real vectors, and the location of these embeddings determines their meaning relative to other embeddings. This suggests that a natural metric for semantic similarity is just the Euclidean distance, which is what we use in this work. We map the optimization problem of determining the minimal-length, meaning-preserving message to a spin glass Hamiltonian and solve the resulting statistical mechanics problem using replica theory. We map out the replica symmetric phase diagram, identifying distinct phases of semantic compression: a first-order transition occurs between lossy and lossless compression, whereas a continuous crossover is seen from extractive to abstractive compression. We conclude by showing numerical simulations of compressions obtained by simulated annealing and greedy algorithms, and argue that while the problem of finding a meaning-preserving compression is computationally hard in the worst case, there exist efficient algorithms which achieve near optimal performance in the typical case.

cond-mat.dis-nn↗

arXiv2024

Random Tree Model of Meaningful Memory

Traditional studies of memory for meaningful narratives focus on specific stories and their semantic structures but do not address common quantitative features of recall across different narratives. We introduce a statistical ensemble of random trees to represent narratives as hierarchies of key points, where each node is a compressed representation of its descendant leaves, which are the original narrative segments. Recall is modeled as constrained by working memory capacity from this hierarchical structure. Our analytical solution aligns with observations from large-scale narrative recall experiments. Specifically, our model explains that (1) average recall length increases sublinearly with narrative length, and (2) individuals summarize increasingly longer narrative segments in each recall sentence. Additionally, the theory predicts that for sufficiently long narratives, a universal, scale-invariant limit emerges, where the fraction of a narrative summarized by a single recall sentence follows a distribution independent of narrative length.

cond-mat.stat-mech↗

arXiv2023

Large-scale study of human memory for meaningful narratives

The statistical study of human memory requires large-scale experiments, involving many stimuli conditions and test subjects. While this approach has proven to be quite fruitful for meaningless material such as random lists of words, naturalistic stimuli, like narratives, have until now resisted such a large-scale study, due to the quantity of manual labor required to design and analyze such experiments. In this work, we develop a pipeline that uses large language models (LLMs) both to design naturalistic narrative stimuli for large-scale recall and recognition memory experiments, as well as to analyze the results. We performed online memory experiments with a large number of participants and collected recognition and recall data for narratives of different sizes. We found that both recall and recognition performance scale linearly with narrative length; however, for longer narratives people tend to summarize the content rather than recalling precise details. To investigate the role of narrative comprehension in memory, we repeated these experiments using scrambled versions of the narratives. Although recall performance declined significantly, recognition remained largely unaffected. Recalls in this condition seem to follow the original narrative order rather than the actual scrambled presentation, pointing to a contextual reconstruction of the story in memory. Finally, using LLM text embeddings, we construct a simple measure for each clause based on semantic similarity to the whole narrative, that shows a strong correlation with recall probability. Overall, our work demonstrates the power of LLMs in accessing new regimes in the study of human memory, as well as suggesting novel psychologically informed benchmarks for LLM performance.

cs.CL↗

arXiv2023

Trainability, Expressivity and Interpretability in Gated Neural ODEs

Understanding how the dynamics in biological and artificial neural networks implement the computations required for a task is a salient open question in machine learning and neuroscience. In particular, computations requiring complex memory storage and retrieval pose a significant challenge for these networks to implement or learn. Recently, a family of models described by neural ordinary differential equations (nODEs) has emerged as powerful dynamical neural network models capable of capturing complex dynamics. Here, we extend nODEs by endowing them with adaptive timescales using gating interactions. We refer to these as gated neural ODEs (gnODEs). Using a task that requires memory of continuous quantities, we demonstrate the inductive bias of the gnODEs to learn (approximate) continuous attractors. We further show how reduced-dimensional gnODEs retain their modeling power while greatly improving interpretability, even allowing explicit visualization of the structure of learned attractors. We introduce a novel measure of expressivity which probes the capacity of a neural network to generate complex trajectories. Using this measure, we explore how the phase-space dimension of the nODEs and the complexity of the function modeling the flow field contribute to expressivity. We see that a more complex function for modeling the flow field allows a lower-dimensional nODE to capture a given target dynamics. Finally, we demonstrate the benefit of gating in nODEs on several real-world tasks.

cs.LG↗

arXiv2022

Flatter, faster: scaling momentum for optimal speedup of SGD

Commonly used optimization algorithms often show a trade-off between good generalization and fast training times. For instance, stochastic gradient descent (SGD) tends to have good generalization; however, adaptive gradient methods have superior training times. Momentum can help accelerate training with SGD, but so far there has been no principled way to select the momentum hyperparameter. Here we study training dynamics arising from the interplay between SGD with label noise and momentum in the training of overparametrized neural networks. We find that scaling the momentum hyperparameter $1-\beta$ with the learning rate to the power of $2/3$ maximally accelerates training, without sacrificing generalization. To analytically derive this result we develop an architecture-independent framework, where the main assumption is the existence of a degenerate manifold of global minimizers, as is natural in overparametrized models. Training dynamics display the emergence of two characteristic timescales that are well-separated for generic values of the hyperparameters. The maximum acceleration of training is reached when these two timescales meet, which in turn determines the scaling limit we propose. We confirm our scaling rule for synthetic regression problems (matrix sensing and teacher-student paradigm) and classification for realistic datasets (ResNet-18 on CIFAR10, 6-layer MLP on FashionMNIST), suggesting the robustness of our scaling rule to variations in architectures and datasets.

cs.LG↗

arXiv2021

Emergence of robust memory manifolds

The ability to store continuous variables in the state of a biological system (e.g. a neural network) is critical for many behaviours. Most models for implementing such a memory manifold require hand-crafted symmetries in the interactions or precise fine-tuning of parameters. We present a general principle that we refer to as {\it frozen stabilisation} (FS), which allows a family of neural networks to self-organise to a critical state exhibiting multiple memory manifolds without parameter fine-tuning or symmetries. Memory manifolds arising from FS exhibit a wide range of emergent relaxational timescales and can be used as general purpose integrators for inputs aligned with the manifold. Moreover, FS allows robust memory manifolds in small networks, and this is relevant to debates of implementing continuous attractors with a small number of neurons in light of recent experimental discoveries.

q-bio.NC↗

arXiv2020

Theory of gating in recurrent neural networks

Recurrent neural networks (RNNs) are powerful dynamical models, widely used in machine learning (ML) and neuroscience. Prior theoretical work has focused on RNNs with additive interactions. However, gating - i.e. multiplicative - interactions are ubiquitous in real neurons and also the central feature of the best-performing RNNs in ML. Here, we show that gating offers flexible control of two salient features of the collective dynamics: i) timescales and ii) dimensionality. The gate controlling timescales leads to a novel, marginally stable state, where the network functions as a flexible integrator. Unlike previous approaches, gating permits this important function without parameter fine-tuning or special symmetries. Gates also provide a flexible, context-dependent mechanism to reset the memory trace, thus complementing the memory function. The gate modulating the dimensionality can induce a novel, discontinuous chaotic transition, where inputs push a stable system to strong chaotic activity, in contrast to the typically stabilizing effect of inputs. At this transition, unlike additive RNNs, the proliferation of critical points (topological complexity) is decoupled from the appearance of chaotic dynamics (dynamical complexity). The rich dynamics are summarized in phase diagrams, thus providing a map for principled parameter initialization choices to ML practitioners.

cond-mat.dis-nn↗

arXiv2020

Spectral transitions and universal steady states in random Kraus maps and circuits

The study of dissipation and decoherence in generic open quantum systems recently led to the investigation of spectral and steady-state properties of random Lindbladian dynamics. A natural question is then how realistic and universal those properties are. Here, we address these issues by considering a different description of dissipative quantum systems, namely, the discrete-time Kraus map representation of completely positive quantum dynamics. Through random matrix theory (RMT) techniques and numerical exact diagonalization, we study random Kraus maps, allowing for a varying dissipation strength, and their local circuit counterpart. We find the spectrum of the random Kraus map to be either an annulus or a disk inside the unit circle in the complex plane, with a transition between the two cases taking place at a critical value of dissipation strength. The eigenvalue distribution and the spectral transition are well described by a simplified RMT model that we can solve exactly in the thermodynamic limit, by means of non-Hermitian RMT and quaternionic free probability. The steady state, on the contrary, is not affected by the spectral transition. It has, however, a perturbative crossover regime at small dissipation, inside which the steady state is characterized by uncorrelated eigenvalues. At large dissipation (or for any dissipation for a large-enough system), the steady state is well described by a random Wishart matrix. The steady-state properties thus coincide with those already observed for random Lindbladian dynamics, indicating their universality. Quite remarkably, the statistical properties of the local Kraus circuit are qualitatively the same as those of the nonlocal Kraus map, indicating that the latter, which is more tractable, already captures the realistic and universal physical properties of generic open quantum systems.

cond-mat.stat-mech↗

arXiv2020

Gating creates slow modes and controls phase-space complexity in GRUs and LSTMs

Recurrent neural networks (RNNs) are powerful dynamical models for data with complex temporal structure. However, training RNNs has traditionally proved challenging due to exploding or vanishing of gradients. RNN models such as LSTMs and GRUs (and their variants) significantly mitigate these issues associated with training by introducing various types of gating units into the architecture. While these gates empirically improve performance, how the addition of gates influences the dynamics and trainability of GRUs and LSTMs is not well understood. Here, we take the perspective of studying randomly initialized LSTMs and GRUs as dynamical systems, and ask how the salient dynamical properties are shaped by the gates. We leverage tools from random matrix theory and mean-field theory to study the state-to-state Jacobians of GRUs and LSTMs. We show that the update gate in the GRU and the forget gate in the LSTM can lead to an accumulation of slow modes in the dynamics. Moreover, the GRU update gate can poise the system at a marginally stable point. The reset gate in the GRU and the output and input gates in the LSTM control the spectral radius of the Jacobian, and the GRU reset gate also modulates the complexity of the landscape of fixed-points. Furthermore, for the GRU we obtain a phase diagram describing the statistical properties of fixed-points. We also provide a preliminary comparison of training performance to the various dynamical regimes realized by varying hyperparameters. Looking to the future, we have introduced a powerful set of techniques which can be adapted to a broad class of RNNs, to study the influence of various architectural choices on dynamics, and potentially motivate the principled discovery of novel architectures.

cs.LG↗

arXiv2019

Hydrodynamics of two-dimensional compressible fluid with broken parity: variational principle and free surface dynamics in the absence of dissipation

We consider an isotropic compressible non-dissipative fluid with broken parity subject to free surface boundary conditions in two spatial dimensions. The hydrodynamic equations describing the bulk dynamics of the fluid as well as the free surface boundary conditions depend explicitly on the parity breaking non-dissipative odd viscosity term. We construct a variational principle in the form of an effective action which gives both bulk hydrodynamic equations and free surface boundary conditions. The free surface boundary conditions require an additional boundary term in the action which resembles a $1+1D$ chiral boson field coupled to the background geometry. We solve the linearized hydrodynamic equations for the deep water case and derive the dispersion of chiral surface waves. We show that in the long wavelength limit the flow profile exhibits an oscillating vortical boundary layer near the free surface. The thickness of the layer is controlled by the length scale given by the ratio of odd viscosity to the sound velocity $\delta \sim \nu_o/c_s$. In the incompressible limit, $c_s\to \infty$ the vortical boundary layer becomes singular with the vorticity within the layer diverging as $\omega \sim c_s$. The boundary layer is formed by odd viscosity coupling the divergence of velocity $\boldsymbol\nabla \cdot \boldsymbol{v}$ to vorticity $\boldsymbol\nabla \times \boldsymbol{v}$. It results in non-trivial chiral free surface dynamics even in the absence of external forces. The structure of the odd viscosity induced boundary layer is very different from the conventional free surface boundary layer associated with dissipative shear viscosity.

physics.flu-dyn↗

arXiv2019

Random Lindblad Dynamics

We study the mixing behavior of random Lindblad generators with no symmetries, using the dynamical map or propagator of the dissipative evolution. In particular, we determine the long-time behavior of a dissipative form factor, which is the trace of the propagator, and use this as a diagnostic for the existence or absence of a spectral gap in the distribution of eigenvalues of the Lindblad generator. We find that simple generators with a single jump operator are slowly mixing, and relax algebraically in time, due to the closing of the spectral gap in the thermodynamic limit. Introducing additional jump operators or a Hamiltonian opens up a spectral gap which remains finite in the thermodynamic limit, leading to exponential relaxation and thus rapid mixing. We use the method of moments and introduce a novel diagrammatic expansion to determine exactly the form factor to leading order in Hilbert space dimension $N$. We also present numerical support for our main results.

quant-ph↗

arXiv2019

Spectral gaps and mid-gap states in random quantum master equations

We discuss the decay rates of chaotic quantum systems coupled to noise. We model both the Hamiltonian and the system-noise coupling by random $N \times N$ Hermitian matrices, and study the spectral properties of the resulting Lindblad superoperator. We consider various random-matrix ensembles, and find that for all of them the asymptotic decay rate remains nonzero in the thermodynamic limit, i.e., the spectrum of the superoperator is gapped as $N \rightarrow \infty$. A sharp spectral transition takes place as the dissipation strength is increased: for weak dissipation, the non-zero eigenvalues of the master equation form a continuum; whereas for strong dissipation, the asymptotic decay rate is an \emph{isolated eigenvalue}, i.e., a `mid-gap state' that is sharply separated from the continuous spectrum of the master equation. For finite $N$, the probability of finding a very small gap vanishes algebraically with a scaling exponent that is extensive in system size, and depends only on the symmetry class of the random matrices and the number of independent decay channels. We comment on experimental implications of our results.

quant-ph↗

arXiv2018

Measuring Electromagnetic and Gravitational Responses of Photonic Landau Levels

The topology of an object describes global properties that are insensitive to local perturbations. Classic examples include string knots and the genus (number of handles) of a surface: no manipulation of a closed string short of cutting it changes its "knottedness"; and no deformation of a closed surface, short of puncturing it, changes how many handles it has. Topology has recently become an intense focus of condensed matter physics, where it arises in the context of the quantum Hall effect [1] and topological insulators [2]. In each case, topology is defined through invariants of the material's bulk [3-5], but experimentally measured through chiral/helical properties of the material's edges. In this work we measure topological invariants of a quantum Hall material through local response of the bulk: treating the material as a many-port circulator enables direct measurement of the Chern number as the spatial winding of the circulator phase; excess density accumulation near spatial curvature quantifies the curvature-analog of charge known as mean orbital spin, while the moment of inertia of this excess density reflects the chiral central charge. We observe that the topological invariants converge to their global values when probed over a few magnetic lengths lB, consistent with intuition that the bulk/edge distinction exists only for samples larger than a few lB. By performing these experiments in photonic Landau levels of a twisted resonator [6], we apply quantum-optics tools to topological matter. Combined with developments in Rydberg-mediated interactions between resonator photons [7], this work augurs an era of precision characterization of topological matter in strongly correlated fluids of light.

cond-mat.quant-gas↗

arXiv2018

Odd surface waves in two-dimensional incompressible fluids

We consider free surface dynamics of a two-dimensional incompressible fluid with odd viscosity. The odd viscosity is a peculiar part of the viscosity tensor which does not result in dissipation and is allowed when parity symmetry is broken. For the case of incompressible fluids, the odd viscosity manifests itself through the free surface (no stress) boundary conditions. We first find the free surface wave solutions of hydrodynamics in the linear approximation and study the dispersion of such waves. As expected, the surface waves are chiral and even exist in the absence of gravity and vanishing shear viscosity. In this limit, we derive effective nonlinear Hamiltonian equations for the surface dynamics, generalizing the linear solutions to the weakly nonlinear case. Within the small surface angle approximation, the equation of motion leads to a new class of non-linear chiral dynamics governed by what we dub the {\it chiral} Burgers equation. The chiral Burgers equation is identical to the complex Burgers equation with imaginary viscosity and an additional analyticity requirement that enforces chirality. We present several exact solutions of the chiral Burgers equation. For generic multiple pole initial conditions, the system evolves to the formation of singularities in a finite time similar to the case of an ideal fluid without odd viscosity. We also obtain a periodic solution to the chiral Burgers corresponding to the non-linear generalization of small amplitude linear waves.

physics.flu-dyn↗

arXiv2016

Particle-Hole Duality in the Lowest Landau Level

We derive a number of exact relations between response functions of holomorphic, chiral fractional quantum Hall states and their particle-hole (PH) conjugates. These exact relations allow one to calculate the Hall conductivity, Hall viscosity, various Berry phases, and the static structure factor of PH-conjugate states from the corresponding properties of the original states. These relations establish a precise duality between chiral quantum Hall states and their PH-conjugates. The key ingredient in the proof of the relations is a generalization of Girvin's construction of PH-conjugate states to inhomogeneous magnetic field and curvature. Finally, we make several non-trivial checks of the relations, including for the Jain states and their PH-conjugates.

cond-mat.str-el↗

arXiv2016

Central charge from adiabatic transport of cusp singularities in the quantum Hall effect

We study quantum Hall (QH) states on a punctured Riemann sphere. We compute the Berry curvature under adiabatic motion in the moduli space in the large N limit. The Berry curvature is shown to be finite in the large N limit and controlled by the conformal dimension of the cusp singularity, a local property of the mean density. Utilizing exact sum rules obtained from a Ward identity, we show that for the Laughlin wave function, the dimension of a cusp singularity is given by the central charge, a robust geometric response coefficient in the QHE. Thus, adiabatic transport of curvature singularities can be used to determine the central charge of QH states. We also consider the effects of threaded fluxes and spin-deformed wave functions. Finally, we give a closed expression for all moments of the mean density in the integer QH state on a punctured disk.

cond-mat.str-el↗