arXiv subjects

Vudtiwat Ngampruetikorn

Publications and source records attributed to Vudtiwat Ngampruetikorn.

17 recordsLinked to original sources

arXiv2026

Fisher information and the geometry of memorization in neural networks

A fruitful approach towards understanding generalization in machine learning involves characterizing how models separate generalizable structure from idiosyncratic or even corrupted features of training data. Fisher information has emerged as a diagnostic tool for this separation, leveraging loss geometry to distinguish generalization from memorization. Here we investigate this connection in vision models trained on CIFAR-10 with controlled noise that can be learned only through memorization. We show that projecting model weights onto the top Fisher eigenvectors decouples general task performance from noisy sample memorization, and we investigate how this decoupling presents differentially across layers. We show that the same phenomena are present in small transformers trained to solve a synthetic in-context learning task, and are reproduced in closed form in a simple, analytically solvable linear model. Our results reveal that Fisher information captures aspects of the geometric structure underlying how neural networks allocate capacity between generalization and memorization.

cs.LG↗

arXiv2026

Stochastic Resetting Accelerates Reinforcement Learning Beyond Random Search

Stochastic resetting -- intermittently returning a process to a fixed reference state -- has emerged as an effective mechanism for optimizing first-passage properties. Existing theory largely treats processes that search but do not learn: the searcher follows fixed dynamics, accumulating no knowledge between resets. Here we ask how stochastic resetting interacts with reinforcement learning, where the underlying dynamics adapt through experience. In tabular grid environments, we find that resetting can accelerate learning even when it does not reduce the search time of a diffusive agent. Our results reveal a distinct additional mechanism through which resetting speeds the propagation of reward information. We show that deterministic, sharp resetting accelerates learning more than the stochastic protocol but over a narrower range of reset rates. In a continuous-state task with neural-network-based value approximation, we demonstrate that resetting speeds up learning when exploration is hard and rewards are sparse. We argue further that, in the tabular tasks, resetting accelerates learning without altering the solution the agent ultimately reaches, unlike other techniques such as temporal discounting, which biases the optimal behavior. Our results establish stochastic resetting as a simple, tunable mechanism for accelerating learning by shaping how experience accumulates, extending a canonical phenomenon of statistical mechanics to adaptive systems.

cs.LG↗

arXiv2025

Understanding temperature tuning in energy-based models

Generative models of complex systems often require post-hoc parameter adjustments to produce useful outputs. For example, energy-based models for protein design are sampled at an artificially low ''temperature'' to generate novel, functional sequences. This temperature tuning is a common yet poorly understood heuristic used across machine learning contexts to control the trade-off between generative fidelity and diversity. Here, we develop an interpretable, physically motivated framework to explain this phenomenon. We demonstrate that in systems with a large ''energy gap'' - separating a small fraction of meaningful states from a vast space of unrealistic states - learning from sparse data causes models to systematically overestimate high-energy state probabilities, a bias that lowering the sampling temperature corrects. More generally, we characterize how the optimal sampling temperature depends on the interplay between data size and the system's underlying energy landscape. Crucially, our results show that lowering the sampling temperature is not always desirable; we identify the conditions where \emph{raising} it results in better generative performance. Our framework thus casts post-hoc temperature tuning as a diagnostic tool that reveals properties of the true data distribution and the limits of the learned model.

q-bio.QM↗

arXiv2025

Data coarse graining can improve model performance

Lossy data transformations by definition lose information. Yet, in modern machine learning, methods like data pruning and lossy data augmentation can help improve generalization performance. We study this paradox using a solvable model of high-dimensional, ridge-regularized linear regression under 'data coarse graining.' Inspired by the renormalization group in statistical physics, we analyze coarse-graining schemes that systematically discard features based on their relevance to the learning task. Our results reveal a nonmonotonic dependence of the prediction risk on the degree of coarse graining. A 'high-pass' scheme--which filters out less relevant, lower-signal features--can help models generalize better. By contrast, a 'low-pass' scheme that integrates out more relevant, higher-signal features is purely detrimental. Crucially, using optimal regularization, we demonstrate that this nonmonotonicity is a distinct effect of data coarse graining and not an artifact of double descent. Our framework offers a clear, analytical explanation for why careful data augmentation works: it strips away less relevant degrees of freedom and isolates more predictive signals. Our results highlight a complex, nonmonotonic risk landscape shaped by the structure of the data, and illustrate how ideas from statistical physics provide a principled lens for understanding modern machine learning phenomena.

cond-mat.stat-mech↗

arXiv2025

When can in-context learning generalize out of task distribution?

In-context learning (ICL) is a remarkable capability of pretrained transformers that allows models to generalize to unseen tasks after seeing only a few examples. We investigate empirically the conditions necessary on the pretraining distribution for ICL to emerge and generalize \emph{out-of-distribution}. Previous work has focused on the number of distinct tasks necessary in the pretraining dataset. Here, we use a different notion of task diversity to study the emergence of ICL in transformers trained on linear functions. We find that as task diversity increases, transformers undergo a transition from a specialized solution, which exhibits ICL only within the pretraining task distribution, to a solution which generalizes out of distribution to the entire task space. We also investigate the nature of the solutions learned by the transformer on both sides of the transition, and observe similar transitions in nonlinear regression problems. We construct a phase diagram to characterize how our concept of task diversity interacts with the number of pretraining tasks. In addition, we explore how factors such as the depth of the model and the dimensionality of the regression problem influence the transition.

cs.LG↗

arXiv2025

Generalization vs. Specialization under Concept Shift

Machine learning models are often brittle under distribution shift, i.e., when data distributions at test time differ from those during training. Understanding this failure mode is central to identifying and mitigating safety risks of mass adoption of machine learning. Here we analyze ridge regression under concept shift -- a form of distribution shift in which the input-label relationship changes at test time. We derive an exact expression for prediction risk in the thermodynamic limit. Our results reveal nontrivial effects of concept shift on generalization performance, including a phase transition between weak and strong concept shift regimes and nonmonotonic data dependence of test performance even when double descent is absent. Our theoretical results are in good agreement with experiments based on transformers pretrained to solve linear regression; under concept shift, too long context length can be detrimental to generalization performance of next token prediction. Finally, our experiments on MNIST and FashionMNIST suggest that this intriguing behavior is present also in classification problems.

stat.ML↗

arXiv2024

Anomalous Hall Effects in Chiral Superconductors

We report theoretical results for the electronic contribution to thermal and electrical transport for chiral superconductors belonging to even or odd-parity E$_1$ and E$_2$ representations of the tetragonal and hexagonal point groups. Chiral superconductors exhibit novel properties that depend on the topology of the order parameter and Fermi surface, and -- as we highlight -- the structure of the impurity potential. An anomalous thermal Hall effect is predicted and shown to be sensitive to the winding number, $ν$, of the chiral order parameter via Andreev scattering that transfers angular momentum from the chiral condensate to excitations that scatter off the random potential. For heat transport in a chiral superconductor with isotropic impurity scattering, i.e., point-like impurities, a transverse heat current is obtained for $ν=\pm 1$, but vanishes for $|ν|>1$. This is not a universal result. For finite-size impurities with radii of order or greater than the Fermi wavelength, $R\ge\hbar/p_f$, the thermal Hall conductivity is finite for chiral order with $|ν|\ge2$, and determined by a specific Fermi-surface average of the differential cross-section for electron-impurity scattering. Our results also provide quantitative formulae for analyzing and interpreting thermal transport measurements for superconductors predicted to exhibit broken time-reversal and mirror symmetries.

cond-mat.supr-con↗

arXiv2023

Random-Energy Secret Sharing via Extreme Synergy

The random-energy model (REM), a solvable spin-glass model, has impacted an incredibly diverse set of problems, from protein folding to combinatorial optimization to many-body localization. Here, we explore a new connection to secret sharing. We formulate a secret-sharing scheme, based on the REM, and analyze its information-theoretic properties. Our analyses reveal that the correlations between subsystems of the REM are highly synergistic and form the basis for secure secret-sharing schemes. We derive the ranges of temperatures and secret lengths over which the REM satisfies the requirement of secure secret sharing. We show further that a special point in the phase diagram exists at which the REM-based scheme is optimal in its information encoding. Our analytical results for the thermodynamic limit are in good qualitative agreement with numerical simulations of finite systems, for which the strict security requirement is replaced by a tradeoff between secrecy and recoverability. Our work offers a further example of information theory as a unifying concept, connecting problems in statistical physics to those in computation.

cond-mat.dis-nn↗

arXiv2023

Extrinsic vs Intrinsic Criticality in Systems with Many Components

Biological systems with many components often exhibit seemingly critical behaviors, characterized by atypically large correlated fluctuations. Yet the underlying causes remain unclear. Here we define and examine two types of criticality. Intrinsic criticality arises from interactions within the system which are fine-tuned to a critical point. Extrinsic criticality, in contrast, emerges without fine tuning when observable degrees of freedom are coupled to unobserved fluctuating variables. We unify both types of criticality using the language of learning and information theory. We show that critical correlations, intrinsic or extrinsic, lead to diverging mutual information between two halves of the system, and are a feature of learning problems, in which the unobserved fluctuations are inferred from the observable degrees of freedom. We argue that extrinsic criticality is equivalent to standard inference, whereas intrinsic criticality describes fractional learning, in which the amount to be learned depends on the system size. We show further that both types of criticality are on the same continuum, connected by a smooth crossover. In addition, we investigate the observability of Zipf's law, a power-law rank-frequency distribution often used as an empirical signature of criticality. We find that Zipf's law is a robust feature of extrinsic criticality but can be nontrivial to observe for some intrinsically critical systems, including critical mean-field models. We further demonstrate that models with global dynamics, such as oscillatory models, can produce observable Zipf's law without relying on either external fluctuations or fine tuning. Our findings suggest that while possible in theory, fine tuning is not the only, nor the most likely, explanation for the apparent ubiquity of criticality in biological systems with many components.

physics.bio-ph↗

arXiv2023

Generalized Information Bottleneck for Gaussian Variables

The information bottleneck (IB) method offers an attractive framework for understanding representation learning, however its applications are often limited by its computational intractability. Analytical characterization of the IB method is not only of practical interest, but it can also lead to new insights into learning phenomena. Here we consider a generalized IB problem, in which the mutual information in the original IB method is replaced by correlation measures based on Renyi and Jeffreys divergences. We derive an exact analytical IB solution for the case of Gaussian correlated variables. Our analysis reveals a series of structural transitions, similar to those previously observed in the original IB case. We find further that although solving the original, Renyi and Jeffreys IB problems yields different representations in general, the structural transitions occur at the same critical tradeoff parameters, and the Renyi and Jeffreys IB solutions perform well under the original IB objective. Our results suggest that formulating the IB method with alternative correlation measures could offer a strategy for obtaining an approximate solution to the original IB problem.

cs.IT↗

arXiv2022

Information bottleneck theory of high-dimensional regression: relevancy, efficiency and optimality

Avoiding overfitting is a central challenge in machine learning, yet many large neural networks readily achieve zero training loss. This puzzling contradiction necessitates new approaches to the study of overfitting. Here we quantify overfitting via residual information, defined as the bits in fitted models that encode noise in training data. Information efficient learning algorithms minimize residual information while maximizing the relevant bits, which are predictive of the unknown generative models. We solve this optimization to obtain the information content of optimal algorithms for a linear regression problem and compare it to that of randomized ridge regression. Our results demonstrate the fundamental trade-off between residual and relevant information and characterize the relative information efficiency of randomized regression with respect to optimal algorithms. Finally, using results from random matrix theory, we reveal the information complexity of learning a linear map in high dimensions and unveil information-theoretic analogs of double and multiple descent phenomena.

cs.IT↗

arXiv2021

Perturbation Theory for the Information Bottleneck

Extracting relevant information from data is crucial for all forms of learning. The information bottleneck (IB) method formalizes this, offering a mathematically precise and conceptually appealing framework for understanding learning phenomena. However the nonlinearity of the IB problem makes it computationally expensive and analytically intractable in general. Here we derive a perturbation theory for the IB method and report the first complete characterization of the learning onset, the limit of maximum relevant information per bit extracted from data. We test our results on synthetic probability distributions, finding good agreement with the exact numerical solution near the onset of learning. We explore the difference and subtleties in our derivation and previous attempts at deriving a perturbation theory for the learning onset and attribute the discrepancy to a flawed assumption. Our work also provides a fresh perspective on the intimate relationship between the IB method and the strong data processing inequality.

cs.LG↗

arXiv2021

Inferring couplings in networks across order-disorder phase transitions

Statistical inference is central to many scientific endeavors, yet how it works remains unresolved. Answering this requires a quantitative understanding of the intrinsic interplay between statistical models, inference methods and data structure. To this end, we characterize the efficacy of direct coupling analysis (DCA)--a highly successful method for analyzing amino acid sequence data--in inferring pairwise interactions from samples of ferromagnetic Ising models on random graphs. Our approach allows for physically motivated exploration of qualitatively distinct data regimes separated by phase transitions. We show that inference quality depends strongly on the nature of generative models: optimal accuracy occurs at an intermediate temperature where the detrimental effects from macroscopic order and thermal noise are minimal. Importantly our results indicate that DCA does not always outperform its local-statistics-based predecessors; while DCA excels at low temperatures, it becomes inferior to simple correlation thresholding at virtually all temperatures when data are limited. Our findings offer new insights into the regime in which DCA operates so successfully and more broadly how inference interacts with data structure.

physics.bio-ph↗

arXiv2020

Impurity-Induced Anomalous Thermal Hall Effect in Chiral Superconductors

Chiral superconductors exhibit novel transport properties that depend on the topology of the order parameter, topology of the Fermi surface, the spectrum of bulk and edge Fermionic excitations, and the structure of the impurity potential. In the case of electronic heat transport, impurities induce an anomalous (zero-field) thermal Hall conductivity that is easily orders of magnitude larger than the quantized edge contribution. The effect originates from branch-conversion scattering of Bogoliubov quasiparticles by the chiral order parameter, induced by potential scattering. The former transfers angular momentum between the condensate and the excitations that transport heat. The anomalous thermal Hall conductivity is shown to depend to the structure of the electron-impurity potential, as well as the winding number, $ν$, of the chiral order parameter, $Δ(p)=|Δ(p)|\,e^{iνϕ_{p}}$. The results provide quantitative formulae for interpreting heat transport experiments seeking to identify broken T and P symmetries, as well as the topology of the order parameter for chiral superconductors.

cond-mat.supr-con↗

arXiv2020

Energy consumption and cooperation for optimal sensing

The reliable detection of environmental molecules in the presence of noise is an important cellular function, yet the underlying computational mechanisms are not well understood. We introduce a model of two interacting sensors which allows for the principled exploration of signal statistics, cooperation strategies and the role of energy consumption in optimal sensing, quantified through the mutual information between the signal and the sensors. Here we report that in general the optimal sensing strategy depends both on the noise level and the statistics of the signals. For joint, correlated signals, energy consuming (nonequilibrium), asymmetric couplings result in maximum information gain in the low-noise, high-signal-correlation limit. Surprisingly we also find that energy consumption is not always required for optimal sensing. We generalise our model to incorporate time integration of the sensor state by a population of readout molecules, and demonstrate that sensor interaction and energy consumption remain important for optimal sensing.

physics.bio-ph↗

arXiv2018

The Effect of Inhomogeneous Surface Disorder on the Superheating Field of Superconducting RF Cavities

Recent advances in surface treatments of Niobium superconducting radio frequency (SRF) cavities have led to substantially increased Q-factors and maximum surface field. This poses theoretical challenges to identify the mechanisms responsible for such performance enhancements. We report theoretical results for the effects of inhomogeneous surface disorder on the superheating field --- the surface magnetic field above which the Meissner state is globally unstable. We find that inhomogeneous disorder, such as that introduced by infusion of Nitrogen into the surface layers of Niobium SRF cavities, can increase the superheating field above the maximum for superconductors in the clean limit or with homogeneously distributed disorder. Homogeneous disorder increases the penetration of screening current, but also suppresses the maximum supercurrent. Inhomogeneous disorder in the form of an impurity diffusion layer biases this trade-off by increasing the penetration of the screening currents into cleaner regions with larger critical currents, thus limiting the suppression of the screening current to a thin dirty region close to the surface. Our results suggest that the impurity diffusion layers play a role in enhancing the maximum accelerating gradient of Nitrogen treated Niobium SRF cavities.

cond-mat.supr-con↗

arXiv2018

Exploiting ecology in drug pulse sequences in favour of population reduction

A deterministic population dynamics model involving birth and death for a two-species system, comprising a wild-type and more resistant species competing via logistic growth, is subjected to two distinct stress environments designed to mimic those that would typically be induced by temporal variation in the concentration of a drug (antibiotic or chemotherapeutic) as it permeates through the population and is progressively degraded. Different treatment regimes, involving single or periodical doses, are evaluated in terms of the minimal population size (a measure of the extinction probability), and the population composition (a measure of the selection pressure for resistance or tolerance during the treatment). We show that there exist timescales over which the low-stress regime is as effective as the high-stress regime, due to the competition between the two species. For multiple periodic treatments, competition can ensure that the minimal population size is attained during the first pulse when the high-stress regime is short, which implies that a single short pulse can be more effective than a more protracted regime. Our results suggest that when the duration of the high-stress environment is restricted, a treatment with one or multiple shorter pulses can produce better outcomes than a single long treatment. If ecological competition is to be exploited for treatments, it is crucial to determine these timescales, and estimate for the minimal population threshold that suffices for extinction. These parameters can be quantified by experiment.

physics.bio-ph↗