arXiv subjects

Chanju Park

Publications and source records attributed to Chanju Park.

8 recordsLinked to original sources

arXiv2026

Spectral phase transitions and trainability in neural network learning dynamics

The emergence of low-dimensional structures in the spectra of neural network weight matrices is a common empirical feature of trained models, but the dynamical origin of this phenomenon during learning remains an open problem. We formulate neural network training as the stochastic evolution of an initially random matrix ensemble, driven by stochastic gradient descent (SGD) updates that reshape the spectral bulk while amplifying signal strength. This induces a Baik-Ben Arous-P\'ech\'e (BBP) transition during training, where isolated eigenvalues detach from the random bulk distribution, providing a dynamical framework for representation formation in high-dimensional learning dynamics. We demonstrate this in a solvable linear teacher-student model, where spectral evolution is analytically tractable and a phase diagram of trainability governed by the step size (or learning rate) and initial weight variance is obtained, and subsequently extend our formalism beyond the linear regime to nonlinear and stochastic settings. Numerical simulations in realistic settings support this picture, showing robust emergence of spectral alignment during training. Our results suggest that spectral analysis may provide a unified perspective of stochastic learning dynamics, linking trainability, optimisation hyperparameters, spectral phase transitions, and representation learning in neural networks.

cond-mat.dis-nn↗

arXiv2025

Phase diagram and eigenvalue dynamics of stochastic gradient descent in multilayer neural networks

Hyperparameter tuning is one of the essential steps to guarantee the convergence of machine learning models. We argue that intuition about the optimal choice of hyperparameters for stochastic gradient descent can be obtained by studying a neural network's phase diagram, in which each phase is characterised by distinctive dynamics of the singular values of weight matrices. Taking inspiration from disordered systems, we start from the observation that the loss landscape of a multilayer neural network with mean squared error can be interpreted as a disordered system in feature space, where the learnt features are mapped to soft spin degrees of freedom, the initial variance of the weight matrices is interpreted as the strength of the disorder, and temperature is given by the ratio of the learning rate and the batch size. As the model is trained, three phases can be identified, in which the dynamics of weight matrices is qualitatively different. Employing a Langevin equation for stochastic gradient descent, previously derived using Dyson Brownian motion, we demonstrate that the three dynamical regimes can be classified effectively, providing practical guidance for the choice of hyperparameters of the optimiser.

cond-mat.dis-nn↗

arXiv2024

Random Matrix Theory for Stochastic Gradient Descent

Investigating the dynamics of learning in machine learning algorithms is of paramount importance for understanding how and why an approach may be successful. The tools of physics and statistics provide a robust setting for such investigations. Here we apply concepts from random matrix theory to describe stochastic weight matrix dynamics, using the framework of Dyson Brownian motion. We derive the linear scaling rule between the learning rate (step size) and the batch size, and identify universal and non-universal aspects of weight matrix dynamics. We test our findings in the (near-)solvable case of the Gaussian Restricted Boltzmann Machine and in a linear one-hidden-layer neural network.

hep-lat↗

arXiv2024

Dyson Brownian motion and random matrix dynamics of weight matrices during learning

During training, weight matrices in machine learning architectures are updated using stochastic gradient descent or variations thereof. In this contribution we employ concepts of random matrix theory to analyse the resulting stochastic matrix dynamics. We first demonstrate that the dynamics can generically be described using Dyson Brownian motion, leading to e.g. eigenvalue repulsion. The level of stochasticity is shown to depend on the ratio of the learning rate and the mini-batch size, explaining the empirically observed linear scaling rule. We verify this linear scaling in the restricted Boltzmann machine. Subsequently we study weight matrix dynamics in transformers (a nano-GPT), following the evolution from a Marchenko-Pastur distribution for eigenvalues at initialisation to a combination with additional structure at the end of learning.

cond-mat.dis-nn↗

arXiv2024

Stochastic weight matrix dynamics during learning and Dyson Brownian motion

We demonstrate that the update of weight matrices in learning algorithms can be described in the framework of Dyson Brownian motion, thereby inheriting many features of random matrix theory. We relate the level of stochasticity to the ratio of the learning rate and the mini-batch size, providing more robust evidence to a previously conjectured scaling relationship. We discuss universal and non-universal features in the resulting Coulomb gas distribution and identify the Wigner surmise and Wigner semicircle explicitly in a teacher-student model and in the (near-)solvable case of the Gaussian restricted Boltzmann machine.

cond-mat.dis-nn↗

arXiv2024

Scalar field Restricted Boltzmann Machine as an ultraviolet regulator

Restricted Boltzmann Machines (RBMs) are well-known tools used in Machine Learning to learn probability distribution functions from data. We analyse RBMs with scalar fields on the nodes from the perspective of lattice field theory. Starting with the simplest case of Gaussian fields, we show that the RBM acts as an ultraviolet regulator, with the cutoff determined by either the number of hidden nodes or a model mass parameter. We verify these ideas in the scalar field case, where the target distribution is known, and explore implications for cases where it is not known using the MNIST data set. We also demonstrate that infrared modes are learnt quickest.

hep-lat↗

arXiv2022

2021 Update on $\varepsilon_K$ with lattice QCD inputs

We present recent updates for $\varepsilon_K$ determined directly from the standard model (SM) with lattice QCD inputs such as $\hat{B}_K$, $|V_{cb}|$, $|V_{us}|$, $ξ_0$, $ξ_2$, $ξ_\text{LD}$, $f_K$, and $m_c$. We find that the standard model with exclusive $|V_{cb}|$ and other lattice QCD inputs describes only 66\% of the experimental value of $|\varepsilon_K|$ and does not explain its remaining 34\%, which leads to a strong tension in $|\varepsilon_K|$ at the $4.5σ\sim 3.7σ$ level between the SM theory and experiment. We also find that this tension disappears when we use the inclusive value of $|V_{cb}|$ obtained using the heavy quark expansion based on the QCD sum rule approach.

hep-lat↗

arXiv2022

Deep learning study on the Dirac eigenvalue spectrum of staggered quarks

We study the chirality of staggered quarks on the Dirac eigenvalue spectrum using deep learning (DL) techniques. The Kluberg-Stern method to construct staggered bilinear operators conserves continuum property such as recursion relations, uniqueness of chirality, and Ward identities, which leads to a unique and characteristic pattern (we call it "leakage pattern (LP)") in the matrix elements of the chirality operator sandwiched between two quark eigenstates of staggered Dirac operator. DL analysis gives $99.4(2)\%$ accuracy on normal gauge configurations and $0.998$ AUC (Area Under ROC Curve) for classifying non-zero mode octets in the Dirac eigenvalue spectrum. It confirms that the leakage pattern is universal on normal gauge configurations. The multi-layer perceptron (MLP) method turns out to be the best DL model for our study on the LP.

hep-lat↗