arXiv subjects

Yeongwoo Song

Publications and source records attributed to Yeongwoo Song.

3 recordsLinked to original sources

arXiv2026

Uncovering Spontaneous Physics Representations in In-Context Learning

In-context learning (ICL) lets large language models (LLMs) solve new tasks from prompts alone, across an ever-widening range of domains, yet the mechanisms underlying this ability remain poorly understood. Physical systems offer a controlled testbed for this question as they provide experimentally controllable data with structured dynamics grounded in fundamental principles. Here we study the ICL ability of LLMs, focusing on physical reasoning. Using dynamics forecasting as a proxy task, we first show that LLMs forecast physical dynamics in context, with accuracy improving as more history is provided. Analyzing the model's residual stream reveals internal activations that correlate with key physical quantities such as energy. These correlations strengthen gradually with context length, indicating that LLMs spontaneously form representations aligned with physical concepts without any physics-specific supervision. To assess whether these representations contribute to the model's predictions, we introduce a layer-wise gradient-based attribution analysis. We find that, residual directions more strongly correlated with energy also receive greater attribution to numerical predictions. This pattern is not observed for features correlated with directly observed quantities such as displacement, suggesting that the energy-related signal is not merely numerical information copied from the input. Our results broaden ICL analysis to structured physical dynamics and give a mechanistic account of how LLMs organize physical structure in context.

cs.CL↗

arXiv2025

Stochastic Resetting Mitigates Latent Gradient Bias of SGD from Label Noise

Giving up and starting over may seem wasteful in many situations such as searching for a target or training deep neural networks (DNNs). Our study, though, demonstrates that resetting from a checkpoint can significantly improve generalization performance when training DNNs with noisy labels. In the presence of noisy labels, DNNs initially learn the general patterns of the data but then gradually memorize the corrupted data, leading to overfitting. By deconstructing the dynamics of stochastic gradient descent (SGD), we identify the behavior of a latent gradient bias induced by noisy labels, which harms generalization. To mitigate this negative effect, we apply the stochastic resetting method to SGD, inspired by recent developments in the field of statistical physics achieving efficient target searches. We first theoretically identify the conditions where resetting becomes beneficial, and then we empirically validate our theory, confirming the significant improvements achieved by resetting. We further demonstrate that our method is both easy to implement and compatible with other methods for handling noisy labels. Additionally, this work offers insights into the learning dynamics of DNNs from an interpretability perspective, expanding the potential to analyze training methods through the lens of statistical physics.

cs.LG↗

arXiv2024

Towards Cross Domain Generalization of Hamiltonian Representation via Meta Learning

Recent advances in deep learning for physics have focused on discovering shared representations of target systems by incorporating physics priors or inductive biases into neural networks. While effective, these methods are limited to the system domain, where the type of system remains consistent and thus cannot ensure the adaptation to new, or unseen physical systems governed by different laws. For instance, a neural network trained on a mass-spring system cannot guarantee accurate predictions for the behavior of a two-body system or any other system with different physical laws. In this work, we take a significant leap forward by targeting cross domain generalization within the field of Hamiltonian dynamics. We model our system with a graph neural network (GNN) and employ a meta learning algorithm to enable the model to gain experience over a distribution of systems and make it adapt to new physics. Our approach aims to learn a unified Hamiltonian representation that is generalizable across multiple system domains, thereby overcoming the limitations of system-specific models. We demonstrate that the meta-trained model captures the generalized Hamiltonian representation that is consistent across different physical domains. Overall, through the use of meta learning, we offer a framework that achieves cross domain generalization, providing a step towards a unified model for understanding a wide array of dynamical systems via deep learning.

cs.LG↗