arXiv subjects

Sihan Feng

Publications and source records attributed to Sihan Feng.

4 recordsLinked to original sources

arXiv2025

Network Dynamics-Based Framework for Understanding Deep Neural Networks

Advancements in artificial intelligence call for a deeper understanding of the fundamental mechanisms underlying deep learning. In this work, we propose a theoretical framework to analyze learning dynamics through the lens of dynamical systems theory. We redefine the notions of linearity and nonlinearity in neural networks by introducing two fundamental transformation units at the neuron level: order-preserving transformations and non-order-preserving transformations. Different transformation modes lead to distinct collective behaviors in weight vector organization, different modes of information extraction, and the emergence of qualitatively different learning phases. Transitions between these phases may occur during training, accounting for key phenomena such as grokking. To further characterize generalization and structural stability, we introduce the concept of attraction basins in both sample and weight spaces. The distribution of neurons with different transformation modes across layers, along with the structural characteristics of the two types of attraction basins, forms a set of core metrics for analyzing the performance of learning models. Hyperparameters such as depth, width, learning rate, and batch size act as control variables for fine-tuning these metrics. Our framework not only sheds light on the intrinsic advantages of deep learning, but also provides a novel perspective for optimizing network architectures and training strategies.

cs.LG↗

arXiv2022

How and what to learn:The modes of machine learning

Despite their great success, neural networks still remain as black-boxes due to the lack of interpretability. Here we propose a new analyzing method, namely the weight pathway analysis (WPA), to make them transparent. We consider weights in pathways that link neurons longitudinally from input neurons to output neurons, or simply weight pathways, as the basic units for understanding a neural network, and decompose a neural network into a series of subnetworks of such weight pathways. A visualization scheme of the subnetworks is presented that gives longitudinal perspectives of the network like radiographs, making the internal structures of the network visible. Impacts of parameter adjustments or structural changes to the network can be visualized via such radiographs. Characteristic maps are established for subnetworks to characterize the enhancement or suppression of the influence of input samples on each output neuron. Using WPA, we discover that neural network store and utilize information in a holographic way, that is, subnetworks encode all training samples in a coherent structure and thus only by investigating the weight pathways can one explore samples stored in the network. Furthermore, with WPA, we reveal fundamental learning modes of a neural network: the linear learning mode and the nonlinear learning mode. The former extracts linearly separable features while the latter extracts linearly inseparable features. The hidden-layer neurons self-organize into different classes for establishing learning modes and for reaching the training goal. The finding of learning modes provides us the theoretical ground for understanding some of the fundamental problems of machine learning, such as the dynamics of learning process, the role of linear and nonlinear neurons, as well as the role of network width and depth.

cs.LG↗

arXiv2022

The anti-Fermi-Pasta-Ulam-Tsingou problem in one-dimensional diatomic lattices

We study the thermalization dynamics of one-dimensional diatomic lattices (which represents the simplest system possessing multi-branch phonons), exemplified by the famous Fermi-Pasta-Ulam-Tsingou (FPUT)-$β$ and the Toda models. Here we focus on how the system relaxes to the equilibrium state when part of highest-frequency optical modes are initially excited, which is called the anti-FPUT problem comparing with the original FPUT problem (low frequency excitations of the monatomic lattice). It is shown numerically that the final thermalization time $T_{\rm eq}$ of the diatomic FPUT-$β$ chain depends on whether its acoustic modes are thermalized, whereas the $T_{\rm eq}$ of the diatomic Toda chain depends on the optical ones; in addition, the metastable state of both models have different energy distributions and lifetimes. Despite these differences, in the near-integrable region, the $T_{\rm eq}$ of both models still follows the same scaling law, i.e., $T_{\rm eq}$ is inversely proportional to the square of the perturbation strength. Finally, comparisons of the thermalization behavior between different models under various initial conditions are briefly summarized.

cond-mat.stat-mech↗

arXiv2021

Thermalization in asymmetric harmonic chains

The symmetry of the interparticle interaction potential (IIP) plays a critical role in determining the thermodynamic and transport properties of solids. This study investigates the isolated effect of IIP asymmetry on thermalization. Asymmetry and nonlinearity are typically intertwined. To isolate the effect of asymmetry, we introduce a one-dimensional asymmetric harmonic (AH) model whose IIP possesses asymmetry but no nonlinearity, evidenced by energy-independent vibrational frequencies. Extensive numerical simulations confirm a power-law relationship between thermalization time ($T_{\rm eq}$) and perturbation strength for the AH chain, revealing an exponent larger than the previously observed inverse-square law in the thermodynamic limit. Upon adding symmetric quartic nonlinearity into the AH model, we systematically study thermalization under combined asymmetry and nonlinearity. Matthiessen's rule provides a good estimate of $T_{\rm eq}$ in this case. Our results demonstrate that asymmetry plays a distinct role in enhancing higher-order effects and governing relaxation dynamics.

cond-mat.stat-mech↗