arXiv subjects

Yikuan Zhang

Publications and source records attributed to Yikuan Zhang.

4 recordsLinked to original sources

arXiv2026

Noise-Driven Exploration and Transient Freezing Select Flat Minima in Stochastic Gradient Descent

Stochastic gradient descent (SGD) is central to deep learning, yet the dynamical origin of its preference for flatter, more generalizable solutions remains unclear. Here, by analyzing SGD learning dynamics, we identify a nonequilibrium mechanism that governs solution selection during training. Numerical experiments reveal a transient exploratory phase in which SGD trajectories repeatedly escape sharp valleys and migrate toward flatter regions of the loss landscape before becoming confined to a final basin. Using a tractable physical model, we show that SGD noise reshapes the loss landscape into an effective potential that preferentially stabilizes flat solutions. We further uncover a transient freezing mechanism: as training progresses, the flattening landscape suppresses transitions between competing valleys. Stronger SGD noise delays this freezing transition, prolonging the exploratory phase and thereby increasing the probability of convergence to flatter minima. Together, these results provide a unified physical framework connecting learning dynamics, loss-landscape geometry, and generalization, and suggest guiding principles for the design of more effective optimization algorithms.

cs.LG↗

arXiv2026

On the Superlinear Relationship between SGD Noise Covariance and Loss Landscape Curvature

Stochastic Gradient Descent (SGD) introduces anisotropic noise that is correlated with the local curvature of the loss landscape, thereby biasing optimization toward flat minima. Prior work often assumes an equivalence between the Fisher Information Matrix and the Hessian for negative log-likelihood losses, leading to the claim that the SGD noise covariance $\mathbf{C}$ is proportional to the Hessian $\mathbf{H}$. We show that this assumption holds only under restrictive conditions that are typically violated in deep neural networks. Using the recently discovered Activity--Weight Duality, we find a more general relationship agnostic to the specific loss formulation, showing that $\mathbf{C} \propto \mathbb{E}_p[\mathbf{h}_p^2]$, where $\mathbf{h}_p$ denotes the per-sample Hessian with $\mathbf{H} = \mathbb{E}_p[\mathbf{h}_p]$. As a consequence, $\mathbf{C}$ and $\mathbf{H}$ commute approximately rather than coincide exactly. We further find that, within the analyzed fully connected layers, their diagonal elements follow per-layer empirical power laws $C_{ii} \propto H_{ii}^γ$, with layer-dependent fitted exponents bounded by $1 \leq γ\leq 2$. Experiments across datasets, architectures, and loss functions support the resulting layerwise bounds, providing a unified characterization of the noise-curvature relationship in deep learning.

cs.LG↗

arXiv2024

Geometry of optimal control in chemical reaction networks

Although optimal control (OC) has been studied in stochastic thermodynamics for systems with continuous state variables, less is known in systems with discrete state variables, such as Chemical Reaction Networks (CRNs). Here, we develop a general theoretical framework to study OC of CRNs for changing the system from an initial distribution of states to a final distribution with minimum dissipation. We derive a ``Kirchhoff's law" for the probability current in the adiabatic limit, from which the optimal kinetic rates are determined analytically for any given probability trajectory. By using the optimal rates, we show that the total dissipation is determined by a $L_2$-distance measure in the probability space and derive an analytical expression for the metric tensor that depends on the probability distribution, network topology, and capacity of each link. Minimizing the total dissipation leads to the geodesic trajectory in the probability space and the corresponding OC protocol is determined by the Kirchhoff's law. To demonstrate our general approach, we use it to find a lower bound for the minimum dissipation that is tighter than existing bounds obtained with only global constraints. We also apply it to simple networks, e.g., fully connected 3-state CRNs with different local constraints and show that indirect pathway and non-functional transient state can play a crucial role in switching between different probability distributions efficiently. Future directions in studying OC in CRNs by using our general framework are discussed.

cond-mat.stat-mech↗

arXiv2022

GPNet: Simplifying Graph Neural Networks via Multi-channel Geometric Polynomials

Graph Neural Networks (GNNs) are a promising deep learning approach for circumventing many real-world problems on graph-structured data. However, these models usually have at least one of four fundamental limitations: over-smoothing, over-fitting, difficult to train, and strong homophily assumption. For example, Simple Graph Convolution (SGC) is known to suffer from the first and fourth limitations. To tackle these limitations, we identify a set of key designs including (D1) dilated convolution, (D2) multi-channel learning, (D3) self-attention score, and (D4) sign factor to boost learning from different types (i.e. homophily and heterophily) and scales (i.e. small, medium, and large) of networks, and combine them into a graph neural network, GPNet, a simple and efficient one-layer model. We theoretically analyze the model and show that it can approximate various graph filters by adjusting the self-attention score and sign factor. Experiments show that GPNet consistently outperforms baselines in terms of average rank, average accuracy, complexity, and parameters on semi-supervised and full-supervised tasks, and achieves competitive performance compared to state-of-the-art model with inductive learning task.

cs.LG↗