arXiv subjects

Sheng Yixuan

Publications and source records attributed to Sheng Yixuan.

3 recordsLinked to original sources

arXiv2026

Control Allocation in Neural Network Optimization: Joint Affine Control of Weight and Bias Updates

Optimization algorithms determine not only the magnitude of a neural-network update but also how that update is distributed across parameter channels. We study whether this distribution can be treated as a controllable quantity independently of global training progress. We define operational update allocation through normalized channel energies and analyze two scalar controls: a coordinate-preconditioning exponent and an affine spectral exponent that scales the bias column of an augmented weight--bias matrix. At a frozen state, a common nonzero step-size multiplier leaves normalized allocation unchanged; the coordinate exponent yields affine pairwise log-odds with an explicit inverse; and the affine exponent induces a rank-one positive-semidefinite Gram perturbation and a logistic raw-participation law. We further separate raw affine participation, spectral gain, and the decoded physical bias update, and show that finite polynomial spectral iterations preserve singular subspaces. Same-state replay verifies the exact control laws. On a five-seed controlled benchmark, intermediate controls improve held-out and worst-group metrics, whereas excessive affine control causes underfitting. A four-task single-seed transfer study provides descriptive corroboration. These results establish instantaneous allocation control and a bounded empirical operating regime, but do not imply a task-independent generalization ordering.

cs.LG↗

arXiv2026

Hidden Boundary Motion in Transformer Optimization: Function-Space Orthogonalization of Affine Weight and Bias Updates

Weights and biases are normally optimized as separate parameter tensors, yet they do not represent separate functions when the input to an affine layer has nonzero mean. For an affine map $z=Wx+b$ with input mean $\mu$, a weight update contains a sample-independent displacement $\Delta W\mu$ that is functionally indistinguishable from a bias update. We call this hidden contribution \emph{boundary motion} and decompose each update into a centered, sample-varying \emph{shape} component and a shared \emph{boundary} component. On a four-layer Transformer trained from scratch on IMDb, the bias-like term $g_b\mu^\top$ has a median norm equal to 0.664 of the raw weight-gradient norm across affine layers and training checkpoints. More strikingly, the median ratio $\norm{\Delta W\mu}/\norm{\Delta b}$ is 134.7, while $\norm{\Delta W\mu}/\norm{\Delta b+\Delta W\mu}$ is 0.994. Thus, under AdamW, the observed boundary motion is almost entirely realized through the weight matrix rather than the explicit bias. We implement a diagnostic optimizer, Shape--Boundary Orthogonal AdamW (SBO-AdamW), that optimizes $g_W-g_b\mu^\top$ and $g_b$ with independent Adam states and compensates the weight-induced boundary displacement. In a single-seed experiment, SBO-AdamW raises validation accuracy from 81.68\% to 85.81\% and validation-selected test accuracy from 78.73\% to 82.73\%, with the best validation checkpoint occurring at step 800 instead of step 3000. However, the moving-batch-center compensation produces severe bias-coordinate drift and strongly reduces boundary energy. The present evidence therefore supports hidden boundary motion as an important optimization mechanism, but it does not yet establish a final general-purpose optimizer. A stable centered-affine parameterization is identified as the required next step.

cs.LG↗

arXiv2026

The Anatomy of Implicit Bias: Information Allocation in Neural Network Training

Implicit bias is usually explained as the preference of an optimization process for certain final solutions and their geometry. This view helps explain where a model finally stops. It gives less direct explanation of how this bias is formed during training. This paper proposes a training-time information allocation view. Under this view, optimization forms a writing pattern for error signals across parameter paths, coordinate channels, and sample regions. This paper builds a set of observable allocation diagnostics. These diagnostics include gradient demand, actual update injection, coordinate gain induced by exponential moving averages, channel-level update ratios, and sample-wise loss distributions. To separate training progress from internal allocation, this paper introduces a collapse--persistence analysis. Under matched training loss, if external loss statistics collapse but internal allocation ratios remain separated, then the factor changes the internal allocation of the training signal. Overall, this paper extends the analysis of implicit bias from final-solution geometry to training-time signal allocation. The main claim is that implicit bias is not only reflected by the final solution. It is also reflected by which parameter paths, coordinate channels, and sample regions receive the error signal first and more strongly during training. Based on this view, this paper places different training factors into a unified information-allocation diagnostic framework. The framework gives a mechanism-level explanation of training-time implicit bias. It also provides a basis for future optimization methods that control training progress and signal allocation separately.

cs.LG↗