arXiv subjects

Yongxin Li

Publications and source records attributed to Yongxin Li.

8 recordsLinked to original sources

arXiv2026

Pre-carved Niches: The Formation Dynamics of Modular Task Partitions in Early LLM Training

Large language models exhibit a modular internal organization that mirrors well-studied functional networks of the human brain, but how this organization forms during training is unknown: prior work has characterized finished models, not the formation process. We track formation step by step: we train a Pythia-410M model from scratch (two trajectories, bf16 and fp32) and run attribution patching at every step, alongside probes for gradient norms, effective updates, weight norms, and first-order loss decomposition across 14 tasks in four cognitive domains. Three findings. First, the modular map is pre-carved: before any learning, the dominant task pair already overlaps at ~3.6x the attribution substrate (a task-independent baseline), and its layer-0 concentration is an architecture-level constant on this model family. Second, the partition locks in through two sharp jumps whose amplitudes do not track the learning-rate schedule (the second reaching 20.4 sigma quiet-window / 6.2 sigma global), accompanied by gradient-level relative deprivation--winners receive 2.25->2.73x the loser's gradient supply, 9.5-11.5 standard deviations below a random control--that does not propagate to updates or weights. Third, deviation from the substrate appears only in the domain being learned, consistent with the hypothesis that modularity tracks learning. We close by separating the feature-level account we can defend from the mechanistic questions we cannot, and we pre-register the scale-threshold hypothesis behind our ongoing 2.8B experiments.

cs.LG↗

arXiv2026

Selection Plateau and a Sparsity-Dependent Hierarchy of Pruning Features

We identify a Selection Plateau phenomenon in one-shot neural network pruning: all rank-monotone weight scorers converge to identical accuracy at fixed sparsity, independent of functional form. We propose the Sparsity-Information-Complexity Spectrum (SICS) hypothesis: a sparsity-dependent minimum feature complexity kappa(S) governs plateau escape, with kappa=0 sufficient at low sparsity (S<0.65), kappa=1 dominant at critical sparsity (S~0.7), and kappa=2 necessary at extreme sparsity (S>0.75). On ViT-Small/CIFAR-10, testing nine feature classes across four sparsities, smooth non-monotone features provide +6.6% escape at S=0.7, while only raw features with high-frequency wiggle escape at S=0.8 (+2.6%). A fake non-monotone scorer underperforms the gradient baseline, indicating the requirement is magnitude-independent non-monotonicity. A handcrafted Gaussian bump achieves only +0.006 escape vs. chaos-derived +0.046, indicating rank-alignment is necessary but insufficient. SICS provides a unifying explanation for the performance clustering of diverse pruning methods and suggests that future selection algorithms should adapt feature complexity to target sparsity.

cs.LG↗

arXiv2025

Learning an Adaptive and View-Invariant Vision Transformer for Real-Time UAV Tracking

Transformer-based models have improved visual tracking, but most still cannot run in real time on resource-limited devices, especially for unmanned aerial vehicle (UAV) tracking. To achieve a better balance between performance and efficiency, we propose AVTrack, an adaptive computation tracking framework that adaptively activates transformer blocks through an Activation Module (AM), which dynamically optimizes the ViT architecture by selectively engaging relevant components. To address extreme viewpoint variations, we propose to learn view-invariant representations via mutual information (MI) maximization. In addition, we propose AVTrack-MD, an enhanced tracker incorporating a novel MI maximization-based multi-teacher knowledge distillation framework. Leveraging multiple off-the-shelf AVTrack models as teachers, we maximize the MI between their aggregated softened features and the corresponding softened feature of the student model, improving the generalization and performance of the student, especially under noisy conditions. Extensive experiments show that AVTrack-MD achieves performance comparable to AVTrack's performance while reducing model complexity and boosting average tracking speed by over 17\%. Codes is available at: https://github.com/wuyou3474/AVTrack.

cs.CV↗

arXiv2024

An Efficient Approach to Regression Problems with Tensor Neural Networks

This paper introduces a tensor neural network (TNN) to address nonparametric regression problems, leveraging its distinct sub-network structure to effectively facilitate variable separation and enhance the approximation of complex, high-dimensional functions. The TNN demonstrates superior performance compared to conventional Feed-Forward Networks (FFN) and Radial Basis Function Networks (RBN) in terms of both approximation accuracy and generalization capacity, even with a comparable number of parameters. A significant innovation in our approach is the integration of statistical regression and numerical integration within the TNN framework. This allows for efficient computation of high-dimensional integrals associated with the regression function and provides detailed insights into the underlying data structure. Furthermore, we employ gradient and Laplacian analysis on the regression outputs to identify key dimensions influencing the predictions, thereby guiding the design of subsequent experiments. These advancements make TNN a powerful tool for applications requiring precise high-dimensional data analysis and predictive modeling.

stat.ML↗

arXiv2024

Tensor Neural Network Interpolation and Its Applications

Based on tensor neural network, we propose an interpolation method for high dimensional non-tensor-product-type functions. This interpolation scheme is designed by using the tensor neural network based machine learning method. This means that we use a tensor neural network to approximate high dimensional functions which has no tensor product structure. In some sense, the non-tenor-product-type high dimensional function is transformed to the tensor neural network which has tensor product structure. It is well known that the tensor product structure can bring the possibility to design highly accurate and efficient numerical methods for dealing with high dimensional functions. In this paper, we will concentrate on computing the high dimensional integrations and solving high dimensional partial differential equations. The corresponding numerical methods and numerical examples will be provided to validate the proposed tensor neural network interpolation.

math.NA↗

arXiv2022

The Infinity of Randomness

This work starts from definition of randomness, the results of algorithmic randomness are analyzed from the perspective of application. Then, the source and nature of randomness is explored, and the relationship between infinity and randomness is found. The properties of randomness are summarized from the perspective of interaction between systems, that is, the set composed of sequences generated by randomness has the property of asymptotic completeness. Finally, the importance of randomness in AI research is emphasized.

math.GM↗

arXiv2022

Estimating Extreme Value Index by Subsampling for Massive Datasets with Heavy-Tailed Distributions

Modern statistical analyses often encounter datasets with massive sizes and heavy-tailed distributions. For datasets with massive sizes, traditional estimation methods can hardly be used to estimate the extreme value index directly. To address the issue, we propose here a subsampling-based method. Specifically, multiple subsamples are drawn from the whole dataset by using the technique of simple random subsampling with replacement. Based on each subsample, an approximate maximum likelihood estimator can be computed. The resulting estimators are then averaged to form a more accurate one. Under appropriate regularity conditions, we show theoretically that the proposed estimator is consistent and asymptotically normal. With the help of the estimated extreme value index, we can estimate high-level quantiles and tail probabilities of a heavy-tailed random variable consistently. Extensive simulation experiments are provided to demonstrate the promising performance of our method. A real data analysis is also presented for illustration purpose.

stat.ME↗

arXiv2020

Ferroelastic-switching-driven colossal shear strain and piezoelectricity in a hybrid ferroelectric

Materials that can produce large controllable strains are widely used in shape memory devices, actuators and sensors. Great efforts have been made to improve the strain outputs of various material systems. Among them, ferroelastic transitions underpin giant reversible strains in electrically-driven ferro/piezoelectrics and thermally- or magneticallydriven shape memory alloys. However, large-strain ferroelastic switching in conventional ferroelectrics is very challenging while magnetic and thermal controls are not desirable for applications. Here, we demonstrate an unprecedentedly large shear strain up to 21.5 % in a hybrid ferroelectric, C6H5N(CH3)3CdCl3. The strain response is about two orders of magnitude higher than those of top-performing conventional ferroelectric polymers and oxides. It is achieved via inorganic bond switching and facilitated by the structural confinement of the large organic moieties, which prevents the undesired 180-degree polarization switching. Furthermore, Br substitution can effectively soften the bonds and result in giant shear piezoelectric coefficient (d35 ~ 4800 pm/V) in Br-rich end of the solid solution, C6H5N(CH3)3CdBr3xCl3(1-x). The superior electromechanical properties of the compounds promise their potential in lightweight and high energy density devices, and the strategy described here should inspire the development of next-generation piezoelectrics and electroactive materials based on hybrid ferroelectrics.

cond-mat.mtrl-sci↗