arXiv subjects

Zhendong Huang

Publications and source records attributed to Zhendong Huang.

9 recordsLinked to original sources

arXiv2026

Deep Simulation-Based Inference for Inhomogeneous Bivariate Log-Gaussian Cox Processes

We propose a computationally efficient simulation-based estimation method with a two-step procedure for inhomogeneous bivariate Log-Gaussian Cox Processes. It combines classical Poisson estimation for the first-order parameters with simulation-based inference using neural networks for the latent field parameters. By separating the estimations, it reduces the complexity of high dimensional parameter estimation and the need for the simulation-based method to specify broad parameter ranges in the presence of covariates. In addition, we introduce two dimensional image inputs that enable the model to learn spatial information directly. Simulation results demonstrate that the proposed approach provides accurate estimates of the latent field parameters. We further illustrate the method's practical applicability using the gorilla dataset.

stat.ME↗

arXiv2026

The Curse and Blessing of Mean Bias in FP4-Quantized LLM Training

FP4 training promises substantial memory and compute savings for large language models, but remains fragile because blockwise quantization is dictated by extreme activation magnitudes, which inflate dynamic range and compress long-tail signals. We identify a counterintuitive source of this failure: dominant activation outliers are not merely arbitrary sparse events, but are largely induced by a coherent rank-one mean bias, whose direction aligns with the leading anisotropic spectral component. This mean component strengthens during training, is amplified and reshaped by attention and FFN operators, and increasingly dominates top activation magnitudes. Crucially, this discovery reveals that a seemingly complex outlier-suppression problem admits a truly simple solution: isolate the coherent mean before quantization. We therefore propose Averis, a mean-residual splitting quantization method that separates the mean component using only reductions and elementwise subtractions before FP4 quantization. Across Qwen3 0.6B Dense trained on 100B tokens and Qwen3 7B A1.5B MoE trained on 50B tokens, Averis enables robust W4A4G4 FP4 training, reducing BF16 loss gaps to 1.19%/0.81% versus 2.05%/1.10% for NVIDIA's recently released Hadamard-based outlier-smoothing method, while limiting downstream gaps to 0.89/0.71 points. With only 2.20% end-to-end overhead over vanilla NVFP4, about 30% of NVIDIA's Hadamard-based design, Averis provides a hardware-efficient path to stable low-bit LLM training. Complementary to Hadamard, Averis further reduces the Qwen3-0.6B loss and downstream gaps to 0.94% and 0.73 points when combined. Code is available at: https://anonymous.4open.science/r/averis-504D.

cs.LG↗

arXiv2026

SD-MoE: Spectral Decomposition for Effective Expert Specialization

Mixture-of-Experts (MoE) architectures scale Large Language Models via expert specialization induced by conditional computation. In practice, however, expert specialization often fails: some experts become functionally similar, while others functioning as de facto shared experts, limiting the effective capacity and model performance. In this work, we analysis from a spectral perspective on parameter and gradient spaces, uncover that (1) experts share highly overlapping dominant spectral components in their parameters, (2) dominant gradient subspaces are strongly aligned across experts, driven by ubiquitous low-rank structure in human corpus, and (3) gating mechanisms preferentially route inputs along these dominant directions, further limiting specialization. To address this, we propose Spectral-Decoupled MoE (SD-MoE), which decomposes both parameter and gradient in the spectral space. SD-MoE improves performance across downstream tasks, enables effective expert specialization, incurring minimal additional computation, and can be seamlessly integrated into a wide range of existing MoE architectures, including Qwen and DeepSeek.

cs.LG↗

arXiv2026

Multi-Head Attention as a Source of Catastrophic Forgetting in MoE Transformers

Mixture-of-Experts (MoE) architectures are often considered a natural fit for continual learning because sparse routing should localize updates and reduce interference, yet MoE Transformers still forget substantially even with sparse, well-balanced expert utilization. We attribute this gap to a pre-routing bottleneck: multi-head attention concatenates head-specific signals into a single post-attention router input, forcing routing to act on co-occurring feature compositions rather than separable head channels. We show that this router input simultaneously encodes multiple separately decodable semantic and structural factors with uneven head support, and that different feature compositions induce weakly aligned parameter-gradient directions; as a result, routing maps many distinct compositions to the same route. We quantify this collision effect via a route-wise effective composition number $N_{eff}$ and find that higher $N_{eff}$ is associated with larger old-task loss increases after continual training. Motivated by these findings, we propose MH-MoE, which performs head-wise routing over sub-representations to increase routing granularity and reduce composition collisions. On TRACE with Qwen3-0.6B/8B, MH-MoE effectively mitigates forgetting, reducing BWT on Qwen3-0.6B from 11.2% (LoRAMoE) to 4.5%.

cs.LG↗

arXiv2026

Spectra: Rethinking Optimizers for LLMs Under Spectral Anisotropy

Gradient signals in LLM training are highly anisotropic: recurrent linguistic structure concentrates energy into a small set of dominant spectral directions, while context specific information resides in a long tail. We show that this spike tail separation persists throughout training, with the spike occupying only about 1.5% of directions yet dominating optimizer statistics. This dominance suppresses tail learning by contracting tail updates through second moment normalization and tightening the globally stable learning rate bound. Motivated by this analysis, we propose Spectra, a spike aware optimizer that suppresses the dominant low rank spike subspace without amplifying the noise sensitive spectral tail. Spectra tracks the spike subspace via cached, warm started power iteration and applies low rank spectral shaping with negligible overhead and substantially reduced optimizer state memory. On LLaMA3 8B trained on 50B tokens, Spectra reaches the same target loss 30% faster than AdamW, reduces per step end to end overhead by 0.7%, cuts optimizer state memory by 49.25%, and improves average downstream accuracy by 1.62%. Compared to Muon, Spectra is 5.1x faster in optimizer processing time, achieves a lower final loss, and improves average accuracy by 0.66%.

cs.LG↗

arXiv2024

High-dimensional Covariance Estimation by Pairwise Likelihood Truncation

Pairwise likelihood is a useful approximation to the full likelihood function for covariance estimation in high-dimensional context. It simplifies high-dimensional dependencies by combining marginal bivariate likelihood objects, thus making estimation more manageable. In certain models, including the Gaussian model, both pairwise and full likelihoods are maximized by the same parameter values, thus retaining optimal statistical efficiency, when the number of variables is fixed. Leveraging on this insight, we introduce estimation of sparse high-dimensional covariance matrices by maximizing a truncated version of the pairwise likelihood function, obtained by including pairwise terms corresponding to nonzero covariance elements. To achieve a meaningful truncation, we propose to minimize the $L_2$-distance between pairwise and full likelihood scores plus an $L_1$-penalty discouraging the inclusion of uninformative terms. Differently from other regularization approaches, our method focuses on selecting whole pairwise likelihood objects rather than shrinking individual covariance parameters, thus retaining the inherent unbiasedness of the pairwise likelihood estimating equations. This selection procedure is shown to have the selection consistency property as the covariance dimension increases exponentially fast. Consequently, the implied pairwise likelihood estimator is consistent and converges to the oracle maximum likelihood estimator assuming knowledge of nonzero covariance entries.

stat.ME↗

arXiv2022

Fast construction of optimal composite likelihoods

A composite likelihood is a combination of low-dimensional likelihood objects useful in applications where the data have complex structure. Although composite likelihood construction is a crucial aspect influencing both computing and statistical properties of the resulting estimator, currently there does not seem to exist a universal rule to combine low-dimensional likelihood objects that is statistically justified and fast in execution. This paper develops a methodology to select and combine the most informative low-dimensional likelihoods from a large set of candidates while carrying out parameter estimation. The new procedure minimizes the distance between composite likelihood and full likelihood scores subject to a constraint representing the afforded computing cost. The selected composite likelihood is sparse in the sense that it contains a relatively small number of informative sub-likelihoods while the noisy terms are dropped. The resulting estimator is found to have asymptotic variance close to that of the minimum-variance estimator constructed using all the low-dimensional likelihoods.

stat.ME↗

arXiv2017

Fast construction of efficient composite likelihood equations

Growth in both size and complexity of modern data challenges the applicability of traditional likelihood-based inference. Composite likelihood (CL) methods address the difficulties related to model selection and computational intractability of the full likelihood by combining a number of low-dimensional likelihood objects into a single objective function used for inference. This paper introduces a procedure to combine partial likelihood objects from a large set of feasible candidates and simultaneously carry out parameter estimation. The new method constructs estimating equations balancing statistical efficiency and computing cost by minimizing an approximate distance from the full likelihood score subject to a L1-norm penalty representing the available computing resources. This results in truncated CL equations containing only the most informative partial likelihood score terms. An asymptotic theory within a framework where both sample size and data dimension grow is developed and finite-sample properties are illustrated through numerical examples.

math.ST↗

arXiv2016

Parsimonious and powerful composite likelihood testing for group difference and genotype-phenotype association

Testing the association between a phenotype and many genetic variants from case-control data is essential in genome-wide association study (GWAS). This is a challenging task as many such variants are correlated or non-informative. Similarities exist in testing the population difference between two groups of high dimensional data with intractable full likelihood function. Testing may be tackled by a maximum composite likelihood (MCL) not entailing the full likelihood, but current MCL tests are subject to power loss for involving non-informative or redundant sub-likelihoods. In this paper, we develop a forward search and test method for simultaneous powerful group difference testing and informative sub-likelihoods composition. Our method constructs a sequence of Wald-type test statistics by including only informative sub-likelihoods progressively so as to improve the test power under local sparsity alternatives. Numerical studies show that it achieves considerable improvement over the available tests as the modeling complexity grows. Our method is further validated by testing the motivating GWAS data on breast cancer with interesting results obtained.

stat.ME↗