arXiv subjects

Dandan Jiang

Publications and source records attributed to Dandan Jiang.

At least 19 recordsLinked to original sources

arXiv2026

High-dimensional ridgeless least squares interpolation under spiked covariance structures

This paper investigates the asymptotic behavior of the out-of-sample prediction risk of the high-dimensional ridgeless least-squares estimator when the feature dimension $p$ and the sample size $n$ grow proportionally. We consider a generalized spiked population covariance model with multiple latent factors, where the number of spiked eigenvalues may remain finite or increase with $n$, and the spiked eigenvalues may be bounded or diverge at arbitrary rates. Beyond characterizing the impact of covariance spectra, we reveal a new mechanism underlying benign overfitting: the prediction behavior of ridgeless interpolation is fundamentally governed by the alignment between the regression coefficient $\boldsymbol\beta$ and the spiked eigenspaces of the population covariance matrix. In particular, we show that the signal energy distributed along latent spike directions determines whether interpolation leads to benign, tempered, or catastrophic overfitting. Our theoretical framework establishes sharp prediction risk limits under minimal moment conditions, requiring only finite fourth moments rather than Gaussianity. We characterize how the number, strength, and geometric structure of the spikes jointly influence the double-descent phenomenon. These results provide a unified understanding of when latent covariance structures facilitate or hinder generalization in overparameterized regression.

math.ST↗

arXiv2026

Double Descent, Ensemble Emergence, and Large Model Averaging in High-Dimensional Multimodel Prediction

This paper investigates the predictive performance of high-dimensional multimodel prediction, where the number of regressors is comparable to the sample size. Leveraging tools from random matrix theory, we derive the exact limiting out-of-sample risk under a nested model setting and comprehensively characterize the risk landscape. This limiting risk helps to reveal two phenomena: simple weighting inherits the double descent trajectory and its associated variance explosion near the interpolation boundary; strategic weighting triggers an ensemble emergence that suppresses the localized risk surge and yields a globally flat risk surface. Building on this limiting risk, we also propose the Large Model Averaging (LaMA) method, in which we consider the discrepancy between in-sample and out-of-sample risks in the high-dimensional regime. Numerical studies and real data applications confirm that LaMA achieves superior predictive accuracy in high-dimensional environments.

stat.ME↗

arXiv2026

Invariant quantile regression for heterogeneous environments

In this paper, we propose an invariant quantile regression (IQR) framework specifically designed for multi-environment datasets, which captures the invariance across different environments. This framework is closely related to transfer learning, causal inference, and fair machine learning, and is motivated by scenarios in which the conditional probability of the response given covariates varies, while certain key variables remain invariant. This perspective differs notably from previous works that restrict attention to the conditional mean, which is often insufficient to capture the full causal relationships between covariates and the response in heterogeneous environments. In contrast, quantile-based invariance naturally accommodates heterogeneity, and aligns more closely with structural causal models, in which variables invariant across environments at one or multiple quantile levels directly indicate potential and stable causal variables. Moreover, we show that IQR may yield a larger set of endogenous variables compared to the conditional mean framework, which in turn promotes more effective exclusion of spurious (non-causal) variables. To achieve this, we introduce a Kernel-Smoothed Invariant Quantile Regression (KS-IQR) estimator, which leverages the underlying invariance structure and heterogeneity among environments, ensuring stable estimation across multiple environments. We establish the causal discovery properties of our method, demonstrate its ability to overcome the ``curse of endogeneity'', and derive an $\ell_2$ error bound for our estimator, all in a non-asymptotic framework. We apply our method to real data for causal discovery and obtain biologically meaningful relationships, recovering known signaling pathways and revealing additional quantile-specific effects.

stat.ME↗

arXiv2026

Asymptotic Theory and Sequential Testing for Adaptive Bandits

Multi-armed bandit (MAB) processes constitute a foundational subclass of reinforcement learning problems and represent a central topic in statistical decision theory. Yet, conducting valid sequential testing under adaptive allocation remains challenging due to the lack of asymptotic theory under non-i.i.d. reward sequences and sublinear sample sizes for some arms. To address this open challenge, we propose an Urn Bandit (UNB) process to integrate the reinforcement mechanism of urn probabilistic models with MAB principles, ensuring almost sure concentration of allocation proportions on optimal arms. We establish a joint functional central limit theorem (FCLT) for consistent estimators of expected rewards under non-i.i.d. reward sequences with non-sub-Gaussian tails and pairwise cross-arm dependence. To overcome the limitations of existing methods that focus mainly on cumulative regret and therefore provide only algorithmic performance guarantees without supporting valid sequential testing, we develop an asymptotic theory for sequential test statistics under the proposed UNB process. The resulting framework enables a broad class of sequential inference procedures, such as A/B testing and policy evaluation. Simulation studies and real data analysis demonstrate that UNB maintains testing performance comparable to that of the equal randomization (ER) design while achieving improved reward accumulation relative to ER.

stat.ME↗

arXiv2026

Phase Transition of Spectral Fluctuations in Large Gram Matrices with a Variance Profile: A Unified Framework for Sparse CLTs

We study the asymptotic spectral behavior of high-dimensional random Gram matrices with sparsity and a variance profile, motivated by applications in wireless communications. Specifically, we consider the Gram matrices $\mathbf S_n=\mathbf Y_n\mathbf Y_n^*$, where the entries of $\mathbf Y_n$ are independent, centered, heteroscedastic, and sparse through Bernoulli masking. The sparsity level is parameterized as $s=q^2/n$, where $q$ ranges from polynomial order up to order $n^{1/2}$. We investigate two asymptotic regimes: a moderate-sparsity regime with fixed $s\in(0,1]$, and a high-sparsity regime where $s\to0$. In both regimes, we establish the convergence of the empirical spectral distribution of $\mathbf S_n$ to a deterministic limit, and further derive central limit theorems for linear spectral statistics using resolvent techniques and martingale difference arguments. Our analysis reveals a phase transition in the fluctuation behavior across the two regimes. In the high-sparsity regime, the asymptotic fluctuations are entirely governed by fourth-moment effects, with sparsity-scaled contributions being suppressed. Moreover, the leading deterministic term and the variance of the linear spectral statistic scale at different rates in $q$, causing the standard centering to fail and necessitating an explicit correction to recover a valid CLT. The results apply to both Gaussian and non-Gaussian entries and are illustrated through applications to hypothesis testing and outage probability analysis in large-scale MIMO systems.

math.ST↗

arXiv2025

Transfer learning for high-dimensional Factor-augmented sparse linear model

In this paper, we study transfer learning for high-dimensional factor-augmented sparse linear models, motivated by applications in economics and finance where strongly correlated predictors and latent factor structures pose major challenges for reliable estimation. Our framework simultaneously mitigates the impact of high correlation and removes the additional contributions of latent factors, thereby reducing potential model misspecification in conventional linear modeling. In such settings, the target dataset is often limited, but multiple heterogeneous auxiliary sources may provide additional information. We develop transfer learning procedures that effectively leverage these auxiliary datasets to improve estimation accuracy, and establish non-asymptotic $\ell_1$- and $\ell_2$-error bounds for the proposed estimators. To prevent negative transfer, we introduce a data-driven source detection algorithm capable of identifying informative auxiliary datasets and prove its consistency. In addition, we provide a hypothesis testing framework for assessing the adequacy of the factor model, together with a procedure for constructing simultaneous confidence intervals for the regression coefficients of interest. Numerical studies demonstrate that our methods achieve substantial gains in estimation accuracy and remain robust under heterogeneity across datasets. Overall, our framework offers a theoretical foundation and a practically scalable solution for incorporating heterogeneous auxiliary information in settings with highly correlated features and latent factor structures.

stat.ME↗

arXiv2025

Unifiedly Efficient Inference on All-Dimensional Targets for Large-Scale GLMs

The scalability of Generalized Linear Models (GLMs) for large-scale, high-dimensional data often forces a trade-off between computational feasibility and statistical accuracy, particularly for inference on pre-specified parameters. While subsampling methods mitigate computational costs, existing estimators are typically constrained by a suboptimal $r^{-1/2}$ convergence rate, where $r$ is the subsample size. This paper introduces a unified framework that systematically breaks this barrier, enabling efficient and precise inference regardless of the dimension of the target parameters. To overcome the accuracy loss and enhance computational efficiency, we propose three estimators tailored to different scenarios. For low-dimensional targets, we propose a de-variance subsampling (DVS) estimator that achieves a sharply improved convergence rate of $\max\{r^{-1}, n^{-1/2}\}$, permitting valid inference even with very small subsamples. As $r$ grows, a multi-step refinement of our estimator is proven to be asymptotically normal and semiparametric efficient when $r/\sqrt{n} \to \infty$, matching the performance of the full-sample estimator-a property confirmed by its Bahadur representation. Critically, we provide an improved principle to high-dimensional targets, developing a novel decorrelated score function that facilitates simultaneous inference for a diverging number of pre-specified parameters. Comprehensive numerical experiments demonstrate that our framework delivers a superior balance of computational efficiency and statistical accuracy across both low- and high-dimensional inferential tasks in large-scale GLM, thereby realizing the promise of unifiedly efficient inference for large-scale GLMs.

stat.ME↗

arXiv2025

Asymptotics for Reinforced Stochastic Processes on Hierarchical Networks

In this paper, we analyze the asymptotic behavior of a system of interacting reinforced stochastic processes $({\bf Z}_n, {\bf N}_n)_n$ on a directed network of $N$ agents. The system is defined by the coupled dynamics ${\bf Z}_{n+1}=(1-r_{n}){\bf Z}_{n}+r_{n}{\bf X}_{n+1}$ and ${\bf N}_{n+1}=(1-\frac{1}{n+1}){\bf N}_n+\frac{1}{n+1}{\bf X}_{n+1}$, where agent actions $\mathbb{P}(X_{n+1,j}=1\mid{\cal F}_n)=\sum_{h} w_{hj}Z_{nh}$ are governed by a column-normalized adjacency matrix ${\bf W}$, and $r_n \sim cn^{-\gamma}$ with $\gamma \in (1/2, 1]$. Existing asymptotic theory has largely been restricted to irreducible and diagonalizable ${\bf W}$. We extend this analysis to the broader and more practical class of reducible and non-diagonalizable matrices ${\bf W}$ possessing a block upper-triangular form, which models hierarchical influence. We first establish synchronization, proving $({\bf Z}^\top_n, {\bf N}^\top_n)^\top \to Z_\infty {\bf 1}$ almost surely, where the distribution of the limit $Z_\infty$ is shown to be determined solely by the internal dynamics of the leading subgroup. Furthermore, we establish a joint central limit theorem for $({\bf Z}_n,{\bf N}_n)_n$, revealing how the spectral properties and Jordan block structure of ${\bf W}$ govern second-order fluctuations. We demonstrate that the convergence rates and the limiting covariance structure exhibit a phase transition dependent on $\gamma$ and the spectral properties of ${\bf W}$. Crucially, we explicitly characterize how the non-diagonalizability of ${\bf W}$ fundamentally alters the asymptotic covariance and introduces new logarithmic scaling factors in the critical case ($\gamma=1$). These results provide a probabilistic foundation for statistical inference on such hierarchical network structures.

math.ST↗

arXiv2025

Detection of subsurface structures with a vehicle-based atom gravity gradiometer

High-precision mobile gravity gradiometers are very useful in geodesy and geophysics. Atom gravity gradiometers (AGGs) could be among the most accurate mobile gravity gradiometers but are currently constrained by the trade-off between portability and sensitivity. Here, we present a high-sensitivity mobile AGG featuring an ultra-compact sensor head with a volume of only 94 L. In the laboratory, it achieves a sensitivity of 77 E/$\sqrt{Hz}$ (1 E=1$\times10^{-9}$/s$^2$) and a long-term stability of better than 0.5 E. We integrated the instrument in a minivan, enabling efficient mobile field surveys with excellent maneuverability in confined spaces. Using this vehicular system, we surveyed the gravitational field over a set of subsurface structures within a small wooded area, successfully resolving their structural signatures with a signal-to-noise ratio of 57 and quantifying the water depth in a reservoir with an accuracy of $\pm$0.23 m. Compared with previous observations using a CG-5 gravimeter, the superior spatial resolution inherent in gradiometry is clearly demonstrated. This work paves the way for bring AGGs to practical field applications.

physics.ins-det↗

arXiv2025

Accelerating Randomized Algorithms for Low-Rank Matrix Approximation

Randomized algorithms are overwhelming methods for low-rank approximation that can alleviate the computational expenditure with great reliability compared to deterministic algorithms. A crucial thought is generating a standard Gaussian matrix $\mathbf{G}$ and subsequently obtaining the orthonormal basis of the range of $\mathbf{AG}$ for a given matrix $\mathbf{A}$. Recently, the \texttt{farPCA} algorithm offers a framework for randomized algorithms, but the dense Gaussian matrix remains computationally expensive. Motivated by this, we introduce the standardized Bernoulli, sparse sign, and sparse Gaussian matrices to replace the standard Gaussian matrix in \texttt{farPCA} for accelerating computation. These three matrices possess a low computational expenditure in matrix-matrix multiplication and converge in distribution to a standard Gaussian matrix when multiplied by an orthogonal matrix under a mild condition. Therefore, the three corresponding proposed algorithms can serve as a superior alternative to fast adaptive randomized PCA (\texttt{farPCA}). Finally, we leverage random matrix theory (RMT) to derive a tighter error bound for \texttt{farPCA} without shifted techniques. Additionally, we extend this improved error bound to the error analysis of our three fast algorithms, ensuring that the proposed methods deliver more accurate approximations for large-scale matrices. Numerical experiments validate that the three algorithms achieve asymptotically the same performance as \texttt{farPCA} but with lower costs, offering a more efficient approach to low-rank matrix approximation.

stat.CO↗

arXiv2025

Nonlinear Principal Component Analysis with Random Bernoulli Features for Process Monitoring

The process generates substantial amounts of data with highly complex structures, leading to the development of numerous nonlinear statistical methods. However, most of these methods rely on computations involving large-scale dense kernel matrices. This dependence poses significant challenges in meeting the high computational demands and real-time responsiveness required by online monitoring systems. To alleviate the computational burden of dense large-scale matrix multiplication, we incorporate the bootstrap sampling concept into random feature mapping and propose a novel random Bernoulli principal component analysis method to efficiently capture nonlinear patterns in the process. We derive a convergence bound for the kernel matrix approximation constructed using random Bernoulli features, ensuring theoretical robustness. Subsequently, we design four fast process monitoring methods based on random Bernoulli principal component analysis to extend its nonlinear capabilities for handling diverse fault scenarios. Finally, numerical experiments and real-world data analyses are conducted to evaluate the performance of the proposed methods. Results demonstrate that the proposed methods offer excellent scalability and reduced computational complexity, achieving substantial cost savings with minimal performance loss compared to traditional kernel-based approaches.

stat.ML↗

arXiv2024

Universal Bootstrap for Spectral Statistics: Beyond Gaussian Approximation

Spectral analysis plays a crucial role in high-dimensional statistics, where determining the asymptotic distribution of various spectral statistics remains a challenging task. Due to the difficulties of deriving the analytic form, recent advances have explored data-driven bootstrap methods for this purpose. However, widely used Gaussian approximation-based bootstrap methods, such as the empirical bootstrap and multiplier bootstrap, have been shown to be inconsistent in approximating the distributions of spectral statistics in high-dimensional settings. To address this issue, we propose a universal bootstrap procedure based on the concept of universality from random matrix theory. Our method consistently approximates a broad class of spectral statistics across both high- and ultra-high-dimensional regimes, accommodating scenarios where the dimension-to-sample-size ratio $p/n$ converges to a nonzero constant or diverges to infinity without requiring structural assumptions on the population covariance matrix, such as eigenvalue decay or low effective rank. We showcase this universal bootstrap method for high-dimensional covariance inference. Extensive simulations and a real-world data study support our findings, highlighting the favorable finite sample performance of the proposed universal bootstrap procedure.

math.ST↗

arXiv2024

Fast Capacity Estimation in Ultra-dense Wireless Networks with Random Interference

In wireless communication systems, the accurate and reliable evaluation of channel capacity is believed to be a fundamental and critical issue for terminals. However, with the rapid development of wireless technology, large-scale communication networks with significant random interference have emerged, resulting in extremely high computational costs for capacity calculation. In ultra-dense wireless networks with extremely large numbers of base stations (BSs) and users, we provide fast estimation methods for determining the capacity. We consider two scenarios according to the ratio of the number of users to the number of BSs, $\beta_m$. First, when $\beta_m\leq1$, the FIsher-Spiked Estimation (FISE) algorithm is proposed to determine the capacity by modeling the channel matrix with random interference as a Fisher matrix. Second, when $\beta_m>1$, based on a closed-form expression for capacity estimation requiring solely simple computations, we prove that this estimation stabilizes and remains invariant with increasing $\beta_m$. Our methods can guarantee high accuracy on capacity estimation with low complexity, which is faster than the existing methods. Furthermore, our approaches exhibit excellent generality, free of network area shapes, BS and user distributions, and sub-network locations. Extensive simulation experiments across various scenarios demonstrate the high accuracy and robustness of our methods.

eess.SP↗

arXiv2024

Inference on testing the number of spikes in a high-dimensional generalized spiked Fisher matrix

The spiked Fisher matrix is a significant topic for two-sample problems in multivariate statistical inference. This paper is dedicated to testing the number of spikes in a high-dimensional generalized spiked Fisher matrix that relaxes the Gaussian population assumption and the diagonal constraints on the population covariance matrices. First, we propose a general test statistic predicated on partial linear spectral statistics to test the number of spikes, then establish the central limit theorem (CLT) for this statistic under the null hypothesis. Second, we apply the CLT to address two statistical problems: variable selection in high-dimensional linear regression and change point detection. For each test problem, we construct new statistics and derive their asymptotic distributions under the null hypothesis. Finally, simulations and empirical analysis are conducted to demonstrate the remarkable effectiveness and generality of our proposed methods across various scenarios.

math.ST↗

arXiv2022

TOSE: A Fast Capacity Determination Algorithm Based on Random Matrix Theory

Wireless network capacity is one of the most important performance metrics for wireless communication networks. Future wireless networks will be composed of extremely large number of base stations (BSs) and users, and organized in the form of multiple clusters. Unfortunately, the determination of average cluster capacity for such future wireless networks is difficult, and lacks of both analytical expressions and fast algorithms. In this paper, we propose a fast algorithm TOSE to estimate the average cluster capacity based on the random matrix theory (RMT). It can avoid the exact eigenvalue derivations of large dimensional matrices, which are complicated and inevitable in conventional capacity determination methods. Instead, fast eigenvalue estimations can be realized based on RMT in our TOSE algorithm. In addition, we derive the analytical upper and lower bounds of the average cluster capacity. Our numerical experiments show that TOSE is faster than the conventional Cholesky decomposition method, by at least three orders of magnitude. Besides, TOSE has superior generality, since it is independent of the distributions of BSs and users, and the shape of network areas.

cs.IT↗

arXiv2022

TOSE: A Fast Capacity Estimation Algorithm Based on Spike Approximations

Capacity is one of the most important performance metrics for wireless communication networks. It describes the maximum rate at which the information can be transmitted of a wireless communication system. To support the growing demand for wireless traffic, wireless networks are becoming more dense and complicated, leading to a higher difficulty to derive the capacity. Unfortunately, most existing methods for the capacity calculation take a polynomial time complexity. This will become unaffordable for future ultra-dense networks, where both the number of base stations (BSs) and the number of users are extremely large. In this paper, we propose a fast algorithm TOSE to estimate the capacity for ultra-dense wireless networks. Based on the spiked model of random matrix theory (RMT), our algorithm can avoid the exact eigenvalue derivations of large dimensional matrices, which are complicated and inevitable in conventional capacity calculation methods. Instead, fast eigenvalue estimations can be realized based on the spike approximations in our TOSE algorithm. Our simulation results show that TOSE is an accurate and fast capacity approximation algorithm. Its estimation error is below 5%, and it runs in linear time, which is much lower than the polynomial time complexity of existing methods. In addition, TOSE has superior generality, since it is independent of the distributions of BSs and users, and the shape of network areas.

eess.SP↗

arXiv2022

The Moment Passing Method for Wireless Channel Capacity Estimation

Wireless network capacity can be regarded as the most important performance metric for wireless communication systems. With the fast development of wireless communication technology, future wireless systems will become more and more complicated. As a result, the channel gain matrix will become a large-dimensional random matrix, leading to an extremely high computational cost to obtain the capacity. In this paper, we propose a moment passing method (MPM) to realize the fast and accurate capacity estimation for future ultra-dense wireless systems. It can determine the capacity with quadratic complexity, which is optimal considering that the cost of a single matrix operation is not less than quadratic complexity. Moreover, it has high accuracy. The simulation results show that the estimation error of this method is below 2 percent. Finally, our method is highly general, as it is independent of the distributions of BSs and users, and the shape of network areas. More importantly, it can be applied not only to the conventional multi-user multiple input and multiple output (MU-MIMO) networks, but also to the capacity-centric networks designed for B5G/6G.

cs.IT↗

arXiv2022

Fisher Matrix Based Fault Detection for PMUs Data in Power Grids

Abnormal event detection is critical in the safe operation of power system. In this paper, using the data collected from phasor measurement units (PMUs), two methods based on Fisher random matrix are proposed to detect faults in power grids. Firstly, the fault detection matrix is constructed and the event detection problem is reformatted as a two-sample covariance matrices test problem. Secondly, the central limit theorem for the linear spectral statistic of the Fisher matrix is derived and a test statistic for testing faults is proposed. To save computing resources, the screening step of fault interval based on the test statistic is designed to check the existence of faults. Then two point-by-point methods are proposed to determine the time of the fault in the interval. One method detects faults by checking whether the largest sample eigenvalue falls outside the supporting set of limiting spectral distribution of the standard Fisher matrix, which can detect the faults with higher accuracy. The other method tests the faults based on the statistic proposed, which has a faster detection speed. Compared with existing works, the simulation results illustrate that two methods proposed in this paper cost less computational time and provide a higher degree of accuracy.

eess.SY↗