arXiv subjects

Yue Ju

Publications and source records attributed to Yue Ju.

15 recordsLinked to original sources

arXiv2026

Bayes Estimators with Performance Comparable to Empirical Bayes Estimators and Improved Local Robustness

Bayes estimation has been extensively studied and widely used in statistics, decision theory, signal processing, machine learning, and system identification. Among its variants, empirical Bayes (EB) estimation has attracted considerable attention due to its favorable estimation performance and computational tractability. However, the direct plug-in dependence of an EB estimator on hyperparameters can make it locally sensitive to hyper-parameter perturbations. This paper considers the linear regression model and focuses on the EB estimator by employing the marginal maximum likelihood hyper-parameter estimator. For conciseness, this estimator is simply referred to as the EB estimator. Given a family of EB weighting functions, a generalized Bayes estimator is constructed with the same excess mean squared error (XMSE) as the corresponding EB estimator. Here, the XMSE is a second-order asymptotic measure of the mean squared error difference between the estimator of interest and the maximum likelihood estimator. Furthermore, the EB estimator is shown to be at most firstorder sensitive to hyper-parameter perturbations, whereas the constructed Bayes estimator is at most second-order sensitive, making it locally more robust. The computational complexities of these two estimators are also analyzed. In some cases, the constructed Bayes estimator can be computationally comparable to, or more efficient than, the EB estimator. These theoretical results are further supported by numerical simulations.

stat.ME↗

arXiv2026

Fundamental Limitations of Data-Driven Control: A Statistical Decision Perspective

Substantial research efforts have been devoted to the design of data-driven controllers; however, comparatively less is known about their statistical performance and fundamental limitations. This contribution develops a statistical decision framework for data-driven control, in which a controller is evaluated by its risk, defined as the expected performance degradation relative to the oracle model-based controller, and by its average risk over the parameter space. Within this framework, we propose a collection of design principles for data-driven controllers. We further derive lower bounds on risks by combining the bias-variance decomposition with the Cram\'er-Rao inequality. In particular, the optimal bias that attains the lower bound for the average risk is determined by calculus of variations, thereby making the bias-variance tradeoff in data-driven control explicit. Moreover, the derived bound reveals a ``waterbed'' effect in data-driven control: any improvement in risk relative to the lower bound over one region of the parameter space must be compensated by deterioration elsewhere. We illustrate the proposed framework on two canonical data-driven control problems: optimal feedforward control and the linear quadratic regulator benchmark. By comparing several representative data-driven controllers with the derived lower bounds, we sharpen the statistical interpretation of existing methods and reveal quantitative limitations that no controller design can avoid.

eess.SY↗

arXiv2025

Range Space or Null Space: Least-Squares Methods for the Realization Problem

This contribution revisits the classical approximate realization problem, which involves determining matrices of a state-space model based on estimates of a truncated series of Markov parameters. A Hankel matrix built up by these Markov parameters plays a fundamental role in this problem, leveraging the fact that both its range space and left null space encode critical information about the state-space model. We examine two prototype realization algorithms based on the Hankel matrix: the classical range-space-based (SVD-based) method and the more recent null-space-based method. It is demonstrated that the range-space-based method corresponds to a total least-squares solution, whereas the null-space-based method corresponds to an ordinary least-squares solution. By analyzing the differences in sensitivity of the two algorithms, we determine the conditions when one or the other realization algorithm is to be preferred, and identify factors that contribute to an ill-conditioned realization problem. Furthermore, recognizing that both methods are suboptimal, we argue that the optimal realization is obtained through a weighted least-squares approach. A statistical analysis of these methods, including their consistency and asymptotic normality is also provided.

eess.SY↗

arXiv2025

Bayes and Biased Estimators Without Hyper-parameter Estimation: Comparable Performance to the Empirical-Bayes-Based Regularized Estimator

Regularized system identification has become a significant complement to more classical system identification. It has been numerically shown that kernel-based regularized estimators often perform better than the maximum likelihood estimator in terms of minimizing mean squared error (MSE). However, regularized estimators often require hyper-parameter estimation. This paper focuses on ridge regression and the regularized estimator by employing the empirical Bayes hyper-parameter estimator. We utilize the excess MSE to quantify the MSE difference between the empirical-Bayes-based regularized estimator and the maximum likelihood estimator for large sample sizes. We then exploit the excess MSE expressions to develop both a family of generalized Bayes estimators and a family of closed-form biased estimators. They have the same excess MSE as the empirical-Bayes-based regularized estimator but eliminate the need for hyper-parameter estimation. Moreover, we conduct numerical simulations to show that the performance of these new estimators is comparable to the empirical-Bayes-based regularized estimator, while computationally, they are more efficient.

stat.ML↗

arXiv2025

Excess Mean Squared Error of Empirical Bayes Estimators

Empirical Bayes estimators are based on minimizing the average risk with the hyper-parameters in the weighting function being estimated from observed data. The performance of an empirical Bayes estimator is typically evaluated by its mean squared error (MSE). However, the explicit expression for its MSE is generally unavailable for finite sample sizes. To address this issue, we define a high-order analytical criterion: the excess MSE. It quantifies the performance difference between the maximum likelihood and empirical Bayes estimators. An explicit expression for the excess MSE of an empirical Bayes estimator employing a general data-dependent hyper-parameter estimator is derived. As specific instances, we provide excess MSE expressions for kernel-based regularized estimators using the scaled empirical Bayes, Stein unbiased risk estimation, and generalized cross-validation hyper-parameter estimators. Moreover, we propose a modification to the excess MSE expressions for regularized estimators for moderate sample sizes and show its improvement on accuracy in numerical simulations.

math.ST↗

arXiv2025

Robust Federated Finetuning of LLMs via Alternating Optimization of LoRA

Parameter-Efficient Fine-Tuning (PEFT) methods like Low-Rank Adaptation (LoRA) optimize federated training by reducing computational and communication costs. We propose RoLoRA, a federated framework using alternating optimization to fine-tune LoRA adapters. Our approach emphasizes the importance of learning up and down projection matrices to enhance expressiveness and robustness. We use both theoretical analysis and extensive experiments to demonstrate the advantages of RoLoRA over prior approaches that either generate imperfect model updates or limit expressiveness of the model. We provide a theoretical analysis on a linear model to highlight the importance of learning both the down-projection and up-projection matrices in LoRA. We validate the insights on a non-linear model and separately provide a convergence proof under general conditions. To bridge theory and practice, we conducted extensive experimental evaluations on language models including RoBERTa-Large, Llama-2-7B on diverse tasks and FL settings to demonstrate the advantages of RoLoRA over other methods.

cs.LG↗

arXiv2024

Robust Federated Finetuning of Foundation Models via Alternating Minimization of LoRA

Parameter-Efficient Fine-Tuning (PEFT) has risen as an innovative training strategy that updates only a select few model parameters, significantly lowering both computational and memory demands. PEFT also helps to decrease data transfer in federated learning settings, where communication depends on the size of updates. In this work, we explore the constraints of previous studies that integrate a well-known PEFT method named LoRA with federated fine-tuning, then introduce RoLoRA, a robust federated fine-tuning framework that utilizes an alternating minimization approach for LoRA, providing greater robustness against decreasing fine-tuning parameters and increasing data heterogeneity. Our results indicate that RoLoRA not only presents the communication benefits but also substantially enhances the robustness and effectiveness in multiple federated fine-tuning scenarios.

cs.LG↗

arXiv2024

Root Cause Analysis of Anomalies in 5G RAN Using Graph Neural Network and Transformer

The emergence of 5G technology marks a significant milestone in developing telecommunication networks, enabling exciting new applications such as augmented reality and self-driving vehicles. However, these improvements bring an increased management complexity and a special concern in dealing with failures, as the applications 5G intends to support heavily rely on high network performance and low latency. Thus, automatic self-healing solutions have become effective in dealing with this requirement, allowing a learning-based system to automatically detect anomalies and perform Root Cause Analysis (RCA). However, there are inherent challenges to the implementation of such intelligent systems. First, there is a lack of suitable data for anomaly detection and RCA, as labelled data for failure scenarios is uncommon. Secondly, current intelligent solutions are tailored to LTE networks and do not fully capture the spatio-temporal characteristics present in the data. Considering this, we utilize a calibrated simulator, Simu5G, and generate open-source data for normal and failure scenarios. Using this data, we propose Simba, a state-of-the-art approach for anomaly detection and root cause analysis in 5G Radio Access Networks (RANs). We leverage Graph Neural Networks to capture spatial relationships while a Transformer model is used to learn the temporal dependencies of the data. We implement a prototype of Simba and evaluate it over multiple failures. The outcomes are compared against existing solutions to confirm the superiority of Simba.

cs.NI↗

arXiv2024

Secure Inference for Vertically Partitioned Data Using Multiparty Homomorphic Encryption

We propose a secure inference protocol for a distributed setting involving a single server node and multiple client nodes. We assume that the observed data vector is partitioned across multiple client nodes while the deep learning model is located at the server node. Each client node is required to encrypt its portion of the data vector and transmit the resulting ciphertext to the server node. The server node is required to collect the ciphertexts and perform inference in the encrypted domain. We demonstrate an application of multi-party homomorphic encryption (MPHE) to satisfy these requirements. We propose a packing scheme, that enables the server to form the ciphertext of the complete data by aggregating the ciphertext of data subsets encrypted using MPHE. While our proposed protocol builds upon prior horizontal federated training protocol~\cite{sav2020poseidon}, we focus on the inference for vertically partitioned data and avoid the transmission of (encrypted) model weights from the server node to the client nodes.

cs.CR↗

arXiv2024

A Calibrated and Automated Simulator for Innovations in 5G

The rise of 5G deployments has created the environment for many emerging technologies to flourish. Self-driving vehicles, Augmented and Virtual Reality, and remote operations are examples of applications that leverage 5G networks' support for extremely low latency, high bandwidth, and increased throughput. However, the complex architecture of 5G hinders innovation due to the lack of accessibility to testbeds or realistic simulators with adequate 5G functionalities. Also, configuring and managing simulators are complex and time consuming. Finally, the lack of adequate representative data hinders the data-driven designs in 5G campaigns. Thus, we calibrated a system-level open-source simulator, Simu5G, following 3GPP guidelines to enable faster innovation in the 5G domain. Furthermore, we developed an API for automatic simulator configuration without knowing the underlying architectural details. Finally, we demonstrate the usage of the calibrated and automated simulator by developing an ML-based anomaly detection in a 5G Radio Access Network (RAN).

cs.NI↗

arXiv2022

An Efficient Implementation for Spatial-Temporal Gaussian Process Regression and Its Applications

Spatial-temporal Gaussian process regression is a popular method for spatial-temporal data modeling. Its state-of-art implementation is based on the state-space model realization of the spatial-temporal Gaussian process and its corresponding Kalman filter and smoother, and has computational complexity $\mathcal{O}(NM^3)$, where $N$ and $M$ are the number of time instants and spatial input locations, respectively, and thus can only be applied to data with large $N$ but relatively small $M$. In this paper, our primary goal is to show that by exploring the Kronecker structure of the state-space model realization of the spatial-temporal Gaussian process, it is possible to further reduce the computational complexity to $\mathcal{O}(M^3+NM^2)$ and thus the proposed implementation can be applied to data with large $N$ and moderately large $M$. The proposed implementation is illustrated over applications in weather data prediction and spatially-distributed system identification. Our secondary goal is to design a kernel for both the Colorado precipitation data and the GHCN temperature data, such that while having more efficient implementation, better prediction performance can also be achieved than the state-of-art result.

eess.SY↗

arXiv2022

Asymptotic Theory for Regularized System Identification Part I: Empirical Bayes Hyper-parameter Estimator

Regularized system identification is the major advance in system identification in the last decade. Although many promising results have been achieved, it is far from complete and there are still many key problems to be solved. One of them is the asymptotic theory, which is about convergence properties of the model estimators as the sample size goes to infinity. The existing related results for regularized system identification are about the almost sure convergence of various hyper-parameter estimators. A common problem of those results is that they do not contain information on the factors that affect the convergence properties of those hyper-parameter estimators, e.g., the regression matrix. In this paper, we tackle problems of this kind for the regularized finite impulse response model estimation with the empirical Bayes (EB) hyper-parameter estimator and filtered white noise input. In order to expose and find those factors, we study the convergence in distribution of the EB hyper-parameter estimator, and the asymptotic distribution of its corresponding model estimator. For illustration, we run Monte Carlo simulations to show the efficacy of our obtained theoretical results.

eess.SY↗

arXiv2021

Tutorial on Asymptotic Properties of Regularized Least Squares Estimator for Finite Impulse Response Model

In this paper, we give a tutorial on asymptotic properties of the Least Square (LS) and Regularized Least Squares (RLS) estimators for the finite impulse response model with filtered white noise inputs. We provide three perspectives: the almost sure convergence, the convergence in distribution and the boundedness in probability. On one hand, these properties deepen our understanding of the LS and RLS estimators. On the other hand, we can use them as tools to investigate asymptotic properties of other estimators, such as various hyper-parameter estimators.

math.ST↗

arXiv2021

ChunkFormer: Learning Long Time Series with Multi-stage Chunked Transformer

The analysis of long sequence data remains challenging in many real-world applications. We propose a novel architecture, ChunkFormer, that improves the existing Transformer framework to handle the challenges while dealing with long time series. Original Transformer-based models adopt an attention mechanism to discover global information along a sequence to leverage the contextual data. Long sequential data traps local information such as seasonality and fluctuations in short data sequences. In addition, the original Transformer consumes more resources by carrying the entire attention matrix during the training course. To overcome these challenges, ChunkFormer splits the long sequences into smaller sequence chunks for the attention calculation, progressively applying different chunk sizes in each stage. In this way, the proposed model gradually learns both local and global information without changing the total length of the input sequences. We have extensively tested the effectiveness of this new architecture on different business domains and have proved the advantage of such a model over the existing Transformer-based models.

cs.LG↗

arXiv2020

Supplementary Material for CDC Submission No. 1461

In this paper, we focus on the influences of the condition number of the regression matrix upon the comparison between two hyper-parameter estimation methods: the empirical Bayes (EB) and the Stein's unbiased estimator with respect to the mean square error (MSE) related to output prediction (SUREy). We firstly show that the greatest power of the condition number of the regression matrix of SUREy cost function convergence rate upper bound is always one larger than that of EB cost function convergence rate upper bound. Meanwhile, EB and SUREy hyper-parameter estimators are both proved to be asymptotically normally distributed under suitable conditions. In addition, one ridge regression case is further investigated to show that when the condition number of the regression matrix goes to infinity, the asymptotic variance of SUREy estimator tends to be larger than that of EB estimator.

math.ST↗