arXiv subjects

Falong Tan

Publications and source records attributed to Falong Tan.

15 recordsLinked to original sources

arXiv2026

A Unified Framework for In-Context Learning with Causal and Masked Language Models

In-context learning (ICL) has emerged as a central capability of pretrained language models, yet its theoretical analysis has focused primarily on causal language models trained by left-to-right autoregressive prediction, such as GPT-style models. Masked language models instead recover masked tokens from bidirectional context, and their role in ICL remains less understood. We develop a statistical learning framework that represents the context examples by their empirical measure and models prediction as a function of the context and the query. This formulation places autoregressive and masked pretraining objectives within a common excess-risk analysis. Under Wasserstein-type regularity conditions, we relate pretraining with T tasks and N samples per task to k-shot excess risk at inference, obtaining same-order upper bounds for masked and autoregressive objectives. We also study task-distribution shift, where pretraining tasks are sampled from P and inference tasks from Q; the resulting bound contains an additional term controlled by the lifted Wasserstein distance between P and Q. The bounds further imply an order-optimal allocation under a fixed pretraining data budget and refined rates under intrinsic low-dimensional structure. Experiments on controlled function-learning tasks show that the Masked Pair Encoder (MPE) can achieve performance comparable to GPT-2-style causal Transformers, suggesting that ICL behavior is not specific to causal language models.

cs.LG↗

arXiv2026

Asymptotic Distribution-Free Tests for Ultra-high Dimensional Parametric Regressions via Projected Empirical Processes and $p$-value Combination

This paper develops a novel methodology for testing the goodness-of-fit of sparse parametric regression models based on projected empirical processes and p-value combination, where the covariate dimension may substantially exceed the sample size. In such ultra-high dimensional settings, traditional empirical process-based tests often fail due to the curse of dimensionality or their reliance on the asymptotic linearity and normality of parameter estimators--properties that may not hold under ultra-high dimensional scenarios. To overcome these challenges, we first extend the classic martingale transformation to ultra-high dimensional settings under mild conditions and construct a Cramer-von Mises type test based on a martingale-transformed, projected residual-marked empirical process for any projection on the unit sphere. The martingale transformation renders this projected test asymptotically distribution-free and enables us to derive its limiting distribution using only standard convergence rates of parameter estimators. While the projected test is consistent for almost all projections on the unit sphere under mild conditions, it may still suffer from power loss for specific projections. Therefore, we further employ powerful p-value combination procedures, such as the Cauchy combination, to aggregate p-values across multiple projections, thereby enhancing overall robustness. Furthermore, recognizing that empirical process-based tests excel at detecting low-frequency signals while local smoothing tests are generally superior for high-frequency alternatives, we propose a novel hybrid test that aggregates both approaches using Cauchy combination. The resulting hybrid test is powerful against both low-frequency and high-frequency alternatives. $\cdots$

math.ST↗

arXiv2025

In-Context Learning as Nonparametric Conditional Probability Estimation: Risk Bounds and Optimality

This paper investigates the expected excess risk of in-context learning (ICL) for multiclass classification. We formalize each task as a sequence of labeled examples followed by a query input; a pretrained model then estimates the query's conditional class probabilities. The expected excess risk is defined as the average truncated Kullback-Leibler (KL) divergence between the predicted and true conditional class distributions over a specified family of tasks. We establish a new oracle inequality for this risk, based on KL divergence, in multiclass classification. This yields tight upper and lower bounds for transformer-based models, showing that the ICL estimator achieves the minimax optimal rate (up to logarithmic factors) for conditional probability estimation. From a technical standpoint, our results introduce a novel method for controlling generalization error via uniform empirical entropy. We further demonstrate that multilayer perceptrons (MLPs) can also perform ICL and attain the same optimal rate (up to logarithmic factors) under suitable assumptions, suggesting that effective ICL need not be exclusive to transformer architectures.

stat.ML↗

arXiv2024

A Two-Step Projection-Based Goodness-of-Fit Test for Ultra-High Dimensional Sparse Regressions

This paper proposes a novel two-step strategy for testing the goodness-of-fit of parametric regression models in ultra-high dimensional sparse settings, where the predictor dimension far exceeds the sample size. This regime usually renders existing goodness-of-fit tests for regressions infeasible, primarily due to the curse of dimensionality or their reliance on the asymptotic linearity and normality of parameter estimators -- properties that may no longer hold under ultra-high dimensional settings. To address these limitations, our strategy first constructs multiple test statistics based on projected predictors from distinct projections and establishes their asymptotic properties under both the null and alternative hypotheses. This projection-based approach significantly mitigates the dimensionality problem, enabling our tests to detect local alternatives converging to the null at the rate as if the predictor were univariate. An important finding is that the resulting test statistics based on linearly independent projections are asymptotically independent under the null hypothesis. Based on this, our second step employs powerful $p$-value combination procedures, such as the minimum $p$-value and the Fisher combination of $p$-value, to form our final tests and enhance power. Theoretically, our tests only require the standard convergence rate of parameter estimators to derive their limiting distributions, thereby circumventing the need for asymptotic linearity or normality of parameter estimators. Simulations and real-data applications confirm that our approach provides robust and powerful goodness-of-fit testing in ultra-high dimensional settings.

stat.ME↗

arXiv2024

Variable Selection for Multi-Source Count Data with Controlled False Discovery Rate

The rapid generation of complex, highly skewed, and zero-inflated multi-source count data poses significant challenges for variable selection, particularly in biomedical domains like tumor development and metabolic dysregulation. To address this, we propose a new variable selection method, Zero-Inflated Poisson-Gamma Simultaneous Knockoff (ZIPG-SK), specifically designed for multi-source count data. Our method leverages a gaussian copula based on the Zero-Inflated Poisson-Gamma (ZIPG) distribution to construct knockoffs that properly account for the properties of count data, including high skewness and zero inflation, while effectively incorporating covariate information. This framework enables the detection of common features across multi-source datasets with guaranteed false discovery rate (FDR) control. Furthermore, we enhance the power of the method by incorporating e-value aggregation, which effectively mitigates the inherent randomness in knockoff generation. Through extensive simulations, we demonstrate that ZIPG-SK significantly outperforms existing methods, achieving superior power across various scenarios. We validate the utility of our method on real-world colorectal cancer (CRC) and type 2 diabetes (T2D) datasets, identifying key variables whose characteristics align with established findings and simultaneously provide new mechanistic insights.

stat.AP↗

arXiv2024

DSDE: Using Proportion Estimation to Improve Model Selection for Out-of-Distribution Detection

Model library is an effective tool for improving the performance of single-model Out-of-Distribution (OoD) detector, mainly through model selection and detector fusion. However, existing methods in the literature do not provide uncertainty quantification for model selection results. Additionally, the model ensemble process primarily focuses on controlling the True Positive Rate (TPR) while neglecting the False Positive Rate (FPR). In this paper, we emphasize the significance of the proportion of models in the library that identify the test sample as an OoD sample. This proportion holds crucial information and directly influences the error rate of OoD detection.To address this, we propose inverting the commonly-used sequential p-value strategies. We define the rejection region initially and then estimate the error rate. Furthermore, we introduce a novel perspective from change-point detection and propose an approach for proportion estimation with automatic hyperparameter selection. We name the proposed approach as DOS-Storey-based Detector Ensemble (DSDE). Experimental results on CIFAR10 and CIFAR100 demonstrate the effectiveness of our approach in tackling OoD detection challenges. Specifically, the CIFAR10 experiments show that DSDE reduces the FPR from 11.07% to 3.31% compared to the top-performing single-model detector.

stat.ML↗

arXiv2024

Enhancing Out-of-Distribution Detection with Multitesting-based Layer-wise Feature Fusion

Deploying machine learning in open environments presents the challenge of encountering diverse test inputs that differ significantly from the training data. These out-of-distribution samples may exhibit shifts in local or global features compared to the training distribution. The machine learning (ML) community has responded with a number of methods aimed at distinguishing anomalous inputs from original training data. However, the majority of previous studies have primarily focused on the output layer or penultimate layer of pre-trained deep neural networks. In this paper, we propose a novel framework, Multitesting-based Layer-wise Out-of-Distribution (OOD) Detection (MLOD), to identify distributional shifts in test samples at different levels of features through rigorous multiple testing procedure. Our approach distinguishes itself from existing methods as it does not require modifying the structure or fine-tuning of the pre-trained classifier. Through extensive experiments, we demonstrate that our proposed framework can seamlessly integrate with any existing distance-based inspection method while efficiently utilizing feature extractors of varying depths. Our scheme effectively enhances the performance of out-of-distribution detection when compared to baseline methods. In particular, MLOD-Fisher achieves superior performance in general. When trained using KNN on CIFAR10, MLOD-Fisher significantly lowers the false positive rate (FPR) from 24.09% to 7.47% on average compared to merely utilizing the features of the last layer.

cs.LG↗

arXiv2023

LRBmat: A Novel Gut Microbial Interaction and Individual Heterogeneity Inference Method for Colorectal Cancer

Many diseases are considered to be closely related to the changes in the gut microbial community, including colorectal cancer (CRC), which is one of the most common cancers in the world. The diagnostic classification and etiological analysis of CRC are two critical issues worthy of attention. Many methods adopt gut microbiota to solve it, but few of them simultaneously take into account the complex interactions and individual heterogeneity of gut microbiota, which are two common and important issues in genetics and intestinal microbiology, especially in high-dimensional cases. In this paper, a novel method with a Binary matrix based on Logistic Regression (LRBmat) is proposed to deal with the above problem. The binary matrix can directly weakened or avoided the influence of heterogeneity, and also contain the information about gut microbial interactions with any order. Moreover, LRBmat has a powerful generalization, it can combine with any machine learning method and enhance them. The real data analysis on CRC validates the proposed method, which has the best classification performance compared with the state-of-the-art. Furthermore, the association rules extracted from the binary matrix of the real data align well with the biological properties and existing literatures, which are helpful for the etiological analysis of CRC. The source codes for LRBmat are available at https://github.com/tsnm1/LRBmat.

q-bio.QM↗

arXiv2022

Boosting Out-of-Distribution Detection with Multiple Pre-trained Models

Out-of-Distribution (OOD) detection, i.e., identifying whether an input is sampled from a novel distribution other than the training distribution, is a critical task for safely deploying machine learning systems in the open world. Recently, post hoc detection utilizing pre-trained models has shown promising performance and can be scaled to large-scale problems. This advance raises a natural question: Can we leverage the diversity of multiple pre-trained models to improve the performance of post hoc detection methods? In this work, we propose a detection enhancement method by ensembling multiple detection decisions derived from a zoo of pre-trained models. Our approach uses the p-value instead of the commonly used hard threshold and leverages a fundamental framework of multiple hypothesis testing to control the true positive rate of In-Distribution (ID) data. We focus on the usage of model zoos and provide systematic empirical comparisons with current state-of-the-art methods on various OOD detection benchmarks. The proposed ensemble scheme shows consistent improvement compared to single-model detectors and significantly outperforms the current competitive methods. Our method substantially improves the relative performance by 65.40% and 26.96% on the CIFAR10 and ImageNet benchmarks.

cs.LG↗

arXiv2022

Testing the parametric form of the conditional variance in regressions based on distance covariance

In this paper, we propose a new test for checking the parametric form of the conditional variance based on distance covariance in nonlinear and nonparametric regression models. Inherit from the nice properties of distance covariance, our test is very easy to implement in practice and less effected by the dimensionality of covariates. The asymptotic properties of the test statistic are investigated under the null and alternative hypotheses. We show that the proposed test is consistent against any alternative and can detect local alternatives converging to the null hypothesis at the parametric rate 1/root(n) in both the nonlinear and nonparametric settings. As the limiting null distribution of the test statistic is intractable, we propose a residual bootstrap to approximate the limiting null distribution. Simulation studies are presented to assess the finite sample performance of the proposed test. We also apply the proposed test to a real data set for illustration.

stat.ME↗

arXiv2022

Weighted residual empirical processes, martingale transformations, and model specification tests for regressions with diverging number of parameters

This paper explores hypothesis testing for the parametric forms of the mean and variance functions in regression models under diverging-dimension settings. To mitigate the curse of dimensionality, we introduce weighted residual empirical process-based tests, both with and without martingale transformations. The asymptotic properties of these tests are derived from the behavior of weighted residual empirical processes and their martingale transformations under the null and alternative hypotheses. The proposed tests without martingale transformations achieve the fastest possible rate of detecting local alternatives, specifically of order $n^{-1/2}$, which is unaffected by dimensionality. However, these tests are not asymptotically distribution-free. To address this limitation, we propose a smooth residual bootstrap approximation and establish its validity in diverging-dimension settings. In contrast, tests incorporating martingale transformations are asymptotically distribution-free but exhibit an unexpected limitation: they can only detect local alternatives converging to the null at a much slower rate of order $n^{-1/4}$, which remains independent of dimensionality. This finding reveals a theoretical advantage in the power of tests based on weighted residual empirical process without martingale transformations over their martingale-transformed counterparts, challenging the conventional wisdom of existing asymptotically distribution-free tests based on martingale transformations. To validate our approach, we conduct simulation studies and apply the proposed tests to a real-world dataset, demonstrating their practical effectiveness.

math.ST↗

arXiv2019

Integrated conditional moment test and beyond: when the number of covariates is divergent

The classic integrated conditional moment test is a promising method for testing regression model misspecification. However, it severely suffers from the curse of dimensionality. To extend it to handle the testing problem for parametric multi-index models with diverging number of covariates, we investigate three issues in inference in this paper. First, we study the consistency and asymptotically linear representation of the least squares estimator of the parameter matrix at faster rates of divergence than those in the literature for nonlinear models. Second, we propose, via sufficient dimension reduction techniques, an adaptive-to-model version of the integrated conditional moment test. We study the asymptotic properties of the new test under both the null and alternative hypothesis to examine its ability of significance level maintenance and its sensitivity to the global and local alternatives that are distinct from the null at the fastest possible rate in hypothesis testing. Third, we derive the consistency of the bootstrap approximation for the new test in the diverging dimension setting. The numerical studies show that the new test can very much enhance the performance of the original ICM test in high-dimensional scenarios. We also apply the test to a real data set for illustrations.

math.ST↗

arXiv2018

Testing heteroscedasticity for regression models based on projections

In this paper we propose a new test of heteroscedasticity for parametric regression models and partial linear regression models in high dimensional settings. When the dimension of covariates is large, existing tests of heteroscedasticity perform badly due to the \curse of dimensionality". To attack this problem, we construct a test of heteroscedasticity by using a projection-based empirical process. We study the asymptotic properties of the test statistic under the null hypothesis and alternative hypotheses. It is shown that the test can detect local alternatives departure from the null hypothesis at the fastest possible rate in hypothesis testing. As the limiting null distribution of the test statistic is not distribution free, we propose a residual-based bootstrap. The validity of the bootstrap approximations is investigated. We present some simulation results to show the finite sample performances of the test. Two real data analyses are conducted for illustration.

stat.ME↗

arXiv2017

Estimation and adaptive-to-model testing for regressions with diverging number of predictors

The research described in this paper is motivated by model checking for parametric single-index models with diverging number of predictors. To construct a test statistic, we first study the asymptotic property of the estimators of involved parameters of interest under the null and alternative hypothesis when the dimension is divergent to infinity as the sample size goes to infinity. For the testing problem, we study an adaptive-to-model residual-marked empirical process as the basis for constructing a test statistic. By modifying the approach in the literature to suit the diverging dimension settings, we construct a martingale transformation. Under the null, local and global alternative hypothesis, the weak limits of the empirical process are derived and then the asymptotic properties of the test statistic are investigated. Simulation studies are carried out to examine the performance of the test.

stat.ME↗

arXiv2016

A projection-based adaptive-to-model test for regressions

A longstanding problem of existing empirical process-based tests for regressions is that when the number of covariates is greater than one, they either have no tractable limiting null distributions or are not omnibus. To attack this problem, we in this paper propose a projection-based adaptive-to-model approach. When the hypothetical model is parametric single-index, the method can fully utilize the dimension reduction model structure under the null hypothesis as if the covariate were one-dimensional such that the martingale transformation-based test can be asymptotically distribution-free. Further, the test can automatically adapt to the underlying model structure such that the test can be omnibus and thus detect alternative models distinct from the hypothetical model at the fastest possible rate in hypothesis testing. The method is examined through simulation studied and is illustrated by a real data analysis.

stat.ME↗