arXiv subjects

Wenge Guo

Publications and source records attributed to Wenge Guo.

At least 19 recordsLinked to original sources

arXiv2026

Weighted Holm Procedures: Theory, Properties, and Recommendations

In many statistical applications, particularly in clinical studies, hypotheses may carry different levels of importance, motivating the use of weighted multiple testing procedures (wMTPs) to control the familywise error rate (FWER). Among these approaches, two weighted Holm procedures are commonly used: the weighted Holm procedure (WHP), which is based on ordered weighted $p$-values, and the weighted alternative Holm procedure (WAP), which relies on ordered raw $p$-values. This paper provides a systematic comparison of these two procedures, along with practical recommendations for their use. We first examine their corresponding closed testing procedures (CTPs) and show that WHP is uniformly more powerful than WAP. We further investigate their structural properties, demonstrating that WAP, while consonant, lacks monotonicity. To facilitate communication with non-statisticians, we introduce graphical representations of both procedures using a common initial graph and distinct updating strategies. In addition, we derive adjusted $p$-values and adjusted weighted $p$-values for both methods. Finally, we establish an optimality result: WHP cannot be improved by enlarging any of its critical values without violating FWER control, whereas WAP is optimal only under specific conditions. Simulation studies support these theoretical findings and highlight the superior FWER control and average power of WHP.

stat.ME↗

arXiv2026

Conformal Risk Control under Non-Monotone Losses: Theory and Finite-Sample Guarantees

Conformal risk control (CRC) provides distribution-free guarantees for controlling the expected loss at a user-specified level. Existing theory typically assumes that the loss decreases monotonically with a tuning parameter that governs the size of the prediction set. However, this assumption is often violated in practice, where losses may behave non-monotonically due to competing objectives such as coverage and efficiency. In this paper, we study CRC under non-monotone loss functions when the tuning parameter is selected from a finite grid, a setting commonly arising in thresholding and discretized decision rules. Revisiting a known counterexample, we show that the validity of CRC without monotonicity depends critically on the relationship between the calibration sample size and the grid resolution. In particular, reliable risk control can still be achieved when the calibration sample is sufficiently large relative to the grid size. We establish a finite-sample guarantee for bounded losses over a grid of size $m$, showing that the excess risk above the target level $\alpha$ scales on the order of $\sqrt{\log(m)/n}$, where $n$ is the calibration sample size. A matching lower bound demonstrates that this rate is minimax optimal. We also derive refined guarantees under additional structural conditions, including Lipschitz continuity and monotonicity, and extend the analysis to settings with distribution shift via importance weighting. Numerical experiments on synthetic multilabel classification and real object detection data illustrate the practical implications of non-monotonicity. Methods that explicitly account for finite-sample uncertainty achieve more stable risk control than approaches based on monotonicity transformations, while maintaining competitive prediction set sizes.

stat.ML↗

arXiv2025

Selective Conformal Risk Control

Reliable uncertainty quantification is essential for deploying machine learning systems in high-stakes domains. Conformal prediction provides distribution-free coverage guarantees but often produces overly large prediction sets, limiting its practical utility. To address this issue, we propose \textit{Selective Conformal Risk Control} (SCRC), a unified framework that integrates conformal prediction with selective classification. The framework formulates uncertainty control as a two-stage problem: the first stage selects confident samples for prediction, and the second stage applies conformal risk control on the selected subset to construct calibrated prediction sets. We develop two algorithms under this framework. The first, SCRC-T, preserves exchangeability by computing thresholds jointly over calibration and test samples, offering exact finite-sample guarantees. The second, SCRC-I, is a calibration-only variant that provides PAC-style probabilistic guarantees while being more computational efficient. Experiments on two public datasets show that both methods achieve the target coverage and risk levels, with nearly identical performance, while SCRC-I exhibits slightly more conservative risk control but superior computational practicality. Our results demonstrate that selective conformal risk control offers an effective and efficient path toward compact, reliable uncertainty quantification.

cs.LG↗

arXiv2024

Uncertainty Quantification With Multiple Sources

Weighted conformal prediction (WCP) has been commonly used to quantify prediction uncertainty under covariate shift. However, the effectiveness of WCP relies heavily on the degree of overlap between the training and test covariate distributions. This challenge is exacerbated in multi-source settings with varying covariate distributions, where direct application of WCP can be impractical. In this paper, we address the multi-source setup by leveraging WCP under the assumption of a shared conditional distribution. We investigate two extensions of WCP: (i) a merge-based aggregation of source-specific weighted conformal prediction sets, and (ii) a data-pooling strategy that jointly reweights samples across all sources. Theoretical guarantees are provided for the proposed approaches, and experiments are conducted based on a synthetic regression task and a multi-domain image classification benchmark to validate our proposed methods.

stat.ME↗

arXiv2024

Conformal Risk Control for Ordinal Classification

As a natural extension to the standard conformal prediction method, several conformal risk control methods have been recently developed and applied to various learning problems. In this work, we seek to control the conformal risk in expectation for ordinal classification tasks, which have broad applications to many real problems. For this purpose, we firstly formulated the ordinal classification task in the conformal risk control framework, and provided theoretic risk bounds of the risk control method. Then we proposed two types of loss functions specially designed for ordinal classification tasks, and developed corresponding algorithms to determine the prediction set for each case to control their risks at a desired level. We demonstrated the effectiveness of our proposed methods, and analyzed the difference between the two types of risks on three different datasets, including a simulated dataset, the UTKFace dataset and the diabetic retinopathy detection dataset.

cs.LG↗

arXiv2024

Multi-label Classification under Uncertainty: A Tree-based Conformal Prediction Approach

Multi-label classification is a common challenge in various machine learning applications, where a single data instance can be associated with multiple classes simultaneously. The current paper proposes a novel tree-based method for multi-label classification using conformal prediction and multiple hypothesis testing. The proposed method employs hierarchical clustering with labelsets to develop a hierarchical tree, which is then formulated as a multiple-testing problem with a hierarchical structure. The split-conformal prediction method is used to obtain marginal conformal $p$-values for each tested hypothesis, and two \textit{hierarchical testing procedures} are developed based on marginal conformal $p$-values, including a hierarchical Bonferroni procedure and its modification for controlling the family-wise error rate. The prediction sets are thus formed based on the testing outcomes of these two procedures. We establish a theoretical guarantee of valid coverage for the prediction sets through proven family-wise error rate control of those two procedures. We demonstrate the effectiveness of our method in a simulation study and two real data analysis compared to other conformal methods for multi-label classification.

stat.ME↗

arXiv2024

Two-stage Risk Control with Application to Ranked Retrieval

Practical machine learning systems often operate in multiple sequential stages, as seen in ranking and recommendation systems, which typically include a retrieval phase followed by a ranking phase. Effectively assessing prediction uncertainty and ensuring effective risk control in such systems pose significant challenges due to their inherent complexity. To address these challenges, we developed two-stage risk control methods based on the recently proposed learn-then-test (LTT) and conformal risk control (CRC) frameworks. Unlike the methods in prior work that address multiple risks, our approach leverages the sequential nature of the problem, resulting in reduced computational burden. We provide theoretical guarantees for our proposed methods and design novel loss functions tailored for ranked retrieval tasks. The effectiveness of our approach is validated through experiments on two large-scale, widely-used datasets: MSLR-Web and Yahoo LTRC.

cs.IR↗

arXiv2024

Distribution-free Conformal Prediction for Ordinal Classification

Conformal prediction is a general distribution-free approach for constructing prediction sets combined with any machine learning algorithm that achieve valid marginal or conditional coverage in finite samples. Ordinal classification is common in real applications where the target variable has natural ordering among the class labels. In this paper, we discuss constructing distribution-free prediction sets for such ordinal classification problems by leveraging the ideas of conformal prediction and multiple testing with FWER control. Newer conformal prediction methods are developed for constructing contiguous and non-contiguous prediction sets based on marginal and conditional (class-specific) conformal $p$-values, respectively. Theoretically, we prove that the proposed methods respectively achieve satisfactory levels of marginal and class-specific conditional coverages. Through simulation study and real data analysis, these proposed methods show promising performance compared to the existing conformal method.

stat.ME↗

arXiv2019

Familywise Error Rate Controlling Procedures for Discrete Data

In applications such as clinical safety analysis, the data of the experiments usually consists of frequency counts. In the analysis of such data, researchers often face the problem of multiple testing based on discrete test statistics, aimed at controlling family-wise error rate (FWER). Most existing FWER controlling procedures are developed for continuous data, which are often conservative when analyzing discrete data. By using minimal attainable $p$-values, several FWER controlling procedures have been specifically developed for discrete data in the literature. In this paper, by utilizing known marginal distributions of true null $p$-values, three more powerful stepwise procedures are developed, which are modified versions of the conventional Bonferroni, Holm and Hochberg procedures, respectively. It is shown that the first two procedures strongly control the FWER under arbitrary dependence and are more powerful than the existing Tarone-type procedures, while the last one only ensures control of the FWER in special settings. Through extensive simulation studies, we provide numerical evidence of superior performance of the proposed procedures in terms of the FWER control and minimal power. A real clinical safety data is used to demonstrate applications of our proposed procedures. An R package "MHTdiscrete" and a web application are developed for implementing the proposed procedures.

stat.ME↗

arXiv2018

A Graphical Framework for Testing Hierarchically Structured Hypothesis Families

In clinical trials, hypotheses are frequently organized into hierarchically ordered families, requiring specialized testing strategies that account for these structured relationships. Existing gatekeeping methods-including serial, parallel, and tree-structured approaches-provide important solutions but are often either too rigid or insufficiently intuitive to accommodate increasingly complex logical dependencies among hypothesis families. To address these limitations, we propose a novel family-based graphical approach that unifies the derivation and visualization of diverse gatekeeping strategies. In this framework, procedures are represented as directed, weighted graphs, where nodes correspond to hypothesis families. Two simple updating rules govern the allocation of significance levels within families and the propagation of significance levels between them. We establish that the proposed method strongly controls the familywise error rate (FWER) at a pre-specified level. Simulation studies under representative configurations indicate that the proposed procedure achieves performance comparable to hypothesis-level graphical approaches and competitive with the superchain procedure, while providing a simpler and more interpretable family-level representation. Case studies and a real clinical trial application further illustrate its flexibility and practical advantages, making it a powerful tool for managing hierarchically structured multiple testing in clinical research.

stat.ME↗

arXiv2018

A New Approach for Large Scale Multiple Testing with Application to FDR Control for Graphically Structured Hypotheses

In many large scale multiple testing applications, the hypotheses often have a known graphical structure, such as gene ontology in gene expression data. Exploiting this graphical structure in multiple testing procedures can improve power as well as aid in interpretation. However, incorporating the structure into large scale testing procedures and proving that an error rate, such as the false discovery rate (FDR), is controlled can be challenging. In this paper, we introduce a new general approach for large scale multiple testing, which can aid in developing new procedures under various settings with proven control of desired error rates. This approach is particularly useful for developing FDR controlling procedures, which is simplified as the problem of developing per-family error rate (PFER) controlling procedures. Specifically, for testing hypotheses with a directed acyclic graph (DAG) structure, by using the general approach, under the assumption of independence, we first develop a specific PFER controlling procedure and based on this procedure, then develop a new FDR controlling procedure, which can preserve the desired DAG structure among the rejected hypotheses. Through a small simulation study and a real data analysis, we illustrate nice performance of the proposed FDR controlling procedure for DAG-structured hypotheses.

stat.ME↗

arXiv2018

Multi-cavity ultra-stable laser towards 10-18

In this letter, we demonstrate a technique of making an ultrastable laser referenced to a multi-cavity, corresponding to a lower thermal noise limit due to the larger equivalent beam size. The multi-cavity consists of several pairs of mirrors and a common spacer. We can stabilize the laser frequencies on these cavities, and average the laser frequencies with synthesizing technique. In comparison with a single cavity system, relative frequency instability of the synthesized laser can be improved by a factor of the squire root of the cavity number (n). In addition, we perform an experiment to simulate a two-cavity system. Experimental results show that frequency instability of the synthesized laser is improved by a factor of 1.4, and discrimination of the laser frequency instability, introduced by the process of lasers synthesizing, is negligible, and can reach a floor at low level 10-18 limited by noise of currently used signal generators. This technique is comparable with other techniques; thus, it can gain a factor of the squre root of n on the frequency instability of an ultrastable laser to an unprecedented level.

physics.ins-det↗

arXiv2017

Control of Directional Errors in Fixed Sequence Multiple Testing

In this paper, we consider the problem of simultaneously testing many two-sided hypotheses when rejections of null hypotheses are accompanied by claims of the direction of the alternative. The fundamental goal is to construct methods that control the mixed directional familywise error rate, which is the probability of making any type 1 or type 3 (directional) error. In particular, attention is focused on cases where the hypotheses are ordered as $H_1 , \ldots, H_n$, so that $H_{i+1}$ is tested only if $H_1 , \ldots, H_i$ have all been previously rejected. In this situation, one can control the usual familywise error rate under arbitrary dependence by the basic procedure which tests each hypothesis at level $α$, and no other multiplicity adjustment is needed. However, we show that this is far too liberal if one also accounts for directional errors. But, by imposing certain dependence assumptions on the test statistics, one can retain the basic procedure. Through a simulation study and a clinical trial example, we numerically illustrate good performance of the proposed procedures compared to the existing mdFWER controlling procedures. The proposed procedures are also implemented in the R-package FixSeqMTP.

math.ST↗

arXiv2017

Analysis of error control in large scale two-stage multiple hypothesis testing

When dealing with the problem of simultaneously testing a large number of null hypotheses, a natural testing strategy is to first reduce the number of tested hypotheses by some selection (screening or filtering) process, and then to simultaneously test the selected hypotheses. The main advantage of this strategy is to greatly reduce the severe effect of high dimensions. However, the first screening or selection stage must be properly accounted for in order to maintain some type of error control. In this paper, we will introduce a selection rule based on a selection statistic that is independent of the test statistic when the tested hypothesis is true. Combining this selection rule and the conventional Bonferroni procedure, we can develop a powerful and valid two-stage procedure. The introduced procedure has several nice properties: (i) it completely removes the selection effect; (ii) it reduces the multiplicity effect; (iii) it does not "waste" data while carrying out both selection and testing. Asymptotic power analysis and simulation studies illustrate that this proposed method can provide higher power compared to usual multiple testing methods while controlling the Type 1 error rate. Optimal selection thresholds are also derived based on our asymptotic analysis.

stat.ME↗

arXiv2016

On Procedures Controlling the FDR for Testing Hierarchically Ordered Hypotheses

Complex large-scale studies, such as those related to microarray data and fMRI studies, often involve testing multiple hierarchically ordered hypotheses. However, most existing false discovery rate (FDR) controlling procedures do not exploit the inherent hierarchical structure among the tested hypotheses. In this paper, we first present a generalized stepwise procedure which generalizes the usual stepwise procedure to the case where each hypothesis is tested with a different set of critical constants. This procedure is helpful in creating a general framework under which our hierarchical testing procedures are developed. Then, we present several hierarchical testing procedures which control the FDR under various forms of dependence such as positive dependence and block dependence. Our simulation studies show that these proposed methods can be more powerful in some situations than alternative methods such as Yekutieli's hierarchical testing procedure (Yekutieli, \emph{JASA} \textbf{103} (2008) 309-316). Finally, we apply our proposed procedures to a real data set involving abundances of microbes in different ecological environments.

stat.ME↗

arXiv2016

The Control of the False Discovery Rate in Fixed Sequence Multiple Testing

Controlling the false discovery rate (FDR) is a powerful approach to multiple testing. In many applications, the tested hypotheses have an inherent hierarchical structure. In this paper, we focus on the fixed sequence structure where the testing order of the hypotheses has been strictly specified in advance. We are motivated to study such a structure, since it is the most basic of hierarchical structures, yet it is often seen in real applications such as statistical process control and streaming data analysis. We first consider a conventional fixed sequence method that stops testing once an acceptance occurs, and develop such a method controlling the FDR under both arbitrary and negative dependencies. The method under arbitrary dependency is shown to be unimprovable without losing control of the FDR and unlike existing FDR methods; it cannot be improved even by restricting to the usual positive regression dependence on subset (PRDS) condition. To account for any potential mistakes in the ordering of the tests, we extend the conventional fixed sequence method to one that allows more but a given number of acceptances. Simulation studies show that the proposed procedures can be powerful alternatives to existing FDR controlling procedures. The proposed procedures are illustrated through a real data set from a microarray experiment.

stat.ME↗

arXiv2016

Adaptive Controls of FWER and FDR Under Block Dependence

Often in multiple testing, the hypotheses appear in non-overlapping blocks with the associated $p$-values exhibiting dependence within but not between blocks. We consider adapting the Benjamini-Hochberg method for controlling the false discovery rate (FDR) and the Bonferroni method for controlling the familywise error rate (FWER) to such dependence structure without losing their ultimate controls over the FDR and FWER, respectively, in a non-asymptotic setting. We present variants of conventional adaptive Benjamini-Hochberg and Bonferroni methods with proofs of their respective controls over the FDR and FWER. Numerical evidence is presented to show that these new adaptive methods can capture the present dependence structure more effectively than the corresponding conventional adaptive methods. This paper offers a solution to the open problem of constructing adaptive FDR and FWER controlling methods under dependence in a non-asymptotic setting and providing real improvements over the corresponding non-adaptive ones.

stat.ME↗

arXiv2016

Bonferroni-based gatekeeping procedure with retesting option

In complex clinical trials, multiple research objectives are often grouped into sets of objectives based on their inherent hierarchical relationships. Consequently, the hypotheses formulated to address these objectives are grouped into ordered families of hypotheses and thus to be tested in a pre-defined sequence. In this paper, we introduce a novel Bonferroni based multiple testing procedure for testing hierarchically ordered families of hypotheses. The proposed procedure allows the families to be sequentially tested more than once with updated local critical values. It is proved to control the global familywise error rate strongly under arbitrary dependence. Implementation of the procedure is illustrated using two examples. Finally, the procedure is extended to testing multiple families of hypotheses with a complex two-layer hierarchical structure.

stat.ME↗