Signal Recovery With Multistage Tests And Without Sparsity Constraints

Abstract

A signal recovery problem is considered, where the same binary testing problem is posed over multiple, independent data streams. The goal is to identify all signals, i.e., streams where the alternative hypothesis is correct, and noises, i.e., streams where the null hypothesis is correct, subject to prescribed bounds on the classical or generalized familywise error probabilities. It is not required that the exact number of signals be a priori known, only upper bounds on the number of signals and noises are assumed instead. A decentralized formulation is adopted, according to which the sample size and the decision for each testing problem must be based only on observations from the corresponding data stream. A novel multistage testing procedure is proposed for this problem and is shown to enjoy a high-dimensional asymptotic optimality property. Specifically, it achieves the optimal, average over all streams, expected sample size, uniformly in the true number of signals, as the maximum possible numbers of signals and noises go to infinity at arbitrary rates, in the class of all sequential tests with the same global error control. In contrast, existing multistage tests in the literature are shown to achieve this high-dimensional asymptotic optimality property only under additional sparsity or symmetry conditions. These results are based on an asymptotic analysis for the fundamental binary testing problem as the two error probabilities go to zero. For this problem, unlike existing multistage tests in the literature, the proposed test achieves the optimal expected sample size under both hypotheses, in the class of all sequential tests with the same error control, as the two error probabilities go to zero at arbitrary rates. These results are further supported by simulation studies and extended to problems with non-iid data and composite hypotheses.