Rare-Feature-Adaptive Subsampling for Big Data Regression

Abstract

Rare binary features cause instability and slow convergence in regression. Subsampling massive datasets may discard observations carrying rare-feature information. We show that rare-feature coefficient convergence rates depend on feature prevalence and the inclusion probabilities assigned to rare observations. This finding motivates subsampling designs that adapt sampling probabilities to feature rarity. We first study optimal rare-census designs, which extend the negative subsampling approach by retaining every rare observation. However, they may face practical constraints. We then propose a computationally efficient rarity-adaptive L-optimal design that incorporates inverse prevalence scaling. Because implementing optimal designs requires pilot estimates whose quality depends on adequate representation of rare features, we propose a balanced design, a one-step rarity-adaptive design based only on binary features. It can be used alone, as a pilot, or combined with designs targeting other objectives. For implementation in two stages, we develop a union-sample estimator that reuses pilot observations while accounting for sample overlap and establish its asymptotic normality. Simulations and applications show improved stability and estimation efficiency over existing methods. One application evaluates designs that jointly protect rare outcomes and rare features.