Recursive random binning to detect and display pairwise dependence

Abstract

Random binnings generated via recursive binary splits are introduced as a way to detect, measure the strength of, and to display the pattern of association between any two variates, whether one or both are continuous or categorical. This provides a single approach to ordering large numbers of variate pairs by their measure of dependence and then to examine any pattern of dependence via a common display, the departure display (colouring bins by a standardized Pearson residual). Continuous variates are first ranked and their rank pairs binned. The Pearson's goodness of fit statistic is applicable but the classic $\chi^2$ approximation to its null distribution is not. Theoretical and empirical investigations motivate several approximations, including a simple $\chi^2$ approximation with real-valued, yet intuitive, degrees of freedom. Alternatively, applying an inverse probability transform from the ranks before binning returns a simple Pearson statistic with the classic degrees of freedom. Recursive random binning with different approximations is compared to recent grid-based methods on a variety of non-null dependence patterns; the method with any of these approximations is found to be well-calibrated and relatively powerful against common test alternatives. Method and displays are illustrated by applying the screening methodology to a publicly available data set having several continuous and categorical measurements of each of 6,497 Portuguese wines. The software is publicly available as the R package AssocBin.