Distinguishable Category Groupings in Complex Surveys: Which Categories Can the Design Tell Apart?

Abstract

Surveys often estimate the share of cases in each of several categories, such as the causes of an event or the reasons for a visit. Readers then compare the categories, but reliability rules usually check each estimate alone, not the differences between them. We ask instead which groups of categories can be told apart under the sampling design. The categories often fall into blocks, such as driver-related and vehicle-related causes. A grouping is called distinguishable when every pair of its groups in the same block differs by more than the design's minimum detectable difference, computed from the primary sampling units at the design's degrees of freedom. The maximal distinguishable groupings are the most detailed and can differ in size, while merging two groups can destroy distinguishability. So even if splitting any one group in two breaks distinguishability, a more detailed distinguishable grouping may still exist. We propose two methods. One merges categories until the grouping is distinguishable, then tests every single split. The other lists every distinguishable grouping of a block with few categories. When groups in different blocks must also differ, there might be no grouping that qualifies. An application to the National Motor Vehicle Crash Causation Survey, with 24 primary sampling units in 12 strata, gives the grouping maxima for two blocks of its categories. Of more than 27 million groupings of its 13 vehicle-related critical reasons, none with more than four groups is distinguishable, and its environment-related reasons resolve into at most four groups. Merging two groups of a distinguishable vehicle grouping destroys distinguishability in 5,239 of 31,773 cases. If groups from different blocks must also be told apart, no grouping qualifies. Analysts can apply these methods before publishing categorical comparisons from designs with few degrees of freedom.