Differences Among Noninformative Stopping Rules Are Often Relevant to Bayesian Decisions

Abstract

L.J. Savage once hoped to show that "the superficially incompatible systems of ideas associated on the one hand with [subjective Bayesianism] and on the other hand with [classical statistics]...lend each other mutual support and clarification." By 1972, however, he had largely "lost faith in the devices" of classical statistics. One aspect of those "devices" that he found objectionable is that differences among the "stopping rules" that are used to decide when to end an experiment which are "noninformative" from a Bayesian perspective can affect decisions made using a classical approach. Two experiments that produce the same data using different stopping rules seem to differ only in the intentions of the experimenters regarding whether or not they would have carried on if the data had been different, which seem irrelevant to the evidential import of the data and thus to facts about what actions the data warrant. I argue that classical and Bayesian ideas about stopping rules do in fact "lend each other" the kind of "mutual support and clarification" that Savage had originally hoped to find. They do so in a kind of case that is common in scientific practice, in which those who design an experiment have different interests from those who will make decisions in light of its results. I show that, in cases of this kind, Bayesian principles provide qualified support for the classical statistical practice of "penalizing" "biased" stopping rules. However, they require this practice in a narrower range of circumstances than classical principles do, and for different reasons. I argue that classical arguments for this practice are compelling in precisely the class of cases in which Bayesian principles also require it, and thus that we should regard Bayesian principles as clarifying classical statistical ideas about stopping rules rather than the reverse.