An Upper Bound on the Probability That a User Encounters an Undiscovered Defect

Abstract

Before releasing software to a general population, a developer must weigh a single question: if we ship now, what fraction of users will still hit a defect? This is not a question about how many defects remain, nor whether any particular defect is present -- the quantities the reliability literature has long estimated -- but about a different and, for a release decision, more consequential one: the probability that a user encounters a defect at all. We give a direct, distribution-free answer. Reading each beta-test report as a draw from the user population and each distinct defect as a class, we show that the fraction of defects reported exactly once, $s/n$, is a conservative upper bound on the probability that a user encounters a defect unseen in testing. This bound is the exact maximum-likelihood estimate of the mass of unseen defects under a general urn construction -- the canonical form -- into which any population of classes embeds; because that construction charges every singleton to the unseen reservoir, $s/n$ overstates the user's risk rather than understating it, the direction a release decision requires. The estimate needs no operational profile, no assumption on the number or frequency of defects, and no model of the program's internal structure -- since a defect's report count already reflects how many users reach it, the estimate is invariant to whether the reachability graph is a tree or a directed acyclic graph, and defects hidden behind other defects are bounded automatically. We validate the estimator against synthetic populations with known ground truth, and discuss the encounter-level data -- beta or crash telemetry -- under which the user-facing reading holds.