Regret in Treatment Choice when Welfare Varies with an Uncertain Event: The Prediction-Threshold Problem

Abstract

We study maximum regret (MR) of binary treatment choice in a population with observed covariates x, when welfare varies with an uncertain binary event. We consider decision making with plug-in probabilistic predictions of the event and pre-specified decision thresholds, which we term the prediction-threshold problem. The optimal treatment for persons with covariate value x is B if the conditional probability P(y=1|x) of a binary outcome y exceeds a particular x-specific threshold and is A otherwise. This structure is common in medical decision making and other contexts. Plug-in prediction uses data to estimate P(y|x) and acts as if the estimate is accurate. However, plug-in prediction is often performed with misspecified prediction models and conventional x-invariant thresholds. We use a combination of algebraic and computational analysis of limit and finite-sample MR to demonstrate how MR depends on the prediction model, the state space, and the thresholds used to choose treatments.