Valid post-selection inference in Robust Q-learning

Abstract

Q-learning facilitates the development of an optimal adaptive treatment strategy through stagewise regression on a pre-specified set of tailoring variables and confounders. Semiparametric robust Q-learning eliminates the residual confounding that can occur when parametric working models for confounding influences are misspecified. However, in the presence of many potential tailoring variables, constructing an optimal adaptive treatment strategy using either approach may lead to including extraneous variables that contribute little or no benefit while increasing implementation costs, thereby placing an undue burden on patients. Using data-driven selection processes to identify a smaller set of informative prognostic factors is straightforward; however, proper statistical inference must account for this selection process. In this paper, we adapt the Universal Post-Selection Inference (UPoSI) procedure to the semiparametric Robust Q-learning method. UPoSI, introduced for use with linear models, allows for very general variable selection mechanisms. Our approach addresses the unique challenges stemming from the use of UPoSI with semiparametric multistage decision methods. Theoretical and simulation results demonstrate the validity of the proposed confidence regions. We illustrate our proposed methods through an application to adaptive treatment strategy estimation for substance abuse.