Evaluating discriminatory accuracy of models using partial risk-scores in two-phase studies

Abstract

Prior to clinical applications, it is critical that risk prediction models are evaluated in independent studies that did not contribute to model development. While prospective cohort studies provide a natural setting for model validation, they often ascertain information on some risk factors (e.g., an expensive biomarker) in a nested sub-study of the original cohort, typically selected based on case-control status, and possibly some additional covariates. In this article, we propose an efficient approach for evaluating discriminatory ability of models using data from all individuals in a cohort study irrespective of whether they were sampled in the nested sub-study for measuring the complete set of risk factors. For evaluation of the Area Under the Curve (AUC) statistics, we estimate probabilities of risk-scores for cases being larger than those in controls conditional on partial risk-scores, the component of the risk-score that could be defined based on partial covariate information. The use of partial risk-scores, as opposed to actual multivariate risk-factor profiles, allows estimation of the underlying conditional expectations using subjects with complete covariate information in a non-parametric fashion even when numerous covariates are involved. We propose an influence function based approach for estimation of the variance of the resulting AUC statistics. We evaluate finite sample performance of the proposed method and compare it to an inverse probability weighted (IPW) estimator through extensive simulation studies. Finally, we illustrate an application of the proposed method for evaluating performance of a lung cancer risk prediction model using data from the Prostate, Lung, Colorectal and Ovarian Cancer (PLCO) trial.