A Novel Tool to Evaluate the Accuracy of Predicting Survival in Cystic Fibrosis

Abstract

Background: Effective allocation of limited donor lungs in cystic fibrosis (CF) requires accurate survival predictions, so that high-risk patients may be prioritized for transplantation. In practice, decisions about allocation are made dynamically, using routinely updated assessments. We present a novel tool for evaluating risk prediction models that, unlike traditional methods, captures the dynamic nature of decision-making. Methods: Predicted risk is used as a score to rank incident deaths versus patients who survive, with the goal of ranking the deaths higher. The mean rank across deaths at a given time measures time-specific predictive accuracy; when assessed over time, it reflects time-varying accuracy. Results: Applying this approach to CF Registry data on patients followed from 1993-2011, we show that traditional methods do not capture the performance of models used dynamically in the clinical setting. Previously proposed multivariate risk scores perform no better than forced expiratory volume in 1 second as a percentage of predicted normal (FEV1%) alone. Despite its value for survival prediction, FEV1% has a low sensitivity of 45% over time (for fixed specificity of 95%), leaving room for improvement in prediction. Finally, prediction accuracy with annually-updated FEV1% shows minor differences compared to FEV1% updated every 2 years, which may have clinical implications regarding the optimal frequency of updating clinical information. Conclusions: It is imperative to continue to develop models that accurately predict survival in CF. Our proposed approach can serve as the basis for evaluating the predictive ability of these models by better accounting for their dynamic clinical use.