How Should We Measure Empirical Risk when Synthesizing Population Data?

Abstract

Synthetic data has become a prominent solution for preserving privacy while sharing data, but current empirical risk assessment frameworks fundamentally assume a sample-based context that fails to translate for the evaluation of synthetic population level datasets. This commentary explores the implications when synthesizing entire populations in order to do population-level data science, arguing that traditional metrics, such as Membership Inference Attacks (MIA) and Attribute Inference Attacks (AIA), require re-examination. First, MIA may be rendered irrelevant in contexts where population membership is public knowledge or not considered sensitive information. Second, the risk of singling out is heightened because the confidential data contain full population information. Additionally, the absence of an "out-of-sample" comparison group for attribute inference means we need to define other policies when defining acceptable inferences. Finally, we cannot rely on simply returning to subsampling prior to generating synthetic data if the use case is truly to enable population-level data science. This commentary highlights the necessity for considering context when generating and evaluating synthetic population data.