arXiv subjects

Joshua Snoke

Publications and source records attributed to Joshua Snoke.

10 recordsLinked to original sources

arXiv2026

How Should We Measure Empirical Risk when Synthesizing Population Data?

Synthetic data has become a prominent solution for preserving privacy while sharing data, but current empirical risk assessment frameworks fundamentally assume a sample-based context that fails to translate for the evaluation of synthetic population level datasets. This commentary explores the implications when synthesizing entire populations in order to do population-level data science, arguing that traditional metrics, such as Membership Inference Attacks (MIA) and Attribute Inference Attacks (AIA), require re-examination. First, MIA may be rendered irrelevant in contexts where population membership is public knowledge or not considered sensitive information. Second, the risk of singling out is heightened because the confidential data contain full population information. Additionally, the absence of an "out-of-sample" comparison group for attribute inference means we need to define other policies when defining acceptable inferences. Finally, we cannot rely on simply returning to subsampling prior to generating synthetic data if the use case is truly to enable population-level data science. This commentary highlights the necessity for considering context when generating and evaluating synthetic population data.

stat.AP↗

arXiv2024

But Can You Use It? Design Recommendations for Differentially Private Interactive Systems

Accessing data collected by federal statistical agencies is essential for public policy research and improving evidence-based decision making, such as evaluating the effectiveness of social programs, understanding demographic shifts, or addressing public health challenges. Differentially private interactive systems, or validation servers, can form a crucial part of the data-sharing infrastructure. They may allow researchers to query targeted statistics, providing flexible, efficient access to specific insights, reducing the need for broad data releases and supporting timely, focused research. However, they have not yet been practically implemented. While substantial theoretical work has been conducted on the privacy and accuracy guarantees of differentially private mechanisms, prior efforts have not considered usability as an explicit goal of interactive systems. This work outlines and considers the barriers to developing differentially private interactive systems for informing public policy and offers an alternative way forward. We propose balancing three design considerations: privacy assurance, statistical utility, and system usability, we develop recommendations for making differentially private interactive systems work in practice, we present an example architecture based on these recommendations, and we provide an outline of how to conduct the necessary user-testing. Our work seeks to move the practical development of differentially private interactive systems forward to better aid public policy making and spark future research.

cs.HC↗

arXiv2024

De-Biasing the Bias: Methods for Improving Disparity Assessments with Noisy Group Measurements

Health care decisions are increasingly informed by clinical decision support algorithms, but these algorithms may perpetuate or increase racial and ethnic disparities in access to and quality of health care. Further complicating the problem, clinical data often have missing or poor quality racial and ethnic information, which can lead to misleading assessments of algorithmic bias. We present novel statistical methods that allow for the use of probabilities of racial/ethnic group membership in assessments of algorithm performance and quantify the statistical bias that results from error in these imputed group probabilities. We propose a sensitivity analysis approach to estimating the statistical bias that allows practitioners to assess disparities in algorithm performance under a range of assumed levels of group probability error. We also prove theoretical bounds on the statistical bias for a set of commonly used fairness metrics and describe real-world scenarios where our theoretical results are likely to apply. We present a case study using imputed race and ethnicity from the Bayesian Improved Surname Geocoding (BISG) algorithm for estimation of disparities in a clinical decision support algorithm used to inform osteoporosis treatment. Our novel methods allow policy makers to understand the range of potential disparities under a given algorithm even when race and ethnicity information is missing and to make informed decisions regarding the implementation of machine learning for clinical decision support.

stat.ME↗

arXiv2023

Incompatibilities Between Current Practices in Statistical Data Analysis and Differential Privacy

The authors discuss their experience applying differential privacy with a complex data set with the goal of enabling standard approaches to statistical data analysis. They highlight lessons learned and roadblocks encountered, distilling them into incompatibilities between current practices in statistical data analysis and differential privacy that go beyond issues which can be solved with a noisy measurements file. The authors discuss how overcoming these incompatibilities require compromise and a change in either our approach to statistical data analysis or differential privacy that should be addressed head-on.

cs.CR↗

arXiv2021

A Feasibility Study of Differentially Private Summary Statistics and Regression Analyses with Evaluations on Administrative and Survey Data

Federal administrative data, such as tax data, are invaluable for research, but because of privacy concerns, access to these data is typically limited to select agencies and a few individuals. An alternative to sharing microlevel data is to allow individuals to query statistics without directly accessing the confidential data. This paper studies the feasibility of using differentially private (DP) methods to make certain queries while preserving privacy. We also include new methodological adaptations to existing DP regression methods for using new data types and returning standard error estimates. We define feasibility as the impact of DP methods on analyses for making public policy decisions and the queries accuracy according to several utility metrics. We evaluate the methods using Internal Revenue Service data and public-use Current Population Survey data and identify how specific data features might challenge some of these methods. Our findings show that DP methods are feasible for simple, univariate statistics but struggle to produce accurate regression estimates and confidence intervals. To the best of our knowledge, this is the first comprehensive statistical study of DP regression methodology on real, complex datasets, and the findings have significant implications for the direction of a growing research field and public policy.

stat.AP↗

arXiv2019

Comparative Study of Differentially Private Synthetic Data Algorithms from the NIST PSCR Differential Privacy Synthetic Data Challenge

Differentially private synthetic data generation offers a recent solution to release analytically useful data while preserving the privacy of individuals in the data. In order to utilize these algorithms for public policy decisions, policymakers need an accurate understanding of these algorithms' comparative performance. Correspondingly, data practitioners require standard metrics for evaluating the analytic qualities of the synthetic data. In this paper, we present an in-depth evaluation of several differentially private synthetic data algorithms using actual differentially private synthetic data sets created by contestants in the 2018-2019 National Institute of Standards and Technology Public Safety Communications Research (NIST PSCR) Division's ``Differential Privacy Synthetic Data Challenge.'' We offer analyses of these algorithms based on both the accuracy of the data they created and their usability by potential data providers. We frame the methods used in the NIST PSCR data challenge within the broader differentially private synthetic data literature. We implement additional utility metrics, including two of our own, on the differentially private synthetic data and compare mechanism utility on three categories. Our comparative assessment of the differentially private data synthesis methods and the quality metrics shows the relative usefulness, the general strengths and weaknesses, and offers preferred choices of algorithms and metrics. Finally we describe the implications of our evaluation for policymakers seeking to implement differentially private synthetic data algorithms on future data products.

stat.AP↗

arXiv2018

Using Neural Generative Models to Release Synthetic Twitter Corpora with Reduced Stylometric Identifiability of Users

We present a method for generating synthetic versions of Twitter data using neural generative models. The goal is protecting individuals in the source data from stylometric re-identification attacks while still releasing data that carries research value. Specifically, we generate tweet corpora that maintain user-level word distributions by augmenting the neural language models with user-specific components. We compare our approach to two standard text data protection methods: redaction and iterative translation. We evaluate the three methods on measures of risk and utility. We define risk following the stylometric models of re-identification, and we define utility based on two general word distribution measures and two common text analysis research tasks. We find that neural models are able to significantly lower risk over previous methods with little cost to utility. We also demonstrate that the neural models allow data providers to actively control the risk-utility trade-off through model tuning parameters. This work presents promising results for a new tool addressing the problem of privacy for free text and sharing social media data in a way that respects privacy and is ethically responsible.

cs.CL↗

arXiv2018

pMSE Mechanism: Differentially Private Synthetic Data with Maximal Distributional Similarity

We propose a method for the release of differentially private synthetic datasets. In many contexts, data contain sensitive values which cannot be released in their original form in order to protect individuals' privacy. Synthetic data is a protection method that releases alternative values in place of the original ones, and differential privacy (DP) is a formal guarantee for quantifying the privacy loss. We propose a method that maximizes the distributional similarity of the synthetic data relative to the original data using a measure known as the pMSE, while guaranteeing epsilon-differential privacy. Additionally, we relax common DP assumptions concerning the distribution and boundedness of the original data. We prove theoretical results for the privacy guarantee and provide simulations for the empirical failure rate of the theoretical results under typical computational limitations. We also give simulations for the accuracy of linear regression coefficients generated from the synthetic data compared with the accuracy of non-differentially private synthetic data and other differentially private methods. Additionally, our theoretical results extend a prior result for the sensitivity of the Gini Index to include continuous predictors.

stat.ME↗

arXiv2017

Providing Accurate Models across Private Partitioned Data: Secure Maximum Likelihood Estimation

This paper focuses on the privacy paradigm of providing access to researchers to remotely carry out analyses on sensitive data stored behind firewalls. We address the situation where the analysis demands data from multiple physically separate databases which cannot be combined. Motivating this problem are analyses using multiple data sources that currently are only possible through extension work creating a trusted user network. We develop and demonstrate a method for accurate calculation of the multivariate normal likelihood equation, for a set of parameters given the partitioned data, which can then be maximized to obtain estimates. These estimates are achieved without sharing any data or any true intermediate statistics of the data across firewalls. We show that under a certain set of assumptions our method for estimation across these partitions achieves identical results as estimation with the full data. Privacy is maintained by adding noise at each partition. This ensures each party receives noisy statistics, such that the noise cannot be removed until the last step to obtain a single value, the true total log-likelihood. Potential applications include all methods utilizing parameter estimation through maximizing the multivariate normal likelihood equation. We give detailed algorithms, along with available software, and both a real data example and simulations estimating structural equation models (SEMs) with partitioned data.

stat.ME↗

arXiv2017

General and specific utility measures for synthetic data

Data holders can produce synthetic versions of datasets when concerns about potential disclosure restrict the availability of the original records. This paper is concerned with methods to judge whether such synthetic data have a distribution that is comparable to that of the original data, what we will term general utility. We consider how general utility compares with specific utility, the similarity of results of analyses from the synthetic data and the original data. We adapt a previous general measure of data utility, the propensity score mean-squared-error (pMSE), to the specific case of synthetic data and derive its distribution for the case when the correct synthesis model is used to create the synthetic data. Our asymptotic results are confirmed by a simulation study. We also consider two specific utility measures, confidence interval overlap and standardized difference in summary statistics, which we compare with the general utility results. We present two examples examining this comparison of general and specific utility to real data syntheses and make recommendations for their use for evaluating synthetic data.

stat.AP↗