arXiv subjects

Zehang Richard Li

Publications and source records attributed to Zehang Richard Li.

At least 19 recordsLinked to original sources

arXiv2026

Prediction-Powered Smoothing and Validation for Disaggregated AI Evaluation

Evaluating an AI system requires disaggregated assessment, as performance varies across domains such as benchmark task types or conversation types in deployed agents. Exhaustive testing is expensive, so evaluation rests on a sample of labeled units. We treat the evaluation set as a finite population and seek accurate point and interval estimates of each domain mean. Direct estimators, including prediction-powered inference (PPI), use only a domain's own labels and are imprecise where labels are few. Small area estimation addresses this problem, and we build on it to develop an integrated workflow for estimation and validation. For estimation, we propose prediction-powered smoothing (PP-S), a Bayesian model fit to each domain's prediction-powered estimate, with an extension that borrows strength across a reporting taxonomy (PP-TS). For validation, we derive a new, approximately unbiased design-based cross-validation score for choosing among direct and smoothed estimators. We study a curated benchmark with verifiable grading and deployed agent traffic graded by humans, each with every outcome observed. In both, the proposed estimators improve on the direct estimators in point and interval estimation, with near-nominal coverage. At the same sampling budget, our score selects as well as an independent validation sample does and estimates the selected estimator's error far more accurately.

stat.ML↗

arXiv2026

Nonprobability Samples for Small Area Estimation: A Review and Comparative Simulation Study

Nonprobability samples (NPS) are attractive because they are less costly to collect, can provide substantially larger sample sizes, and may reach populations that traditional probability surveys do not. As response rates for traditional surveys fall, interest in NPS has grown rapidly within the field of survey statistics. These methods are especially relevant for small area estimation (SAE), where there is ever-present demand for estimates at fine geographic scales and detailed demographic domains. Despite rapid methodological development, there remains limited understanding of which approaches perform best under different conditions. In this paper, we review recent developments in NPS methodology, including the concept of data defect correlation (DDC) as a measure of data quality and as a tool for categorizing the various NPS methods. We then present a comprehensive simulation study that evaluates a range of NPS approaches under varying levels of DDC and extend several existing methods to the SAE setting.

stat.ME↗

arXiv2026

Bayesian Federated Cause-of-Death Classification and Quantification Under Distribution Shift

In regions lacking medically certified causes of death, verbal autopsy (VA) is a widely used tool to ascertain the cause of death through interviews with caregivers. Data collected by VAs are often analyzed using probabilistic algorithms. The performance of these algorithms often degrades due to distribution shift across populations. Most existing VA algorithms rely on centralized training, requiring full access to training data for joint modeling. This can be infeasible due to privacy and logistical constraints. In this paper, we propose a novel Bayesian Federated Learning (BFL) framework that avoids data sharing across multiple training sources. Our method supports individual-level cause-of-death classification and population-level quantification of cause-specific mortality fractions in a target domain with limited or no local labeled data. The proposed framework is modular, computationally efficient, and compatible with a wide range of existing VA algorithms as base models, facilitating flexible deployment in real-world mortality surveillance systems. We validate the performance of BFL through extensive experiments on two real-world VA datasets under varying levels of distribution shift scenarios. Our results show that BFL significantly outperforms single-domain base models and performs comparably to or better than joint modeling.

stat.ME↗

arXiv2026

On Data Thinning for Model Validation in Small Area Estimation

Small area estimation produces estimates of population parameters for geographic and demographic subgroups with limited sample sizes. Such estimates are critical for policy decisions, yet principled validation of these models remains a challenge. Unlike conventional predictive settings, validation data are rarely available. Data thinning splits a single observation into independent training and test components. It enables out-of-sample validation using only the area-level summary statistics routinely available, requiring only their Gaussianity and known sampling variances. However, the properties of thinning-based model comparison have not been formally studied. In this paper, we develop these properties. We construct an unbiased estimator of thinned-data mean squared error and show that it differs systematically from its full-data counterpart; for the standard Fay-Herriot model, the gap admits a closed-form expression that depends on the candidate model's shrinkage behavior. We further show that the estimator variance increases sharply as the training fraction approaches one, producing a bias-variance tradeoff with no universally optimal thinning parameter. Practical recommendations balancing these forces are informed by theory and verified empirically. Design-based simulations using American Community Survey microdata show that the recommended data thinning approach is competitive with information-criterion and simulation-based methods, and substantially more stable across heterogeneous sampling designs.

stat.ME↗

arXiv2026

Design-Based Cross-Validation for Comparing Small Area Estimators

Subnational monitoring of public health and development indicators often relies on household surveys where data are sparse at the desired spatial resolution. Small area estimation (SAE) methods address this challenge by borrowing strength across areas and incorporating auxiliary information. However, comparing these estimators remains difficult in the absence of ground truth. We propose a design-based cross-validation framework for evaluating small area estimators that accommodates complex survey designs. Our approach enables model-agnostic comparisons between area-level and unit-level SAE models. We derive a decomposition of the conditional mean squared error that yields a consistent cross-validation score, show that finite-sample comparisons carry an unidentifiable bias that can be bounded, and use this bound as a principled threshold for ranking models. We further show that leave-one-area-out cross-validation, a popular alternative, targets extrapolation rather than smoothing error and can reverse the correct ranking. We evaluate the framework through extensive design-based simulations. We apply the framework to compare subnational female literacy estimators in Zambia using the 2024 Demographic and Health Survey. The framework applies broadly across prevalence mapping and other SAE problems and is applicable to any small area estimator irrespective of the underlying model class.

stat.ME↗

arXiv2025

sae4health: An R Shiny Application for Small Area Estimation in Low- and Middle-Income Countries

Accurate subnational estimation of health indicators is critical for public health planning, particularly in low- and middle-income countries (LMICs), where data and analytic tools are often limited. sae4health is an open-access Shiny application (https://rsc.stat.washington.edu/sae4health/) that generates small area estimates for more than 150 demographic and health indicators, based on over 150 Demographic and Health Surveys (DHS) from 60 countries. The platform offers both area- and unit-level models with spatial random effects, implemented through fast Bayesian inference using Integrated Nested Laplace Approximation (INLA). The app is fully browser-based and requires no data input, programming skills, or statistical modeling expertise, making advanced methods accessible to a wide range of users. Estimates are processed in real time and presented as interactive maps, tables, and downloadable reports. A companion website (https://sae4health.stat.uw.edu) provides documentation and methodological background to support the app. Together, these resources enhance access to subnational health data and facilitate the use of DHS surveys for evidence-based decision making.

stat.CO↗

arXiv2025

The Two Cultures of Prevalence Mapping: Small Area Estimation and Model-Based Geostatistics

In low- and middle-income countries (LMICs), accurate estimates of subnational health and demographic indicators are critical for guiding policy and identifying disparities. Many indicators of interest are proportions of binary outcomes and the task of estimating these fractions is often called prevalence mapping. In LMICs, health and vital records data are limited, so prevalence mapping relies on data from household surveys with complex sampling designs. However, estimates are often desired at spatial resolutions at which data are insufficient. We review two families of approaches to prevalence mapping: small area estimation (SAE) methods (from the survey statistics literature) and model-based geostatistics (MBG) methods (from the spatial statistics literature). SAE models can be ``area-level" or ``unit-level" and commonly use area-specific random effects and rely upon high-quality covariate data from administrative sources. Unit-level models for binary responses are relatively underdeveloped. MBG approaches explicitly specify binary response models, incorporate continuous spatial random effects, and leverage alternative data sources, e.g., satellite imagery. SAE methods often address the design by incorporating sampling weights or modeling the sampling mechanism. Two delicate issues arise when using MBG methods. First, aggregating unit level predictions to create area-level summaries requires population-level information that is rarely available. Second, MBG approaches typically assume the sampling design is ignorable. We review both approaches, and argue that binary response models can be improved using insights from both the survey sampling and the spatial statistics literature. We highlight these issues using household survey data from the Zambia 2018 Demographic Health Survey to estimate subnational HIV prevalence for woman aged 15--49.

stat.AP↗

arXiv2025

Hierarchical Latent Class Models for Mortality Surveillance Using Partially Verified Verbal Autopsies

Monitoring cause-of-death data is an important part of understanding disease burdens and effects of public health interventions. Verbal autopsy (VA) is a well-established method for gathering information about deaths outside of hospitals by conducting an interview to caregivers of a deceased person. It is usually the only tool for cause-of-death surveillance in low-resource settings. A critical limitation with current practices of VA analysis is that all algorithms require either domain knowledge about symptom-cause relationships or large labeled datasets for model training. Therefore, they cannot be easily adopted during public health emergencies when new diseases emerge with rapidly evolving epidemiological patterns. In this paper, we consider estimating the fraction of deaths due to an emerging disease. We develop a novel Bayesian framework using hierarchical latent class models to account for the informative cause-of-death verification process. Our model flexibly captures the joint distribution of symptoms and how they change over time in different sub-populations. We also propose structured priors to improve the precision of the cause-specific mortality estimates for small sub-populations. Our model is motivated by mortality surveillance of COVID-19 related deaths in low-resource settings. We apply our method to a dataset that includes suspected COVID-19 related deaths in Brazil in 2021.

stat.ME↗

arXiv2025

Toward a Principled Workflow for Prevalence Mapping Using Household Survey Data

Understanding the prevalence of key demographic and health indicators in small geographic areas and domains is of global interest, especially in low- and middle-income countries (LMICs), where vital registration data is sparse and household surveys are the primary source of information. Recent advances in computation and the increasing availability of spatially detailed datasets have led to much progress in sophisticated statistical modeling of prevalence. As a result, high-resolution prevalence maps for many indicators are routinely produced in the literature. However, statistical and practical guidance for producing prevalence maps in LMICs has been largely lacking. In particular, advice in choosing and evaluating models and interpreting results is needed, especially when data is limited. Software and analysis tools are also usually inaccessible to researchers in low-resource settings to conduct their own analysis or reproduce findings in the literature. In this paper, we propose a general workflow for prevalence mapping using household survey data. We consider all stages of the analysis pipeline, with particular emphasis on model choice and interpretation. We illustrate the proposed workflow using a case study mapping the proportion of pregnant women who had at least four antenatal care visits in Kenya. Reproducible code is provided in the Supplementary Materials and can be readily extended to a broad collection of indicators.

stat.AP↗

arXiv2025

Flexible Bayesian Tensor Decomposition for Verbal Autopsy Data

Cause-of-death data is fundamental for understanding population health trends and inequalities as well as designing and evaluating public health interventions. A significant proportion of global deaths, particularly in low- and middle-income countries (LMICs), do not have medically certified causes assigned. In such settings, verbal autopsy (VA) is a widely adopted approach to estimate disease burdens by interviewing caregivers of the deceased. Recently, latent class models have been developed to model the joint distribution of symptoms and perform probabilistic cause-of-death assignment. A large number of latent classes are usually needed in order to characterize the complex dependence among symptoms, making the estimated symptom profiles challenging to summarize and interpret. In this paper, we propose a flexible Bayesian tensor decomposition framework that balances the predictive accuracy of the cause-of-death assignment task and the interpretability of the latent structures. The key to our approach is to partition symptoms into groups and model the joint distributions of group-level symptom sub-profiles. The proposed methods achieve better predictive accuracy than existing VA methods and provide a more parsimonious representation of the symptom distributions. We show our methods provide new insights into the clustering patterns of both symptoms and causes using the PHMRC gold-standard VA dataset.

stat.AP↗

arXiv2025

Space-Time Smoothing of Survey Outcomes using the R Package SUMMER

The increasing availability of complex survey data, and the continued need for estimates of demographic and health indicators at a fine spatial and temporal scale, which leads to issues of data sparsity, has led to the need for spatio-temporal smoothing methods that acknowledge the manner in which the data were collected. The open source R package SUMMER implements a variety of methods for spatial or spatio-temporal smoothing of survey data. The emphasis is on small-area estimation. We focus primarily on indicators in a low and middle-income countries context. Our methods are particularly useful for data from Demographic Health Surveys and Multiple Indicator Cluster Surveys. We build upon functions within the survey package, and use INLA for fast Bayesian computation. This paper includes a brief overview of these methods and illustrates the workflow of accessing and processing surveys, estimating subnational child mortality rates, and visualizing results with both simulated data and DHS surveys.

stat.AP↗

arXiv2024

Spatially Selected and Dependent Random Effects for Small Area Estimation with Application to Rent Burden

Area-level models for small area estimation typically rely on areal random effects to shrink design-based direct estimates towards a model-based predictor. Incorporating the spatial dependence of the random effects into these models can further improve the estimates when there are not enough covariates to fully account for spatial dependence of the areal means. A number of recent works have investigated models that include random effects for only a subset of areas, in order to improve the precision of estimates. However, such models do not readily handle spatial dependence. In this paper, we introduce a model that accounts for spatial dependence in both the random effects as well as the latent process that selects the effects. We show how this model can significantly improve predictive accuracy via an empirical simulation study based on data from the American Community Survey, and illustrate its properties via an application to estimate county-level median rent burden.

stat.ME↗

arXiv2024

Bayesian analysis of verbal autopsy data using factor models with age- and sex-dependent associations between symptoms

Verbal autopsies (VAs) are extensively used to investigate the population-level distributions of deaths by cause in low-resource settings without well-organized vital statistics systems. Computer-based methods are often adopted to assign causes of death to deceased individuals based on the interview responses of their family members or caregivers. In this article, we develop a new Bayesian approach that extracts information about cause-of-death distributions from VA data considering the age- and sex-related variation in the associations between symptoms. Its performance is compared with that of existing approaches using gold-standard data from the Population Health Metrics Research Consortium. In addition, we compute the relevance of predictors to causes of death based on information-theoretic measures.

stat.AP↗

arXiv2024

Treatment Effect Estimation Amidst Dynamic Network Interference in Online Gaming Experiments

The evolving landscape of online multiplayer gaming presents unique challenges in assessing the causal impacts of game features. Traditional A/B testing methodologies fall short due to complex player interactions, leading to violations of fundamental assumptions like the Stable Unit Treatment Value Assumption (SUTVA). Unlike traditional social networks with stable and long-term connections, networks in online games are often dynamic and short-lived. Players are temporarily teamed up for the duration of a game, forming transient networks that dissolve once the game ends. This fleeting nature of interactions presents a new challenge compared with running experiments in a stable social network. This study introduces a novel framework for treatment effect estimation in online gaming environments, considering the dynamic and ephemeral network interference that occurs among players. We propose an innovative estimator tailored for scenarios where a completely randomized experimental design is implemented without explicit knowledge of network structures. Notably, our method facilitates post-hoc interference adjustment on experimental data, significantly reducing the complexities and costs associated with intricate experimental designs and randomization strategies. The proposed framework stands out for its ability to accommodate varying levels of interference, thereby yielding more accurate and robust estimations. Through comprehensive simulations set against a variety of interference scenarios, along with empirical validation using real-world data from a mobile gaming environment, we demonstrate the efficacy of our approach. This study represents a pioneering effort in exploring causal inference in user-randomized experiments impacted by dynamic network effects.

stat.AP↗

arXiv2023

Bayesian Nested Latent Class Models for Cause-of-Death Assignment using Verbal Autopsies Across Multiple Domains

Understanding cause-specific mortality rates is crucial for monitoring population health and designing public health interventions. Worldwide, two-thirds of deaths do not have a cause assigned. Verbal autopsy (VA) is a well-established tool to collect information describing deaths outside of hospitals by conducting surveys to caregivers of a deceased person. It is routinely implemented in many low- and middle-income countries. Statistical algorithms to assign cause of death using VAs are typically vulnerable to the distribution shift between the data used to train the model and the target population. This presents a major challenge for analyzing VAs as labeled data are usually unavailable in the target population. This article proposes a Latent Class model framework for VA data (LCVA) that jointly models VAs collected over multiple heterogeneous domains, assign cause of death for out-of-domain observations, and estimate cause-specific mortality fractions for a new domain. We introduce a parsimonious representation of the joint distribution of the collected symptoms using nested latent class models and develop an efficient algorithm for posterior inference. We demonstrate that LCVA outperforms existing methods in predictive performance and scalability. Supplementary materials for this article and the R package to implement the model are available online.

stat.ME↗

arXiv2023

Bayesian Active Questionnaire Design for Cause-of-Death Assignment Using Verbal Autopsies

Only about one-third of the deaths worldwide are assigned a medically-certified cause, and understanding the causes of deaths occurring outside of medical facilities is logistically and financially challenging. Verbal autopsy (VA) is a routinely used tool to collect information on cause of death in such settings. VA is a survey-based method where a structured questionnaire is conducted to family members or caregivers of a recently deceased person, and the collected information is used to infer the cause of death. As VA becomes an increasingly routine tool for cause-of-death data collection, the lengthy questionnaire has become a major challenge to the implementation and scale-up of VAs. In this paper, we propose a novel active questionnaire design approach that optimizes the order of the questions dynamically to achieve accurate cause-of-death assignment with the smallest number of questions. We propose a fully Bayesian strategy for adaptive question selection that is compatible with any existing probabilistic cause-of-death assignment methods. We also develop an early stopping criterion that fully accounts for the uncertainty in the model parameters. We also propose a penalized score to account for constraints and preferences of existing question structures. We evaluate the performance of our active designs using both synthetic and real data, demonstrating that the proposed strategy achieves accurate cause-of-death assignment using considerably fewer questions than the traditional static VA survey instruments.

stat.AP↗

arXiv2022

The openVA Toolkit for Verbal Autopsies

Verbal autopsy (VA) is a survey-based tool widely used to infer cause of death (COD) in regions without complete-coverage civil registration and vital statistics systems. In such settings, many deaths happen outside of medical facilities and are not officially documented by a medical professional. VA surveys, consisting of signs and symptoms reported by a person close to the decedent, are used to infer the cause of death for an individual, and to estimate and monitor the cause of death distribution in the population. Several classification algorithms have been developed and widely used to assign cause of death using VA data. However, The incompatibility between different idiosyncratic model implementations and required data structure makes it difficult to systematically apply and compare different methods. The openVA package provides the first standardized framework for analyzing VA data that is compatible with all openly available methods and data structure. It provides an open-sourced, R implementation of several most widely used VA methods. It supports different data input and output formats, and customizable information about the associations between causes and symptoms. The paper discusses the relevant algorithms, their implementations in R packages under the openVA suite, and demonstrates the pipeline of model fitting, summary, comparison, and visualization in the R environment.

stat.AP↗

arXiv2021

Tree-informed Bayesian multi-source domain adaptation: cross-population probabilistic cause-of-death assignment using verbal autopsy

Determining causes of deaths (COD) occurred outside of civil registration and vital statistics systems is challenging. A technique called verbal autopsy (VA) is widely adopted to gather information on deaths in practice. A VA consists of interviewing relatives of a deceased person about symptoms of the deceased in the period leading to the death, often resulting in multivariate binary responses. While statistical methods have been devised for estimating the cause-specific mortality fractions (CSMFs) for a study population, continued expansion of VA to new populations (or "domains") necessitates approaches that recognize between-domain differences while capitalizing on potential similarities. In this paper, we propose such a domain-adaptive method that integrates external between-domain similarity information encoded by a pre-specified rooted weighted tree. Given a cause, we use latent class models to characterize the conditional distributions of the responses that may vary by domain. We specify a logistic stick-breaking Gaussian diffusion process prior along the tree for class mixing weights with node-specific spike-and-slab priors to pool information between the domains in a data-driven way. Posterior inference is conducted via a scalable variational Bayes algorithm. Simulation studies show that the domain adaptation enabled by the proposed method improves CSMF estimation and individual COD assignment. We also illustrate and evaluate the method using a validation data set. The paper concludes with a discussion on limitations and future directions.

stat.ME↗