arXiv subjects

Siu-Ming Tam

Publications and source records attributed to Siu-Ming Tam.

10 recordsLinked to original sources

arXiv2026

National Versus Domain: Coverage Properties of HB Credible Intervals Under Survey Redesign

A companion paper to Tam (2026) reports an extended Monte Carlo (MC) study examining the frequentist coverage of 95% hierarchical Bayes (HB) credible intervals at the national and domain levels under four stress-test scenarios. The extended study covers 140 strata and 13 estimation domains separately, adds a classical direct-estimator benchmark, and tests two strategies for restoring domain coverage: prior sensitivity and Prasad and Rao (PR) MSE correction. At the national level, HB credible intervals achieve near-nominal coverage across all four scenarios and all three labour force variables (Employment 93 to 96%, Unemployment 87 to 97%, Hours Worked 99.5 to 100%). At the domain level, Hours Worked coverage is nearnominal (94 to 98%) in all scenarios; Employment and Unemployment coverage is below nominal for scenarios with low between-domain heterogeneity, a direct consequence of HB shrinkage toward the national mean. A key operational finding emerges from the Rare Event scenario (D): the classical direct estimator collapses to 0% national coverage for Employment and Hours Worked because five unsampled strata introduce a systematic bias; the HB estimator achieves 96 to 100% national coverage at roughly 15% of the classical sample cost. Neither a weaker prior nor PR MSE correction reliably restores domain coverage, confirming that the failure is bias-driven and cannot be remedied by variance inflation alone.

stat.AP↗

arXiv2026

Responsible AI and Algorithmic Adoption in Methodology Development for National Statistical Offices

To meet growing demand for granular demographic and socioeconomic indicators under tighter budgets, national statistical offices must continually develop new methods. These include using big data, satellite imagery, and transactional sources to improve or redesign data collection. Artificial intelligence can support this work, but algorithms generated with AI should not be trusted for production without rigorous verification. This paper focuses on two foundations of trust in the use of AI in official statistics: independent statistical verification before production use, and disciplined protection of respondent confidentiality during development and testing. The approach is illustrated through the author's experience directing AI to construct and implement a Mini Max Hierarchical Bayes sampling algorithm. Applied to a synthetic labour force population, the method met all specified precision targets while reducing the required sample size by 80 percent, as confirmed by a Monte Carlo study with 1000 replications. Applied to 2021 Australian Census microdata, it achieved a 90 percent reduction while producing national point estimates accurate to well below 1 percent. The paper concludes with a practical evaluation checklist aligned with the UN Fundamental Principles of Official Statistics and the HLG MOS Quality Framework for Statistical Algorithms.

stat.OT↗

arXiv2026

Bayesian Seasonal Adjustment for Survey Time Series

Seasonal adjustment procedures used by national statistical offices -- X-11 and X-12-ARIMA -- treat each survey estimate as an exact observation, discarding the accompanying standard errors that survey methodologists routinely compute. This paper closes that gap by embedding time-varying sampling error variances into a Basic Structural Model (BSM), extending a recently proposed Dynamic Mini-Max (DMM) Bayesian framework for survey estimation. Via the Harvey-Todd equivalence, BSM with zero measurement error variance reduces to X-11-style seasonal adjustment, so DMM-BSM is a principled Bayesian generalisation of existing practice rather than a departure from it. A two-block Gibbs sampler delivers the full joint smoothing posterior of the latent state trajectory. Exact credible intervals for the trend level, k-step trend movements (k=1,2,...), and seasonally adjusted estimates follow directly, together with posterior probabilities of directional change -- outputs that X-11 cannot provide. Simulation studies with within-replication parameter estimation confirm substantially higher credible interval coverage than the X-11-equivalent model across both large and small survey domains. Applied to 120 months of Australian Bureau of Statistics Labour Force Survey data, once sampling variance is modelled the maximum likelihood estimate of stochastic seasonal variance collapses to zero -- evidence that apparent seasonal fluctuations in the published series are largely attributable to measurement noise rather than genuine seasonal drift, a distinction X-11 cannot make.

stat.ME↗

arXiv2026

More with Less -- Bethel Allocation and Precision-Preserving Sample Size Reduction via Hierarchical Bayes Modelling

Statistical offices face a familiar and intensifying dilemma: rising demand for detailed regional and domain-level estimates under budgets that are fixed or shrinking. National statistical offices (NSOs) either ignore the problem of optimal sample allocation for multiple target variables when designing a multi-purpose survey, or address it incorrectly - relying on ad hoc approaches such as computing Neyman allocations separately per variable and taking the element-wise maximum, a practice that simultaneously wastes budget and fails to guarantee precision across all domains. This paper presents a practical two-stage strategy that reframes the question: not how to allocate a given sample, but how small the sample can be made while still meeting pre-defined precision targets for all target variables across all geographic domains at once. The innovation lies not in inventing new methods, but in the novel combination of two well-established techniques applied to this cost-reduction problem: (i) multivariate constrained optimisation via Bethel allocation, which finds the globally minimum sample satisfying all precision constraints simultaneously; and (ii) Hierarchical Bayes (HB) small area modelling, which borrows strength across strata and permits a further reduction of the Bethel sample. The approach is validated using a Monte Carlo study (B = 1,000 replications) based on a synthetic labour-force population of one million individuals, where known population truth allows rigorous evaluation of precision, accuracy, and credible-interval coverage. Keywords: Bethel allocation; Hierarchical Bayes; small area estimation; sample size reduction; multivariate optimisation; labour force survey; coefficient of variation.

stat.ME↗

arXiv2026

On design-unbiased algorithmic Machine Learning

Machine Learning (ML) algorithms, such as k-Nearest Neighbours (kNN) or random forest, eschew the ideal of true data models in favour of predictive performance. However, minimising the MSE or F-score cannot lead to unbiasedness directly, which is important in many situations such as official statistics. We study the conditions of algorithmic ML, other than the existence and knowledge of true data models, which lead to unbiased prediction or classification for a given finite population, including how the training data may be sampled from the population, how a trained prediction algorithm can be tuned to achieve unbiased prediction or classification for that population, and how the performance of out-of-sample prediction or classification can be assessed unbiasedly. The inference is based on the known probability design of samples and training sets, rather than any assumed distributions or models.

cs.LG↗

arXiv2026

Dynamic Mini Max Design and Sequential HB Inference for Repeated Surveys

TThis paper develops a Dynamic Mini-Max (DMM) framework for repeated surveys comprising a Dynamic Mini-Max Design and a Sequential Hierarchical Bayes Update (SHBU). The DMM jointly optimizes sample size and wave overlap subject to simultaneous precision constraints for levels and movements, a respondent burden limit, and a fieldwork budget. The methods are illustrated using 2021 Australian Census data (t = 1) and simulated waves t = 2, 3, 4. Both the DMM and the classical design start from the same 5% proportional allocation of n_A = 42,018 units. The DMM reduces this to n* = 40,251 while meeting all precision constraints, achieving a cost saving of approximately 6.3%. Level coverage is comparable between the two designs (maximum absolute relative error (MARE) ratio 0.844--1.263). Movement coverage diverges markedly: the DMM achieves 100% across all 27 domain-variable cells, while the classical design achieves only 82%--96% (87.5%--95.0% nationally). The classical confidence interval understates movement uncertainty because it addresses sampling variance only and does not account for the model variance component V_mod_hat. Additional benefits of the DMM framework -- including coherent joint inference for levels and movements, sequential updating without ad hoc composite-estimator chaining, and small area estimation -- are outlined in the paper.

stat.ME↗

arXiv2026

Post-Hoc Inference of Cross-Classified Statistics from Hierarchical Bayes Survey Weights

Tam [2026] shows that combining Bethel multivariate allocation with Hierarchical Bayes (HB) small area models can substantially reduce survey sample sizes while maintaining domain-level precision and near-nominal coverage of posterior credible intervals (CrIs). This paper extends that framework to cross-classified statistics derived from HBcalibrated unit record data. Its central contribution is a Post-Hoc Inference Engine (PHIE) that propagates uncertainty from HB domain posterior draws to arbitrary cross-tabulations. PHIE transforms each MCMC draw via chi-square calibration to produce replicate survey weights, from which CrIs are obtained. Three tiers of statistics are identified. Tier 1-E cells reproduce calibration totals and yield exact posterior CrIs. Tier 2 cells involve filtered sums of calibration variables; PHIE alone undercovers, but a Calibrated Bayes interval (CBI), augmenting PHIE with design-based compositional variance, restores near-nominal coverage. Tier 3-NCV cells involve non-calibration variables; a ratio-based CBI linked to a correlated calibration variable achieves reliable coverage even under weak correlation. A key empirical finding is that uncertainty in cross-tabulations is driven primarily by compositional sampling variability rather than HB model uncertainty. Resulting CBI-based coefficients of variation remain within standard publication thresholds.

stat.ME↗

arXiv2025

On linkage bias-correction for estimators using iterated bootstraps

By amalgamating data from disparate sources, the resulting integrated dataset becomes a valuable resource for statistical analysis. In probabilistic record linkage, the effectiveness of such integration relies on the availability of linkage variables free from errors. Where this is lacking, the linked data set would suffer from linkage errors and the resultant analyses, linkage bias. This paper proposes a methodology leveraging the bootstrap technique to devise linkage bias-corrected estimators. Additionally, it introduces a test to assess whether increasing the number of bootstrap iterations meaningfully reduces linkage bias or merely inflates variance without further improving accuracy. An application of these methodologies is demonstrated through the analysis of a simulated dataset featuring hormone information, along with a dataset obtained from linking two data sets from the Australian Bureau of Statistics' labour mobility surveys.

stat.ME↗

arXiv2024

A Calibrated Data-Driven Approach for Small Area Estimation using Big Data

Where the response variable in a big data set is consistent with the variable of interest for small area estimation, the big data by itself can provide the estimates for small areas. These estimates are often subject to the coverage and measurement error bias inherited from the big data. However, if a probability survey of the same variable of interest is available, the survey data can be used as a training data set to develop an algorithm to impute for the data missed by the big data and adjust for measurement errors. In this paper, we outline a methodology for such imputations based on an kNN algorithm calibrated to an asymptotically design-unbiased estimate of the national total and illustrate the use of a training data set to estimate the imputation bias and the fixed - asymptotic bootstrap to estimate the variance of the small area hybrid estimator. We illustrate the methodology of this paper using a public use data set and use it to compare the accuracy and precision of our hybrid estimator with the Fay-Harriot (FH) estimator. Finally, we also examine numerically the accuracy and precision of the FH estimator when the auxiliary variables used in the linking models are subject to under-coverage errors.

stat.ME↗

arXiv2020

Data Integration by combining big data and survey sample data for finite population inference

The statistical challenges in using big data for making valid statistical inference in the finite population have been well documented in literature. These challenges are due primarily to statistical bias arising from under-coverage in the big data source to represent the population of interest and measurement errors in the variables available in the data set. By stratifying the population into a big data stratum and a missing data stratum, we can estimate the missing data stratum by using a fully responding probability sample, and hence the population as a whole by using a data integration estimator. By expressing the data integration estimator as a regression estimator, we can handle measurement errors in the variables in big data and also in the probability sample. We also propose a fully nonparametric classification method for identifying the overlapping units and develop a bias-corrected data integration estimator under misclassification errors. Finally, we develop a two-step regression data integration estimator to deal with measurement errors in the probability sample. An advantage of the approach advocated in this paper is that we do not have to make unrealistic missing-at-random assumptions for the methods to work. The proposed method is applied to the real data example using 2015-16 Australian Agricultural Census data.

stat.ME↗