arXiv subjects

Joseph Lam

Publications and source records attributed to Joseph Lam.

5 recordsLinked to original sources

arXiv2026

Designing Ambiguity-Aware Clerical Review: A Stratified Sampling Framework for Record Linkage and Deduplication

Clerical review of candidate record pairs remains the de facto gold standard for evaluating record linkage, but it is resource-intensive and often designed informally. We propose a design-based framework that treats clerical review as finite-population sampling over fine-grained strata defined by match weight, comparison pattern, record-level ambiguity, and demographic group. Match-probability bands are constructed from model-based score deciles. Within bands, strata combine comparison patterns with an ambiguity factor derived from matchability and conditional candidate perplexity. A band-specific margin-of-error profile encodes substantive priorities, such as tighter precision in high-score bands, while a single scaling parameter enforces the overall clerical budget. We evaluate the framework using a labelled dataset deduplicated in Splink, comprising 50,000 records and approximately 478,000 candidate pairs. We compare a baseline design reviewing about 23% of pairs with a budget-constrained design reviewing about 7%. The baseline accurately estimates global and band-specific match rates, while sampled distributions of comparison patterns, gender, and ambiguity broadly track the population. Under the budget design, global error approximately doubles, with the largest band-level errors in middle-score bands where matches, non-matches, and ambiguous cases are intermixed. Accuracy in the highest-score bands and the band-wise ambiguity profile are largely preserved, although representativeness by comparison pattern and gender declines. The framework generalises to other clerical-review objectives and can incorporate gold-standard data as prior information for design and calibration. It makes explicit the negotiable trade-offs between workload, precision, representativeness, and coverage of linkage uncertainty.

stat.ME↗

arXiv2026

What does ethnic density represent? Spatial co-occurrence networks of a widely used contextual measure using harmonised UK small-area census data

Ethnic density is widely used in epidemiology and health geography as a contextual exposure, yet it is rarely examined as a measurement problem in its own right. Equivalent percentage values may represent different neighbourhood contexts across groups and places, particularly where migration, religion, language, household structure and socioeconomic conditions are spatially co-located. Using the harmonised Unified UK Census Data release, I analysed 239,023 small-area census data to examine ethnic density as an exploratory contextual co-occurrence construct. I estimated UK-wide mixed graphical models (MGM) for eight ethnic-density targets using 239,019 complete cases and 32 nodes per target-specific model. England-only spatial analyses then used k-nearest-neighbour Output Area centroids (k = 8) to estimate LISA and spatially adjusted residual networks. Ethnic density did not behave as a single contextual scalar. In the UK-wide MGM, the strongest retained target-neighbour edges differed across groups. Asian density was linked most strongly with Middle East/Asia-born share (0.59), Indian density with Hindu share (0.55), Pakistani density with Muslim share (0.47), Bangladeshi density with Muslim share (0.23), Black density with Africa-born share (0.42), and White density with Middle East/Asia-born share (0.35). England-only ethnic-density measures were strongly spatially autocorrelated, with Global Moran's I ranging from 0.57 for Mixed share to 0.90 for White share. After residualising against English region and local spatial lag, 64.3% to 96.4% of original target-node edges persisted across ethnic-density networks. Equivalent percentage values are not necessarily comparable across ethnic groups. This has implications for estimand definition, adjustment strategies, and the interpretation of ethnic density and other bundled contextual measures in urban health research.

stat.ME↗

arXiv2026

Cluster-based name embeddings reduce ethnic disparities in record linkage quality under realistic name corruption: evidence from the North Carolina Voter Registry

Differential ethnic-based record linkage errors can bias epidemiologic estimates. Prior evidence often conflates heterogeneity in error mechanisms with unequal exposure to error. Using snapshots of the North Carolina Voter Registry (Oct 2011-Oct 2022), we derived empirical name-discrepancy profiles to parameterise realistic corruptions. From an Oct 2022 extract (n=848,566), we generated five replicate corrupted datasets under three settings that separately varied mechanism heterogeneity and exposure inequality, and linked records back to originals using unadjusted Jaro-Winkler, Term Frequency (TF)-adjusted Jaro-Winkler, and a cluster-based forename-embedding comparator combined with TF-adjusted surname comparison. We evaluated false match rate (FMR), missed match rate (MMR) and white-centric disparities. At a fixed MMR near 0.20, overall error rates and ethnic disparities diverged substantially by model under disproportionate exposure to corruption. Term-frequency (TF)-adjusted Jaro-Winkler achieved very low overall FMR (0.55% (95% CI 0.54-0.57)) at overall MMR 20.34% (20.30-20.39), but large white-centric under-linkage disparities persisted: Hispanic voters had 36.3% (36.1-36.6) and Non-Hispanic Black voters 8.6% (8.6-8.7) higher FMRs compared to Non-Hispanic White groups. Relative to unadjusted string similarity, TF adjustment reduced these disparities (Hispanic: +60.4% (60.1-60.7) to +36.3%; Black: +13.1% (13.0-13.2) to +8.6%). The cluster-based forename-embedding model reduced missed-match disparities further (Hispanic: +10.2% (9.8-10.3); Black: +0.6% (0.4-0.7)), but at a cost of increasing overall FMR (4.28% (4.22-4.35)) at the same threshold. Unequal exposure to identifier error drove substantially larger disparities than mechanism heterogeneity alone; cluster-based embeddings markedly narrowed under-linkage disparities beyond TF adjustment.

stat.ME↗

arXiv2025

A comparison of the Alkire-Foster method and a Markov random field approach in the analysis of multidimensional poverty

Multidimensional poverty measurement is crucial for capturing deprivation beyond income-based metrics. This study compares the Alkire-Foster (AF) method and a Markov Random Field (MRF) approach for classifying multidimensional poverty using a simulation-based analysis. The AF method applies a deterministic threshold-based classification, while the MRF approach leverages probabilistic graphical modelling to account for correlations between deprivation indicators. Using a synthetic dataset of 50,000 individuals with ten binary deprivation indicators, we assess classification accuracy, false positive/negative trade-offs, and agreement between the methods. Results show that AF achieves higher classification accuracy (89.5%) compared to MRF (75.4%), with AF minimizing false negatives and MRF reducing false positives. The overall agreement between the two methods is 65%, with discrepancies primarily occurring when AF classifies individuals as poor while MRF does not. While AF is transparent and easy to implement, it does not capture interdependencies among indicators, potentially leading to misclassification. MRF, though computationally intensive, offers a more nuanced understanding of deprivation clusters. These findings highlight the trade-offs in multidimensional poverty measurement and provide insights for policymakers on method selection based on data availability and policy objectives. Future research should extend these approaches to non-binary indicators and real-world datasets.

stat.ME↗

arXiv2024

Decolonising Data Systems: Using Jyutping or Pinyin as tonal representations of Chinese names for data linkage

Data linkage is increasingly used in health research and policy making and is relied on for understanding health inequalities. However, linked data is only as useful as the underlying data quality, and differential linkage rates may induce selection bias in the linked data. A mechanism that selectively compromises data quality is name romanisation. Converting text of a different writing system into Latin based writing, or romanisation, has long been the standard process of representing names in character-based writing systems such as Chinese, Vietnamese, and other languages such as Swahili. Unstandardised romanisation of Chinese characters, due in part to problems of preserving the correct name orders the lack of proper phonetic representation of a tonal language, has resulted in poor linkage rates for Chinese immigrants. This opinion piece aims to suggests that the use of standardised romanisation systems for Cantonese (Jyutping) or Mandarin (Pinyin) Chinese, which incorporate tonal information, could improve linkage rates and accuracy for individuals with Chinese names. We used 771 Chinese and English names scraped from openly available sources, and compared the utility of Jyutping, Pinyin and the Hong Kong Government Romanisation system (HKG-romanisation) for representing Chinese names. We demonstrate that both Jyutping and Pinyin result in fewer errors compared with the HKG-romanisation system. We suggest that collecting and preserving people's names in their original writing systems is ethically and socially pertinent. This may inform development of language-specific pre-processing and linkage paradigms that result in more inclusive research data which better represents the targeted populations.

cs.CL↗