arXiv subjects

Partha Lahiri

Publications and source records attributed to Partha Lahiri.

17 recordsLinked to original sources

arXiv2026

Reconciliation of Bayes and empirical Bayes interval estimation with application to small area estimation

Multi-level normal hierarchical models, also interpreted as mixed effects models, play an important role in developing statistical theory in multi-parameter estimation for a wide range of applications, including small area estimation. In this article, we propose a novel reconciliation framework of the empirical Bayes (EB) and hierarchical Bayes approaches for interval estimation of random effects under a two-level normal model. Our framework shows that a second-order efficient empirical Bayes confidence interval, with EB coverage error of order $O(m^{-3/2})$, $m$ being the number of areas in the area-level model, can also be viewed as a credible interval whose expected posterior coverage is close to the nominal level, provided a carefully chosen prior, referred to as a "matching prior", is placed on the hyperparameters. While existing literature has examined matching priors that reconcile frequentist and Bayesian inference in various settings, this paper is the first to study matching priors with the goal of interval estimation of random effects in a two-level model. We obtain a matching prior on the variance component that achieves a proper posterior under mild regularity conditions. The theoretical results in the paper are corroborated through a Monte Carlo simulation study and real data analysis.

math.ST↗

arXiv2026

Impact of existence and nonexistence of pivot on the coverage of empirical best linear prediction intervals for small areas

We advance the theory of parametric bootstrap in constructing highly efficient empirical best (EB) prediction intervals of small area means. The coverage error of such a prediction interval is of the order $O(m^{-3/2})$, where $m$ is the number of small areas to be pooled using a linear mixed normal model. In the context of an area level model where the random effects follow a non-normal known distribution except possibly for unknown hyperparameters, we analytically show that the order of coverage error of empirical best linear (EBL) prediction interval remains the same even if we relax the normality of the random effects by the existence of pivot for a suitably standardized random effects when hyperpameters are known. Recognizing the challenge of showing existence of a pivot, we develop a simple moment-based method to claim non-existence of pivot. We show that existing parametric bootstrap EBL prediction interval fails to achieve the desired order of the coverage error, i.e. $O(m^{-3/2})$, in absence of a pivot. We obtain a surprising result that the order $O(m^{-1})$ term is always positive under certain conditions indicating possible overcoverage of the existing parametric bootstrap EBL prediction interval. In general, we analytically show for the first time that the coverage problem can be corrected by adopting a suitably devised double parametric bootstrap. Our Monte Carlo simulations show that our proposed single bootstrap method performs reasonably well when compared to rival methods.

math.ST↗

arXiv2026

Empirical best prediction of poverty indicators via nested error regression with high dimensional parameters

The Nested Error Regression Model with High-Dimensional Parameters (NERHDP) is extended to address challenges in small area poverty estimation. A robust and flexible framework is proposed to derive empirical best predictors (EBPs) of small area poverty indicators while accommodating heterogeneity in regression coefficients and sampling variances across areas. To mitigate the computational limitations of the existing algorithm, an efficient estimation procedure is introduced, substantially reducing computation time and enhancing scalability for large datasets. A novel approach for generating area-specific poverty estimates in out-of-sample areas is also developed, improving the reliability of synthetic estimates. Uncertainty is quantified through a parametric bootstrap method specifically tailored to the extended model. Under heterogeneous data-generating scenarios, the proposed method yields lower relative bias and relative root mean squared prediction error than existing approaches. The methodology is further illustrated using data from the 2002 Albania Living Standards Measurement Survey, combined with auxiliary information from the 2001 census, to estimate poverty indicators for 374 municipalities.

stat.ME↗

arXiv2025

Improving measurement error and representativeness in nonprobability surveys

In the age of big data, nonprobability surveys are becoming increasingly abundant. Data integration techniques involving both probability and nonprobability surveys are being extensively used for providing improved estimates for finite population estimation. While much of the existing research has focused on mitigating selection bias in nonprobability surveys, the issue of measurement error within these surveys remains relatively unexplored. Statistical methods devised with the purpose of reducing selection bias are appropriate for reliable estimation, only under the assumption of accuracy of survey responses. Motivated by a recent case study of Kennedy, Mercer, and Lau (2024), our research addresses bias from both measurement and sampling errors in nonprobability surveys. In this article, we propose a new data integration method that uses multiple probability and nonprobability surveys and leverages machine learning models to construct a composite estimator. The proposed composite estimator integrates probability and nonprobability surveys, when both contain response variables of interest. We analyze the performance of this estimator in comparison to an existing composite estimator in literature, analytically as well as empirically, using multiple survey data from Kennedy et al. (2024). Finally, we identify conditions under which the proposed estimator outperforms estimators based solely on probability surveys.

stat.AP↗

arXiv2025

Multidimensional Poverty Mapping for Small Areas

Many countries measure poverty based only on income or consumption. However, there is a growing awareness of measuring poverty through multiple dimensions that captures a more reasonable status of poverty. Estimating poverty measure(s) for small geographical areas, commonly referred to as poverty mapping, is challenging due to small or no sample for the small areas. While there is a huge literature available on unidimensional poverty mapping, only a limited effort has been made to address special challenges that arise only in the multidimensional poverty mapping. For example, in multidimensional poverty mapping, a new problem arises involving estimation of relative contributions of different dimensions to overall poverty for small areas. This problem has been grossly ignored in the small area estimation (SAE) literature. We address this issue using a multivariate hierarchical model implemented via a Bayesian method. Moreover, we demonstrate how a multidimensional poverty composite measure can be estimated for small areas. In this paper, we demonstrate our proposed methodology using a survey data specially designed by one of us for multidimensional poverty mapping. This paper adds a new direction to poverty mapping literature.

stat.ME↗

arXiv2025

Approximate hierarchical Bayes small area estimation using NEF-QVF and poststratification

We propose an approximate hierarchical Bayes approach that uses the Natural Exponential Family with Quadratic Variance Function in combining information from multiple sources to improve traditional survey estimates of finite population means for small areas. Unlike other Bayesian approaches in finite population sampling, we do not assume a model for all units of the finite population and do not require linking sampled units to the finite population frame. We assume a model only for the finite population units in which the outcome variable is observed; because, for these units, the assumed model can be checked using existing statistical tools. We do not posit an elaborate model on the true means for unobserved units. Instead, we assume that population means of cells with the same combination of factor levels are identical across small areas, and that the population mean for a cell is identical to the mean of the observed units in that cell. We apply our proposed methodology to a real-life survey, linking information from multiple disparate data sources. We also provide practical ways of model selection that can be applied to a wider class of models under similar setting but for a diverse range of scientific problems.

stat.ME↗

arXiv2024

Effects of model misspecification on small area estimators

Nested error regression models are commonly used to incorporate observational unit specific auxiliary variables to improve small area estimates. When the mean structure of this model is misspecified, there is generally an increase in the mean square prediction error (MSPE) of Empirical Best Linear Unbiased Predictors (EBLUP). Observed Best Prediction (OBP) method has been proposed with the intent to improve on the MSPE over EBLUP. We conduct a Monte Carlo simulation experiment to understand the effect of mispsecification of mean structures on different small area estimators. Our simulation results lead to an unexpected result that OBP may perform very poorly when observational unit level auxiliary variables are used and that OBP can be improved significantly when population means of those auxiliary variables (area level auxiliary variables) are used in the nested error regression model or when a corresponding area level model is used. Our simulation also indicates that the MSPE of OBP in an increasing function of the difference between the sample and population means of the auxiliary variables.

stat.ME↗

arXiv2024

Estimation of finite population proportions for small areas -- a statistical data integration approach

Empirical best prediction (EBP) is a well-known method for producing reliable proportion estimates when the primary data source provides only small or no sample from finite populations. There are potential challenges in implementing existing EBP methodology such as limited auxiliary variables in the frame (not adequate for building a reasonable working predictive model) or unable to accurately link the sample to the finite population frame due to absence of identifiers. In this paper, we propose a new data linkage approach where the finite population frame is replaced by a big probability sample, having a large set of auxiliary variables but not the outcome binary variable of interest. We fit an assumed model on the small probability sample and then impute the outcome variable for all units of the big sample to obtain standard weighted proportions. We develop a new adjusted maximum likelihood (ML) method so that the estimate of model variance doesn't fall on the boundary, which is otherwise encountered in commonly used ML method. We also propose an estimator of the mean squared prediction error using a parametric bootstrap method and address computational issues by developing an efficient Expectation Maximization algorithm. The proposed methodology is illustrated in the context of election projection for small areas.

stat.ME↗

arXiv2022

A Nested Error Regression Model with High Dimensional Parameter for Small Area Estimation

In this paper we propose a flexible nested error regression small area model with high dimensional parameter that incorporates heterogeneity in regression coefficients and variance components. We develop a new robust small area specific estimating equations method that allows appropriate pooling of a large number of areas in estimating small area specific model parameters. We propose a parametric bootstrap and jackknife method to estimate not only the mean squared errors but also other commonly used uncertainty measures such as standard errors and coefficients of variation. We conduct both modelbased and design-based simulation experiments and real-life data analysis to evaluate the proposed methodology

stat.ME↗

arXiv2021

Estimation of mask effectiveness perception for small domains using multiple data sources

All pandemics are local; so learning about the impacts of pandemics on public health and related societal issues at granular levels is of great interest. COVID-19 is affecting everyone in the globe and mask wearing is one of the few precautions against it. To quantify people's perception of mask effectiveness and to prevent the spread of COVID-19 for small areas, we use Understanding America Study's (UAS) survey data on COVID-19 as our primary data source. Our data analysis shows that direct survey-weighted estimates for small areas could be highly unreliable. In this paper we develop a synthetic estimation method to estimate proportions of mask effectiveness for small areas using a logistic model that combines information from multiple data sources. We select our working model using an extensive data analysis facilitated by a new variable selection criterion for survey data and benchmarking ratios. We propose a Jackknife method to estimate variance of our proposed estimator. From our data analysis. it is evident that our proposed synthetic method outperforms direct survey-weighted estimator with respect to commonly used evaluation measures.

stat.AP↗

arXiv2020

Parametric Bootstrap Confidence Intervals for the Multivariate Fay-Herriot Model

The multivariate Fay-Herriot model is quite effective in combining information through correlations among small area survey estimates of related variables or historical survey estimates of the same variable or both. Though the literature on small area estimation is already very rich, construction of second-order efficient confidence intervals from multivariate models have so far received very little attention. In this paper, we develop a parametric bootstrap method for constructing a second-order efficient confidence interval for a general linear combination of small area means using the multivariate Fay-Herriot normal model. The proposed parametric bootstrap method replaces difficult and tedious analytical derivations by the power of efficient algorithm and high speed computer. Moreover, the proposed method is more versatile than the analytical method because the parametric bootstrap method can be easily applied to any method of model parameter estimation and any specific structure of the variance-covariance matrix of the multivariate Fay-Herriot model avoiding all the cumbersome and time-consuming calculations required in the analytical method. We apply our proposed methodology in constructing confidence intervals for the median income of four-person families for the fifty states and the District of Columbia in the United States. Our data analysis demonstrates that the proposed parametric bootstrap method generally provides much shorter confidence intervals compared to the corresponding traditional direct method. Moreover, the confidence intervals obtained from the multivariate model is generally shorter than the corresponding univariate model indicating the potential advantage of exploiting correlations of median income of four-person families with median incomes of three and five person families.

stat.ME↗

arXiv2020

An Evaluation of Design-based Properties of Different Composite Estimators

For the last several decades, the US Census Bureau has been using the AK composite estimation method to produce statistics on employment from the Current Population Survey (CPS) data. The CPS uses a rotating design and AK estimators are linear combinations of monthly survey weighted averages (called month-in-sample estimates) in each rotation groups. Denoting by $X$ the vector of month-in-sample estimates and by $Σ$ its design based variance, the coefficients of the linear combination were optimized by the Census Bureau after substituting $Σ$ by an estimate and under unrealistic stationarity assumptions. To show the limits of this approach, we compared the AK estimator with different competitors using three different synthetic populations that mimics the Current Population Survey (CPS) data and a simplified sample design that mimics the CPS design. In our simulation setup, empirically best estimators have larger mean square error than simple averages. In the real data analysis, the AK estimates are constantly below the survey-weighted estimates, indicating potential bias. Any attempt to improve on the estimated optimal estimator in either class would require a thorough investigation of the highly non-trivial problem of estimation of $Σ$ for a complex setting like the CPS (we did not entertain this problem in this paper). A different approach is to use a variant of the regression composite estimator used by Statistics Canada. The regression composite estimator does not require estimation of $Σ$ and is less sensitive to the rotation group bias in our simulations. Our study demonstrates that there is a great potential for improving the estimation of levels and month to month changes in the unemployment rates by using the regression composite estimator.

stat.ME↗

arXiv2019

Multi-Goal Prior Selection: A Way to Reconcile Bayesian and Classical Approaches for Random Effects Models

The two-level normal hierarchical model has played an important role in statistical theory and applications. In this paper, we first introduce a general adjusted maximum likelihood method for estimating the unknown variance component of the model and the associated empirical best linear unbiased predictor of the random effects. We then discuss a new idea for selecting prior for the hyperparameters. The prior, called a multi-goal prior, produces Bayesian solutions for hyperparmeters and random effects that match (in the higher order asymptotic sense) the corresponding classical solution in linear mixed model with respect to several properties. Moreover, we establish for the first time an analytical equivalence of the posterior variances under the proposed multi-goal prior and the corresponding parametric bootstrap second-order mean squared error estimates in the context of a random effects model.

stat.ME↗

arXiv2018

A General Bayesian Approach to Meet Different Inferential Goals in Poverty Research for Small Areas

Poverty mapping that displays spatial distribution of various poverty indices is most useful to policymakers and researchers when they are disaggregated into small geographic units, such as cities, municipalities or other administrative partitions of a country. Typically, national household surveys that contain welfare variables such as income and expenditures provide limited or no data for small areas. It is well-known that while direct survey-weighted estimates are quite reliable for national or large geographical areas they are unreliable for small geographic areas. If the objective is to find areas with extreme poverty, these direct estimates will often select small areas due to the high variabilities in the estimates. Empirical best prediction and Bayesian methods have been proposed to improve on the direct point estimates. However, these estimates are not appropriate for different inferential purposes. For example, for identifying areas with extreme poverty, these estimates would often select areas with large sample sizes. In this paper, using databases used by the Chilean Ministry for their Small Area Estimation production, we illustrate how appropriate Bayesian methodology can be developed to address different inferential problems.

stat.AP↗

arXiv2017

A New Model Variance Estimator for an Area Level Small Area Model to Solve Multiple Problems Simultaneously

The two-level normal hierarchical model (NHM) has played a critical role in the theory of small area estimation (SAE), one of the growing areas in statistics with numerous applications in different disciplines. In this paper, we address major well-known shortcomings associated with the empirical best linear unbiased prediction (EBLUP) of a small area mean and its mean squared error (MSE) estimation by considering an appropriate model variance estimator that satisfies multiple properties. The proposed model variance estimator simultaneously (i) improves on the estimation of the related shrinkage factors, (ii) protects EBLUP from the common overshrinkage problem, (iii) avoids complex bias correction in generating strictly positive second-order unbiased mean square error (MSE) estimator either by the Taylor series or single parametric bootstrap method. The idea of achieving multiple desirable properties in an EBLUP method through a suitably devised model variance estimator is the first of its kind and holds promise in providing good inferences for small area means under the classical linear mixed model prediction framework. The proposed methodology is also evaluated using a Monte Carlo simulation study and real data analysis.

math.ST↗

arXiv2014

A second-order efficient empirical Bayes confidence interval

We introduce a new adjusted residual maximum likelihood method (REML) in the context of producing an empirical Bayes (EB) confidence interval for a normal mean, a problem of great interest in different small area applications. Like other rival empirical Bayes confidence intervals such as the well-known parametric bootstrap empirical Bayes method, the proposed interval is second-order correct, that is, the proposed interval has a coverage error of order $O(m^{-{3}/{2}})$. Moreover, the proposed interval is carefully constructed so that it always produces an interval shorter than the corresponding direct confidence interval, a property not analytically proved for other competing methods that have the same coverage error of order $O(m^{-{3}/{2}})$. The proposed method is not simulation-based and requires only a fraction of computing time needed for the corresponding parametric bootstrap empirical Bayes confidence interval. A Monte Carlo simulation study demonstrates the superiority of the proposed method over other competing methods.

math.ST↗

arXiv2008

Parametric bootstrap approximation to the distribution of EBLUP and related prediction intervals in linear mixed models

Empirical best linear unbiased prediction (EBLUP) method uses a linear mixed model in combining information from different sources of information. This method is particularly useful in small area problems. The variability of an EBLUP is traditionally measured by the mean squared prediction error (MSPE), and interval estimates are generally constructed using estimates of the MSPE. Such methods have shortcomings like under-coverage or over-coverage, excessive length and lack of interpretability. We propose a parametric bootstrap approach to estimate the entire distribution of a suitably centered and scaled EBLUP. The bootstrap histogram is highly accurate, and differs from the true EBLUP distribution by only $O(d^3n^{-3/2})$, where $d$ is the number of parameters and $n$ the number of observations. This result is used to obtain highly accurate prediction intervals. Simulation results demonstrate the superiority of this method over existing techniques of constructing prediction intervals in linear mixed models.

math.ST↗