arXiv subjects

Abhik Ghosh

Publications and source records attributed to Abhik Ghosh.

At least 73 records · Page 4Linked to original sources

arXiv2020

Ultrahigh-dimensional Robust and Efficient Sparse Regression using Non-Concave Penalized Density Power Divergence

We propose a sparse regression method based on the non-concave penalized density power divergence loss function which is robust against infinitesimal contamination in very high dimensionality. Present methods of sparse and robust regression are based on $\ell_1$-penalization, and their theoretical properties are not well-investigated. In contrast, we use a general class of folded concave penalties that ensure sparse recovery and consistent estimation of regression coefficients. We propose an alternating algorithm based on the Concave-Convex procedure to obtain our estimate, and demonstrate its robustness properties using influence function analysis. Under some conditions on the fixed design matrix and penalty function, we prove that this estimator possesses large-sample oracle properties in an ultrahigh-dimensional regime. The performance and effectiveness of our proposed method for parameter estimation and prediction compared to state-of-the-art are demonstrated through simulation studies.

stat.ME↗

arXiv2020

Interpreting LOFAR 21-cm signal upper limits at z~9.1 in the context of high-z galaxy and reionisation observations

Using the latest upper limits on the 21-cm power spectrum at $z\approx9.1$ from the Low Frequency Array (LOFAR), we explore regions of parameter space which are inconsistent with the data. We use 21CMMC, a Monte Carlo Markov Chain sampler of 21cmFAST which directly forward models the 3D cosmic 21-cm signal in a fully Bayesian framework. We use the astrophysical parameterisation from 21cmFAST, which includes mass-dependent star formation rates and ionising escape fractions as well as soft-band X-ray luminosities to place limits on the properties of the high-$z$ galaxies. Further, we connect the disfavoured regions of parameter space with existing observational constraints on the Epoch of Reionisation such as ultra-violet (UV) luminosity functions, background UV photoionisation rate, intergalactic medium (IGM) neutral fraction and the electron scattering optical depth. We find that all models exceeding the 21-cm signal limits set by LOFAR at $z\approx9.1$ are excluded at $\gtrsim2σ$ by other probes. Finally, we place limits on the IGM spin temperature from LOFAR, disfavouring at 95 per cent confidence spin temperatures below $\sim2.6$ K across an IGM neutral fraction range of $0.15 \lesssim \bar{x}_{H{\scriptscriptstyle I}} \lesssim 0.6$. Note, these limits are only obtained from 141 hrs of data in a single redshift bin. With tighter upper limits, across multiple redshift bins expected in the near future from LOFAR, more viable models will be ruled out. Our approach demonstrates the potential of forward modelling tools such as 21CMMC in combining 21-cm observations with other high-$z$ probes to constrain the astrophysics of galaxies.

astro-ph.CO↗

arXiv2020

Foreground modelling via Gaussian process regression: an application to HERA data

The key challenge in the observation of the redshifted 21-cm signal from cosmic reionization is its separation from the much brighter foreground emission. Such separation relies on the different spectral properties of the two components, although, in real life, the foreground intrinsic spectrum is often corrupted by the instrumental response, inducing systematic effects that can further jeopardize the measurement of the 21-cm signal. In this paper, we use Gaussian Process Regression to model both foreground emission and instrumental systematics in $\sim 2$ hours of data from the Hydrogen Epoch of Reionization Array. We find that a simple co-variance model with three components matches the data well, giving a residual power spectrum with white noise properties. These consist of an "intrinsic" and instrumentally corrupted component with a coherence-scale of 20 MHz and 2.4 MHz respectively (dominating the line of sight power spectrum over scales $k_{\parallel} \le 0.2$ h cMpc$^{-1}$) and a baseline dependent periodic signal with a period of $\sim 1$ MHz (dominating over $k_{\parallel} \sim 0.4 - 0.8$h cMpc$^{-1}$) which should be distinguishable from the 21-cm EoR signal whose typical coherence-scales is $\sim 0.8$ MHz.

astro-ph.CO↗

arXiv2020

A robust variable screening procedure for ultra-high dimensional data

Variable selection in ultra-high dimensional regression problems has become an important issue. In such situations, penalized regression models may face computational problems and some pre screening of the variables may be necessary. A number of procedures for such pre-screening has been developed; among them the sure independence screening (SIS) enjoys some popularity. However, SIS is vulnerable to outliers in the data, and in particular in small samples this may lead to faulty inference. In this paper, we develop a new robust screening procedure. We build on the density power divergence (DPD) estimation approach and introduce DPD-SIS and its extension iterative DPD-SIS. We illustrate the behavior of the methods through extensive simulation studies and show that they are superior to both the original SIS and other robust methods when there are outliers in the data. We demonstrate the claimed robustness through use of influence functions, and we discuss appropriate choice of the tuning parameter $α$. Finally, we illustrate its use on a small dataset from a study on regulation of lipid metabolism.

stat.ME↗

arXiv2020

Consistent Fixed-Effects Selection in Ultra-high dimensional Linear Mixed Models with Error-Covariate Endogeneity

Recently, applied sciences, including longitudinal and clustered studies in biomedicine require the analysis of ultra-high dimensional linear mixed effects models where we need to select important fixed effect variables from a vast pool of available candidates. However, all existing literature assume that all the available covariates and random effect components are independent of the model error which is often violated (endogeneity) in practice. In this paper, we first investigate this important issue in ultra-high dimensional linear mixed effects models with particular focus on the fixed effects selection. We study the effects of different types of endogeneity on existing regularization methods and prove their inconsistencies. Then, we propose a new profiled focused generalized method of moments (PFGMM) approach to consistently select fixed effects under 'error-covariate' endogeneity, i.e., in the presence of correlation between the model error and covariates. Our proposal is proved to be oracle consistent with probability tending to one and works well under most other type of endogeneity too. Additionally, we also propose and illustrate a few consistent parameter estimators, including those of the variance components, along with variable selection through PFGMM. Empirical simulations and an interesting real data example further support the claimed utility of our proposal.

stat.ME↗

arXiv2020

Robust Generalised Quadratic Discriminant Analysis

Quadratic discriminant analysis (QDA) is a widely used statistical tool to classify observations from different multivariate Normal populations. The generalized quadratic discriminant analysis (GQDA) classification rule/classifier, which generalizes the QDA and the minimum Mahalanobis distance (MMD) classifiers to discriminate between populations with underlying elliptically symmetric distributions competes quite favorably with the QDA classifier when it is optimal and performs much better when QDA fails under non-Normal underlying distributions, e.g. Cauchy distribution. However, the classification rule in GQDA is based on the sample mean vector and the sample dispersion matrix of a training sample, which are extremely non-robust under data contamination. In real world, since it is quite common to face data highly vulnerable to outliers, the lack of robustness of the classical estimators of the mean vector and the dispersion matrix reduces the efficiency of the GQDA classifier significantly, increasing the misclassification errors. The present paper investigates the performance of the GQDA classifier when the classical estimators of the mean vector and the dispersion matrix used therein are replaced by various robust counterparts. Applications to various real data sets as well as simulation studies reveal far better performance of the proposed robust versions of the GQDA classifier. A Comparative study has been made to advocate the appropriate choice of the robust estimators to be used in a specific situation of the degree of contamination of the data sets.

stat.ME↗

arXiv2020

Robust Wald-type test in GLM with random design based on minimum density power divergence estimators

We consider the problem of robust inference under the generalized linear model (GLM) with stochastic covariates. We derive the properties of the minimum density power divergence estimator of the parameters in GLM with random design and use this estimator to propose robust Wald-type tests for testing any general composite null hypothesis about the GLM. The asymptotic and robustness properties of the proposed tests are also examined for the GLM with random design. Application of the proposed robust inference procedures to the popular Poisson regression model for analyzing count data is discussed in detail both theoretically and numerically through simulation studies and real data examples.

stat.ME↗

arXiv2020

All sky angular power spectrum: I. Estimating brightness temperature fluctuations using TGSS 150 MHz survey

Measurements of the Galactic synchrotron emission is relevant for the 21-cm studies from the Epoch of Reionization. The study of the synchrotron emission is also useful to quantify the fluctuations in the magnetic field and the cosmic ray electron density of the turbulent interstellar medium (ISM) of our Galaxy. Here, we present the all-sky angular power spectrum $(C_{\ell})$ measurements of the diffuse synchrotron emission using the TIFR GMRT Sky Survey (TGSS) at 150 {\rm MHz}. We estimate $C_{\ell}$ using visibility data both before and after subtracting the modelled point sources. The amplitude of the measured $C_{\ell}$ falls significantly after subtracting the point sources, and it is also slightly higher in the Galactic plane for the residual data. The residual $C_{\ell}$ is most likely to be dominated by the Galactic synchrotron emission. The amplitude of the residual $C_{\ell}$ falls significantly away from the Galactic plane. We find the measurements are quite symmetric in the Northern and Southern hemispheres except in the latitude range $15-30^{\circ}$ which is the transition region from the disk dominated to diffuse halo dominated region. The comparison between this interferometric measurement with the scaled version of the Haslam rms map at 150 {\rm MHz} shows that the correlation coefficient $(r)$ is more than 0.5 for most of the latitude ranges considered here. This signifies the TGSS survey is quite sensitive to the diffuse Galactic synchrotron radiation.

astro-ph.CO↗

arXiv2019

On Robust Pseudo-Bayes Estimation for the Independent Non-homogeneous Set-up

The ordinary Bayes estimator based on the posterior density suffers from the potential problems of non-robustness under data contamination or outliers. In this paper, we consider the general set-up of independent but non-homogeneous (INH) observations and study a robustified pseudo-posterior based estimation for such parametric INH models. In particular, we focus on the $R^{(α)}$-posterior developed by Ghosh and Basu (2016) for IID data and later extended by Ghosh and Basu (2017) for INH set-up, where its usefulness and desirable properties have been numerically illustrated. In this paper, we investigate the detailed theoretical properties of this robust pseudo Bayes $R^{(α)}$-posterior and associated $R^{(α)}$-Bayes estimate under the general INH set-up with applications to fixed-design regressions. We derive a Bernstein von-Mises types asymptotic normality results and Laplace type asymptotic expansion of the $R^{(α)}$-posterior as well as the asymptotic distributions of the expected $R^{(α)}$-posterior estimators. The required conditions and the asymptotic results are simplified for linear regressions with known or unknown error variance and logistic regression models with fixed covariates. The robustness of the $R^{(α)}$-posterior and associated estimators are theoretically examined through appropriate influence function analyses under general INH set-up; illustrations are provided for the case of linear regression. A high breakdown point result is derived for the expected $R^{(α)}$-posterior estimators of the location parameter under a location-scale type model. Some interesting real life data examples illustrate possible applications.

math.ST↗

arXiv2019

Maximum Entropy Framework for a Universal Rank Order distribution with Socio-economic Applications

In this paper we derive the maximum entropy characteristics of a particular rank order distribution, namely the discrete generalized beta distribution, which has recently been observed to be extremely useful in modelling many several rank-size distributions from different context in Arts and Sciences, as a two-parameter generalization of Zipf's law. Although it has been seen to provide excellent fits for several real world empirical datasets, the underlying theory responsible for the success of this particular rank order distribution is not explored properly. Here we, for the first time, provide its generating process which describes it as a natural maximum entropy distribution under an appropriate bivariate utility constraint. Further, considering the similarity of the proposed utility function with the usual logarithmic utility function from economic literature, we have also explored its acceptability in universal modeling of different types of socio-economic factors within a country as well as across the countries. The values of distributional parameters estimated through a rigorous statistical estimation method, along with the $entropy$ values, are used to characterize the distributions of all these socio-economic factors over the years.

physics.soc-ph↗

arXiv2019

Robust Inference for Skewed data in Health Sciences

Health data are often not symmetric to be adequately modeled through the usual normal distributions; most of them exhibit skewed patterns. They can indeed be modeled better through the larger family of skew-normal distributions covering both skewed and symmetric cases. However, the existing likelihood based inference, that is routinely performed in these cases, is extremely non-robust against data contamination/outliers. Since outliers are not uncommon in complex real-life experimental datasets, a robust methodology automatically taking care of the noises in the data would be of great practical value to produce stable and more precise research insights leading to better policy formulation. In this paper, we develop a class of robust estimators and testing procedures for the family of skew-normal distributions using the minimum density power divergence approach with application to health data. In particular, a robust procedure for testing of symmetry is discussed in the presence of outliers. Two efficient computational algorithms are discussed. Besides deriving the asymptotic and robustness theory for the proposed methods, their advantages and utilities are illustrated through simulations and a couple of real-life applications for health data of athletes from Australian Institute of Sports and AIDS clinical trial data.

stat.AP↗

arXiv2019

A Robust Generalization of the Rao Test

This paper presents new families of Rao-type test statistics based on the minimum density power divergence estimators which provide robust generalizations for testing simple and composite null hypotheses. The asymptotic null distributions of the proposed tests are obtained and their robustness properties are also theoretically studied. Numerical illustrations are provided to substantiate the theory developed. On the whole, the proposed tests are seen to be excellent alternatives to the classical Rao test.

stat.ME↗

arXiv2019

Optimal Guessing under Nonextensive Framework and associated Moment Bounds

We consider the problem of guessing the realization of a random variable but under more general Tsallis' non-extensive entropic framework rather than the classical Maxwell-Boltzman-Gibbs-Shannon framework. We consider both the conditional guessing problem in the presence of some related side information, and the unconditional one where no such side-information is available. For both types of the problem, the non-extensive moment bounds of the required number of guesses are derived; here we use the $q$-normalized expectation in place of the usual (linear) expectation to define the non-extensive moments. These moment bounds are seen to be a function of the logarithmic norm entropy measure, a recently developed two-parameter generalization of the Renyi entropy, and hence provide their information theoretic interpretation. We have also considered the case of uncertain source distribution and derived the non-extensive moment bounds for the corresponding mismatched guessing function. These mismatched bounds are interestingly seen to be linked with an important robust statistical divergence family known as the relative $(α,β)$-entropies; similar link is discussed between the optimum mismatched guessing with the extremes of these relative entropy measures.

cs.IT↗

arXiv2019

Universal City-size distributions through rank ordering

We consider a two-parameter discrete generalized beta (DGB) distribution and propose its universal applications to study the size-distribution of the urban agglomerations across various countries in the world, where the urban agglomerations include the small and mid-sized cities along with the heavily populated cities. Our proposition is validated by an exhaustive study with the 3 decades' census data for India and China and census data of USA for a time window of 8 years. Moreover, we have studied the city size distributions for many different countries, like Brazil, Italy, Sweden, Australia, Uganda etc., from all the continents around the world according to the availability of the data. The detailed analyses exhibit a unique global pattern for the city size distributions, from low to high size, across the world with various geographic and economic conditions. Further analyses based on the entropy of the distribution provide insights on the underlying randomness and spreads of the city sizes within a country. The DGB distribution, a typical rank order (RO) distribution, through its two parameters, not only fits the data on wider range of city sizes better than the well-known power law for all the countries considered, it also helps us to characterize, discriminate and study their evolution over time.

physics.soc-ph↗

arXiv2019

Robust semiparametric inference for polytomous logistic regression with complex survey design

Analyzing polytomous response from a complex survey scheme, like stratified or cluster sampling is very crucial in several socio-economics applications. We present a class of minimum quasi weighted density power divergence estimators for the polytomous logistic regression model with such a complex survey. This family of semiparametric estimators is a robust generalization of the maximum quasi weighted likelihood estimator exploiting the advantages of the popular density power divergence measure. Accordingly robust estimators for the design effects are also derived. Robust testing of general linear hypotheses on the regression coefficients are proposed using the new estimators. Their asymptotic distributions and robustness properties are theoretically studied and also empirically validated through a numerical example and an extensive Monte Carlo study.

stat.ME↗

arXiv2019

Robust Wald-Type Tests under Random Censoring with Applications to Clinical Trial Analyses

Randomly censored survival data are frequently encountered in applied sciences including biomedical or reliability applications and clinical trial analyses. Testing the significance of statistical hypotheses is crucial in such analyses to get conclusive inference but the existing likelihood based tests, under a fully parametric model, are extremely non-robust against outliers in the data. Although, there exists a few robust parameter estimators (e.g., M-estimators and minimum density power divergence estimators) given randomly censored data, there is hardly any robust testing procedure available in the literature in this context. One of the major difficulties in this context is the construction of a suitable consistent estimator of the asymptotic variance of M estimators; the latter is a function of the unknown censoring distribution. In this paper, we take the first step in this direction by proposing a consistent estimator of asymptotic variance of the M-estimators based on randomly censored data without any assumption on the form of the censoring scheme. We then describe and study a class of robust Wald-type tests for parametric statistical hypothesis, both simple as well as composite, under such set-up, along with their general asymptotic and robustness properties. Robust tests for comparing two independent randomly censored samples and robust tests against one sided alternatives are also discussed. Their advantages and usefulness are demonstrated for the tests based on the minimum density power divergence estimators with specific attention to clinical trial analyses.

stat.ME↗

arXiv2018

Robust and Efficient Estimation in the Parametric Cox Regression Model under Random Censoring

Cox proportional hazard regression model is a popular tool to analyze the relationship between a censored lifetime variable with other relevant factors. The semi-parametric Cox model is widely used to study different types of data arising from applied disciplines like medical science, biology, reliability studies and many more. A fully parametric version of the Cox regression model, if properly specified, can yield more efficient parameter estimates leading to better insight generation. However, the existing maximum likelihood approach of generating inference under the fully parametric Cox regression model is highly non-robust against data-contamination which restricts its practical usage. In this paper we develop a robust estimation procedure for the parametric Cox regression model based on the minimum density power divergence approach. The proposed minimum density power divergence estimator is seen to produce highly robust estimates under data contamination with only a slight loss in efficiency under pure data. Further, they are always seen to generate more precise inference than the likelihood based estimates under the semi-parametric Cox models or their existing robust versions. We also sketch the derivation of the asymptotic properties of the proposed estimator using the martingale approach and justify their robustness theoretically through the influence function analysis. The practical applicability and usefulness of the proposal are illustrated through simulations and a real data example.

stat.ME↗

arXiv2018

Power and Level Robustness of A Composite Hypothesis Testing under Independent Non-Homogeneous Data

Robust tests of general composite hypothesis under non-identically distributed observations is always a challenge. Ghosh and Basu (2018, Statistica Sinica, 28, 1133--1155) have proposed a new class of test statistics for such problems based on the density power divergence, but their robustness with respect to the size and power are not studied in detail. This note fills this gap by providing a rigorous derivation of power and level influence functions of these tests to theoretically justify their robustness. Applications to the fixed-carrier linear regression model are also provided with empirical illustrations.

math.ST↗