arXiv subjects

Abhik Ghosh

Publications and source records attributed to Abhik Ghosh.

At least 37 records · Page 2Linked to original sources

arXiv2025

A demonstration of the effect of fringe-rate filtering in the Hydrogen Epoch of Reionization Array delay power spectrum pipeline

Radio interferometers targeting the 21cm brightness temperature fluctuations at high redshift are subject to systematic effects that operate over a range of different timescales. These can be isolated by designing appropriate Fourier filters that operate in fringe-rate (FR) space, the Fourier pair of local sidereal time (LST). Applications of FR filtering include separating effects that are correlated with the rotating sky vs. those relative to the ground, down-weighting emission in the primary beam sidelobes, and suppressing noise. FR filtering causes the noise contributions to the visibility data to become correlated in time however, making interpretation of subsequent averaging and error estimation steps more subtle. In this paper, we describe fringe rate filters that are implemented using discrete prolate spheroidal sequences, and designed for two different purposes -- beam sidelobe/horizon suppression (the `mainlobe' filter), and ground-locked systematics removal (the `notch' filter). We apply these to simulated data, and study how their properties affect visibilities and power spectra generated from the simulations. Included is an introduction to fringe-rate filtering and a demonstration of fringe-rate filters applied to simple situations to aid understanding.

astro-ph.CO↗

arXiv2025

Probing Equatorial Ionospheric TEC at Sub-GHz Frequencies with Wide-Band (B4) uGMRT Interferometric Data

Phase stability at low radio frequencies is severely impacted by ionospheric propagation delays. Radio interferometers such as the Giant Metrewave Radio Telescope (GMRT) are capable of detecting changes in the ionosphere's total electron content (TEC) over larger spatial scales and with greater sensitivity compared to conventional tools like the Global Navigation Satellite System (GNSS). Thanks to its unique design featuring both a dense central array and long outer arms-and its strategic location, the GMRT is particularly well-suited for studying the sensitive ionospheric region located between the northern peak of the Equatorial Ionization Anomaly (EIA) and the magnetic equator. In this study, we observe the bright flux calibrator 3C48 for ten hours to characterize and study the low-latitude ionosphere with the upgraded GMRT (uGMRT). We outline the methods used for wideband data reduction and processing to accurately measure differential TEC (dTEC) between antenna pairs, achieving a precision of less than 1 mTECU for the central square antennas and approximately 1 mTECU for the arm antennas. The measured dTEC values are used to estimate the TEC gradient across the GMRT arm antennas. We measure the ionospheric phase structure function and find a power-law slope of $β= 1.72$, indicating deviations from pure Kolmogorov turbulence. The inferred diffractive scale the spatial separation over which the phase variance reaches $1 \text{rad}^2$ is 6.66 km. A small diffractive scale implies high phase variability across the field of view and reduced temporal coherence, which poses challenges for calibration and imaging.

astro-ph.IM↗

arXiv2025

Exploring Citation Diversity in Scholarly Literature: An Entropy-Based Approach

This study explores the citation diversity in scholarly literature, analyzing different patterns of citations observed within different countries and academic disciplines. We examine citation distributions across top institutions within certain countries and find that the higher end of the distribution follows a Power Law or Pareto Law pattern; the scaling exponent of the Pareto Law varies depending on the number of top institutions included in the analysis. By adopting a novel entropy-based diversity measure, our findings reveal that countries with both small and large economies tend to cluster similarly in terms of citation diversity. The composition of countries within each group changes as the number of top institutions considered in the analysis varies. Moreover, we analyze citation diversity among award-winning scientists across six scientific disciplines, finding significant variations. We also explore the evolution of citation diversity over the past century across multiple fields. A gender-based study in several disciplines confirms varying citation diversities among male and female scientists. Our innovative citation diversity measure stands out as a valuable tool for assessing the unevenness of citation distributions, providing deeper insights that go beyond what traditional citation counts alone can reveal. This comprehensive analysis enhances our understanding of global scientific contributions and fosters a more equitable view of academic achievements.

physics.soc-ph↗

arXiv2025

Asymptotic Breakdown Point Analysis for a General Class of Minimum Divergence Estimators

Robust inference based on the minimization of statistical divergences has proved to be a useful alternative to classical techniques based on maximum likelihood and related methods. Basu et al. (1998) introduced the density power divergence (DPD) family as a measure of discrepancy between two probability density functions and used this family for robust estimation of the parameter for independent and identically distributed data. Ghosh et al. (2017) proposed a more general class of divergence measures, namely the S-divergence family and discussed its usefulness in robust parametric estimation through several asymptotic properties and some numerical illustrations. In this paper, we develop the results concerning the asymptotic breakdown point for the minimum S-divergence estimators (in particular the minimum DPD estimator) under general model setups. The primary result of this paper provides lower bounds to the asymptotic breakdown point of these estimators which are independent of the dimension of the data, in turn corroborating their usefulness in robust inference under high dimensional data.

math.ST↗

arXiv2025

Conditional variable screening for ultra-high dimensional longitudinal data with time interactions

In recent years we have been able to gather large amounts of genomic data at a fast rate, creating situations where the number of variables greatly exceeds the number of observations. In these situations, most models that can handle a moderately high dimension will now become computationally infeasible or unstable. Hence, there is a need for a pre-screening of variables to reduce the dimension efficiently and accurately to a more moderate scale. There has been much work to develop such screening procedures for independent outcomes. However, much less work has been done for high-dimensional longitudinal data in which the observations can no longer be assumed to be independent. In addition, it is of interest to capture possible interactions between the genomic variable and time in many of these longitudinal studies. In this work, we propose a novel conditional screening procedure that ranks variables according to the likelihood value at the maximum likelihood estimates in a marginal linear mixed model, where the genomic variable and its interaction with time are included in the model. This is to our knowledge the first conditional screening approach for clustered data. We prove that this approach enjoys the sure screening property, and assess the finite sample performance of the method through simulations.

stat.ME↗

arXiv2025

matvis: A matrix-based visibility simulator for fast forward modelling of many-element 21 cm arrays

Detection of the faint 21 cm line emission from the Cosmic Dawn and Epoch of Reionisation will require not only exquisite control over instrumental calibration and systematics to achieve the necessary dynamic range of observations but also validation of analysis techniques to demonstrate their statistical properties and signal loss characteristics. A key ingredient in achieving this is the ability to perform high-fidelity simulations of the kinds of data that are produced by the large, many-element, radio interferometric arrays that have been purpose-built for these studies. The large scale of these arrays presents a computational challenge, as one must simulate a detailed sky and instrumental model across many hundreds of frequency channels, thousands of time samples, and tens of thousands of baselines for arrays with hundreds of antennas. In this paper, we present a fast matrix-based method for simulating radio interferometric measurements (visibilities) at the necessary scale. We achieve this through judicious use of primary beam interpolation, fast approximations for coordinate transforms, and a vectorised outer product to expand per-antenna quantities to per-baseline visibilities, coupled with standard parallelisation techniques. We validate the results of this method, implemented in the publicly-available matvis code, against a high-precision reference simulator, and explore its computational scaling on a variety of problems.

astro-ph.IM↗

arXiv2025

Robust inference for linear regression models with possibly skewed error distribution

Traditional methods for linear regression generally assume that the underlying error distribution, equivalently the distribution of the responses, is normal. Yet, sometimes real life response data may exhibit a skewed pattern, and assuming normality would not give reliable results in such cases. This is often observed in cases of some biomedical, behavioral, socio-economic and other variables. In this paper, we propose to use the class of skew normal (SN) distributions, which also includes the ordinary normal distribution as its special case, as the model for the errors in a linear regression setup and perform subsequent statistical inference using the popular and robust minimum density power divergence approach to get stable insights in the presence of possible data contamination (e.g., outliers). We provide the asymptotic distribution of the proposed estimator of the regression parameters and also propose robust Wald-type tests of significance for these parameters. We provide an influence function analysis of these estimators and test statistics, and also provide level and power influence functions. Numerical verification including simulation studies and real data analysis is provided to substantiate the theory developed.

stat.ME↗

arXiv2024

Robust Genomic Prediction and Heritability Estimation using Density Power Divergence

This manuscript delves into the intersection of genomics and phenotypic prediction, focusing on the statistical innovation required to navigate the complexities introduced by noisy covariates and confounders. The primary emphasis is on the development of advanced robust statistical models tailored for genomic prediction from single nucleotide polymorphism data in plant and animal breeding and multi-field trials. The manuscript highlights the significance of incorporating all estimated effects of marker loci into the statistical framework and aiming to reduce the high dimensionality of data while preserving critical information. This paper introduces a new robust statistical framework for genomic prediction, employing one-stage and two-stage linear mixed model analyses along with utilizing the popular robust minimum density power divergence estimator (MDPDE) to estimate genetic effects on phenotypic traits. The study illustrates the superior performance of the proposed MDPDE-based genomic prediction and associated heritability estimation procedures over existing competitors through extensive empirical experiments on artificial datasets and application to a real-life maize breeding dataset. The results showcase the robustness and accuracy of the proposed MDPDE-based approaches, especially in the presence of data contamination, emphasizing their potential applications in improving breeding programs and advancing genomic prediction of phenotyping traits.

stat.ME↗

arXiv2024

Towards $21$-cm intensity mapping at $z=2.28$ with uGMRT using the tapered gridded estimator -- IV. Wideband analysis

We present a Wideband Tapered Gridded Estimator (TGE), which incorporates baseline migration and variation of the primary beam pattern for neutral hydrogen (${\rm H\hspace{0.5mm}}{\scriptsize {\rm I}}$) 21-cm intensity mapping (IM) with large frequency bandwidth radio-interferometric observations. Here we have analysed $394-494 \, {\rm MHz}$ $(z = 1.9 - 2.6)$ uGMRT data to estimate the Multi-frequency Angular Power Spectrum (MAPS) $C_\ell(Δν)$ from which we have removed the foregrounds using the polynomial fitting (PF) and Gaussian Process Regression (GPR) methods developed in our earlier work. Using the residual $C_\ell(Δν)$ to estimate the mean squared 21-cm brightness temperature fluctuation $Δ^2(k)$, we find that this is consistent with $0 \pm 2 σ$ in several $k$ bins. The resulting $2σ$ upper limit $Δ^2(k) < (4.68)^2 \, \rm{mK^2}$ at $k=0.219\,\rm{Mpc^{-1}}$ is nearly $15$ times tighter than earlier limits obtained from a smaller bandwidth ($24.4 \, {\rm MHz}$) of the same data. The $2σ$ upper limit $[Ω_{\rm H\hspace{0.5mm}{\scriptsize {\rm I}}} b_{\rm H\hspace{0.5mm}{\scriptsize {\rm I}}}] < 1.01 \times 10^{-2}$ is within an order of magnitude of the value expected from independent estimates of the ${\rm H\hspace{0.5mm}}{\scriptsize {\rm I}}$ mass density $Ω_{\rm H\hspace{0.5mm}{\scriptsize {\rm I}}}$ and the ${\rm H\hspace{0.5mm}}{\scriptsize {\rm I}}$ bias $b_{\rm H\hspace{0.5mm}{\scriptsize {\rm I}}}$. The techniques used here can be applied to other telescopes and frequencies, including $\sim 150 \, {\rm MHz}$ Epoch of Reionization observations.

astro-ph.CO↗

arXiv2024

Robust statistical modeling of monthly rainfall: The minimum density power divergence approach

Statistical modeling of monthly, seasonal, or annual rainfall data is an important research area in meteorology. These models play a crucial role in rainfed agriculture, where a proper assessment of the future availability of rainwater is necessary. The rainfall amount during a rainy month or a whole rainy season} can take any positive value and some simple (one or two-parameter) probability models supported over the positive real line that are generally used for rainfall modeling are exponential, gamma, Weibull, lognormal, Pearson Type-V/VI, log-logistic, etc., where the unknown model parameters are routinely estimated using the maximum likelihood estimator (MLE). However, the presence of outliers or extreme observations is a common issue in rainfall data and the MLEs being highly sensitive to them often leads to spurious inference. Here, we discuss a robust parameter estimation approach based on the minimum density power divergence estimator (MDPDE). We fit the above four parametric models to the detrended areally-weighted monthly rainfall data from the 36 meteorological subdivisions of India for the years 1951-2014 and compare the fits based on MLE and the proposed optimum MDPDE; the superior performance of MDPDE is showcased for several cases. For all month-subdivision combinations, we discuss the best-fit models and median rainfall amounts.

stat.AP↗

arXiv2024

Hydrogen Epoch of Reionization Array (HERA) Phase II Deployment and Commissioning

This paper presents the design and deployment of the Hydrogen Epoch of Reionization Array (HERA) phase II system. HERA is designed as a staged experiment targeting 21 cm emission measurements of the Epoch of Reionization. First results from the phase I array are published as of early 2022, and deployment of the phase II system is nearing completion. We describe the design of the phase II system and discuss progress on commissioning and future upgrades. As HERA is a designated Square Kilometer Array (SKA) pathfinder instrument, we also show a number of "case studies" that investigate systematics seen while commissioning the phase II system, which may be of use in the design and operation of future arrays. Common pathologies are likely to manifest in similar ways across instruments, and many of these sources of contamination can be mitigated once the source is identified.

astro-ph.IM↗

arXiv2024

Robust and Efficient Estimation in Ordinal Response Models using the Density Power Divergence

In real life, we frequently come across data sets that involve some independent explanatory variable(s) generating a set of ordinal responses. These ordinal responses may correspond to an underlying continuous latent variable, which is linearly related to the covariate(s), and takes a particular (ordinal) label depending on whether this latent variable takes value in some suitable interval specified by a pair of (unknown) cut-offs. The most efficient way of estimating the unknown parameters (i.e., the regression coefficients and the cut-offs) is the method of maximum likelihood (ML). However, contamination in the data set either in the form of misspecification of ordinal responses, or the unboundedness of the covariate(s), might destabilize the likelihood function to a great extent where the ML based methodology might lead to completely unreliable inferences. In this paper, we explore a minimum distance estimation procedure based on the popular density power divergence (DPD) to yield robust parameter estimates for the ordinal response model. This paper highlights how the resulting estimator, namely the minimum DPD estimator (MDPDE), can be used as a practical robust alternative to the classical procedures based on the ML. We rigorously develop several theoretical properties of this estimator, and provide extensive simulations to substantiate the theory developed.

stat.ME↗

arXiv2023

Robust Clustering with Normal Mixture Models: A Pseudo $β$-Likelihood Approach

As in other estimation scenarios, likelihood based estimation in the normal mixture set-up is highly non-robust against model misspecification and presence of outliers (apart from being an ill-posed optimization problem). A robust alternative to the ordinary likelihood approach for this estimation problem is proposed which performs simultaneous estimation and data clustering and leads to subsequent anomaly detection. To invoke robustness, the methodology based on the minimization of the density power divergence (or alternatively, the maximization of the $β$-likelihood) is utilized under suitable constraints. An iteratively reweighted least squares approach has been followed in order to compute the proposed estimators for the component means (or equivalently cluster centers) and component dispersion matrices which leads to simultaneous data clustering. Some exploratory techniques are also suggested for anomaly detection, a problem of great importance in the domain of statistics and machine learning. The proposed method is validated with simulation studies under different set-ups; it performs competitively or better compared to the popular existing methods like K-medoids, TCLUST, trimmed K-means and MCLUST, especially when the mixture components (i.e., the clusters) share regions with significant overlap or outlying clusters exist with small but non-negligible weights (particularly in higher dimensions). Two real datasets are also used to illustrate the performance of the newly proposed method in comparison with others along with an application in image processing. The proposed method detects the clusters with lower misclassification rates and successfully points out the outlying (anomalous) observations from these datasets.

stat.ME↗

arXiv2023

Multi-scale analysis of rural and urban areas: A case study of Indian districts

This paper explores the extension of the idea of allometric urban scaling law to study the scaling behaviour of Indian districts, with both the urban and rural population. To proceed, we have chosen districts (both rural and urban) of India, a relatively larger local administrative units, which are more or less independently functional within a country. This interdisciplinary work focus on the scaling analysis of various socio-economic indicators (SEIs) corresponding to the size (population) of four distinct urbanization classes, namely rural, semi-rural, semi-urban and urban districts. The scaling exponents were estimated for each classes for the years 2001 and 2011 along with their goodness-of-fit measured by the $R^2$ values. Our rigorous statistical analysis indicates that the scaling laws indeed exist even at the district level for most of the SEIs considered, related to education, employment, housing, health, etc.; the $R^2$ values obtained for these SEIs are very high (often greater than 0.8 or 0.9) in both the the years. Moreover, linearity of the scaling factors have been statistically tested and it has been found, at 95% level of confidence, that not all the SEIs behave linearly; some of them are characterized by super-linear behaviour and some behave sub-linearly. Statistical hypothesis tests have also been performed to test the equality of two scaling factors corresponding to two distinct classes and two different years to understand the differences in scaling relationships among increasing urbanisation classes and their changes over time.

physics.soc-ph↗

arXiv2023

Towards $21$-cm intensity mapping at $z=2.28$ with uGMRT using the tapered gridded estimator III: Foreground removal

Neutral hydrogen (HI) $21$-cm intensity mapping (IM) is a promising probe of the large-scale structures in the Universe. However, a few orders of magnitude brighter foregrounds obscure the IM signal. Here we use the Tapered Gridded Estimator (TGE) to estimate the multi-frequency angular power spectrum (MAPS) $C_{\ell}(Δν)$ from a $24.4\,\rm{MHz}$ bandwidth uGMRT Band $3$ data at $432.8\,\rm{MHz}$. In $C_{\ell}(Δν)$ foregrounds remain correlated across the entire $Δν$ range, whereas the $21$-cm signal is localized within $Δν\le[Δν]$ (typically $0.5-1\,\rm{MHz}$). Assuming the range $Δν>[Δν]$ to have minimal $21$-cm signal, we use $C_{\ell}(Δν)$ in this range to model the foregrounds. This foreground model is extrapolated to $Δν\leq[Δν]$, and subtracted from the measured $C_{\ell}(Δν)$. The residual $[C_{\ell}(Δν)]_{\rm res}$ in the range $Δν\le[Δν]$ is used to constrain the $21$-cm signal, compensating for the signal loss from foreground subtraction. $[C_{\ell}(Δν)]_{\rm{res}}$ is found to be noise-dominated without any trace of foregrounds. Using $[C_{\ell}(Δν)]_{\rm res}$ we constrain the $21$-cm brightness temperature fluctuations $Δ^2(k)$, and obtain the $2σ$ upper limit $Δ_{\rm UL}^2(k)\leq(18.07)^2\,\rm{mK^2}$ at $k=0.247\,\rm{Mpc}^{-1}$. We further obtain the $2σ$ upper limit $ [Ω_{\rm HI}b_{\rm HI}]_{\rm UL}\leq0.022$ where $Ω_{\rm HI}$ and $b_{\rm HI}$ are the comoving HI density and bias parameters respectively. Although the upper limit is nearly $10$ times larger than the expected $21$-cm signal, it is $3$ times tighter over previous works using foreground avoidance on the same data.

astro-ph.CO↗

arXiv2023

Robust Principal Component Analysis using Density Power Divergence

Principal component analysis (PCA) is a widely employed statistical tool used primarily for dimensionality reduction. However, it is known to be adversely affected by the presence of outlying observations in the sample, which is quite common. Robust PCA methods using M-estimators have theoretical benefits, but their robustness drop substantially for high dimensional data. On the other end of the spectrum, robust PCA algorithms solving principal component pursuit or similar optimization problems have high breakdown, but lack theoretical richness and demand high computational power compared to the M-estimators. We introduce a novel robust PCA estimator based on the minimum density power divergence estimator. This combines the theoretical strength of the M-estimators and the minimum divergence estimators with a high breakdown guarantee regardless of data dimension. We present a computationally efficient algorithm for this estimate. Our theoretical findings are supported by extensive simulations and comparisons with existing robust PCA methods. We also showcase the proposed algorithm's applicability on two benchmark datasets and a credit card transactions dataset for fraud detection.

stat.ME↗

arXiv2023

Robust adaptive variable selection in ultra-high dimensional linear regression models

We consider the problem of simultaneous variable selection and estimation of the corresponding regression coefficients in an ultra-high dimensional linear regression models, an extremely important problem in the recent era. The adaptive penalty functions are used in this regard to achieve the oracle variable selection property along with easier computational burden. However, the usual adaptive procedures (e.g., adaptive LASSO) based on the squared error loss function is extremely non-robust in the presence of data contamination which are quite common with large-scale data (e.g., noisy gene expression data, spectra and spectral data). In this paper, we present a regularization procedure for the ultra-high dimensional data using a robust loss function based on the popular density power divergence (DPD) measure along with the adaptive LASSO penalty. We theoretically study the robustness and the large-sample properties of the proposed adaptive robust estimators for a general class of error distributions; in particular, we show that the proposed adaptive DPD-LASSO estimator is highly robust, satisfies the oracle variable selection property, and the corresponding estimators of the regression coefficients are consistent and asymptotically normal under easily verifiable set of assumptions. Numerical illustrations are provided for the mostly used normal error density. Finally, the proposal is applied to analyze an interesting spectral dataset, in the field of chemometrics, regarding the electron-probe X-ray microanalysis (EPXMA) of archaeological glass vessels from the 16th and 17th centuries.

stat.ME↗

arXiv2023

rSVDdpd: A Robust Scalable Video Surveillance Background Modelling Algorithm

A basic algorithmic task in automated video surveillance is to separate background and foreground objects. Camera tampering, noisy videos, low frame rate, etc., pose difficulties in solving the problem. A general approach that classifies the tampered frames, and performs subsequent analysis on the remaining frames after discarding the tampered ones, results in loss of information. Several robust methods based on robust principal component analysis (PCA) have been introduced to solve this problem. To date, considerable effort has been expended to develop robust PCA via Principal Component Pursuit (PCP) methods with reduced computational cost and visually appealing foreground detection. However, the convex optimizations used in these algorithms do not scale well to real-world large datasets due to large matrix inversion steps. Also, an integral component of these foreground detection algorithms is singular value decomposition which is nonrobust. In this paper, we present a new video surveillance background modelling algorithm based on a new robust singular value decomposition technique rSVDdpd which takes care of both these issues. We also demonstrate the superiority of our proposed algorithm on a benchmark dataset and a new real-life video surveillance dataset in the presence of camera tampering. Software codes and additional illustrations are made available at the accompanying website rSVDdpd Homepage (https://subroy13.github.io/rsvddpd-home/)

stat.AP↗