arXiv subjects

Subhabrata Majumdar

Publications and source records attributed to Subhabrata Majumdar.

At least 55 records · Page 3Linked to original sources

arXiv2022

Local Dampening: Differential Privacy for Non-numeric Queries via Local Sensitivity

Differential privacy is the state-of-the-art formal definition for data release under strong privacy guarantees. A variety of mechanisms have been proposed in the literature for releasing the output of numeric queries (e.g., the Laplace mechanism and smooth sensitivity mechanism). Those mechanisms guarantee differential privacy by adding noise to the true query's output. The amount of noise added is calibrated by the notions of global sensitivity and local sensitivity of the query that measure the impact of the addition or removal of an individual on the query's output. Mechanisms that use local sensitivity add less noise and, consequently, have a more accurate answer. However, although there has been some work on generic mechanisms for releasing the output of non-numeric queries using global sensitivity (e.g., the Exponential mechanism), the literature lacks generic mechanisms for releasing the output of non-numeric queries using local sensitivity to reduce the noise in the query's output. In this work, we remedy this shortcoming and present the local dampening mechanism. We adapt the notion of local sensitivity for the non-numeric setting and leverage it to design a generic non-numeric mechanism. We provide theoretical comparisons to the exponential mechanism and show under which conditions the local dampening mechanism is more accurate than the exponential mechanism. We illustrate the effectiveness of the local dampening mechanism by applying it to three diverse problems: (i) percentile selection problem. We report the p-th element in the database; (ii) Influential node analysis. Given an influence metric, we release the top-k most influential nodes while preserving the privacy of the relationship between nodes in the network; (iii) Decision tree induction. We provide a private adaptation to the ID3 algorithm to build decision trees from a given tabular dataset.

cs.CR↗

arXiv2022

Non-thermal Sunyaev-Zeldovich signal from radio galaxy lobes

Energetic electrons in the lobes of radio galaxies make them potential sources for not only radio and X-rays but also Sunyaev-Zeldovich (SZ) distortions in the cosmic microwave background (CMB) radiation. Previous works have discussed the energetics of radio galaxy lobes, but assuming thermal SZ effect, coming from the non-thermal electron population. We use an improved evolutionary model for radio galaxy lobes to estimate the observed parameters such as the radio luminosity and intensity of SZ-distortions at the redshifts of observation. We, further, quantify the effects of various relevant physical parameters of the radio galaxies, such as the jet power, the time scale over which the jet is active, the evolutionary time scale for the lobe, etc on the observed parameters. For current SZ observations towards galaxy clusters, we find that the non-thermal SZ distortions from radio lobes embedded in galaxy clusters can be non-negligible compared to the amount of thermal SZ distortion from the intra-cluster medium and, hence, can not be neglected. We show that small and young (and preferably residing in a cluster environment) radio galaxies offer better prospects for the detection of the non-thermal SZ signal from these sources. We further discuss the limits on different physical parameters for some sources for which SZ effect has been either detected or upper limits are available. The evolutionary models enable us to obtain limits, previously unavailable, on the low energy cut-off of electron spectrum ($p_{min} \sim 1\hbox{--}2$) in order to explain the recent non-thermal SZ detection. Finally, we discuss how future CMB experiments, which would cover higher frequency bands ($>$400 GHz), may provide clear signatures for non-thermal SZ effect.

astro-ph.HE↗

arXiv2022

Detecting Bias in the Presence of Spatial Autocorrelation

In spite of considerable practical importance, current algorithmic fairness literature lacks technical methods to account for underlying geographic dependency while evaluating or mitigating bias issues for spatial data. We initiate the study of bias in spatial applications in this paper, taking the first step towards formalizing this line of quantitative methods. Bias in spatial data applications often gets confounded by underlying spatial autocorrelation. We propose hypothesis testing methodology to detect the presence and strength of this effect, then account for it by using a spatial filtering-based approach -- in order to enable application of existing bias detection metrics. We evaluate our proposed methodology through numerical experiments on real and synthetic datasets, demonstrating that in the presence of several types of confounding effects due to the underlying spatial structure our testing methods perform well in maintaining low type-II errors and nominal type-I errors.

stat.AP↗

arXiv2022

Joint Estimation and Inference for Data Integration Problems based on Multiple Multi-layered Gaussian Graphical Models

The rapid development of high-throughput technologies has enabled the generation of data from biological or disease processes that span multiple layers, like genomic, proteomic or metabolomic data, and further pertain to multiple sources, like disease subtypes or experimental conditions. In this work, we propose a general statistical framework based on Gaussian graphical models for horizontal (i.e. across conditions or subtypes) and vertical (i.e. across different layers containing data on molecular compartments) integration of information in such datasets. We start with decomposing the multi-layer problem into a series of two-layer problems. For each two-layer problem, we model the outcomes at a node in the lower layer as dependent on those of other nodes in that layer, as well as all nodes in the upper layer. We use a combination of neighborhood selection and group-penalized regression to obtain sparse estimates of all model parameters. Following this, we develop a debiasing technique and asymptotic distributions of inter-layer directed edge weights that utilize already computed neighborhood selection coefficients for nodes in the upper layer. Subsequently, we establish global and simultaneous testing procedures for these edge weights. Performance of the proposed methodology is evaluated on synthetic and real data.

stat.ML↗

arXiv2021

Scaling New Peaks: A Viewership-centric Approach to Automated Content Curation

Summarizing video content is important for video streaming services to engage the user in a limited time span. To this end, current methods involve manual curation or using passive interest cues to annotate potential high-interest segments to form the basis of summarized videos, and are costly and unreliable. We propose a viewership-driven, automated method that accommodates a range of segment identification goals. Using satellite television viewership data as a source of ground truth for viewer interest, we apply statistical anomaly detection on a timeline of viewership metrics to identify 'seed' segments of high viewer interest. These segments are post-processed using empirical rules and several sources of content metadata, e.g. shot boundaries, adding in personalization aspects to produce the final highlights video. To demonstrate the flexibility of our approach, we present two case studies, on the United States Democratic Presidential Debate on 19th December 2019, and Wimbledon Women's Final 2019. We perform qualitative comparisons with their publicly available highlights, as well as early vs. late viewership comparisons for insights into possible media and social influence on viewing behavior.

cs.MM↗

arXiv2021

Generalized Multivariate Signs for Nonparametric Hypothesis Testing in High Dimensions

High-dimensional data, where the dimension of the feature space is much larger than sample size, arise in a number of statistical applications. In this context, we construct the generalized multivariate sign transformation, defined as a vector divided by its norm. For different choices of the norm function, the resulting transformed vector adapts to certain geometrical features of the data distribution. Building up on this idea, we obtain one-sample and two-sample testing procedures for mean vectors of high-dimensional data using these generalized sign vectors. These tests are based on U-statistics using kernel inner products, do not require prohibitive assumptions, and are amenable to a fast randomization-based implementation. Through experiments in a number of data settings, we show that tests using generalized signs display higher power than existing tests, while maintaining nominal type-I error rates. Finally, we provide example applications on the MNIST and Minnesota Twin Studies genomic data.

stat.ME↗

arXiv2021

An Interpretable Graph-based Mapping of Trustworthy Machine Learning Research

There is an increasing interest in ensuring machine learning (ML) frameworks behave in a socially responsible manner and are deemed trustworthy. Although considerable progress has been made in the field of Trustworthy ML (TwML) in the recent past, much of the current characterization of this progress is qualitative. Consequently, decisions about how to address issues of trustworthiness and future research goals are often left to the interested researcher. In this paper, we present the first quantitative approach to characterize the comprehension of TwML research. We build a co-occurrence network of words using a web-scraped corpus of more than 7,000 peer-reviewed recent ML papers -- consisting of papers both related and unrelated to TwML. We use community detection to obtain semantic clusters of words in this network that can infer relative positions of TwML topics. We propose an innovative fingerprinting algorithm to obtain probabilistic similarity scores for individual words, then combine them to give a paper-level relevance score. The outcomes of our analysis inform a number of interesting insights on advancing the field of TwML research.

cs.SI↗

arXiv2021

Towards an Open Global Air Quality Monitoring Platform to Assess Children's Exposure to Air Pollutants in the Light of COVID-19 Lockdowns

This ongoing work attempts to understand and address the requirements of UNICEF, a leading organization working in children's welfare, where they aim to tackle the problem of air quality for children at a global level. We are motivated by the lack of a proper model to account for heavily fluctuating air quality levels across the world in the wake of the COVID-19 pandemic, leading to uncertainty among public health professionals on the exact levels of children's exposure to air pollutants. We create an initial model as per the agency's requirement to generate insights through a combination of virtual meetups and online presentations. Our research team comprised of UNICEF's researchers and a group of volunteer data scientists. The presentations were delivered to a number of scientists and domain experts from UNICEF and community champions working with open data. We highlight their feedback and possible avenues to develop this research further.

cs.HC↗

arXiv2021

Towards Integrating Fairness Transparently in Industrial Applications

Numerous Machine Learning (ML) bias-related failures in recent years have led to scrutiny of how companies incorporate aspects of transparency and accountability in their ML lifecycles. Companies have a responsibility to monitor ML processes for bias and mitigate any bias detected, ensure business product integrity, preserve customer loyalty, and protect brand image. Challenges specific to industry ML projects can be broadly categorized into principled documentation, human oversight, and need for mechanisms that enable information reuse and improve cost efficiency. We highlight specific roadblocks and propose conceptual solutions on a per-category basis for ML practitioners and organizational subject matter experts. Our systematic approach tackles these challenges by integrating mechanized and human-in-the-loop components in bias detection, mitigation, and documentation of projects at various stages of the ML lifecycle. To motivate the implementation of our system -- SIFT (System to Integrate Fairness Transparently) -- we present its structural primitives with an example real-world use case on how it can be used to identify potential biases and determine appropriate mitigation strategies in a participatory manner.

cs.CY↗

arXiv2020

On Weighted Multivariate Sign Functions

Multivariate sign functions are often used for robust estimation and inference. We propose using data dependent weights in association with such functions. The proposed weighted sign functions retain desirable robustness properties, while significantly improving efficiency in estimation and inference compared to unweighted multivariate sign-based methods. Using weighted signs, we demonstrate methods of robust location estimation and robust principal component analysis. We extend the scope of using robust multivariate methods to include robust sufficient dimension reduction and functional outlier detection. Several numerical studies and real data applications demonstrate the efficacy of the proposed methodology.

stat.ME↗

arXiv2020

Ultrahigh-dimensional Robust and Efficient Sparse Regression using Non-Concave Penalized Density Power Divergence

We propose a sparse regression method based on the non-concave penalized density power divergence loss function which is robust against infinitesimal contamination in very high dimensionality. Present methods of sparse and robust regression are based on $\ell_1$-penalization, and their theoretical properties are not well-investigated. In contrast, we use a general class of folded concave penalties that ensure sparse recovery and consistent estimation of regression coefficients. We propose an alternating algorithm based on the Concave-Convex procedure to obtain our estimate, and demonstrate its robustness properties using influence function analysis. Under some conditions on the fixed design matrix and penalty function, we prove that this estimator possesses large-sample oracle properties in an ultrahigh-dimensional regime. The performance and effectiveness of our proposed method for parameter estimation and prediction compared to state-of-the-art are demonstrated through simulation studies.

stat.ME↗

arXiv2019

Observationally inferred dark matter phase-space distribution and direct detection experiments

We present a detailed analysis of the effect of an observationally determined dark matter (DM) velocity distribution function (VDF) of the Milky Way (MW) on DM direct detection rates. We go beyond local kinematic tracers and use rotation curve data up to 200 kpc to construct a MW mass model and self-consistently determine the local phase-space distribution of DM. This approach mitigates any incomplete understanding of local dark matter-visible matter degeneracies that can affect the determination of the VDF. Comparing with the oft used Standard Halo Model (SHM), which assumes an isothermal VDF, we look at how the tail of the empirically determined VDF alters our interpretation of the present direct detection WIMP DM cross section exclusion limits. While previous studies have suggested a very large difference (of more than an order of magnitude) in the bounds at low DM masses, we show that accounting for the detector response at low threshold energies, the difference is still significant although less extreme. The change in the number of signal events, when using the empirically determined DM VDF in contrast to the SHM VDF, is most prominent for low DM masses for which the shape of the recoil energy spectrum depends sensitively on the detector threshold energy as well as detector response near the threshold. We demonstrate that these trends carry over to the respective DM exclusion limits, modulo detailed understanding of the experimental backgrounds. With the unprecedented precision of astrometric data in the GAIA era, use of observationally determined DM phase-space will become a critical and necessary ingredient for DM searches. We provide an accurate fit to the current best observationally determined DM VDF (and self-consistent local DM density) for use in analyzing current DM direct detection data by the experimental community.

hep-ph↗

arXiv2018

Correlations of the feedback energy and BCG radio luminosity in galaxy clusters

We study the excess entropy and the corresponding non-gravitational feedback energy ($E_{feedback}$) in the intra-cluster medium (ICM) by considering a sample of 38 galaxy clusters using Chandra X-ray and NRAO VLA Sky Survey (NVSS)/Giant Metrewave Radio Telescope (GMRT) radio observations. We find moderate correlation of the feedback energy and brightest cluster galaxy (BCG) radio luminosity ($L_R$) with the various cluster thermal properties. We show conclusively that the active galactic nucleus (AGN) is more effective in transferring feedback energy to the ICM in less massive clusters. We find that within $0.3r_{500}$, the feedback energy correlates with cluster temperature as $E_{feedback}\propto T_{obs}^{0.98\pm0.37}$. Moreover, for radio detected BCG sample we find that BCG radio luminosity at 1.4 GHz scales with gas mass as $L_R\propto m_{g,obs}^{ 1.76\pm0.71}$ and with X-ray luminosity as $L_R\propto L_{X,obs}^{0.94\pm0.35}$. Finally, we discuss the implications of our results with regard to feedback in clusters.

astro-ph.CO↗

arXiv2018

X-ray and SZ constraints on the properties of hot CGM

We use observations of stacked X-ray luminosity and Sunyaev-Zel'dovich (SZ) signal from a cosmological sample of $\sim 80,000$ and $104,000$ massive galaxies, respectively, with $ 10^{12.6}\lesssim M_{500} \lesssim 10^{13} M_{\odot}$ and mean redshift, z $\sim$ 0.1 - 0.14 to constrain the hot Circumgalactic Medium (CGM) density and temperature. The X-ray luminosities constrain the density and hot CGM mass, while the SZ signal helps in breaking the density-temperature degeneracy. We consider a simple power-law density distribution ($n_e \propto r^{-3β}$) as well as a hydrostatic hot halo model, with the gas assumed to be isothermal in both cases. The datasets are best described by the mean hot CGM profile $\propto r^{-1.2}$, which is shallower than an NFW profile. For halo virial mass $\sim 10^{12}$ - $10^{13} M_{\odot}$, the hot CGM contains $\sim$ 20 - 30\% of galactic baryonic mass for the power-law model and 4 - 11\% for the hydrostatic halo model, within the virial radii. For the power-law model, the hot CGM profile broadly agrees with observations of the Milky Way. The mean hot CGM mass is comparable to or larger than the mass contained in other phases of the CGM for $L^*$ galaxies.

astro-ph.GA↗

arXiv2017

Fast and General Model Selection using Data Depth and Resampling

We present a technique using data depth functions and resampling to perform best subset variable selection for a wide range of statistical models. We do this by assigning a score, called an $e$-value, to a candidate model, and use a fast bootstrap method to approximate sample versions of these $e$-values. Under general conditions, $e$-values can separate statistical models that adequately explain properties of the data from those that do not. This results in a fast algorithm that fits only a single model and evaluates $p +1$ models, $p$ being the number of predictors under consideration, as opposed to the traditional requirement of fitting and evaluating $2^{p}$ models. We illustrate in simulation experiments that our proposed method typically performs better than an array of currently used methods for variable selection in linear models and fixed effect selection in linear mixed models. As a real data application, we use our procedure to elicit climatic drivers of Indian summer monsoon precipitation.

stat.ME↗

arXiv2017

AGN feedback with the Square Kilometer Array (SKA) and implications for cluster physics and cosmology

AGN feedback is regarded as an important non-gravitational process in galaxy clusters, providing useful constraints on large-scale structure formation. It modifies the structure and energetics of the intra-cluster medium (ICM) and hence its understanding is crucially needed in order to use clusters as high precision cosmological probes. In this context, particularly keeping in mind the upcoming high quality radio data expected from radio surveys like SKA with its higher sensitivity, high spatial and spectral resolutions, we review our current understanding of AGN feedback, its cosmological implications and the impact that SKA can have in revolutionizing our understanding of AGN feedback in large-scale structures. Recent developments regarding the AGN outbursts and its possible contribution to excess entropy in the hot atmospheres of groups and clusters, its correlation with the feedback energy in ICM, quenching of cooling flows and the possible connection between cool core clusters and radio mini-halos, are discussed. We describe current major issues regarding modeling of AGN feedback and its impact on the surrounding medium. With regard to the future of AGN feedback studies, we examine the possible breakthroughs that can be expected from SKA observations. In the context of cluster cosmology, for example, we point out the importance of SKA observations for cluster mass calibration by noting that most of $z>1$ clusters discovered by eROSITA X-ray mission can be expected to be followed up through a 1000 hour SKA-1 mid programme. Moreover, approximately $1000$ radio mini halos and $\sim 2500$ radio halos at $z<0.6$ can be potentially detected by SKA1 and SKA2 and used as tracers of galaxy clusters and determination of cluster selection function.

astro-ph.CO↗

arXiv2017

Excess entropy and energy feedback from within cluster cores up to r$_{200}$

We estimate the "non-gravitational" entropy-injection profiles, $ΔK$, and the resultant energy feedback profiles, $ΔE$, of the intracluster medium for 17 clusters using their Planck SZ and ROSAT X-Ray observations, spanning a large radial range from $0.2r_{500}$ up to $r_{200}$. The feedback profiles are estimated by comparing the observed entropy, at fixed gas mass shells, with theoretical entropy profiles predicted from non-radiative hydrodynamic simulations. We include non-thermal pressure and gas clumping in our analysis. The inclusion of non-thermal pressure and clumping results in changing the estimates for $r_{500}$ and $r_{200}$ by 10\%-20\%. When clumpiness is not considered it leads to an under-estimation of $ΔK\approx300$ keV cm$^2$ at $r_{500}$ and $ΔK\approx1100$ keV cm$^2$ at $r_{200}$. On the other hand, neglecting non-thermal pressure results in an over-estimation of $ΔK\approx 100$ keV cm$^2$ at $r_{500}$ and under-estimation of $ΔK\approx450$ keV cm$^2$ at $r_{200}$. For the estimated feedback energy, we find that ignoring clumping leads to an under-estimation of energy per particle $ΔE\approx1$ keV at $r_{500}$ and $ΔE\approx1.5$ keV at $r_{200}$. Similarly, neglect of the non-thermal pressure results in an over-estimation of $ΔE\approx0.5$ keV at $r_{500}$ and under-estimation of $ΔE\approx0.25$ keV at $r_{200}$. We find entropy floor of $ΔK\approx300$ keV cm$^2$ is ruled out at $\approx3σ$ throughout the entire radial range and $ΔE\approx1$ keV at more than 3$σ$ beyond $r_{500}$, strongly constraining ICM pre-heating scenarios. We also demonstrate robustness of results w.r.t sample selection, X-Ray analysis procedures, entropy modeling etc.

astro-ph.CO↗

arXiv2017

Theia: Faint objects in motion or the new astrometry frontier

In the context of the ESA M5 (medium mission) call we proposed a new satellite mission, Theia, based on relative astrometry and extreme precision to study the motion of very faint objects in the Universe. Theia is primarily designed to study the local dark matter properties, the existence of Earth-like exoplanets in our nearest star systems and the physics of compact objects. Furthermore, about 15 $\%$ of the mission time was dedicated to an open observatory for the wider community to propose complementary science cases. With its unique metrology system and "point and stare" strategy, Theia's precision would have reached the sub micro-arcsecond level. This is about 1000 times better than ESA/Gaia's accuracy for the brightest objects and represents a factor 10-30 improvement for the faintest stars (depending on the exact observational program). In the version submitted to ESA, we proposed an optical (350-1000nm) on-axis TMA telescope. Due to ESA Technology readiness level, the camera's focal plane would have been made of CCD detectors but we anticipated an upgrade with CMOS detectors. Photometric measurements would have been performed during slew time and stabilisation phases needed for reaching the required astrometric precision.

astro-ph.IM↗