arXiv subjects

Subhadeep Mukhopadhyay

Publications and source records attributed to Subhadeep Mukhopadhyay.

18 recordsLinked to original sources

arXiv2021

Density Sharpening: Principles and Applications to Discrete Data Analysis

This article introduces a general statistical modeling principle called "Density Sharpening" and applies it to the analysis of discrete count data. The underlying foundation is based on a new theory of nonparametric approximation and smoothing methods for discrete distributions which play a useful role in explaining and uniting a large class of applied statistical methods. The proposed modeling framework is illustrated using several real applications, from seismology to healthcare to physics.

stat.ME↗

arXiv2021

InfoGram and Admissible Machine Learning

We have entered a new era of machine learning (ML), where the most accurate algorithm with superior predictive power may not even be deployable, unless it is admissible under the regulatory constraints. This has led to great interest in developing fair, transparent and trustworthy ML methods. The purpose of this article is to introduce a new information-theoretic learning framework (admissible machine learning) and algorithmic risk-management tools (InfoGram, L-features, ALFA-testing) that can guide an analyst to redesign off-the-shelf ML methods to be regulatory compliant, while maintaining good prediction accuracy. We have illustrated our approach using several real-data examples from financial sectors, biomedical research, marketing campaigns, and the criminal justice system.

stat.ML↗

arXiv2021

On The Problem of Relevance in Statistical Inference

This paper is dedicated to the "50 Years of the Relevance Problem" - a long-neglected topic that begs attention from practical statisticians who are concerned with the problem of drawing inference from large-scale heterogeneous data.

stat.ME↗

arXiv2019

Nonparametric Universal Copula Modeling

To handle the ubiquitous problem of "dependence learning," copulas are quickly becoming a pervasive tool across a wide range of data-driven disciplines encompassing neuroscience, finance, econometrics, genomics, social science, machine learning, healthcare and many more. Copula (or connection) functions were invented in 1959 by Abe Sklar in response to a query of Maurice Frechet. After 60 years, where do we stand now? This article provides a history of the key developments and offers a unified perspective.

stat.ME↗

arXiv2019

Spectral Graph Analysis: A Unified Explanation and Modern Perspectives

Complex networks or graphs are ubiquitous in sciences and engineering: biological networks, brain networks, transportation networks, social networks, and the World Wide Web, to name a few. Spectral graph theory provides a set of useful techniques and models for understanding `patterns of interconnectedness' in a graph. Our prime focus in this paper is on the following question: Is there a unified explanation and description of the fundamental spectral graph methods? There are at least two reasons to be interested in this question. Firstly, to gain a much deeper and refined understanding of the basic foundational principles, and secondly, to derive rich consequences with practical significance for algorithm design. However, despite half a century of research, this question remains one of the most formidable open issues, if not the core problem in modern network science. The achievement of this paper is to take a step towards answering this question by discovering a simple, yet universal statistical logic of spectral graph analysis. The prescribed viewpoint appears to be good enough to accommodate almost all existing spectral graph techniques as a consequence of just one single formalism and algorithm.

math.ST↗

arXiv2018

Decentralized Nonparametric Multiple Testing

Consider a big data multiple testing task, where, due to storage and computational bottlenecks, one is given a very large collection of p-values by splitting into manageable chunks and distributing over thousands of computer nodes. This paper is concerned with the following question: How can we find the full data multiple testing solution by operating completely independently on individual machines in parallel, without any data exchange between nodes? This version of the problem tends naturally to arise in a wide range of data-intensive science and industry applications whose methodological solution has not appeared in the literature to date; therefore, we feel it is necessary to undertake such analysis. Based on the nonparametric functional statistical viewpoint of large-scale inference, started in Mukhopadhyay (2016), this paper furnishes a new computing model that brings unexpected simplicity to the design of the algorithm which might otherwise seem daunting using classical approach and notations.

stat.ME↗

arXiv2018

Nonparametric Distributed Learning Architecture for Big Data: Algorithm and Applications

Dramatic increases in the size and complexity of modern datasets have made traditional "centralized" statistical inference prohibitive. In addition to computational challenges associated with big data learning, the presence of numerous data types (e.g. discrete, continuous, categorical, etc.) makes automation and scalability difficult. A question of immediate concern is how to design a data-intensive statistical inference architecture without changing the basic statistical modeling principles developed for "small" data over the last century. To address this problem, we present MetaLP, a flexible, distributed statistical modeling framework.

stat.AP↗

arXiv2017

Nonlinear Time Series Modeling: A Unified Perspective, Algorithm, and Application

A new comprehensive approach to nonlinear time series analysis and modeling is developed in the present paper. We introduce novel data-specific mid-distribution based Legendre Polynomial (LP) like nonlinear transformations of the original time series Y(t) that enables us to adapt all the existing stationary linear Gaussian time series modeling strategy and made it applicable for non-Gaussian and nonlinear processes in a robust fashion. The emphasis of the present paper is on empirical time series modeling via the algorithm LPTime. We demonstrate the effectiveness of our theoretical framework using daily S&P 500 return data between Jan/2/1963 - Dec/31/2009. Our proposed LPTime algorithm systematically discovers all the `stylized facts' of the financial time series automatically all at once, which were previously noted by many researchers one at a time.

math.ST↗

arXiv2017

Statistics Educational Challenge in the 21st Century

What do we teach and what should we teach? An honest answer to this question is painful, very painful--what we teach lags decades behind what we practice. How can we reduce this `gap' to prepare a data science workforce of trained next-generation statisticians? This is a challenging open problem that requires many well-thought-out experiments before finding the secret sauce. My goal in this article is to lay out some basic principles and guidelines (rather than creating a pseudo-curriculum based on cherry-picked topics) to expedite this process for finding an `objective' solution.

stat.OT↗

arXiv2017

Large Scale Signal Detection: A Unified Perspective

There is an overwhelmingly large literature and algorithms already available on `large scale inference problems' based on different modeling techniques and cultures. Our primary goal in this paper is \emph{not to add one more new methodology} to the existing toolbox but instead (a) to clarify the mystery how these different simultaneous inference methods are \emph{connected}, (b) to provide an alternative more intuitive derivation of the formulas that leads to \emph{simpler} expressions, and (c) to develop a \emph{unified} algorithm for practitioners. A detailed discussion on representation, estimation, inference, and model selection is given. Applications to a variety of real and simulated datasets show promise. We end with several future research directions.

math.ST↗

arXiv2016

Large-Scale Mode Identification and Data-Driven Sciences

Bump-hunting or mode identification is a fundamental problem that arises in almost every scientific field of data-driven discovery. Surprisingly, very few data modeling tools are available for automatic (not requiring manual case-by-base investigation), objective (not subjective), and nonparametric (not based on restrictive parametric model assumptions) mode discovery, which can scale to large data sets. This article introduces LPMode--an algorithm based on a new theory for detecting multimodality of a probability density. We apply LPMode to answer important research questions arising in various fields from environmental science, ecology, econometrics, analytical chemistry to astronomy and cancer genomics.

stat.ME↗

arXiv2016

Unified Statistical Theory of Spectral Graph Analysis

The goal of this paper is to show that there exists a simple, yet universal statistical logic of spectral graph analysis by recasting it into a nonparametric function estimation problem. The prescribed viewpoint appears to be good enough to accommodate most of the existing spectral graph techniques as a consequence of just one single formalism and algorithm.

math.ST↗

arXiv2015

Strength of Connections in a Random Graph: Definition, Characterization, and Estimation

How can the `affinity' or `strength' of ties of a random graph be characterized and compactly represented? How can concepts like Fourier and inverse-Fourier like transform be developed for graph data? To do so, we introduce a new graph-theoretic function called `Graph Correlation Density Field' (or in short GraField), which differs from the traditional edge probability density-based approaches, to completely characterize tie-strength between graph nodes. Our approach further allows frequency domain analysis, applicable for both directed and undirected random graphs.

math.ST↗

arXiv2014

LP Approach to Statistical Modeling

We present an approach to statistical data modeling and exploratory data analysis called `LP Statistical Data Science.' It aims to generalize and unify traditional and novel statistical measures, methods, and exploratory tools. This article outlines fundamental concepts along with real-data examples to illustrate how the `LP Statistical Algorithm' can systematically tackle different varieties of data types, data patterns, and data structures under a coherent theoretical framework. A fundamental role is played by specially designed orthonormal basis of a random variable X for linear (Hilbert space theory) representation of a general function of X, such as $\mbox{E}[Y \mid X]$.

math.ST↗

arXiv2013

LP Mixed Data Science : Outline of Theory

This article presents the theoretical foundation of a new frontier of research-`LP Mixed Data Science'-that simultaneously extends and integrates the practice of traditional and novel statistical methods for nonparametric exploratory data modeling, and is applicable to the teaching and training of statistics. Statistics journals have great difficulty accepting papers unlike those previously published. For statisticians with new big ideas a practical strategy is to publish them in many small applied studies which enables one to provide references to work of others. This essay outlines the many concepts, new theory, and important algorithms of our new culture of statistical science called LP MIXED DATA SCIENCE. It provides comprehensive solutions to problems of data analysis and nonparametric modeling of many variables that are continuous or discrete, which does not yet have a large literature. It develops a new modeling approach to nonparametric estimation of the multivariate copula density. We discuss the theory which we believe is very elegant (and can provide a framework for United Statistical Algorithms, for traditional Small Data methods and Big Data methods).

math.ST↗

arXiv2013

CDfdr: A Comparison Density Approach to Local False Discovery Rate Estimation

Efron et al. (2001) proposed empirical Bayes formulation of the frequentist Benjamini and Hochbergs False Discovery Rate method (Benjamini and Hochberg,1995). This article attempts to unify the `two cultures' using concepts of comparison density and distribution function. We have also shown how almost all of the existing local fdr methods can be viewed as proposing various model specification for comparison density - unifies the vast literature of false discovery methods under one concept and notation.

stat.ME↗

arXiv2013

United Statistical Algorithm, Small and Big Data: Future OF Statistician

This article provides the role of big idea statisticians in future of Big Data Science. We describe the `United Statistical Algorithms' framework for comprehensive unification of traditional and novel statistical methods for modeling Small Data and Big Data, especially mixed data (discrete, continuous).

math.ST↗

arXiv2012

Modeling, dependence, classification, united statistical science, many cultures

Breiman (2001) proposed to statisticians awareness of two cultures: 1. Parametric modeling culture, pioneered by R.A.Fisher and Jerzy Neyman; 2. Algorithmic predictive culture, pioneered by machine learning research. Parzen (2001), as a part of discussing Breiman (2001), proposed that researchers be aware of many cultures, including the focus of our research: 3. Nonparametric, quantile based, information theoretic modeling. We provide a unification of many statistical methods for traditional small data sets and emerging big data sets in terms of comparison density, copula density, measure of dependence, correlation, information, new measures (called LP score comoments) that apply to long tailed distributions with out finite second order moments. A very important goal is to unify methods for discrete and continuous random variables. Our research extends these methods to modern high dimensional data modeling.

math.ST↗