SEARCH · Searcharxiv

Results for “stat.OT”

Search indexed arXiv papers on artificial intelligence, large language models, computer vision and robotics. Read source abstracts and follow links to arXiv.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 829 records · Page 46Linked to original sources

arXiv2022

Alternative approaches for analysing repeated measures data that are missing not at random

We consider studies where multiple measures on an outcome variable are collected over time, but some subjects drop out before the end of follow up. Analyses of such data often proceed under either a 'last observation carried forward' or 'missing at random' assumption. We consider two alternative strategies for identification; the first is closely related to the difference-in-differences methodology in the causal inference literature. The second enables correction for violations of the parallel trend assumption, so long as one has access to a valid 'bespoke instrumental variable'. These are compared with existing approaches, first conceptually and then in an analysis of data from the Framingham Heart Study.

stat.ME↗

arXiv2022

Model-Free, Monotone Invariant and Computationally Efficient Feature Screening with Data-adaptive Threshold

Feature screening for ultrahigh-dimension, in general, proceeds with two essential steps. The first step is measuring and ranking the marginal dependence between response and covariates, and the second is determining the threshold. We develop a new screening procedure, called SIT-BY procedure, that possesses appealing statistical properties in both steps. By employing sliced independence estimates in the measuring and ranking stage, our proposed procedure requires no model assumptions, remains invariant to monotone transformation, and achieves almost linear computation complexity. Inspired by false discovery rate (FDR) control procedures, we offer a data-adaptive threshold benefit from the asymptotic normality of test statistics. Under moderate conditions, we demonstrate that our procedure can asymptotically control the FDR while maintaining the sure screening property. We investigate the finite sample performance of our proposed procedure via extensive simulations and an application to genome-wide dataset.

stat.ME↗

arXiv2022

InsightiGen: a versatile tool to generate insight for an academic systematic literature review

A comprehensive literature review has always been an essential first step of every meaningful research. In recent years, however, the availability of a vast amount of information in both open-access and subscription-based literature in every field has made it difficult, if not impossible, to be certain about the comprehensiveness of one's survey. This subsequently can lead to reviewers' questioning of the novelties of the research directions proposed, regardless of the quality of the actual work presented. In this situation, statistics derived from the published literature data can provide valuable quantitative and visual information about research trends, knowledge gaps, and research networks and hubs in different fields. Our tool provides an automatic and rapid way of generating insight for systematic reviews in any research area.

stat.OT↗

arXiv2023

Customs Import Declaration Datasets

Given the huge volume of cross-border flows, effective and efficient control of trade becomes more crucial in protecting people and society from illicit trade. However, limited accessibility of the transaction-level trade datasets hinders the progress of open research, and lots of customs administrations have not benefited from the recent progress in data-based risk management. In this paper, we introduce an import declaration dataset to facilitate the collaboration between domain experts in customs administrations and researchers from diverse domains, such as data science and machine learning. The dataset contains 54,000 artificially generated trades with 22 key attributes, and it is synthesized with conditional tabular GAN while maintaining correlated features. Synthetic data has several advantages. First, releasing the dataset is free from restrictions that do not allow disclosing the original import data. The fabrication step minimizes the possible identity risk which may exist in trade statistics. Second, the published data follow a similar distribution to the source data so that it can be used in various downstream tasks. Hence, our dataset can be used as a benchmark for testing the performance of any classification algorithm. With the provision of data and its generation process, we open baseline codes for fraud detection tasks, as we empirically show that more advanced algorithms can better detect fraud.

cs.LG↗

arXiv2022

Atomist or Holist? A Diagnosis and Vision for More Productive Interdisciplinary AI Ethics Dialogue

In response to growing recognition of the social impact of new AI-based technologies, major AI and ML conferences and journals now encourage or require papers to include ethics impact statements and undergo ethics reviews. This move has sparked heated debate concerning the role of ethics in AI research, at times devolving into name-calling and threats of "cancellation." We diagnose this conflict as one between atomist and holist ideologies. Among other things, atomists believe facts are and should be kept separate from values, while holists believe facts and values are and should be inextricable from one another. With the goal of reducing disciplinary polarization, we draw on numerous philosophical and historical sources to describe each ideology's core beliefs and assumptions. Finally, we call on atomists and holists within the ever-expanding data science community to exhibit greater empathy during ethical disagreements and propose four targeted strategies to ensure AI research benefits society.

cs.CY↗

arXiv2024

Optimal Pre-Analysis Plans: Statistical Decisions Subject to Implementability

What is the purpose of pre-analysis plans, and how should they be designed? We model the interaction between an agent who analyzes data and a principal who makes a decision based on agent reports. The agent could be the manufacturer of a new drug, and the principal a regulator deciding whether the drug is approved. Or the agent could be a researcher submitting a research paper, and the principal an editor deciding whether it is published. The agent decides which statistics to report to the principal. The principal cannot verify whether the analyst reported selectively. Absent a pre-analysis message, if there are conflicts of interest, then many desirable decision rules cannot be implemented. Allowing the agent to send a message before seeing the data increases the set of decision rules that can be implemented, and allows the principal to leverage agent expertise. The optimal mechanisms that we characterize require pre-analysis plans. Applying these results to hypothesis testing, we show that optimal rejection rules pre-register a valid test, and make worst-case assumptions about unreported statistics. Optimal tests can be found as a solution to a linear-programming problem.

econ.EM↗

arXiv2022

Race and ethnicity data for first, middle, and last names

We provide the largest compiled publicly available dictionaries of first, middle, and last names for the purpose of imputing race and ethnicity using, for example, Bayesian Improved Surname Geocoding (BISG). The dictionaries are based on the voter files of six Southern states that collect self-reported racial data upon voter registration. Our data cover a much larger scope of names than any comparable dataset, containing roughly one million first names, 1.1 million middle names, and 1.4 million surnames. Individuals are categorized into five mutually exclusive racial and ethnic groups -- White, Black, Hispanic, Asian, and Other -- and racial/ethnic counts by name are provided for every name in each dictionary. Counts can then be normalized row-wise or column-wise to obtain conditional probabilities of race given name or name given race. These conditional probabilities can then be deployed for imputation in a data analytic task for which ground truth racial and ethnic data is not available.

stat.OT↗

arXiv2022

A Bayesian nonparametric approach for causal inference with multiple mediators

Mediation analysis with contemporaneously observed multiple mediators is an important area of causal inference. Recent approaches for multiple mediators are often based on parametric models and thus may suffer from model misspecification. Also, much of the existing literature either only allow estimation of the joint mediation effect, or, estimate the joint mediation effect as the sum of individual mediator effects, which often is not a reasonable assumption. In this paper, we propose a methodology which overcomes the two aforementioned drawbacks. Our method is based on a novel Bayesian nonparametric (BNP) approach, wherein the joint distribution of the observed data (outcome, mediators, treatment, and confounders) is modeled flexibly using an enriched Dirichlet process mixture with three levels: the first level characterizing the conditional distribution of the outcome given the mediators, treatment and the confounders, the second level corresponding to the conditional distribution of each of the mediators given the treatment and the confounders, and the third level corresponding to the distribution of the treatment and the confounders. We use standardization (g-computation) to compute causal mediation effects under three uncheckable assumptions that allow identification of the individual and joint mediation effects. The efficacy of our proposed method is demonstrated with simulations. We apply our proposed method to analyze data from a study of Ventilator-associated Pneumonia (VAP) co-infected patients, where the effect of the abundance of Pseudomonas on VAP infection is suspected to be mediated through antibiotics.

stat.ME↗

arXiv2022

Model Transparency and Interpretability : Survey and Application to the Insurance Industry

The use of models, even if efficient, must be accompanied by an understanding at all levels of the process that transforms data (upstream and downstream). Thus, needs increase to define the relationships between individual data and the choice that an algorithm could make based on its analysis (e.g. the recommendation of one product or one promotional offer, or an insurance rate representative of the risk). Model users must ensure that models do not discriminate and that it is also possible to explain their results. This paper introduces the importance of model interpretation and tackles the notion of model transparency. Within an insurance context, it specifically illustrates how some tools can be used to enforce the control of actuarial models that can nowadays leverage on machine learning. On a simple example of loss frequency estimation in car insurance, we show the interest of some interpretability methods to adapt explanation to the target audience.

stat.ML↗

arXiv2022

Testing for the Important Components of Posterior Predictive Variance

We give a decomposition of the posterior predictive variance using the law of total variance and conditioning on a finite dimensional discrete random variable. This random variable summarizes various features of modeling that are used to form the prediction for a future outcome. Then, we test which terms in this decomposition are small enough to ignore. This allows us identify which of the discrete random variables are most important to prediction intervals. The terms in the decomposition admit interpretations based on conditional means and variances and are analogous to the terms in a Cochran's theorem decomposition of squared error often used in analysis of variance. Thus, the modeling features are treated as factors in completely randomized design. In cases where there are multiple decompositions we suggest choosing the one that that gives the best predictive coverage with the smallest variance.

stat.ME↗

arXiv2022

elhmc: An R Package for Hamiltonian Monte Carlo Sampling in Bayesian Empirical Likelihood

In this article, we describe a {\tt R} package for sampling from an empirical likelihood-based posterior using a Hamiltonian Monte Carlo method. Empirical likelihood-based methodologies have been used in Bayesian modeling of many problems of interest in recent times. This semiparametric procedure can easily combine the flexibility of a non-parametric distribution estimator together with the interpretability of a parametric model. The model is specified by estimating equations-based constraints. Drawing an inference from a Bayesian empirical likelihood (BayesEL) posterior is challenging. The likelihood is computed numerically, so no closed expression of the posterior exists. Moreover, for any sample of finite size, the support of the likelihood is non-convex, which hinders the fast mixing of many Markov Chain Monte Carlo (MCMC) procedures. It has been recently shown that using the properties of the gradient of log empirical likelihood, one can devise an efficient Hamiltonian Monte Carlo (HMC) algorithm to sample from a BayesEL posterior. The package requires the user to specify only the estimating equations, the prior, and their respective gradients. An MCMC sample drawn from the BayesEL posterior of the parameters, with various details required by the user is obtained.

stat.OT↗

arXiv2022

Biblio-Analysis of Cohort Intelligence (CI) Algorithm and its allied applications from Scopus and Web of Science Perspective

Cohort Intelligence or CI is one of its kind of novel optimization algorithm. Since its inception, in a very short span it is applied successfully in various domains and its results are observed to be effectual in contrast to algorithm of its kind. Till date, there is no such type of bibliometric analysis carried out on CI and its related applications. So, this research paper in a way will be an ice breaker for those who want to take up CI to a new level. In this research papers, CI publications available in Scopus are analyzed through graphs, networked diagrams about authors, source titles, keywords over the years, journals over the time. In a way this bibliometric paper showcase CI, its applications and detail outs systematic review in terms its bibliometric details.

cs.DL↗

arXiv2023

Hitting a prime in 2.43 dice rolls (on average)

What is the number of rolls of fair 6-sided dice until the first time the total sum of all rolls is a prime? We compute the expectation and the variance of this random variable up to an additive error of less than 10^{-4}. This is a solution to a puzzle suggested by DasGupta (2017) in the Bulletin of the Institute of Mathematical Statistics, where the published solution is incomplete. The proof is simple, combining a basic dynamic programming algorithm with a quick Matlab computation and basic facts about the distribution of primes.

math.PR↗

arXiv2025

Finite- and Large- Sample Inference for Model and Coefficients in High-dimensional Linear Regression with Repro Samples

In this paper, we present a novel and effective inference approach to conduct both finite- and large-sample inference for high-dimensional linear regression models. This approach is developed under the so-called repro samples framework, in which we conduct statistical inference by creating and studying the behavior of artificial samples that are obtained by mimicking the sampling mechanism of the data. We construct confidence sets for (a) the true model corresponding to the nonzero coefficients, (b) a single or any collection of regression coefficients, and (c) both the model and regression coefficients jointly. To facilitate the constructions of these confidence sets and overcome computational difficulties of searching all possible models, we use an innovative Fisher inversion technique to construct a model candidate set that includes the true sparse model with the probability close to 1 for models with both Gaussian and non-Gaussian errors. The proposed approach fills in two major gaps in the high-dimensional regression literature: (1) lack of effective approaches to addressing model selection uncertainty and providing valid inference for the underlying true model; (2) lack of effective inference approaches to guaranteeing finite-sample performance. We provide both finite-sample and asymptotic results to theoretically guarantee the performance of the proposed methods. In addition, our numerical results demonstrate that the proposed methods are valid and achieve better coverage with smaller confidence sets than the current state-of-the-art approaches, such as debiasing and bootstrap approaches.

stat.ME↗

arXiv2022

Predictive density estimators with integrated $L_1$ loss

This paper addresses the problem of an efficient predictive density estimation for the density $q(\|y-θ\|^2)$ of $Y$ based on $X \sim p(\|x-θ\|^2)$ for $y, x, θ\in \mathbb{R}^d$. The chosen criteria are integrated $L_1$ loss given by $L(θ, \hat{q}) \, =\, \int_{\mathbb{R}^d} \big|\hat{q}(y)- q(\|y-θ\|^2) \big| \, dy$, and the associated frequentist risk, for $θ\in Θ$. For absolutely continuous and strictly decreasing $q$, we establish the inevitability of scale expansion improvements $\hat{q}_c(y;X)\,=\, \frac{1}{c^d} q\big(\|y-X\|^2/c^2 \big) $ over the plug-in density $\hat{q}_1$, for a subset of values $c \in (1,c_0)$. The finding is universal with respect to $p,q$, and $d \geq 2$, and extended to loss functions $γ\big(L(θ, \hat{q} ) \big)$ with strictly increasing $γ$. The finding is also extended to include scale expansion improvements of more general plug-in densities $q(\|y-\hatθ(X)\|^2 \big)$, when the parameter space $Θ$ is a compact subset of $\mathbb{R}^d$. Numerical analyses illustrative of the dominance findings are presented and commented upon. As a complement, we demonstrate that the unimodal assumption on $q$ is necessary with a detailed analysis of cases where the distribution of $Y|θ$ is uniformly distributed on a ball centered about $θ$. In such cases, we provide a univariate ($d=1$) example where the best equivariant estimator is a plug-in estimator, and we obtain cases (for $d=1,3$) where the plug-in density $\hat{q}_1$ is optimal among all $\hat{q}_c$.

math.ST↗

arXiv2022

Design Amortization for Bayesian Optimal Experimental Design

Bayesian optimal experimental design is a sub-field of statistics focused on developing methods to make efficient use of experimental resources. Any potential design is evaluated in terms of a utility function, such as the (theoretically well-justified) expected information gain (EIG); unfortunately however, under most circumstances the EIG is intractable to evaluate. In this work we build off of successful variational approaches, which optimize a parameterized variational model with respect to bounds on the EIG. Past work focused on learning a new variational model from scratch for each new design considered. Here we present a novel neural architecture that allows experimenters to optimize a single variational model that can estimate the EIG for potentially infinitely many designs. To further improve computational efficiency, we also propose to train the variational model on a significantly cheaper-to-evaluate lower bound, and show empirically that the resulting model provides an excellent guide for more accurate, but expensive to evaluate bounds on the EIG. We demonstrate the effectiveness of our technique on generalized linear models, a class of statistical models that is widely used in the analysis of controlled experiments. Experiments show that our method is able to greatly improve accuracy over existing approximation strategies, and achieve these results with far better sample efficiency.

cs.LG↗

arXiv2026

Type $1$, $2$, $3$ and $4$ $q$-negative binomial distribution of order $k$

We study the distributions of waiting times in variations of the negative binomial distribution of order $k$. One variation apply different enumeration scheme on the runs of successes. Another case considers binary trials for which the probability of ones is geometrically varying. We investigate the exact distribution of the waiting time for the $r$-th occurrence of success run of a specified length (non-overlapping, overlapping, at least, exactly, $\ell$-overlapping) in a $q$-sequence of binary trials. The main theorems are Type $1$, $2$, $3$ and $4$ $q$-negative binomial distribution of order $k$ and $q$-negative binomial distribution of order $k$ in the $\ell$-overlapping case. In the present work, we consider a sequence of independent binary zero and one trials with not necessarily identical distribution with the probability of ones varying according to a geometric rule. Exact formulae for the distributions obtained by means of enumerative combinatorics.

math.PR↗

arXiv2022

On Success runs of a fixed length defined on a $q$-sequence of binary trials

We study the exact distributions of runs of a fixed length in variation which considers binary trials for which the probability of ones is geometrically varying. The random variable $E_{n,k}$ denote the number of success runs of a fixed length $k$, $1\leq k \leq n$. Theorem 3.1 gives an closed expression for the probability mass function (PMF) of the Type4 $q$-binomial distribution of order $k$. Theorem 3.2 and Corollary 3.1 gives an recursive expression for the probability mass function (PMF) of the Type4 $q$-binomial distribution of order $k$. The probability generating function and moments of random variable $E_{n,k}$ are obtained as a recursive expression. We address the parameter estimation in the distribution of $E_{n,k}$ by numerical techniques. In the present work, we consider a sequence of independent binary zero and one trials with not necessarily identical distribution with the probability of ones varying according to a geometric rule. Exact and recursive formulae for the distribution obtained by means of enumerative combinatorics.

math.PR↗