SEARCH · Searcharxiv

Results for “stat.OT”

Search indexed arXiv papers on artificial intelligence, large language models, computer vision and robotics. Read source abstracts and follow links to arXiv.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 991 records · Page 55Linked to original sources

arXiv2024

An Investigation into Distance Measures in Cluster Analysis

This report provides an exploration of different distance measures that can be used with the $K$-means algorithm for cluster analysis. Specifically, we investigate the Mahalanobis distance, and critically assess any benefits it may have over the more traditional measures of the Euclidean, Manhattan and Maximum distances. We perform this by first defining the metrics, before considering their advantages and drawbacks as discussed in literature regarding this area. We apply these distances, first to some simulated data and then to subsets of the Dry Bean dataset [1], to explore if there is a better quality detectable for one metric over the others in these cases. One of the sections is devoted to analysing the information obtained from ChatGPT in response to prompts relating to this topic.

stat.OT↗

arXiv2024

Percentage Coefficient (bp) -- Effect Size Analysis (Theory Paper 1)

Percentage coefficient (bp) has emerged in recent publications as an additional and alternative estimator of effect size for regression analysis. This paper retraces the theory behind the estimator. It's posited that an estimator must first serve the fundamental function of enabling researchers and readers to comprehend an estimand, the target of estimation. It may then serve the instrumental function of enabling researchers and readers to compare two or more estimands. Defined as the regression coefficient when dependent variable (DV) and independent variable (IV) are both on conceptual 0-1 percentage scales, percentage coefficients (bp) feature 1) clearly comprehendible interpretation and 2) equitable scales for comparison. The coefficient (bp) serves the two functions effectively and efficiently. It thus serves needs unserved by other indicators, such as raw coefficient (bw) and standardized beta. Another premise of the functionalist theory is that "effect" is not a monolithic concept. Rather, it is a collection of concepts, each of which measures a component of the conglomerate called "effect", thereby serving a subfunction. Regression coefficient (b), for example, indicates the unit change in DV associated with a one-unit increase in IV, thereby measuring one aspect called unit effect, aka efficiency. Percentage coefficient (bp) indicates the percentage change in DV associated with a whole scale increase in IV. It is not meant to be an all-encompassing indicator of an all-encompassing concept, but rather a comprehendible and comparable indicator of efficiency, a key aspect of effect.

stat.AP↗

arXiv2024

What's So Hard about the Monty Hall Problem?

The Monty Hall problem is notorious for its deceptive simplicity. Although today it is widely used as a provocative thought experiment to introduce Bayesian thinking to students of probability, in the not so distant past it was rejected by established mathematicians. This essay provides some historical background to the problem and explains why it is considered so counter-intuitive to many. It is argued that the main barrier to understanding the problem is the back-grounding of the concept of dependence in probability theory as it is commonly taught. To demonstrate this, a Bayesian solution is provided and augmented with a probabilistic graphical model (PGM) inspired by the work of Pearl (1988, 1998). Although the Bayesian approach produces the correct answer, without a representation of the dependency structure of events implied by the problem, the salient fact that motivates the problem's solution remains hidden.

stat.OT↗

arXiv2026

Data-Driven Strategies for Detecting and Sampling Misrepresented Subgroups

Economic policy research frequently examines population well-being, with a particular focus on the relationships between unequal living conditions, low educational attainment, and social exclusion. Sample surveys, such as EU-SILC, are widely used for this purpose and inform public policy; yet, their sampling designs may fail to adequately represent rare, hard-to-sample, or under-covered subgroups. This limitation can hinder socio-demographic analyses and evidence-based policy design. We propose a generalisable approach based on univariate and multivariate unsupervised learning techniques to detect outliers in survey data that may signal under-represented subgroups. Identified groups can then be characterised to inform targeted resampling strategies that improve survey inclusiveness. An empirical application using the 2019 EU-SILC data for the Italian region of Liguria shows that citizenship, material deprivation, large household size, and economic vulnerability are key indicators of under-representation.

stat.AP↗

arXiv2024

Which Identities Are Mobilized: Towards an automated detection of social group appeals in political texts

This paper proposes a computational text classification strategy to identify references to social groups in European party manifestos and beyond. Our methodology uses machine learning techniques, including BERT and large language models, to capture group-based appeals in texts. We propose to combine automated identification of social groups using the Mistral-7B-v0.1 Large Language Model with Embedding Space-based filtering to extend a sample of core social groups to all social groups mentioned in party manifestos. By applying this approach to RRP's and mainstream parties' group images in manifestos, we explore whether electoral dynamics explain similarities in group appeals and potential convergence or divergence in party strategies. Contrary to expectations, increasing RRP support or mainstream parties' vote loss does not necessarily lead to convergence in group appeals. Nonetheless, our methodology enables mapping similarities in group appeals across time and space in 15 European countries from 1980 to 2021 and can be transferred to other use cases as well.

cs.SI↗

arXiv2026

Nested Instrumental Variables Analysis: Switcher Average Treatment Effect, Identification, Efficient Estimation and Generalizability

Instrumental variables (IVs) are widely used to estimate causal effects from non-randomized data. A canonical example is a randomized trial with noncompliance, in which the randomized treatment assignment serves as an IV for the non-ignorable treatment received. Under a monotonicity assumption, a valid IV nonparametrically identifies the average treatment effect among a latent complier subgroup, whose generalizability is often under debate. In many studies, there exist multiple versions of an IV, for instance, different nudges to take the same treatment in different study sites in a multicenter clinical trial. These different versions of an IV may result in different compliance rates and offer a unique opportunity to study IV estimates' generalizability. In this article, we introduce a novel nested IV assumption and study identification of the average treatment effect among two latent subgroups: always-compliers and switchers, who are defined based on the joint potential treatment received under two versions of a binary IV. We derive the efficient influence function for the SWitcher Average Treatment Effect (SWATE) under a nonparametric model and propose efficient estimators. We then propose formal statistical tests of the generalizability of IV estimates under the nested IV framework. The proposed tests are flexible nonparametric generalizations of classical overidentification tests that allow estimating nuisance parameters using machine learning tools. We apply the proposed method to the Prostate, Lung, Colorectal and Ovarian (PLCO) Cancer Screening Trial and study the causal effect of colorectal cancer screening and its generalizability.

stat.ME↗

arXiv2024

Sequential Maximal Updated Density Parameter Estimation for Dynamical Systems with Parameter Drift

We present a novel method for generating sequential parameter estimates and quantifying epistemic uncertainty in dynamical systems within a data-consistent (DC) framework. The DC framework differs from traditional Bayesian approaches due to the incorporation of the push-forward of an initial density, which performs selective regularization in parameter directions not informed by the data in the resulting updated density. This extends a previous study that included the linear Gaussian theory within the DC framework and introduced the maximal updated density (MUD) estimate as an alternative to both least squares and maximum a posterior (MAP) estimates. In this work, we introduce algorithms for operational settings of MUD estimation in real or near-real time where spatio-temporal datasets arrive in packets to provide updated estimates of parameters and identify potential parameter drift. Computational diagnostics within the DC framework prove critical for evaluating (1) the quality of the DC update and MUD estimate and (2) the detection of parameter value drift. The algorithms are applied to estimate (1) wind drag parameters in a high-fidelity storm surge model, (2) thermal diffusivity field for a heat conductivity problem, and (3) changing infection and incubation rates of an epidemiological model.

stat.ME↗

arXiv2024

Predicting Short Response Ratings with Non-Content Related Features: A Hierarchical Modeling Approach

We explore whether the human ratings of open ended responses can be explained with non-content related features, and if such effects vary across different mathematics-related items. When scoring is rigorously defined and rooted in a measurement framework, educators intend that the features of a response which are indicative of the respondent's level of ability are contributing to scores. However, we find that features such as response length, a grammar score of the response, and a metric relating to key phrase frequency are significant predictors for response ratings. Although our findings are not causally conclusive, they may propel us to be more critical of he way in which we assess open ended responses, especially in high stakes scenarios. Educators take great care to provide unbiased, consistent ratings, but it may be that extraneous features unrelated to those which were intended to be rated are being evaluated.

stat.AP↗

arXiv2026

Extrapolating Single-Treatment Effects Out of Factorial Experiments

Despite their cost, randomized controlled trials (RCTs) are widely regarded as gold-standard evidence in disciplines ranging from social science to medicine. In recent decades, researchers have increasingly sought to reduce the resource burden of repeated RCTs with factorial designs that simultaneously test multiple hypotheses, e.g. experiments that evaluate the effects of many medications or products simultaneously. Here I show that when multiple interventions are randomized in experiments, the effect any single intervention would have outside the experimental setting is not identified absent heroic assumptions, even if otherwise perfectly realistic conditions are achieved. This happens because single-treatment effects involve a counterfactual world with a single focal intervention, allowing other variables to take their natural values (which may be confounded or modified by the focal intervention). In contrast, observational studies and factorial experiments provide information about potential-outcome distributions with zero and multiple interventions, respectively. In this paper, I formalize sufficient conditions for the identifiability of those isolated quantities. I show that researchers who rely on this type of design have to justify either linearity of functional forms or -- in the nonparametric case -- specify with Directed Acyclic Graphs how variables are related in the real world. Finally, I develop nonparametric sharp bounds -- i.e., maximally informative best-/worst-case estimates consistent with limited RCT data -- that show when extrapolations about effect signs are empirically justified. These new results are illustrated with simulated data.

stat.ME↗

arXiv2025

Expected Points Above Average: A Novel NBA Player Metric Based on Bayesian Hierarchical Modeling

In this paper, we propose two novel basketball metrics: ``expected points'' for team-based comparisons and ``expected points above average (EPAA)'' as a player-evaluation tool. Established within the Bayesian hierarchical model framework, teams and players are clustered based on their shooting propensities and abilities using posterior predictive distributions. We illustrate the concepts for the top 100 shot takers over the last decade and offer our metric as an additional metric for evaluating players. We compare our metrics to two traditional NBA player evaluation metrics: player efficiency rating and box plus/minus. Finally, we develop a Shiny web application that allows interested readers to make additional team and player comparisons.

stat.OT↗

arXiv2025

The Logic of Counterfactuals and the Epistemology of Causal Inference

The 2021 Nobel Prize in Economics recognized an epistemology of causal inference based on the Rubin causal model (Rubin 1974), which merits broader attention in philosophy. This model, in fact, presupposes a logical principle of counterfactuals, Conditional Excluded Middle (CEM), the locus of a pivotal debate between Stalnaker (1968) and Lewis (1973) on the semantics of counterfactuals. Proponents of CEM should recognize that this connection points to a new argument for CEM -- a Quine-Putnam indispensability argument grounded in the Nobel-winning applications of the Rubin model in health and social sciences. To advance the dialectic, I challenge this argument with an updated Rubin causal model that retains its successes while dispensing with CEM. This novel approach combines the strengths of the Rubin causal model and a causal model familiar in philosophy, the causal Bayes net. The takeaway: deductive logic and inductive inference, often studied in isolation, are deeply interconnected.

cs.AI↗

arXiv2024

The Epistemology behind Covariate Adjustment

It is often asserted that to control for the effects of confounders, one should include the confounding variables of concern in a statistical model as a covariate. Conversely, it is also asserted that control can only be concluded by design, where the results from an analysis can only be interpreted as evidence of an effect because the design controlled for the cause. To suggest otherwise is said to be a fallacy of cum hoc ergo propter hoc. Obviously, these two assertions create a conundrum: How can the effect of confounder be controlled for with analysis instead of by design without committing cum hoc ergo propter hoc? The present manuscript answers this conundrum.

stat.OT↗

arXiv2024

Scaling up archival text analysis with the blockmodeling of n-gram networks -- A case study of Bulgaria's representation in the Osservatore Romano (January-May 1877)

This paper seeks to bridge the gap between archival text analysis and network analysis by applying network clustering methods to analyze the coverage of Bulgaria in 123 issues of the newspaper Osservatore Romano published between January and May 1877. Utilizing optical character recognition and generalized homogeneity blockmodeling, the study constructs networks of relevant keywords. Those including the sets Bulgaria and Russia are rather isomorphic and they largely overlap with those for Germany, Britain, and War. In structural terms, the blockmodel of the two networks exhibits a clear core-semiperiphery-periphery structure that reflects relations between concepts in the newpaper's coverage. The newspaper's lexical choices effectively delegitimised the Bulgarian national revival, highlighting the influence of the Holy See on the newspaper's editorial line.

cs.DL↗

arXiv2025

Differentially Private Boxplots

Despite the potential of differentially private data visualization to harmonize data analysis and privacy, research in this area remains underdeveloped. Boxplots are a widely popular visualization used for summarizing a dataset and for comparison of multiple datasets. Consequentially, we introduce a differentially private boxplot. We evaluate its effectiveness for displaying location, scale, skewness and tails of a given empirical distribution. In our theoretical exposition, we show that the location and scale of the boxplot are estimated with optimal sample complexity, and the skewness and tails are estimated consistently, which is not always the case for a boxplot naively constructed from a single existing differentially private quantile algorithm. As a byproduct of this exposition, we introduce several new results concerning private quantile estimation. In simulations, we show that this boxplot performs similarly to a non-private boxplot, and it outperforms the naive boxplot. Additionally, we conduct a real data analysis of Airbnb listings, which shows that comparable analysis can be achieved through differentially private boxplot visualization.

stat.ME↗

arXiv2024

Bayesian Nonparametric Quasi Likelihood

A recent trend in Bayesian research has been revisiting generalizations of the likelihood that enable Bayesian inference without requiring the specification of a model for the data generating mechanism. This paper focuses on a Bayesian nonparametric extension of Wedderburn's quasi-likelihood, using Bayesian additive regression trees to model the mean function. Here, the analyst posits only a structural relationship between the mean and variance of the outcome. We show that this approach provides a unified, computationally efficient, framework for extending Bayesian decision tree ensembles to many new settings, including simplex-valued and heavily heteroskedastic data. We also introduce Bayesian strategies for inferring the dispersion parameter of the quasi-likelihood, a task which is complicated by the fact that the quasi-likelihood itself does not contain information about this parameter; despite these challenges, we are able to inject updates for the dispersion parameter into a Markov chain Monte Carlo inference scheme in a way that, in the parametric setting, leads to a Bernstein-von Mises result for the stationary distribution of the resulting Markov chain. We illustrate the utility of our approach on a variety of both synthetic and non-synthetic datasets.

stat.ME↗

arXiv2024

Advances in Machine Learning, Statistical Methods, and AI for Single-Cell RNA Annotation Using Raw Count Matrices in scRNA-seq Data

Single-cell RNA sequencing (scRNA-seq) has revolutionized our ability to analyze gene expression at the resolution of individual cells, providing unprecedented insights into cellular heterogeneity and complex biological systems. This paper reviews various advanced computational and machine learning techniques tailored for the analysis of scRNA-seq data, emphasizing their roles in different stages of the data processing pipeline.

q-bio.OT↗

arXiv2025

Ride-sharing Determinants: Spatial and Spatio-temporal Bayesian Analysis for Chicago Service in 2022

The rapid expansion of ride-sharing services has caused significant disruptions in the transpor-tation industry and fundamentally altered the way individuals move from one place to another. Accurate estimation of ride-sharing improves service utilization and reliability and reduces travel time and traffic congestion. In this study, we employ two Bayesian models to estimate ride-sharing demand in the 77 Chicago community areas. We consider demographic, scoio-economic, transportation factors as well as land-use characteristics as explanatory variables. Our models assume conditional autoregression (CAR) prior for the explanatory variables. Moreover, the Bayesian frameworks estimate both the unstructured random error and the struc-tured errors for the spatial and the spatiotemporal correlation. We assessed the performance of the estimated models and the residuals of the spatial regression model have no left-over spatial structure. For the spatiotemporal model, the squared correlation between actual ride-shares and the fitted values is 0.95. Our analysis revealed that the demographic factors (populations size and registered crimes) positively impact the ride-sharing demand. Additionally, the ride-sharing demand increases with higher income and increase in the economically active propor-tion of the population as well as the residents with no cars. Moreover, the transit availability and the walkability indices are crucial determinants for the ridesharing in Chicago.

stat.AP↗

arXiv2024

Model-Based Inference and Experimental Design for Interference Using Partial Network Data

The stable unit treatment value assumption states that the outcome of an individual is not affected by the treatment statuses of others, however in many real world applications, treatments can have an effect on many others beyond the immediately treated. Interference can generically be thought of as mediated through some network structure. In many empirically relevant situations however, complete network data (required to adjust for these spillover effects) are too costly or logistically infeasible to collect. Partially or indirectly observed network data (e.g., subsamples, aggregated relational data (ARD), egocentric sampling, or respondent-driven sampling) reduce the logistical and financial burden of collecting network data, but the statistical properties of treatment effect adjustments from these design strategies are only beginning to be explored. In this paper, we present a framework for the estimation and inference of treatment effect adjustments using partial network data through the lens of structural causal models. We also illustrate procedures to assign treatments using only partial network data, with the goal of either minimizing estimator variance or optimally seeding. We derive single network asymptotic results applicable to a variety of choices for an underlying graph model. We validate our approach using simulated experiments on observed graphs with applications to information diffusion in India and Malawi.

stat.ME↗