SEARCH · Searcharxiv

Results for “stat.OT”

Search indexed arXiv papers on artificial intelligence, large language models, computer vision and robotics. Read source abstracts and follow links to arXiv.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 1,135 records · Page 63Linked to original sources

arXiv2025

Data Verbalisation: What is Text Doing in a Data Visualisation?

This article discusses the role that text elements play in a data visualisation. We argue that there is a need for a simple, coherent explanation of text elements similar to the understanding that already exists for non-text elements like bars, points, and lines. We explore examples of how text is used within a data visualisation and use existing knowledge and assessment techniques to evaluate when text is effective and when it is not. The result is a framework that aims to be easy to understand and easy to apply in order to understand the purpose and effectiveness of the text elements in any data visualisation.

cs.HC↗

arXiv2025

Valid Selection among Conformal Sets

Conformal prediction offers a distribution-free framework for constructing prediction sets with coverage guarantees. In practice, multiple valid conformal prediction sets may be available, arising from different models or methodologies. However, selecting the most desirable set, such as the smallest, can invalidate the coverage guarantees. To address this challenge, we propose a stability-based approach that ensures coverage for the selected prediction set. We extend our results to the online conformal setting, propose several refinements in settings where additional structure is available, and demonstrate its effectiveness through experiments.

stat.ML↗

arXiv2025

Average-case complexity in statistical inference: A puzzle-driven research seminar

These notes describe our experience with running a student seminar on average-case complexity in statistical inference using the jigsaw learning format at ETH Zurich in Fall of 2024. The jigsaw learning technique is an active learning technique where students work in groups on independent parts of the task and then reassemble the groups to combine all the parts together. We implemented this technique for the proofs of various recent research developments, combined with a presentation by one of the students in the beginning of the session. We describe our experience and thoughts on such a format applied in a student research seminar: including, but not limited to, higher engagement, more accessible talks by the students, and increased student participation in discussions. In the Appendix, we include all the exercises sheets for the topic, which may be of independent interest for courses on statistical-to-computational gaps and average-case complexity.

stat.OT↗

arXiv2025

A Plea for History and Philosophy of Statistics and Machine Learning

The integration of the history and philosophy of statistics was initiated at least by Hacking (1975) and advanced by Hacking (1990), Mayo (1996), and Zabell (2005), but it has not received sustained follow-up. Yet such integration is more urgent than ever, as the recent success of artificial intelligence has been driven largely by machine learning -- a field historically developed alongside statistics. Today, the boundary between statistics and machine learning is increasingly blurred. What we now need is integration, twice over: of history and philosophy, and of two fields they engage -- statistics and machine learning. I present a case study of a philosophical idea in machine learning (and in formal epistemology) whose root can be traced back to an often under-appreciated insight in Neyman and Pearson's 1936 work (a follow-up to their 1933 classic). This leads to the articulation of an epistemological principle -- largely implicit in, but shared by, the practices of frequentist statistics and machine learning -- which I call achievabilism: the thesis that the correct standard for assessing non-deductive inference methods should not be fixed, but should instead be sensitive to what is achievable in specific problem contexts. Another integration also emerges at the level of methodology, combining two ends of the philosophy of science spectrum: history and philosophy of science on the one hand, and formal epistemology on the other hand.

stat.OT↗

arXiv2026

Treatment, evidence, imitation, and chat

Large language models are thought to have the potential to aid in medical decision making. This work investigates the degree to which this might be the case. We start with the treatment problem, the patient's core medical decision-making task, which is solved in collaboration with a clinician. We discuss different approaches to solving it, including, within evidence-based medicine, experimental and observational data. We then discuss the chat problem, and how this differs from the treatment problem -- in particular with respect to imitation (and how imitation alone cannot solve the true treatment problem, although this does not mean it is not useful). We then discuss how a large-language-model-based system might be trained to solve the treatment problem, highlighting that the major challenges relate to the ethics of experimentation and the assumptions associated with observation. We finally discuss how these challenges relate to evidence-based medicine and how this might inform the efforts of the medical research community to solve the treatment problem. Throughout, we illustrate our arguments with the cholesterol medications, statins.

stat.OT↗

arXiv2025

BACTA-GPT: An AI-Based Bayesian Adaptive Clinical Trial Architect

Bayesian adaptive clinical trials offer a flexible and efficient alternative to traditional fixed-design trials, but their implementation is often hindered by the complexity of Bayesian computations and the need for advanced statistical programming expertise. The authors introduce a custom fine-tuned LLM designed to assist with this and lower barriers to adoption of Bayesian methods for adaptive clinical trials. This paper describes the development and fine-tuning of BACTA-GPT, a Large Language Model (LLM)-based tool designed to assist in the implementation of Bayesian Adaptive Clinical Trials. This engine uses GPT-3.5 as the underlying model and takes in Natural Language input from the Statistician or the Trialist. The fine-tuned model demonstrates a viable proof-of-concept in its objectives. Test case evaluations show that the model is capable of generating a fit-for-purpose Bayesian model for an adaptive trial and evaluate its operating characteristics via simulations using R and JAGS. The integration of AI code generation has significant potential to lower technical barriers for the design and implementation of Bayesian Adaptive trials. But they also require attention to important considerations regarding validation and quality control.

stat.AP↗

arXiv2025

When Numbers Mislead Us

The belief that numbers offer a single, objective description of reality overlooks a crucial truth: data does not speak for itself. Every dataset results from choices-what to measure, how, when, and with whom-which inevitably reflect implicit, and sometimes ideological, assumptions about what is worth quantifying. Moreover, in any analysis, what remains unmeasured can be just as significant as what is captured. When a key variable is omitted-whether by neglect, design, or ignorance-it can distort the observed relationships between other variables. This phenomenon, known as omitted variable bias, may produce misleading correlations or conceal genuine effects. In some cases, accounting for this hidden factor can completely overturn the conclusions drawn from a superficial analysis. This is precisely the mechanism behind Simpson's paradox.

stat.OT↗

arXiv2025

Are Betting Markets Better than Polling in Predicting Political Elections?

Political elections are one of the most significant aspects of what constitutes the fabric of the United States. In recent history, typical polling estimates have largely lacked precision in predicting election outcomes, which has not only caused uncertainty for American voters, but has also impacted campaign strategies, spending, and fundraising efforts. One intriguing aspect of traditional polling is the types of questions that are asked -- the questions largely focus on asking individuals who they intend to vote for. However, they don't always probe who voters think will win -- regardless of who they want to win. In contrast, online betting markets allow individuals to wager money on who they expect to win, which may capture who individuals think will win in an especially salient manner. The current study used both descriptive and predictive analytics to determine whether data from Polymarket, the world's largest online betting market, provided insights that differed from traditional presidential polling. Overall, findings suggest that Polymarket was superior to polling in predicting the outcome of the 2024 presidential election, particularly in swing states. Results are in alignment with research on ''Wisdom of Crowds'' theory, which suggests a large group of people are often accurate in predicting outcomes, even if they are not necessarily experts or closely aligned with the issue at hand. Overall, our results suggest that betting markets, such as Polymarket, could be employed to predict presidential elections and/or other real-world events. However, future investigations are needed to fully unpack and understand the current study's intriguing results, including alignment with Wisdom of Crowds theory and portability to other events.

stat.OT↗

arXiv2025

R2 priors for Grouped Variance Decomposition in High-dimensional Regression

We introduce the Group-R2 decomposition prior, a hierarchical shrinkage prior that extends R2-based priors to structured regression settings with known groups of predictors. By decomposing the prior distribution of the coefficient of determination R2 in two stages, first across groups, then within groups, the prior enables interpretable control over model complexity and sparsity. We derive theoretical properties of the prior, including marginal distributions of coefficients, tail behavior, and connections to effective model complexity. Through simulation studies, we evaluate the conditions under which grouping improves predictive performance and parameter recovery compared to priors that do not account for groups. Our results provide practical guidance for prior specification and highlight both the strengths and limitations of incorporating grouping into R2-based shrinkage priors.

stat.ME↗

arXiv2026

Robust Causal Discovery in Real-World Time Series with Power-Laws

Exploring causal relationships in stochastic time series is a challenging yet crucial task with a vast range of applications, including finance, economics, neuroscience, and climate science. Many algorithms for Causal Discovery (CD) have been proposed; however, they often exhibit a high sensitivity to noise, resulting in spurious causal inferences in real data. In this paper, we observe that the frequency spectra of many real-world time series follow a power-law distribution, notably due to an inherent self-organizing behavior. Leveraging this insight, we build a robust CD method based on the extraction of power-law spectral features that amplify genuine causal signals. Our method consistently outperforms state-of-the-art alternatives on both synthetic benchmarks and real-world datasets with known causal structures, demonstrating its robustness and practical relevance.

cs.LG↗

arXiv2025

Hierarchical Temporal Point Process Modeling of Aggressive Behavior Onset in Psychiatric Inpatient Youth with Autism for Branching Factor Estimation

Aggressive behavior in autistic inpatient youth often arises in temporally clustered bursts complicating efforts to distinguish external triggers from internal escalation. The sample population branching factor-the expected number of new onsets triggered by a given event-is a key summary of self-excitation in behavior dynamics. Prior pooled models overestimate this quantity by ignoring patient-specific variability. We addressed this using a hierarchical Hawkes process with an exponential kernel and edge-effect correction allowing partial pooling across patients. This approach reduces bias from high-frequency individuals and stabilizes estimates for those with sparse data. Bayesian inference was performed using the No U-Turn Sampler with model evaluation via convergence diagnostics, power-scaling sensitivity analysis, and multiple Goodness-of-Fit (GOF) metrics: PSIS-LOO the Lewis test with Durbin's modification and residual analysis based on the Random Time Change Theorem (RTCT). The hierarchical model yielded a significantly lower and more precise branching factor estimate mean (0.742 +- 0.026) than the pooled model (0.899 +- 0.015) and narrower intervals than the unpooled model (0.717 +- 0.139). This led to a threefold smaller cascade of events per onset under the hierarchical model. Sensitivity analyses confirmed robustness to prior and likelihood perturbations while the unpooled model showed instability for sparse individuals. GOF measures consistently favored or on par to the hierarchical model. Hierarchical Hawkes modeling with edge-effect correction provides robust estimation of branching dynamics by capturing both within- and between-patient variability. This enables clearer separation of endogenous from exogenous events supports linkage to physiological signals and enhances early warning systems individualized treatment and resource allocation in inpatient care.

stat.AP↗

arXiv2026

Multivariate Standardized Residuals for Conformal Prediction

While split conformal prediction guarantees marginal coverage, approaching the stronger property of conditional coverage is essential for reliable uncertainty quantification. Naive conformal scores, however, suffer from poor conditional coverage in heteroskedastic settings. In univariate regression, this is commonly addressed by normalizing non-conformity scores using an estimated local score variance. In this work, we propose a natural extension of this normalization to the multivariate setting, effectively whitening the residuals to decouple output correlations and standardize local variance. Furthermore, we derive a sufficient condition characterizing a broad class of distributions for which standardized residuals yield asymptotic conditional coverage. We demonstrate that using the Mahalanobis distance induced by a learned local covariance as a non-conformity score provides a closed-form, computationally efficient mechanism for capturing inter-output correlations and heteroskedasticity, avoiding the expensive sampling required by previous methods based on cumulative distribution functions. This structure unlocks several practical extensions, including the handling of missing output values, the refinement of conformal sets when partial information is revealed, and the construction of valid conformal sets for transformations of the output. Finally, we provide extensive empirical evidence on both synthetic and real-world datasets showing that our approach yields conformal sets that improve upon the conditional coverage of existing multivariate baselines.

stat.ML↗

arXiv2025

A Generalized Cramér-Rao Bound Using Information Geometry

In information geometry, statistical models are considered as differentiable manifolds, where each probability distribution represents a unique point on the manifold. A Riemannian metric can be systematically obtained from a divergence function using Eguchi's theory (1992); the well-known Fisher-Rao metric is obtained from the Kullback-Leibler (KL) divergence. The geometric derivation of the classical Cramér-Rao Lower Bound (CRLB) by Amari and Nagaoka (2000) is based on this metric. In this paper, we study a Riemannian metric obtained by applying Eguchi's theory to the Basu-Harris-Hjort-Jones (BHHJ) divergence (1998) and derive a generalized Cramér-Rao bound using Amari-Nagaoka's approach. There are potential applications for this bound in robust estimation.

math.ST↗

arXiv2025

An alternative method of adjusting for multiple comparison in medical research

Background Most methods of adjusting for multiplicity focus primarily on controlling type I errors and rarely consider type II errors. We propose a new method that considers controlling for false-positive findings while ensuring sufficient statistical power. Methods We proposed a new method for multiple corrections called (Beta-exponential Adjustment, BEA) that considered the statistical power to control for type I errors while also considering the probability of type II errors. We conducted simulation studies to evaluate the performance characteristic of multiple testing correction procedures. We calculated sensitivity, specificity, and power separately for different sample sizes and number of biomarkers and compared them with the Bonferroni, Holm, and Benjamini-Hochberg (BH) correction methods. Results The results demonstrated that our proposed BEA correction method exhibited the highest sensitivity at different sample sizes and biomarkers (e.g., sensitivity: BEA 0.8 versus BH 0.62 at sample size at 1000, tested biomarkers at 1000 and positive rate at 30%). With different sample sizes and number of biomarkers, the BEA correction method demonstrated comparable specificity compared with traditional methods. Moreover, we observed that the BEA-corrected had the highest statistical power than other methods, when the outcome was relatively rare. Conclusion We proposed the BEA multiple correction method to adjust for multiple comparisons while considering statistical power. The BEA method demonstrated a higher sensitivity, comparable specificity, and higher statistical power, compared with traditional correction methods in different conditions. The BEA correction method can be an alternative of traditional methods of adjusting for multiplicity, especially in studies with small sample size, rare outcomes, or substantial number of biomarkers.

stat.AP↗

arXiv2025

Efficient inference of dynamic gene regulatory networks using discrete penalty

Gene regulatory networks (GRNs) orchestrate cellular decision making and survival strategies. Inferring the structure of these networks from high-dimensional transcriptomics data is a central challenge in systems biology. Traditional approaches to GRN inference, such as the graphical lasso and its joint extensions, rely on $\ell_1$ penalty to induce sparsity but can bias network recovery and require extensive hyperparameter tuning. Here, we present a scalable framework for the joint inference of dynamic GRNs using a discrete $\ell_0$ penalty, enabling direct and unbiased control over network sparsity. Leveraging recent algorithmic advances, we efficiently solve the resulting mixed-integer optimization problem for populations structured as arbitrary tree hypergraphs, accommodating both continuous and categorical distinctions among biological samples. After validating our method on synthetic benchmarks, we apply it to single-cell and spatial transcriptomics data from glioblastoma (GBM) patient tumors. Our approach reconstructs gene networks across tumor clusters, maps network rewiring along hypoxia gradients, and reveals niche-specific differences between primary and recurrent tumors. By providing a robust and interpretable tool for GRN inference in complex tissues, our work facilitates high-resolution dissection of tumor heterogeneity and adaptation, with broad applicability to emerging large-scale transcriptomic datasets.

stat.AP↗

arXiv2025

Measuring Human Leadership Skills with Artificially Intelligent Agents

We show that the ability to lead groups of humans is predicted by leadership skill with Artificially Intelligent agents. In a large pre-registered lab experiment, human leaders worked with AI agents to solve problems. Their performance on this 'AI leadership test' was strongly correlated with their causal impact on human teams, which we estimate by repeatedly randomly assigning leaders to groups of human followers and measuring team performance. Successful leaders of both humans and AI agents ask more questions and engage in more conversational turn-taking; they score higher on measures of social intelligence, fluid intelligence, and decision-making skill, but do not differ in gender, age, ethnicity or education. Our findings indicate that AI agents can be effective proxies for human participants in social experiments, which greatly simplifies the measurement of leadership and teamwork skills.

econ.GN↗

arXiv2025

A Blueprint to Design Curriculum and Pedagogy for Introductory Data Science

As the demand for jobs in data science increases, so does the demand for universities to develop and facilitate modernized data science curricula to train students for these positions. Yet, the development of these courses remains challenging, especially at the introductory level. To help instructors to meet this demand, we present a flexible blueprint that supports the development of a modernized introductory data science curriculum. This blueprint is narrated through the lens and experience in teaching the introductory data science course at \university{}. This is a large course that serves both STEM and non-STEM majors and includes the incorporation and facilitation of technologies such as R, RStudio, Quarto, Git, and GitHub. We identify and provide discussion around common challenges in teaching a modernized introductory data science course, detail a learning model for students to grow their understanding of data science concepts, and provide reproducible materials to help empower teachers to adopt and adapt such curriculum at their universities.

stat.OT↗

arXiv2026

GeoGR^2:Zero-Shot Geospatial Inference via Geostatistically-Guided Iterative Refinement with LLMs

Standard large language model prompting treats geospatial inference as independent, instance-wise prediction, ignoring the fundamental spatial dependencies that govern geographic reality. Consequently, even advanced models struggle with spatial consistency and exhibit severe biases toward populous regions. To bridge this gap, we propose GeoGR^2 (Geospatial Graph Refine Reasoning), a framework that formalizes zero-shot geospatial prediction as an iterative message-passing process on a dynamically constructed graph. Unlike static retrieval methods, GeoGR^2 instantiates three dynamic operators via collaborating operators: (1) a Topology Operator that constructs graph topology to enforce the Spatial Markov property; (2) a Feature Operator that enriches nodes with task-relevant semantic covariates; and (3) an Update Operator that performs natural language message passing to iteratively minimize spatial discrepancy. Theoretically, we frame this refinement as a contraction mapping that approximates the fixed point of a global consistency equation. Empirically, we validate GeoGR^2 on diverse physical and socioeconomic tasks. Results demonstrate that by explicitly embedding geostatistical inductive biases, GeoGR^2 significantly outperforms standard prompting baselines, while effectively mitigating systematic geographic bias. Our framework leverages large language models' intrinsic capacity for understanding spatial correlations through explicit topological scaffolding, without resorting to general graph reasoning paradigms. The code of GeoGR^2 is available at https://github.com/JinfanTang/GeoGRR.

cs.AI↗