SEARCH · Searcharxiv

Results for “stat.OT”

Search indexed arXiv papers on artificial intelligence, large language models, computer vision and robotics. Read source abstracts and follow links to arXiv.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 973 records · Page 54Linked to original sources

arXiv2024

Gerontologic Biostatistics 2.0: Developments over 10+ years in the age of data science

Background: Introduced in 2010, the sub-discipline of gerontologic biostatistics (GBS) was conceptualized to address the specific challenges in analyzing data from research studies involving older adults. However, the evolving technological landscape has catalyzed data science and statistical advancements since the original GBS publication, greatly expanding the scope of gerontologic research. There is a need to describe how these advancements enhance the analysis of multi-modal data and complex phenotypes that are hallmarks of gerontologic research. Methods: This paper introduces GBS 2.0, an updated and expanded set of analytical methods reflective of the practice of gerontologic biostatistics in contemporary and future research. Results: GBS 2.0 topics and relevant software resources include cutting-edge methods in experimental design; analytical techniques that include adaptations of machine learning, quantifying deep phenotypic measurements, high-dimensional -omics analysis; the integration of information from multiple studies, and strategies to foster reproducibility, replicability, and open science. Discussion: The methodological topics presented here seek to update and expand GBS. By facilitating the synthesis of biostatistics and data science in gerontology, we aim to foster the next generation of gerontologic researchers.

stat.ME↗

arXiv2025

Using Mathlink Cubes to Introduce Data Wrangling with Examples in R

This paper explores an innovative approach to teaching data wrangling skills to students through hands-on activities before transitioning to coding. Data wrangling, a critical aspect of data analysis, involves cleaning, transforming, and restructuring data. We introduce the use of a physical tool, mathlink cubes, to facilitate a tangible understanding of data sets. This approach helps students grasp the concepts of data wrangling before implementing them in coding languages such as R. We detail a classroom activity that includes hands-on tasks paralleling common data wrangling processes such as filtering, selecting, and mutating, followed by their coding equivalents using R's `dplyr` package.

cs.HC↗

arXiv2024

A scalable, synergy-first backbone decomposition of higher-order structures in complex systems

Since its introduction in 2011, the partial information decomposition (PID) has triggered an explosion of interest in the field of multivariate information theory and the study of emergent, higher-order ("synergistic") interactions in complex systems. Despite its power, however, the PID has a number of limitations that restrict its general applicability: it scales poorly with system size and the standard approach to decomposition hinges on a definition of "redundancy", leaving synergy only vaguely defined as "that information not redundant." Other heuristic measures, such as the O-information, have been introduced, although these measures typically only provided a summary statistic of redundancy/synergy dominance, rather than direct insight into the synergy itself. To address this issue, we present an alternative decomposition that is synergy-first, scales much more gracefully than the PID, and has a straightforward interpretation. Our approach defines synergy as that information in a set that would be lost following the minimally invasive perturbation on any single element. By generalizing this idea to sets of elements, we construct a totally ordered "backbone" of partial synergy atoms that sweeps systems scales. Our approach starts with entropy, but can be generalized to the Kullback-Leibler divergence, and by extension, to the total correlation and the single-target mutual information. Finally, we show that this approach can be used to decompose higher-order interactions beyond just information theory: we demonstrate this by showing how synergistic combinations of pairwise edges in a complex network supports signal communicability and global integration. We conclude by discussing how this perspective on synergistic structure (information-based or otherwise) can deepen our understanding of part-whole relationships in complex systems.

cs.IT↗

arXiv2024

A computed 95% confidence interval does cover the true value with probability 0.95 if epistemically interpreted

Suppose the lifetime of a large sample of batteries in routine use is measured. A confidence interval is computed to 394 plus/minus 1.96 times 4.6 days. The standard interpretation is that if we repeatedly draw samples and compute confidence intervals, about 95% of the intervals will cover the unknown true lifetime. What can be said about the particular interval 394 plus/minus 1.96 times 4.6 has not been clear. We clarify this by using an epistemic interpretation of probability. The conclusion is that a realised (computed) confidence interval covers the parameter with the probability given by the confidence level is a valid statement, unless there are relevant and recognisable subsets of the sample.

stat.OT↗

arXiv2024

A framework for understanding data science

The objective of this research is to provide a framework with which the data science community can understand, define, and develop data science as a field of inquiry. The framework is based on the classical reference framework (axiology, ontology, epistemology, methodology) used for 200 years to define knowledge discovery paradigms and disciplines in the humanities, sciences, algorithms, and now data science. I augmented it for automated problem-solving with (methods, technology, community). The resulting data science reference framework is used to define the data science knowledge discovery paradigm in terms of the philosophy of data science addressed in previous papers and the data science problem-solving paradigm, i.e., the data science method, and the data science problem-solving workflow, both addressed in this paper. The framework is a much called for unifying framework for data science as it contains the components required to define data science. For insights to better understand data science, this paper uses the framework to define the emerging, often enigmatic, data science problem-solving paradigm and workflow, and to compare them with their well-understood scientific counterparts, scientific problem-solving paradigm and workflow.

stat.OT↗

arXiv2024

An analysis of the NCAA college football playoff team selections using an Elo ratings model

In December 2023 the Florida State Seminoles became the first Power 5 school to have an undefeated season and miss selection for the College Football Playoff. In order to assess this decision, we employed an Elo ratings model to rank the teams and found that the selection committee's decision was justified and that Florida State were not one of the four best teams in college football in that season (ranking only 11th!). We extended this analysis to all other years of the CFP and found that the top four teams by Elo ratings differ greatly from the four teams selected in almost every year of the CFP's existence. Furthermore, we found that there have been more egregious non-selections including when Alabama was ranked first by Elo ratings in 2022 and were not selected. The analysis suggests that the current criteria are too subjective and a ratings model should be implemented to provide transparency for the sport, its teams, and its fans.

stat.OT↗

arXiv2024

Improving the Equation of Exchange for Cryptoasset Valuation Using Empirical Data

In the evolving domain of cryptocurrency markets, accurate token valuation remains a critical aspect influencing investment decisions and policy development. Whilst the prevailing equation of exchange pricing model offers a quantitative valuation approach based on the interplay between token price, transaction volume, supply, and either velocity or holding time, it exhibits intrinsic shortcomings. Specifically, the model may not consistently delineate the relationship between average token velocity and holding time. This paper aims to refine this equation, enhancing the depth of insight into token valuation methodologies.

cs.CE↗

arXiv2024

Guidelines for the Creation of Analysis Ready Data

Globally, there is an increased need for guidelines to produce high-quality data outputs for analysis. No framework currently exists that provides guidelines for a comprehensive approach to producing analysis ready data (ARD). Through critically reviewing and summarising current literature, this paper proposes such guidelines for the creation of ARD. The guidelines proposed in this paper inform ten steps in the generation of ARD: ethics, project documentation, data governance, data management, data storage, data discovery and collection, data cleaning, quality assurance, metadata, and data dictionary. These steps are illustrated through a substantive case study that aimed to create ARD for a digital spatial platform: the Australian Child and Youth Wellbeing Atlas (ACYWA).

cs.DB↗

arXiv2024

Risk Quadrangle and Robust Optimization Based on Extended $φ$-Divergence

The Fundamental Risk Quadrangle (FRQ) is a unified framework linking risk management, statistical estimation, and optimization. Distributionally robust optimization (DRO) based on $φ$-divergence minimizes the maximal expected loss, where the maximum is over a $φ$-divergence ambiguity set. This paper introduces the \emph{extended} $φ$-divergence and the extended $φ$-divergence quadrangle, which integrates DRO into the FRQ framework. We derive the primal and dual representations of the quadrangle elements (risk, deviation, regret, error, and statistic). The dual representation provides an interpretation of classification, portfolio optimization, and regression as robust optimization based on the extended $φ$-divergence. The primal representation offers tractable formulations of these robust optimizations as convex optimization. We provide illustrative examples showing that many common problems, such as least-squares regression, quantile regression, support vector machines, and CVaR optimization, fall within this framework. Additionally, we conduct a case study to visualize the optimal solution of the inner maximization in robust optimization.

math.OC↗

arXiv2024

Robust estimations from distribution structures: I. Mean

As the most fundamental problem in statistics, robust location estimation has many prominent solutions, such as the trimmed mean, Winsorized mean, Hodges Lehmann estimator, Huber M estimator, and median of means. Recent studies suggest that their maximum biases concerning the mean can be quite different, but the underlying mechanisms largely remain unclear. This study exploited a semiparametric method to classify distributions by the asymptotic orderliness of quantile combinations with varying breakdown points, showing their interrelations and connections to parametric distributions. Further deductions explain why the Winsorized mean typically has smaller biases compared to the trimmed mean; two sequences of semiparametric robust mean estimators emerge, particularly highlighting the superiority of the median Hodges Lehmann mean. This article sheds light on the understanding of the common nature of probability distributions.

math.ST↗

arXiv2024

Quantitative Analysis of AI-Generated Texts in Academic Research: A Study of AI Presence in Arxiv Submissions using AI Detection Tool

Many people are interested in ChatGPT since it has become a prominent AIGC model that provides high-quality responses in various contexts, such as software development and maintenance. Misuse of ChatGPT might cause significant issues, particularly in public safety and education, despite its immense potential. The majority of researchers choose to publish their work on Arxiv. The effectiveness and originality of future work depend on the ability to detect AI components in such contributions. To address this need, this study will analyze a method that can see purposely manufactured content that academic organizations use to post on Arxiv. For this study, a dataset was created using physics, mathematics, and computer science articles. Using the newly built dataset, the following step is to put originality.ai through its paces. The statistical analysis shows that Originality.ai is very accurate, with a rate of 98%.

cs.DL↗

arXiv2024

A machine learning approach to predict university enrolment choices through students' high school background in Italy

This paper explores the influence of Italian high school students' proficiency in mathematics and the Italian language on their university enrolment choices, specifically focusing on STEM (Science, Technology, Engineering, and Mathematics) courses. We distinguish between students from scientific and humanistic backgrounds in high school, providing valuable insights into their enrolment preferences. Furthermore, we investigate potential gender differences in response to similar previous educational choices and achievements. The study employs gradient boosting methodology, known for its high predicting performance and ability to capture non-linear relationships within data, and adjusts for variables related to the socio-demographic characteristics of the students and their previous educational achievements. Our analysis reveals significant differences in the enrolment choices based on previous high school achievements. The findings shed light on the complex interplay of academic proficiency, gender, and high school background in shaping students' choices regarding university education, with implications for educational policy and future research endeavours.

cs.LG↗

arXiv2024

Why Name Popularity is a Good Test of Historicity

Are name statistics in the Gospels and Acts a good test of historicity? Kamil Gregor and Brian Blais, in a recent article in The Journal for the Study of the Historical Jesus, argue that the sample of name occurrences in the Gospels and Acts is too small to be determinative and that several statistical anomalies weigh against a positive verdict. Unfortunately, their conclusions result directly from improper testing and questionable data selection. Chi-squared goodness-of-fit testing establishes that name occurrences in the Gospels and Acts fit into their historical context at least as good as those in the works of Josephus. Additionally, they fit better than occurrences derived from ancient fictional sources and occurrences from modern, well-researched historical novels.

stat.OT↗

arXiv2024

Robust estimations from distribution structures: V. Non-asymptotic

Due to the complexity of order statistics, the finite sample behaviour of robust statistics is generally not analytically solvable. While the Monte Carlo method can provide approximate solutions, its convergence rate is typically very slow, making the computational cost to achieve the desired accuracy unaffordable for ordinary users. In this paper, we propose an approach analogous to the Fourier transformation to decompose the finite sample structure of the uniform distribution. By obtaining sets of sequences that are consistent with parametric distributions for the first four sample moments, we can approximate the finite sample behavior of other estimators with significantly reduced computational costs. This article reveals the underlying structure of randomness and presents a novel approach to integrate multiple assumptions.

stat.ME↗

arXiv2024

Best Subset Solution Path for Linear Dimension Reduction Models using Continuous Optimization

The selection of best variables is a challenging problem in supervised and unsupervised learning, especially in high dimensional contexts where the number of variables is usually much larger than the number of observations. In this paper, we focus on two multivariate statistical methods: principal components analysis and partial least squares. Both approaches are popular linear dimension-reduction methods with numerous applications in several fields including in genomics, biology, environmental science, and engineering. In particular, these approaches build principal components, new variables that are combinations of all the original variables. A main drawback of principal components is the difficulty to interpret them when the number of variables is large. To define principal components from the most relevant variables, we propose to cast the best subset solution path method into principal component analysis and partial least square frameworks. We offer a new alternative by exploiting a continuous optimization algorithm for best subset solution path. Empirical studies show the efficacy of our approach for providing the best subset solution path. The usage of our algorithm is further exposed through the analysis of two real datasets. The first dataset is analyzed using the principle component analysis while the analysis of the second dataset is based on partial least square framework.

stat.ME↗

arXiv2024

Efficient estimation for a smoothing thin plate spline in a two-dimensional space

Using a deterministic framework allows us to estimate a function with the purpose of interpolating data in spatial statistics. Radial basis functions are commonly used for scattered data interpolation in a d-dimensional space, however, interpolation problems have to deal with dense matrices. For the case of smoothing thin plate splines, we propose an efficient way to address this problem by compressing the dense matrix by an hierarchical matrix ($\mathcal{H}$-matrix) and using the conjugate gradient method to solve the linear system of equations. A simulation study was conducted to assess the effectiveness of the spatial interpolation method. The results indicated that employing an $\mathcal{H}$-matrix along with the conjugate gradient method allows for efficient computations while maintaining a minimal error. We also provide a sensitivity analysis that covers a range of smoothing and compression parameter values, along with a Monte Carlo simulation aimed at quantifying uncertainty in the approximated function. Lastly, we present a comparative study between the proposed approach and thin plate regression using the "mgcv" package of the statistical software R. The comparison results demonstrate similar interpolation performance between the two methods.

stat.CO↗

arXiv2025

Improved Semi-Parametric Bounds for Tail Probability and Expected Loss: Theory and Applications

Many management decisions involve accumulated random realizations for which only the first and second moments of their distribution are available. The sharp Chebyshev-type bound for the tail probability and Scarf bound for the expected loss are widely used in this setting. We revisit the tail behavior of such quantities with a focus on independence. Conventional primal-dual approaches from optimization are ineffective in this setting. Instead, we use probabilistic inequalities to derive new bounds and offer new insights. For non-identical distributions attaining the tail probability bounds, we show that the extreme values are equidistant regardless of the distributional differences. For the bound on the expected loss, we show that the impact of each random variable on the expected sum can be isolated using an extension of the Korkine identity. We illustrate how these new results open up abundant practical applications, including improved pricing of product bundles, more precise option pricing, more efficient insurance design, and better inventory management. For example, we establish a new solution to the optimal bundling problem, yielding a 17% uplift in per-bundle profits, and a new solution to the inventory problem, yielding a 5.6% cost reduction for a model with 20 retailers.

econ.EM↗

arXiv2024

Non-Parametric Estimation of Multiple Periodic Components in Turkey's Electricity Consumption

Electric generation and consumption are an essential component of contemporary living, influencing diverse facets of our daily routines, convenience, and economic progress. There is a high demand for characterizing the periodic pattern of electricity consumption. VBPBB employs a bandpass filter aligned to retain the frequency of a PC component and eliminating interference from other components. This leads to a significant reduction in the size of bootstrapped confidence intervals. Furthermore, other PC bootstrap methods preserve one but not multiple periodically correlated components, resulting in superior performance compared to other methods by providing a more precise estimation of the sampling distribution for the desired characteristics. The study of the periodic means of Turkey electricity consumption using VBPBB is presented and compared with outcomes from alternative bootstrapping approaches. These findings offer significant evidence supporting the existence of daily, weekly, and annual PC patterns, along with information on their timing and confidence intervals for their effects. This information is valuable for enhancing predictions and preparations for future responses to electricity consumption.

stat.AP↗