Searcharxiv⌕ Search

SEARCH · Searcharxiv

Results for “stat.OT”

Search indexed arXiv papers on artificial intelligence, large language models, computer vision and robotics. Read source abstracts and follow links to arXiv.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 865 records · Page 48Linked to original sources
arXiv2023

Mortality modeling at old-age: a mixture model approach

This paper presents a novel approach for modeling mortality rates above age 70 by proposing a mixture-based model. This model is compared to four other widely used models: the Beard, Gompertz, Makeham, and Perks models. Our model can capture the complex behavior of mortality rates at all ages, providing a more accurate representation of the data. To evaluate the performance of our model, we applied it to two countries with different data quality: Japan and Brazil. Our results show that the proposed model outperforms the other models in both countries, particularly in Japan where it obtained an absolute mean percentage error of less than 7%, while the other models presented values greater than 30%. This highlights the ability of our model to adapt to different data quality and country-specific mortality patterns. In summary, this paper presents a mixture-based model that captures the behavior of mortality rates at all ages and outperforms other widely used models in both high- and low-quality data settings. This model can improve mortality prediction and inform public health policy.

stat.AP↗
arXiv2022

mRpostman: An IMAP Client for R

Internet Message Access Protocol (IMAP) clients are a common feature in several programming languages. Despite having some packages for electronic messages retrieval, the R language, until recently, lacked a broader solution, capable of coping with different IMAP servers and providing a wide spectrum of features. mRpostman covers most of the IMAP 4rev1 functionalities by implementing tools for message searching, selective fetching of message attributes, mailbox management, attachment extraction, and several other IMAP features that can be executed in virtually any mail provider. By doing so, it enables users to perform data analysis based on e-mail content. The goal of this article is to showcase the toolkit provided with the mRpostman package, to describe its key features and provide some application examples.

cs.NI↗
arXiv2023

Open Case Studies: Statistics and Data Science Education through Real-World Applications

With unprecedented and growing interest in data science education, there are limited educator materials that provide meaningful opportunities for learners to practice statistical thinking, as defined by Wild and Pfannkuch (1999), with messy data addressing real-world challenges. As a solution, Nolan and Speed (1999) advocated for bringing applications to the forefront in undergraduate statistics curriculum with the use of in-depth case studies to encourage and develop statistical thinking in the classroom. Limitations to this approach include the significant time investment required to develop a case study -- namely, to select a motivating question and to create an illustrative data analysis -- and the domain expertise needed. As a result, case studies based on realistic challenges, not toy examples, are scarce. To address this, we developed the Open Case Studies (https://www.opencasestudies.org) project, which offers a new statistical and data science education case study model. This educational resource provides self-contained, multimodal, peer-reviewed, and open-source guides (or case studies) from real-world examples for active experiences of complete data analyses. We developed an educator's guide describing how to most effectively use the case studies, how to modify and adapt components of the case studies in the classroom, and how to contribute new case studies. (https://www.opencasestudies.org/OCS_Guide).

stat.AP↗
arXiv2023

Correlation-Based And-Operations Can Be Copulas: A Proof

In many practical situations, we know the probabilities $a$ and $b$ of two events $A$ and $B$, and we want to estimate the joint probability ${\rm Prob}(A\,\&\,B)$. The algorithm that estimates the joint probability based on the known values $a$ and $b$ is called an and-operation. An important case when such a reconstruction is possible is when we know the correlation between $A$ and $B$; we call the resulting and-operation correlation-based. On the other hand, in statistics, there is a widely used class of and-operations known as copulas. Empirical evidence seems to indicate that the correlation-based and-operation derived in https://doi.org/10.1007/978-3-031-08971-8_64 is a copula, but until now, no proof of this statement was available. In this paper, we provide such a proof.

stat.OT↗
arXiv2023

Factors other than climate change are currently more important in predicting how well fruit farms are doing financially

Machine learning and statistical modeling methods were used to analyze the impact of climate change on financial wellbeing of fruit farmers in Tunisia and Chile. The analysis was based on face to face interviews with 801 farmers. Three research questions were investigated. First, whether climate change impacts had an effect on how well the farm was doing financially. Second, if climate change was not influential, what factors were important for predicting financial wellbeing of the farm. And third, ascertain whether observed effects on the financial wellbeing of the farm were a result of interactions between predictor variables. This is the first report directly comparing climate change with other factors potentially impacting financial wellbeing of farms. Certain climate change factors, namely increases in temperature and reductions in precipitation, can regionally impact self-perceived financial wellbeing of fruit farmers. Specifically, increases in temperature and reduction in precipitation can have a measurable negative impact on the financial wellbeing of farms in Chile. This effect is less pronounced in Tunisia. Climate impact differences were observed within Chile but not in Tunisia. However, climate change is only of minor importance for predicting farm financial wellbeing, especially for farms already doing financially well. Factors that are more important, mainly in Tunisia, included trust in information sources and prior farm ownership. Other important factors include farm size, water management systems used and diversity of fruit crops grown. Moreover, some of the important factors identified differed between farms doing and not doing well financially. Interactions between factors may improve or worsen farm financial wellbeing.

cs.LG↗
arXiv2023

Matlab routines for centrality in directed acyclic graphs

New Matlab functions for network centrality are introduced. Instead of the mean distance, the generalized mean distance is used. If closer relationships are prioritized, this closeness measure is also defined for unconnected graphs. Instead of distance to all nodes, distance to selected nodes is considered. Besides the vertical in- and out-closeness measures, horizontal cross-closeness is proposed.

stat.OT↗
arXiv2023

Sharing information across patient subgroups to draw conclusions from sparse treatment networks

Network meta-analysis (NMA) usually provides estimates of the relative effects with the highest possible precision. However, sparse networks with few available studies and limited direct evidence can arise, threatening the robustness and reliability of NMA estimates. In these cases, the limited amount of available information can hamper the formal evaluation of the underlying NMA assumptions of transitivity and consistency. In addition, NMA estimates from sparse networks are expected to be imprecise and possibly biased as they rely on large sample approximations which are invalid in the absence of sufficient data. We propose a Bayesian framework that allows sharing of information between two networks that pertain to different population subgroups. Specifically, we use the results from a subgroup with a lot of direct evidence (a dense network) to construct informative priors for the relative effects in the target subgroup (a sparse network). This is a two-stage approach where at the first stage we extrapolate the results of the dense network to those expected from the sparse network. This takes place by using a modified hierarchical NMA model where we add a location parameter that shifts the distribution of the relative effects to make them applicable to the target population. At the second stage, these extrapolated results are used as prior information for the sparse network. We illustrate our approach through a motivating example of psychiatric patients. Our approach results in more precise and robust estimates of the relative effects and can adequately inform clinical practice in presence of sparse networks.

stat.ME↗
arXiv2023

Challenges and Successes of Emergency Online Teaching in Statistics Courses

As the COVID-19 pandemic took hold in early months of 2020, education at all levels was pushed to emergency fully remote, online formats. This emergency shift affected all aspects of teaching and learning with very little notice and often with limited resources. Educators were required to convert entire courses online and shift to remote instructional approaches practically overnight. Students found themselves enrolled in online courses without choice and struggling to adjust to their new learning environments. This article highlights some of the challenges and successes of teaching emergency online undergraduate statistics courses. In particular, we discuss challenges and successes related to (1) technology, (2) classroom community and feedback, and (3) student-content engagement. We also reflect on the opportunity to continue to enhance and enrich the learning experiences of our students by utilizing some of the lessons learned from emergency online teaching as new permanent online statistics courses are developed and/or moved back into the classroom.

stat.OT↗
arXiv2023

A practically efficient fixed-pivot selection algorithm and its extensible MATLAB suite

Many statistical problems and applications require repeated computation of order statistics, such as the median, but most statistical and programming environments do not offer in their main distribution linear selection algorithms. We introduce one, formally equivalent to quickselect, which keeps the position of the pivot fixed. This makes the implementation simpler and much practical compared with the best known solutions. It also enables an "oracular" pivot position option that can reduce a lot the convergence time of certain statistical applications. We have extended the algorithm to weighted percentiles such as the weighted median, applicable to data associated with varying precision measurements, image filtering, descriptive statistics like the medcouple and for combining multiple predictors in boosting algorithms. We provide the new functions in MATLAB, C and R. We have packaged them in a broad MATLAB toolbox addressing robust statistical methods, many of which can be now optimised by means of efficient (weighted) selections.

stat.ME↗
arXiv2023

A Graphical Point Process Framework for Understanding Removal Effects in Multi-Touch Attribution

Marketers employ various online advertising channels to reach customers, and they are particularly interested in attribution for measuring the degree to which individual touchpoints contribute to an eventual conversion. The availability of individual customer-level path-to-purchase data and the increasing number of online marketing channels and types of touchpoints bring new challenges to this fundamental problem. We aim to tackle the attribution problem with finer granularity by conducting attribution at the path level. To this end, we develop a novel graphical point process framework to study the direct conversion effects and the full relational structure among numerous types of touchpoints simultaneously. Utilizing the temporal point process of conversion and the graphical structure, we further propose graphical attribution methods to allocate proper path-level conversion credit, called the attribution score, to individual touchpoints or corresponding channels for each customer's path to purchase. Our proposed attribution methods consider the attribution score as the removal effect, and we use the rigorous probabilistic definition to derive two types of removal effects. We examine the performance of our proposed methods in extensive simulation studies and compare their performance with commonly used attribution models. We also demonstrate the performance of the proposed methods in a real-world attribution application.

stat.ME↗
arXiv2026

Piecewise Deterministic Markov Processes for Bayesian Neural Networks

Inference on modern Bayesian Neural Networks (BNNs) often relies on a variational inference treatment, imposing violated assumptions of independence and the form of the posterior. Traditional MCMC approaches avoid these assumptions at the cost of increased computation due to its incompatibility to subsampling of the likelihood. New Piecewise Deterministic Markov Process (PDMP) samplers permit subsampling, though introduce a model specific inhomogenous Poisson Process (IPPs) which is difficult to sample from. This work introduces a new generic and adaptive thinning scheme for sampling from these IPPs, and demonstrates how this approach can accelerate the application of PDMPs for inference in BNNs. Experimentation illustrates how inference with these methods is computationally feasible, can improve predictive accuracy, MCMC mixing performance, and provide informative uncertainty measurements when compared against other approximate inference schemes.

stat.ML↗
arXiv2023

Against normality testing

I reject the following null hypothesis: {H0: your data are normal}. Such drastic decision is motivated by theoretical reasons, and applies to your current data, the past ones, and the future ones. While this situation may appear embarrassing, it does not invalidate any of your results. Moreover, it allows to save time and energy that are currently spent in vain by performing the following unnecessary tasks: (i) carrying out normality tests; (ii) pretending to do something if normality is rejected; and (iii) arguing about normality with Referee #2.

stat.OT↗
arXiv2023

The Matrix-variate Dirichlet Averages and Its Applications

This paper is about Dirichlet averages in the matrix-variate case or averages of functions over the Dirichlet measure in the complex domain. The classical power mean contains the harmonic mean, arithmetic mean and geometric mean (Hardy, Littlewood and Polya), which is generalized to $y$-mean by deFinetti and hypergeometric mean by Carlson, see the references herein. Carlson's hypergeometric mean is to average a scalar function over a real scalar variable type-1 Dirichlet measure and this in the current literature is known as Dirichlet average of that function. The idea is examined when there is a type-1 or type-2 Dirichlet density in the complex domain. Averages of several functions are computed in such Dirichlet densities in the complex domain. Dirichlet measures are defined when the matrices are Hermitian positive definite. Some applications are also discussed.

math.ST↗
arXiv2023

Reimagining Doctoral Training in Statistics: Is There a Role for a Professional Doctorate?

Modern demands of the statistics profession call for reimagining statistics training. The discipline needs to attract and develop students who are effective as real-world problem solvers, interdisciplinary collaborators, communicators, leaders, and teachers. Demand for statistics professionals with broad technical and non-technical skills has grown in a variety of settings, but especially in business and industry. Academic curricula, though, remain primarily oriented around a narrow, technical conception of statistics. Advanced graduate-level training essentially is limited to research doctorate (PhD) programs which tend to prioritize theoretical and methodological research over development of effective applied statisticians. Other professions, such as those of physicians and surgeons, have training oriented around a professional doctorate, as opposed to a research doctorate. The statistics profession should consider not only changes to PhD curricula, but also the potential for a professional doctorate, drawing ideas from the curricula of other professional degrees such as the MD.

stat.OT↗
arXiv2023

Successful and sustainable undergraduate research in data science

Undergraduate research experiences hold many potential benefits. Students can learn about new areas opening up previously unknown paths in academia and industry. The hands-on experience often provides a deeper understanding of what science, research, and data analysis is and, importantly, is not. While numerous studies have provided information about the benefits and challenges of undergraduate research, many still find it difficult to start an undergraduate research group. Here, we provide a roadmap and resources to help faculty of all levels create and sustain an undergraduate research group in quantitative areas such as statistics, informatics, and data science. While we focus on undergraduate research in data science, many of the recommendations may be generally useful to research mentoring of all levels and other domains.

stat.OT↗
arXiv2023

Why must every data scientist be a Platonist

Data scientists are not mathematicians, but they make heavy use of mathematics in their daily work. While mathematicians can study a mathematical object which is inaccessible to our five senses, data scientists must deal with real-world data which are observable to us. This fine line suggests that a data scientist's philosophical position on mathematics might have a nontrivial impact on their work. By examining how different philosophical views of mathematics affect the interpretation of the basic model assumption in data science, we arrive at the conclusion that a data scientist, who uses modern probabilistic and statistical tools, must be a Platonist.

math.HO↗
arXiv2023

Soft Skills Centrality in Graduate Studies Offerings

Is it possible to measure how critical soft skills like leadership or teamwork are from the viewpoint of graduate studies offerings? This paper provides a conceptual and methodological framework that introduces the concept of a bipartite network as a practical way to estimate the importance of soft skills as socio-emotional abilities trained in graduate studies. We examined 230 graduate programs offered by 49 higher education institutions in Colombia to estimate the empirical importance of soft skills from the viewpoint of graduate studies offerings. The results show that: a) graduate programs in Colombia share 31 soft skills in their intended learning outcomes; b) the centrality of these skills varies as a function of the graduate program, although this variation was not statistically significant; and c) while most central soft skills tend to be those related to creativity (i.e., creation or generation of ideas or projects), leadership (to lead or teamwork), and analytical orientation (e.g., evaluating situations and solving problems), less central were those related to empathy (i.e., understanding others and acknowledgment of others), ethical thinking, and critical thinking, posing the question if too much emphasis on most visible skills might imply an unbalance in the opportunities to enhancing other soft skills such as ethical thinking.

cs.SI↗
arXiv2023

The Value of Information and Circular Settings

We present a universal concept for the Value of Information (VoI), based on the works of Claude Shannon's and Ruslan Stratonovich that can take into account very general preferences of the agents and results in a single number. As such it is convenient for applications and also has desirable properties for decision theory and demand analysis. The Shannon/Stratonovich VoI concept is compared to alternatives and applied in examples. In particular we apply the concept to a circular spatial structure well known from many economic models and allow for various economic transport costs.

econ.TH↗