SEARCH · Searcharxiv

Results for “stat.OT”

Search indexed arXiv papers on artificial intelligence, large language models, computer vision and robotics. Read source abstracts and follow links to arXiv.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 235 records · Page 13Linked to original sources

arXiv2016

A note on basis dimension selection in generalized additive modelling

Two new approaches for checking the dimension of the basis functions when using penalized regression smoothers are presented. The first approach is a test for adequacy of the basis dimension based on an estimate of the residual variance calculated by differencing residuals that are neighbours according to the smooth covariates. The second approach is based on estimated degrees of freedom for a smooth of the model residuals with respect to the model covariates. In comparison with basis dimension selection algorithms based on smoothness selection criterion (GCV, AIC, REML) the above procedures are computationally efficient enough for routine use as part of model checking.

stat.ME↗

arXiv2016

A Bayesian baseline for belief in uncommon events

The plausibility of uncommon events and miracles based on testimony of such an event has been much discussed. When analyzing the probabilities involved, it has mostly been assumed that the common events can be taken as data in the calculations. However, we usually have only testimonies for the common events. While this difference does not have a significant effect on the inductive part of the inference, it has a large influence on how one should view the reliability of testimonies. In this work, a full Bayesian solution is given for the more realistic case, where one has a large number of testimonies for a common event and one testimony for an uncommon event. It is seen that, in order for there to be a large amount of testimonies for a common event, the testimonies will probably be quite reliable. For this reason, because the testimonies are quite reliable based on the testimonies for the common events, the probability for the uncommon event, given a testimony for it, is also higher. Hence, one should be more open-minded when considering the plausibility of uncommon events.

stat.AP↗

arXiv2016

On the capacity of information processing systems

We propose and analyze a family of information processing systems, where a finite set of experts or servers are employed to extract information about a stream of incoming jobs. Each job is associated with a hidden label drawn from some prior distribution. An inspection by an expert produces a noisy outcome that depends both on the job's hidden label and the type of the expert, and occupies the expert for a finite time duration. A decision maker's task is to dynamically assign inspections so that the resulting outcomes can be used to accurately recover the labels of all jobs, while keeping the system stable. Among our chief motivations are applications in crowd-sourcing, diagnostics, and experiment designs, where one wishes to efficiently learn the nature of a large number of items, using a finite pool of computational resources or human agents. We focus on the capacity of such an information processing system. Given a level of accuracy guarantee, we ask how many experts are needed in order to stabilize the system, and through what inspection architecture. Our main result provides an adaptive inspection policy that is asymptotically optimal in the following sense: the ratio between the required number of experts under our policy and the theoretical optimal converges to one, as the probability of error in label recovery tends to zero.

math.PR↗

arXiv2018

On point processes in multitarget tracking

The finite-set statistics (FISST) approach to multitarget tracking was introduced in the mid-1990s. Its current extended form dates from 2001. In 2008, an "elementary" alternative to FISST was proposed, based on "finite point processes" rather than RFS's. This was accompanied by single-sensor and multisensor versions of a claimed generalization of the PHD filter, the "iFilter." Then in 2013 in the Journal of Advances in Information Fusion (JAIF) and elsewhere, the same author went on to claim that the FISST p.g.fl./functional derivative approach is actually "due to" (a "corollary" of) a 50-year-old pure-mathematics paper by Moyal; and described a "point process" p.g.fl./functional derivative approach to multitarget tracking supposedly based on it. In this paper it is shown that: (1)non-RFS point processes are a phenomenologically erroneous foundation for multitarget tracking; (2) nearly every equation, concept, discussion, derivation, and methodology in the JAIF paper originally appeared in FISST publications, without being so attributed; (3) FISST cannot possibly be "due to Moyal"; (4) the "point process" approach described in JAIF differs from FISST only in regard to terminology and notation, and thus in this sense appears to be an obscured, phenomenologically erroneous, and improperly attributed copy of FISST. It is also shown that the derivations of the single-sensor and multisensory iFilter appear to have had major errors, as did a subsequent recasting of the multisensor iFilter as a "traffic mapping filter."

stat.OT↗

arXiv2018

Dynamic Data in the Statistics Classroom

The call for using real data in the classroom has long meant using datasets which are culled, cleaned, and wrangled prior to any student working with the observations. However, an important part of teaching statistics should include actually retrieving data from the Internet. Nowadays, there are many different sources of data that are continually updated by the organization hosting the data website. The R tools to download such dynamic data have improved in such a way to make accessing the data possible even in an introductory statistics class. We provide five full analyses on dynamic data as well as an additional nine sources of dynamic data that can be brought into the classroom. The goal of our work is to demonstrate that using dynamic data can have a short learning curve, even for introductory students or faculty unfamiliar with the landscape. The examples provided are unlikely to create expert data scrapers, but they should help motivate students and faculty toward more engaged use of online data sources.

stat.OT↗

arXiv2018

Reduced Perplexity: A simplified perspective on assessing probabilistic forecasts

A simple, intuitive approach to the assessment of probabilistic inferences is introduced. The Shannon information metrics are translated to the probability domain. The translation shows that the negative logarithmic score and the geometric mean are equivalent measures of the accuracy of a probabilistic inference. Thus there is both a quantitative reduction in perplexity, which is the inverse of the geometric mean of the probabilities, as good inference algorithms reduce the uncertainty and a qualitative reduction due to the increased clarity between the original set of probabilistic forecasts and their central tendency, the geometric mean. Further insight is provided by showing that the Rényi and Tsallis entropy functions translated to the probability domain are both the weighted generalized mean of the distribution. The generalized mean of probabilistic forecasts forms a spectrum of performance metrics referred to as a Risk Profile. The arithmetic mean is used to measure the decisiveness, while the -2/3 mean is used to measure the robustness.

stat.OT↗

arXiv2017

Risk contagion under regular variation and asymptotic tail independence

Risk contagion concerns any entity dealing with large scale risks. Suppose (X,Y) denotes a risk vector pertaining to two components in some system. A relevant measurement of risk contagion would be to quantify the amount of influence of high values of Y on X. This can be measured in a variety of ways. In this paper, we study two such measures: the quantity E[max(X-t,0)|Y > t] called Marginal Mean Excess (MME) as well as the related quantity E[X|Y > t] called Marginal Expected Shortfall (MES). Both quantities are indicators of risk contagion and useful in various applications ranging from finance, insurance and systemic risk to environmental and climate risk. We work under the assumptions of multivariate regular variation, hidden regular variation and asymptotic tail independence for the risk vector (X,Y). Many broad and useful model classes satisfy these assumptions. We present several examples and derive the asymptotic behavior of both MME and MES as the threshold t tends to infinity. We observe that although we assume asymptotic tail independence in the models, MME and MES converge to 1 under very general conditions; this reflects that the underlying weak dependence in the model still remains significant. Besides the consistency of the empirical estimators, we introduce an extrapolation method based on extreme value theory to estimate both MME and MES for high thresholds t where little data are available. We show that these estimators are consistent and illustrate our methodology in both simulated and real data sets.

math.ST↗

arXiv2016

An overview and perspective on social network monitoring

In this expository paper we give an overview of some statistical methods for the monitoring of social networks. We discuss the advantages and limitations of various methods as well as some relevant issues. One of our primary contributions is to give the relationships between network monitoring methods and monitoring methods in engineering statistics and public health surveillance. We encourage researchers in the industrial process monitoring area to work on developing and comparing the performance of social network monitoring methods. We also discuss some of the issues in social network monitoring and give a number of research ideas.

stat.OT↗

arXiv2019

Picking Winners in Daily Fantasy Sports Using Integer Programming

We consider the problem of selecting a portfolio of entries of fixed cardinality for contests with top-heavy payoff structures, i.e. most of the winnings go to the top-ranked entries. This framework is general and can be used to model a variety of problems, such as movie studios selecting movies to produce, venture capital firms picking start-up companies to invest in, or individuals selecting lineups for daily fantasy sports contests, which is the example we focus on here. We model the portfolio selection task as a combinatorial optimization problem with a submodular objective function, which is given by the probability of at least one entry winning. We then show that this probability can be approximated using only pairwise marginal probabilities of the entries winning when there is a certain structure on their joint distribution. We consider a model where the entries are jointly Gaussian random variables and present a closed form approximation to the objective function. Building on this, we then consider a scenario where the entries are given by sums of constrained resources and present an integer programming formulation to construct the entries. Our formulation uses principles based on our theoretical analysis to construct entries: we maximize the expected score of an entry subject to a lower bound on its variance and an upper bound on its correlation with previously constructed entries. To demonstrate the effectiveness of our integer programming approach, we apply it to daily fantasy sports contests that have top-heavy payoff structures. We find that our approach performs well in practice. Using our integer programming approach, we are able to rank in the top-ten multiple times in hockey and baseball contests with thousands of competing entries. Our approach can easily be extended to other problems with constrained resources and a top-heavy payoff structure.

stat.OT↗

arXiv2016

Statistical sensitiveness for science

Research often necessitates of samples, yet obtaining large enough samples is not always possible. When it is, the researcher may use one of two methods for deciding upon the required sample size: rules-of-thumb, quick yet uncertain, and estimations for power, mathematically precise yet with the potential to overestimate or underestimate sample sizes when effect sizes are unknown. Misestimated sample sizes have negative repercussions in the form of increased costs, abandoned projects or abandoned publication of non-significant results. Here I describe a procedure for estimating sample sizes adequate for the testing approach which is most common in the behavioural, social, and biomedical sciences, that of tests of significance developed by Fisher. The procedure focuses on a desired minimum effect size for the research at hand and finds the minimum sample size required for capturing such effect size as a statistically significant result. In a similar fashion than power analyses, sensitiveness analyses can also be extended to finding the minimum effect for a given sample size a priori as well as to calculating sensitiveness a posteriori. The article provides a full tutorial for carrying out a sensitiveness analysis, as well as empirical support via simulation

stat.ME↗

arXiv2017

BFDA: A Matlab Toolbox for Bayesian Functional Data Analysis

We provide a MATLAB toolbox, BFDA, that implements a Bayesian hierarchical model to smooth multiple functional data with the assumptions of the same underlying Gaussian process distribution, a Gaussian process prior for the mean function, and an Inverse-Wishart process prior for the covariance function. This model-based approach can borrow strength from all functional data to increase the smoothing accuracy, as well as estimate the mean-covariance functions simultaneously. An option of approximating the Bayesian inference process using cubic B-spline basis functions is integrated in BFDA, which allows for efficiently dealing with high-dimensional functional data. Examples of using BFDA in various scenarios and conducting follow-up functional regression are provided. The advantages of BFDA include: (1) Simultaneously smooths multiple functional data and estimates the mean-covariance functions in a nonparametric way; (2) flexibly deals with sparse and high-dimensional functional data with stationary and nonstationary covariance functions, and without the requirement of common observation grids; (3) provides accurately smoothed functional data for follow-up analysis.

stat.OT↗

arXiv2016

Its All on the Square- The Importance of the Sum of Squares and Making the General Linear Model Simple

Statistics is one of the most valuable of disciplines. Science is based on proof and it alone produces results, other approaches are not, and do not. Statistics is the only acceptable language of proof in science. Yet statistics is difficult to understand for a large percentage of those who will be evaluating and even doing research. Reasons for this difficulty may be that statistics operates counter to the way people think, as well as the widespread phobia of numeracy. Adding to the difficulty is that undergraduate textbooks tend to make statistical tests seem to be an unorganized conglomeration of unrelated procedures, and this leads to a failure of students to understand that all of the parametric procedures they are studying in an introductory course are ultimately doing the same thing and stem from common sources. In statistics, precisely because the material is complex, the presentation must be simple! This article endeavors to do just that.

stat.OT↗

arXiv2016

Teaching Data Science

We describe an introductory data science course, entitled Introduction to Data Science, offered at the University of Illinois at Urbana-Champaign. The course introduced general programming concepts by using the Python programming language with an emphasis on data preparation, processing, and presentation. The course had no prerequisites, and students were not expected to have any programming experience. This introductory course was designed to cover a wide range of topics, from the nature of data, to storage, to visualization, to probability and statistical analysis, to cloud and high performance computing, without becoming overly focused on any one subject. We conclude this article with a discussion of lessons learned and our plans to develop new data science courses.

stat.OT↗

arXiv2017

Bayesian Lower Bounds for Dense or Sparse (Outlier) Noise in the RMT Framework

Robust estimation is an important and timely research subject. In this paper, we investigate performance lower bounds on the mean-square-error (MSE) of any estimator for the Bayesian linear model, corrupted by a noise distributed according to an i.i.d. Student's t-distribution. This class of prior parametrized by its degree of freedom is relevant to modelize either dense or sparse (accounting for outliers) noise. Using the hierarchical Normal-Gamma representation of the Student's t-distribution, the Van Trees' Bayesian Cramér-Rao bound (BCRB) on the amplitude parameters is derived. Furthermore, the random matrix theory (RMT) framework is assumed, i.e., the number of measurements and the number of unknown parameters grow jointly to infinity with an asymptotic finite ratio. Using some powerful results from the RMT, closed-form expressions of the BCRB are derived and studied. Finally, we propose a framework to fairly compare two models corrupted by noises with different degrees of freedom for a fixed common target signal-to-noise ratio (SNR). In particular, we focus our effort on the comparison of the BCRBs associated with two models corrupted by a sparse noise promoting outliers and a dense (Gaussian) noise, respectively.

stat.ME↗

arXiv2016

Sobol' indices for problems defined in non-rectangular domains

A novel theoretical and numerical framework for the estimation of Sobol sensitivity indices for models in which inputs are confined to a non-rectangular domain (e.g., in presence of inequality constraints) is developed. Two numerical methods, namely the quadrature integration method which may be very efficient for problems of low and medium dimensionality and the MC/QMC estimators based on the acceptance-rejection sampling method are proposed for the numerical estimation of Sobol sensitivity indices. Several model test functions with constraints are considered for which analytical solutions for Sobol sensitivity indices were found. These solutions were used as benchmarks for verifying numerical estimates. The method is shown to be general and efficient.

math.ST↗

arXiv2016

Some Mathematical Aspects of Price Optimisation

Calculation of an optimal tariff is a principal challenge for pricing actuaries. In this contribution we are concerned with the renewal insurance business discussing various mathematical aspects of calculation of an optimal renewal tariff. Our motivation comes from two important actuarial tasks, namely a) construction of an optimal renewal tariff subject to business and technical constraints, and b) determination of an optimal allocation of certain premium loadings. We consider both continuous and discrete optimisation and then present several algorithmic sub-optimal solutions. Additionally, we explore some simulation techniques. Several illustrative examples show both the complexity and the importance of the optimisation approach.

q-fin.CP↗

arXiv2016

Fairly Random: The Impact of Winning the Toss on the Probability of Winning

In a competitive sport, every little thing matters. Yet, many sports leave some large levers out of the reach of the teams, and in the hands of fate. In cricket, world's second most popular sport by some measures, one such lever---the toss---has been subject to much recent attention. Using a large novel dataset of 44,224 cricket matches, we estimate the impact of winning the toss on the probability of winning. The data suggest that winning the toss increases the chance of winning by a small ($\sim$ 2.8\%) but significant margin. The advantage varies heftily and systematically, by how closely matched the competing teams are, and by playing conditions---tautologically, winning the toss in conditions where the toss grants a greater advantage, for e.g., in day and night matches, has a larger impact on the probability of winning.

stat.AP↗

arXiv2016

Understanding Convolutional Neural Networks

Convoulutional Neural Networks (CNNs) exhibit extraordinary performance on a variety of machine learning tasks. However, their mathematical properties and behavior are quite poorly understood. There is some work, in the form of a framework, for analyzing the operations that they perform. The goal of this project is to present key results from this theory, and provide intuition for why CNNs work.

stat.OT↗