SEARCH · Searcharxiv

Results for “stat.OT”

Search indexed arXiv papers on artificial intelligence, large language models, computer vision and robotics. Read source abstracts and follow links to arXiv.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 757 records · Page 42Linked to original sources

arXiv2021

Trading styles and long-run variance of asset prices

Trading styles can be classified into either trend-following or mean-reverting. If the net trading style is trend-following the traded asset is more likely to move in the same direction it moved previously (the opposite is true if the net style is mean-reverting). The result of this is to introduce positive (or negative) correlations into the time series. We here explore the effect of these correlations on the long-run variance of the series through probabilistic models designed to explicitly capture the direction of trading. Our theoretical insights suggests that relative to random walk models of asset prices the long-run variance is increased under trend-following strategies and can actually be reduced under mean-reversal conditions. We apply these models to some of the largest US stocks by market capitalisation as well as high-frequency EUR/USD data and show that in both these settings, the ability to predict the asset price is generally increased relative to a random walk.

q-fin.GN↗

arXiv2021

Interactive Probing of Multivariate Time Series Prediction Models: A Case of Freight Rate Analysis

We present an interactive probing tool to create, modify and analyze what-if scenarios for multivariate time series models. The solution is applied to freight trading, where analysts can carry out sensitivity analysis on freight rates by changing demand and supply-related econometric variables and observing their resultant effects on freight indexes. We utilize various visualization techniques to enable intuitive scenario creation, alteration, and comprehension of time series inputs and model predictions. Our tool proved to be useful to the industry practitioners, demonstrated by a case study where freight traders are given hypothetical market scenarios and successfully generated quantitative freight index projection with confidence.

stat.OT↗

arXiv2021

Preemptive Two-stage Goal-Programming Formulation of a Strict Version of the Unbounded Knapsack Problem with Bounded Weights

The unbounded knapsack problem with bounded weights is a variant of the well-studied variant of the traditional binary knapsack problem; key changes being the relaxation of the binary constraint and allowing the unit weights of each item to fall within a range. In this paper, we formulate a variant of this problem, which we call the strict unbounded knapsack problem with bounded weights, by replacing the inequality constraint on the total weight with an equality. We show that this problem can be decomposed into a two-stage, pre-emptive goal programming problem, with the first stage being a 2-dimensional knapsack problem and the second being either a linear feasibility program (per canonical formulation) or simply a linearly-constrained program in the general case. This reformulation is shown to be equivalent to the original formulation but allows the use of well-studied, efficient algorithms for multidimensional knapsack problems. In addition, it separates the modeling effort around what to put in the knapsack from considerations around what unit weight one should assign to each item type, providing substantially more flexibility to the modeler without adding complexity to the choice of knapsack configuration. Finally, we show that for the feasibility version of the second stage, one can immediately get a feasible solution to the first stage solution.

cs.DS↗

arXiv2022

Bernoulli Sums: The only random variables that count

A novel multinomial theorem for commutative idempotents is shown to lead to new results about the moments, central moments, factorial moments, and their generating functions for any random variable $X = \sum_{i} Y_i $ expressible as a sum of Bernoulli indicator random variables $Y_i$. The resulting expressions are functions of the expectation of products of the Bernoulli indicator random variables. These results are used to derive novel expressions for the various moments in a large number of familiar examples and classic problems including: the binomial, hypergeometric, Poisson limit of the binomial, Poisson, Conway-Maxwell-Poisson binomial, Ideal Soliton, and Benford distributions as well as the empty urns and the matching problems. Other general results include expressions for the moments of an arbitrary count random variable in terms of its upper tail probabilities.

math.PR↗

arXiv2022

Generating MCMC proposals by randomly rotating the regular simplex

We present the simplicial sampler, a class of parallel MCMC methods that generate and choose from multiple proposals at each iteration. The algorithm's multiproposal randomly rotates a simplex connected to the current Markov chain state in a way that inherently preserves symmetry between proposals. As a result, the simplicial sampler leads to a simplified acceptance step: it simply chooses from among the simplex nodes with probability proportional to their target density values. We also investigate a multivariate Gaussian-based symmetric multiproposal mechanism and prove that it also enjoys the same simplified acceptance step. This insight leads to significant theoretical and practical speedups. While both algorithms enjoy natural parallelizability, we show that conventional implementations are sufficient to confer efficiency gains across an array of dimensions and a number of target distributions.

stat.CO↗

arXiv2022

DMS, AE, DAA: methods and applications of adaptive time series model selection, ensemble, and financial evaluation

We introduce three adaptive time series learning methods, called Dynamic Model Selection (DMS), Adaptive Ensemble (AE), and Dynamic Asset Allocation (DAA). The methods respectively handle model selection, ensembling, and contextual evaluation in financial time series. Empirically, we use the methods to forecast the returns of four key indices in the US market, incorporating information from the VIX and Yield curves. We present financial applications of the learning results, including fully-automated portfolios and dynamic hedging strategies. The strategies strongly outperform long-only benchmarks over our testing period, spanning from Q4 2015 to the end of 2021. The key outputs of the learning methods are interpreted during the 2020 market crash.

stat.AP↗

arXiv2021

On rereading Savage

If we accept Savage's set of axioms, then all uncertainties must be treated like ordinary probability. Savage espoused subjective probability, allowing, for example, the probability of Donald Trump's re-election. But Savage's probability also covers the objective version, such as the probability of heads in a fair toss of a coin. In other words, there is no distinction between objective and subjective probability. Savage's system has great theoretical implications; for example, prior probabilities can be elicited from subjective preferences, and then get updated by objective evidence, a learning step that forms the basis of Bayesian computations. Non-Bayesians have generally refused to accept the subjective aspect of probability or to allow priors in formal statistical modelling. As demanded, for example, by the late Dennis Lindley, since Bayesian probability is axiomatic, it is the non-Bayesians' duty to point out which axioms are not acceptable to them. This is not a simple request, since the Bayesian axioms are not commonly covered in our professional training, even in the Bayesian statistics courses. So our aim is to provide a readable exposition the Bayesian axioms from a close rereading Savage's classic book.

stat.OT↗

arXiv2021

Principal Component Pursuit for Pattern Identification in Environmental Mixtures

Environmental health researchers often aim to identify sources/behaviors that give rise to potentially harmful exposures. We adapted principal component pursuit (PCP)-a robust technique for dimensionality reduction in computer vision and signal processing-to identify patterns in environmental mixtures. PCP decomposes the exposure mixture into a low-rank matrix containing consistent exposure patterns across pollutants and a sparse matrix isolating unique exposure events. We adapted PCP to accommodate non-negative and missing data, and values below a given limit of detection (LOD). We simulated data to represent environmental mixtures of two sizes with increasing proportions <LOD and three noise structures. We compared PCP-LOD to principal component analysis (PCA) to evaluate performance. We next applied PCP-LOD to a mixture of 21 persistent organic pollutants (POPs) measured in 1,000 U.S. adults from the 2001-2002 National Health and Nutrition Examination Survey. We applied singular value decomposition to the estimated low-rank matrix to characterize the patterns. PCP-LOD recovered the true number of patterns through cross-validation for all simulations; based on an a priori specified criterion, PCA recovered the true number of patterns in 32% of simulations. PCP-LOD achieved lower relative predictive error than PCA for all simulated datasets with up to 50% of the data <LOD. When 75% of values were <LOD, PCP-LOD outperformed PCA only when noise was low. In the POP mixture, PCP-LOD identified a rank-three underlying structure and separated 6% of values as unique events. One pattern represented comprehensive exposure to all POPs. The other patterns grouped chemicals based on known structure and toxicity. PCP-LOD serves as a useful tool to express multi-dimensional exposures as consistent patterns that, if found to be related to adverse health, are amenable to targeted interventions.

stat.AP↗

arXiv2021

A revision to the theory of organic fiducial inference

A principle is modified that underlies the theory of organic fiducial inference as this theory was presented in an earlier paper. This modification, which is arguably a natural one to make, allows Bayesian inference to sometimes have a minor role within the theory in question and, as a consequence, allows more information from the data to be incorporated into the way a full conditional fiducial density is defined in certain cases. The new version of the principle concerned is applied to examples that were analysed previously using the older version of this principle.

stat.OT↗

arXiv2022

History and Nature of the Jeffreys-Lindley Paradox

The Jeffreys-Lindley paradox exposes a rift between Bayesian and frequentist hypothesis testing that strikes at the heart of statistical inference. Contrary to what most current literature suggests, the paradox was central to the Bayesian testing methodology developed by Sir Harold Jeffreys in the late 1930s. Jeffreys showed that the evidence against a point-null hypothesis $\mathcal{H}_0$ scales with $\sqrt{n}$ and repeatedly argued that it would therefore be mistaken to set a threshold for rejecting $\mathcal{H}_0$ at a constant multiple of the standard error. Here we summarize Jeffreys's early work on the paradox and clarify his reasons for including the $\sqrt{n}$ term. The prior distribution is seen to play a crucial role; by implicitly correcting for selection, small parameter values are identified as relatively surprising under $\mathcal{H}_1$. We highlight the general nature of the paradox by presenting both a fully frequentist and a fully Bayesian version. We also demonstrate that the paradox does not depend on assigning prior mass to a point hypothesis, as is commonly believed.

stat.ME↗

arXiv2021

The Practical Scope of the Central Limit Theorem

The \textit{Central Limit Theorem (CLT)} is at the heart of a great deal of applied problem-solving in statistics and data science, but the theorem is silent on an important implementation issue: \textit{how much data do you need for the CLT to give accurate answers to practical questions?} Here we examine several approaches to addressing this issue -- along the way reviewing the history of this problem over the last 290 years -- and we illustrate the calculations with case-studies from finite-population sampling and gambling. A variety of surprises emerge.

stat.OT↗

arXiv2021

Bayesian Estimation Approach for Linear Regression Models with Linear Inequality Restrictions

Univariate and multivariate general linear regression models, subject to linear inequality constraints, arise in many scientific applications. The linear inequality restrictions on model parameters are often available from phenomenological knowledge and motivated by machine learning applications of high-consequence engineering systems (Agrell, 2019; Veiga and Marrel, 2012). Some studies on the multiple linear models consider known linear combinations of the regression coefficient parameters restricted between upper and lower bounds. In the present paper, we consider both univariate and multivariate general linear models subjected to this kind of linear restrictions. So far, research on univariate cases based on Bayesian methods is all under the condition that the coefficient matrix of the linear restrictions is a square matrix of full rank. This condition is not, however, always feasible. Another difficulty arises at the estimation step by implementing the Gibbs algorithm, which exhibits, in most cases, slow convergence. This paper presents a Bayesian method to estimate the regression parameters when the matrix of the constraints providing the set of linear inequality restrictions undergoes no condition. For the multivariate case, our Bayesian method estimates the regression parameters when the number of the constrains is less than the number of the regression coefficients in each multiple linear models. We examine the efficiency of our Bayesian method through simulation studies for both univariate and multivariate regressions. After that, we illustrate that the convergence of our algorithm is relatively faster than the previous methods. Finally, we use our approach to analyze two real datasets.

stat.ME↗

arXiv2024

The Importance of Discussing Assumptions when Teaching Bootstrapping

Bootstrapping and other resampling methods are increasingly appearing in the textbooks and curricula of courses that introduce undergraduate students to statistical methods. In order to teach the bootstrap well, students and instructors need to be aware of the assumptions behind these intervals. In this article we discuss important assumptions about simple non-parametric bootstrap intervals and their corresponding hypothesis tests. We present simulations that instructors can use to help students understand some of the assumptions behind these methods. The simulations will be especially relevant to instructors who desire to increase accessibility for students from non-mathematical backgrounds, including those with math anxiety.

stat.OT↗

arXiv2022

Temporal and spectral governing dynamics of Australian hydrological streamflow time series

We use new and established methodologies in multivariate time series analysis to study the dynamics of 414 Australian hydrological stations' streamflow. First, we analyze our collection of time series in the temporal domain, and compare the similarity in hydrological stations' candidate trajectories. Then, we introduce a Whittle Likelihood-based optimization framework to study the collective similarity in periodic phenomena among our collection of stations. Having identified noteworthy similarity in the temporal and spectral domains, we introduce an algorithmic procedure to estimate a governing hydrological streamflow process across Australia. To determine the stability of such behaviours over time, we then study the evolution of the governing dynamics and underlying time series with time-varying applications of principal components analysis (PCA) and spectral analysis.

stat.AP↗

arXiv2021

Efficient Estimation of the Maximal Association between Multiple Predictors and a Survival Outcome

This paper develops a new approach to post-selection inference for screening high-dimensional predictors of survival outcomes. Post-selection inference for right-censored outcome data has been investigated in the literature, but much remains to be done to make the methods both reliable and computationally-scalable in high-dimensions. Machine learning tools are commonly used to provide {\it predictions} of survival outcomes, but the estimated effect of a selected predictor suffers from confirmation bias unless the selection is taken into account. The new approach involves construction of semi-parametrically efficient estimators of the linear association between the predictors and the survival outcome, which are used to build a test statistic for detecting the presence of an association between any of the predictors and the outcome. Further, a stabilization technique reminiscent of bagging allows a normal calibration for the resulting test statistic, which enables the construction of confidence intervals for the maximal association between predictors and the outcome and also greatly reduces computational cost. Theoretical results show that this testing procedure is valid even when the number of predictors grows superpolynomially with sample size, and our simulations support that this asymptotic guarantee is indicative the performance of the test at moderate sample sizes. The new approach is applied to the problem of identifying patterns in viral gene expression associated with the potency of an antiviral drug.

stat.ME↗

arXiv2023

Data fission: splitting a single data point

Suppose we observe a random vector $X$ from some distribution $P$ in a known family with unknown parameters. We ask the following question: when is it possible to split $X$ into two parts $f(X)$ and $g(X)$ such that neither part is sufficient to reconstruct $X$ by itself, but both together can recover $X$ fully, and the joint distribution of $(f(X),g(X))$ is tractable? As one example, if $X=(X_1,\dots,X_n)$ and $P$ is a product distribution, then for any $m<n$, we can split the sample to define $f(X)=(X_1,\dots,X_m)$ and $g(X)=(X_{m+1},\dots,X_n)$. Rasines and Young (2022) offers an alternative approach that uses additive Gaussian noise -- this enables post-selection inference in finite samples for Gaussian distributed data and asymptotically when errors are non-Gaussian. In this paper, we offer a more general methodology for achieving such a split in finite samples by borrowing ideas from Bayesian inference to yield a (frequentist) solution that can be viewed as a continuous analog of data splitting. We call our method data fission, as an alternative to data splitting, data carving and p-value masking. We exemplify the method on a few prototypical applications, such as post-selection inference for trend filtering and other regression problems.

stat.ME↗

arXiv2021

A generalised matching distribution for the problem of coincidences

This paper examines the classical matching distribution arising in the "problem of coincidences". We generalise the classical matching distribution with a preliminary round of allocation where items are correctly matched with some fixed probability, and remaining non-matched items are allocated using simple random sampling without replacement. Our generalised matching distribution is a convolution of the classical matching distribution and the binomial distribution. We examine the properties of this latter distribution and show how its probability functions can be computes. We also show how to use the distribution for matching tests and inferences of matching ability.

stat.OT↗

arXiv2022

Source Reconstruction for Spatio-Temporal Physical Statistical Models

In many applications, a signal is deformed by well-understood dynamics before it can be measured. For example, when a pollutant enters a river, it immediately begins dispersing, flowing, settling, and reacting. If the pollutant enters at a single point, its concentration can be measured before it enters the complex dynamics of the river system. However, in the case of a non-point source pollutant, it is not clear how to efficiently measure its source. One possibility is to record concentration measurements in the river, but this signal is masked by the fluid dynamics of the river. Specifically, concentration is governed by the advection-diffusion-reaction PDE, with an unknown source term. We propose a method to statistically reconstruct a source term from these PDE-deformed measurements. Our method is general and applies to any linear PDE. This method has important applications in the study of environmental DNA and non-point source pollution.

stat.OT↗