arXiv subjects

Vladimir Vovk

Publications and source records attributed to Vladimir Vovk.

At least 55 records · Page 3Linked to original sources

arXiv2020

Testing for concept shift online

This note continues study of exchangeability martingales, i.e., processes that are martingales under any exchangeable distribution for the observations. Such processes can be used for detecting violations of the IID assumption, which is commonly made in machine learning. Violations of the IID assumption are sometimes referred to as dataset shift, and dataset shift is sometimes subdivided into concept shift, covariate shift, etc. Our primary interest is in concept shift, but we will also discuss exchangeability martingales that decompose perfectly into two components one of which detects concept shift and the other detects what we call label shift. Our methods will be based on techniques of conformal prediction.

cs.LG↗

arXiv2020

Negative probabilities

We explain, on the example of Wigner's quasiprobability distribution, how negative probabilities may be used in the foundations of probability.

quant-ph↗

arXiv2020

E-values: Calibration, combination, and applications

Multiple testing of a single hypothesis and testing multiple hypotheses are usually done in terms of p-values. In this paper we replace p-values with their natural competitor, e-values, which are closely related to betting, Bayes factors, and likelihood ratios. We demonstrate that e-values are often mathematically more tractable; in particular, in multiple testing of a single hypothesis, e-values can be merged simply by averaging them. This allows us to develop efficient procedures using e-values for testing multiple hypotheses.

math.ST↗

arXiv2020

A note on data splitting with e-values: online appendix to my comment on Glenn Shafer's "Testing by betting"

This note reanalyzes Cox's idealized example of testing with data splitting using e-values (Shafer's betting scores). Cox's exciting finding was that the method of data splitting, while allowing flexible data analysis, achieves quite high efficiencies, of about 80%. The most serious objection to the method was that it involves splitting data at random, and so different people analyzing the same data may get very different answers. Using e-values instead of p-values remedies this disadvantage.

stat.ME↗

arXiv2020

Training conformal predictors

Efficiency criteria for conformal prediction, such as \emph{observed fuzziness} (i.e., the sum of p-values associated with false labels), are commonly used to \emph{evaluate} the performance of given conformal predictors. Here, we investigate whether it is possible to exploit efficiency criteria to \emph{learn} classifiers, both conformal predictors and point classifiers, by using such criteria as training objective functions. The proposed idea is implemented for the problem of binary classification of hand-written digits. By choosing a 1-dimensional model class (with one real-valued free parameter), we can solve the optimization problems through an (approximate) exhaustive search over (a discrete version of) the parameter space. Our empirical results suggest that conformal predictors trained by minimizing their observed fuzziness perform better than conformal predictors trained in the traditional way by minimizing the \emph{prediction error} of the corresponding point classifier. They also have a reasonable performance in terms of their prediction error on the test set.

cs.LG↗

arXiv2020

Testing randomness

The hypothesis of randomness is fundamental in statistical machine learning and in many areas of nonparametric statistics; it says that the observations are assumed to be independent and coming from the same unknown probability distribution. This hypothesis is close, in certain respects, to the hypothesis of exchangeability, which postulates that the distribution of the observations is invariant with respect to their permutations. This paper reviews known methods of testing the two hypotheses concentrating on the online mode of testing, when the observations arrive sequentially. All known online methods for testing these hypotheses are based on conformal martingales, which are defined and studied in detail. The paper emphasizes conceptual and practical aspects and states two kinds of results. Validity results limit the probability of a false alarm or the frequency of false alarms for various procedures based on conformal martingales, including conformal versions of the CUSUM and Shiryaev-Roberts procedures. Efficiency results establish connections between randomness, exchangeability, and conformal martingales.

math.PR↗

arXiv2019

Computationally efficient versions of conformal predictive distributions

Conformal predictive systems are a recent modification of conformal predictors that output, in regression problems, probability distributions for labels of test observations rather than set predictions. The extra information provided by conformal predictive systems may be useful, e.g., in decision making problems. Conformal predictive systems inherit the relative computational inefficiency of conformal predictors. In this paper we discuss two computationally efficient versions of conformal predictive systems, which we call split conformal predictive systems and cross-conformal predictive systems. The main advantage of split conformal predictive systems is their guaranteed validity, whereas for cross-conformal predictive systems validity only holds empirically and in the absence of excessive randomization. The main advantage of cross-conformal predictive systems is their greater predictive efficiency.

cs.LG↗

arXiv2019

Combining p-values via averaging

This paper proposes general methods for the problem of multiple testing of a single hypothesis, with a standard goal of combining a number of p-values without making any assumptions about their dependence structure. An old result by Rüschendorf and, independently, Meng implies that the p-values can be combined by scaling up their arithmetic mean by a factor of 2 (and no smaller factor is sufficient in general). A similar result about the geometric mean (Mattner) replaces 2 by $e$. Based on more recent developments in mathematical finance, specifically, robust risk aggregation techniques, we extend these results to generalized means; in particular, we show that $K$ p-values can be combined by scaling up their harmonic mean by a factor of $\ln K$ (asymptotically as $K\to\infty$). This leads to a generalized version of the Bonferroni-Holm procedure. We also explore methods using weighted averages of p-values. Finally, we discuss the efficiency of various methods of combining p-values and how to choose a suitable method in light of data and prior information.

math.ST↗

arXiv2019

Non-algorithmic theory of randomness

This paper proposes an alternative language for expressing results of the algorithmic theory of randomness. The language is more precise in that it does not involve unspecified additive or multiplicative constants, making mathematical results, in principle, applicable in practice. Our main testing ground for the proposed language is the problem of defining Bernoulli sequences, which was of great interest to Andrei Kolmogorov and his students.

math.ST↗

arXiv2019

Conformal calibrators

Most existing examples of full conformal predictive systems, split-conformal predictive systems, and cross-conformal predictive systems impose severe restrictions on the adaptation of predictive distributions to the test object at hand. In this paper we develop split-conformal and cross-conformal predictive systems that are fully adaptive. Our method consists in calibrating existing predictive systems; the input predictive system is not supposed to satisfy any properties of validity, whereas the output predictive system is guaranteed to be calibrated in probability. It is interesting that the method may also work without the IID assumption, standard in conformal prediction.

cs.LG↗

arXiv2018

Non-stochastic portfolio theory

This paper studies a non-stochastic version of Fernholz's stochastic portfolio theory for a simple model of stock markets with continuous price paths. It establishes non-stochastic versions of the most basic results of stochastic portfolio theory and discusses connections with Stroock-Varadhan martingales.

q-fin.PM↗

arXiv2018

Game-Theoretic Capital Asset Pricing in Continuous Time

We derive formulas for the performance of capital assets in continuous time from an efficient market hypothesis, with no stochastic assumptions and no assumptions about the beliefs or preferences of investors. Our efficient market hypothesis says that a speculator with limited means cannot beat a particular index by a substantial factor. Our results include a formula that resembles the classical CAPM formula for the expected simple return of a security or portfolio. This version of the article was essentially written in December 2001 but remains a working paper.

q-fin.PR↗

arXiv2018

The origins and legacy of Kolmogorov's Grundbegriffe

April 25, 2003, marked the 100th anniversary of the birth of Andrei Nikolaevich Kolmogorov, the twentieth century's foremost contributor to the mathematical and philosophical foundations of probability. The year 2003 was also the 70th anniversary of the publication of Kolmogorov's Grundbegriffe der Wahrscheinlichkeitsrechnung. Kolmogorov's Grundbegriffe put probability's modern mathematical formalism in place. It also provided a philosophy of probability - an explanation of how the formalism can be connected to the world of experience. In this article, we examine the sources of these two aspects of the Grundbegriffe - the work of the earlier scholars whose ideas Kolmogorov synthesized.

math.HO↗

arXiv2018

A game-theoretic derivation of the $\sqrt{dt}$ effect

We study the origins of the $\sqrt{dt}$ effect in finance and SDE. In particular, we show, in the game-theoretic framework, that market volatility is a consequence of the absence of riskless opportunities for making money and that too high volatility is also incompatible with such opportunities. More precisely, riskless opportunities for making money arise whenever a traded security has fractal dimension below or above that of the Brownian motion and its price is not almost constant and does not become extremely large. This is a simple observation known in the measure-theoretic mathematical finance. At the end of the article we also consider the case of non-zero interest rate. This version of the article was essentially written in March 2005 but remains a working paper.

q-fin.MF↗

arXiv2017

Conformal predictive distributions with kernels

This paper reviews the checkered history of predictive distributions in statistics and discusses two developments, one from recent literature and the other new. The first development is bringing predictive distributions into machine learning, whose early development was so deeply influenced by two remarkable groups at the Institute of Automation and Remote Control. The second development is combining predictive distributions with kernel methods, which were originated by one of those groups, including Emmanuel Braverman.

cs.LG↗

arXiv2017

Universally consistent predictive distributions

This paper describes simple universally consistent procedures of probability forecasting that satisfy a natural property of small-sample validity, under the assumption that the observations are produced independently in the IID fashion.

cs.LG↗

arXiv2017

Inductive Conformal Martingales for Change-Point Detection

We consider the problem of quickest change-point detection in data streams. Classical change-point detection procedures, such as CUSUM, Shiryaev-Roberts and Posterior Probability statistics, are optimal only if the change-point model is known, which is an unrealistic assumption in typical applied problems. Instead we propose a new method for change-point detection based on Inductive Conformal Martingales, which requires only the independence and identical distribution of observations. We compare the proposed approach to standard methods, as well as to change-point detection oracles, which model a typical practical situation when we have only imprecise (albeit parametric) information about pre- and post-change data distributions. Results of comparison provide evidence that change-point detection based on Inductive Conformal Martingales is an efficient tool, capable to work under quite general conditions unlike traditional approaches.

stat.ML↗

arXiv2017

Test statistics and p-values

We point out that the traditional notion of test statistic is too narrow, and we propose a natural generalization that is arguably maximal. The study is restricted to simple statistical hypotheses.

stat.ME↗