SearcharxivSearch

arXiv subjects

Jasper De Bock

Publications and source records attributed to Jasper De Bock.

At least 19 recordsLinked to original sources

Local Robustness Quantification for Naive Bayes Classifiers and Generative Forests: a General Approach

We provide methods for calculating the robustness of the predictions of two types of generative classifiers whose underlying distribution is a Probabilistic Graphical Model (PGM): naive Bayes classifiers and generative forests (a probabilistic extension of random forests). Following the paradigm of robustness quantification, we define the robustness of a prediction as the extent to which the distribution of the classifier can be perturbed without changing this prediction. We consider perturbations obtained by varying the local models of the PGMs within general neighborhoods and focus in particular on epsilon-contamination, total variation distance and chi-squared divergence balls. We test our methods on benchmark datasets, demonstrate that the robustness value of a prediction serves as an indicator for its trustworthiness and compare our approach with other such indicators.

cs.LG

Robustness Quantification for Discriminative Models: a New Robustness Metric and its Application to Dynamic Classifier Selection

Among the different possible strategies for evaluating the reliability of individual predictions of classifiers, robustness quantification stands out as a method that evaluates how much uncertainty a classifier could cope with before changing its prediction. However, its applicability is more limited than some of its alternatives, since it requires the use of generative models and restricts the analyses either to specific model architectures or discrete features. In this work, we propose a new robustness metric applicable to any probabilistic discriminative classifier and any type of features. We demonstrate that this new metric is capable of distinguishing between reliable and unreliable predictions, and use this observation to develop new strategies for dynamic classifier selection.

cs.LG

A decision-theoretic approach to dealing with uncertainty in quantum mechanics

We provide a decision-theoretic framework for dealing with uncertainty in quantum mechanics. This uncertainty is two-fold: on the one hand there may be uncertainty about the state the quantum system is in, and on the other hand, as is essential to quantum mechanical uncertainty, even if the quantum state is known, measurements may still produce an uncertain outcome. In our framework, measurements therefore play the role of acts with an uncertain outcome and our simple decision-theoretic postulates ensure that Born's rule is encapsulated in the utility functions associated with such acts. This approach allows us to uncouple (precise) probability theory from quantum mechanics, in the sense that it leaves room for a more general, so-called imprecise probabilities approach. We discuss the mathematical implications of our findings, which allow us to give a decision-theoretic foundation to recent seminal work by Benavoli, Facchini and Zaffalon, and we compare our approach to earlier and different approaches by Deutsch and Wallace.

quant-ph

Robustness Quantification and Uncertainty Quantification: Comparing Two Methods for Assessing the Reliability of Classifier Predictions

We consider two approaches for assessing the reliability of the individual predictions of a classifier: Robustness Quantification (RQ) and Uncertainty Quantification (UQ). We explain the conceptual differences between the two approaches, compare both approaches on a number of benchmark datasets and show that RQ is capable of outperforming UQ, both in a standard setting and in the presence of distribution shift. Beside showing that RQ can be competitive with UQ, we also demonstrate the complementarity of RQ and UQ by showing that a combination of both approaches can lead to even better reliability assessments.

cs.LG

Robustness and uncertainty: two complementary aspects of the reliability of the predictions of a classifier

We consider two conceptually different approaches for assessing the reliability of the individual predictions of a classifier: Robustness Quantification (RQ) and Uncertainty Quantification (UQ). We compare both approaches on a number of benchmark datasets and show that there is no clear winner between the two, but that they are complementary and can be combined to obtain a hybrid approach that outperforms both RQ and UQ. As a byproduct of our approach, for each dataset, we also obtain an assessment of the relative importance of uncertainty and robustness as sources of unreliability.

cs.LG

Robustness quantification: a new method for assessing the reliability of the predictions of a classifier

Based on existing ideas in the field of imprecise probabilities, we present a new approach for assessing the reliability of the individual predictions of a generative probabilistic classifier. We call this approach robustness quantification, compare it to uncertainty quantification, and demonstrate that it continues to work well even for classifiers that are learned from small training sets that are sampled from a shifted distribution.

cs.LG

A convenient characterisation of convergent upper transition operators

Motivated by its connection to the limit behaviour of imprecise Markov chains, we introduce and study the so-called convergence of upper transition operators: the condition that for any function, the orbit resulting from iterated application of this operator converges. In contrast, the existing notion of `ergodicity' requires convergence of the orbit to a constant. We derive a very general (and practically verifiable) sufficient condition for convergence in terms of accessibility and lower reachability, and prove that this sufficient condition is also necessary whenever (i) all transient states are absorbed or (ii) the upper transition operator is finitely generated.

math.PR

Extending choice assessments to choice functions: An algorithm for computing the natural extension

We study how to infer new choices from prior choices using the framework of choice functions, a unifying mathematical framework for decision-making based on sets of preference orders. In particular, we define the natural (most conservative) extension of a given choice assessment to a coherent choice function -- whenever possible -- and use this natural extension to make new choices. We provide a practical algorithm for computing this natural extension and various ways to improve scalability. Finally, we test these algorithms for different types of choice assessments.

cs.AI

The logic behind desirable sets of things, and its filter representation

We identify the (filter representation of the) logic behind the recent theory of coherent sets of desirable (sets of) things, which generalise coherent sets of desirable (sets of) gambles as well as coherent choice functions, and show that this identification allows us to establish various representation results for such coherent models in terms of simpler ones.

math.LO

Randomness and imprecision: from supermartingales to randomness tests

We generalise the randomness test definitions in the literature for both the Martin-Löf and Schnorr randomness of a series of binary outcomes, in order to allow for interval-valued rather than merely precise forecasts for these outcomes, and prove that under some computability conditions on the forecasts, our definition of Martin-Löf test randomness can be seen as a special case of Levin's uniform randomness. We show that the resulting randomness notions are, under some computability and non-degeneracy conditions on the forecasts, equivalent to the martingale-theoretic versions we introduced in earlier papers. In addition, we prove that our generalised notion of Martin-Löf randomness can be characterised by universal supermartingales and universal randomness tests.

math.PR

A theory of desirable things

Inspired by the theory of desirable gambles that is used to model uncertainty in the field of imprecise probabilities, I present a theory of desirable things. Its aim is to model a subject's beliefs about which things are desirable. What the things are is not important, nor is what it means for them to be desirable. It can be applied to gambles, calling them desirable if a subject accepts them, but it can just as well be applied to pizzas, calling them desirable if my friend Arthur likes to eat them. Other useful examples of things one might apply this theory to are propositions, horse lotteries, or preferences between any of the above. Regardless of the particular things that are considered, inference rules are imposed by means of an abstract closure operator, and models that adhere to these rules are called coherent. I consider two types of models, each of which can capture a subject's beliefs about which things are desirable: sets of desirable things and sets of desirable sets of things. A crucial result is that the latter type can be represented by a set of the former.

cs.AI

Countable-state stochastic processes with càdlàg sample paths

The Daniell-Kolmogorov Extension Theorem is a fundamental result in the theory of stochastic processes, as it allows one to construct a stochastic process with prescribed finite-dimensional distributions. However, it is well-known that the domain of the constructed probability measure - the product sigma-algebra in the set of all paths - is not sufficiently rich. This problem is usually dealt with through a modification of the stochastic process, essentially changing the sample paths so that they become càdlàg. Assuming a countable state space, we provide an alternative version of the Daniell-Kolmogorov Extension Theorem that does not suffer from this problem, in that the domain is sufficiently rich and we do not need a subsequent modification step: we assume a rather weak regularity condition on the finite-dimensional distributions, and directly obtain a probability measure on the product sigma-algebra in the set of all càdlàg paths.

math.PR

On the (dis)similarities between stationary imprecise and non-stationary precise uncertainty models in algorithmic randomness

The field of algorithmic randomness studies what it means for infinite binary sequences to be random for some given uncertainty model. Classically, martingale-theoretic notions of such randomness involve precise uncertainty models, and it is only recently that imprecision has been introduced into this context. As a consequence, the investigation into how imprecision alters our view on martingale-theoretic random sequences has only just begun. In this contribution, where we allow for non-computable uncertainty models, we establish a close and surprising connection between precise and imprecise uncertainty models in this randomness context. In particular, we show that there are stationary imprecise models and non-computable non-stationary precise models that have the exact same set of random sequences. We also give a preliminary discussion of the possible implications of our result for a statistics based on imprecise probabilities, and shed some light on the practical (ir)relevance of both imprecise and non-computable precise uncertainty models in that context.

math.PR

Decision-making with E-admissibility given a finite assessment of choices

Given information about which options a decision-maker definitely rejects from given finite sets of options, we study the implications for decision-making with E-admissibility. This means that from any finite set of options, we reject those options that no probability mass function compatible with the given information gives the highest expected utility. We use the mathematical framework of choice functions to specify choices and rejections, and specify the available information in the form of conditions on such functions. We characterise the most conservative extension of the given information to a choice function that makes choices based on E-admissibility, and provide an algorithm that computes this extension by solving linear feasibility problems.

cs.AI

The Smallest Probability Interval a Sequence Is Random for: A Study for Six Types of Randomness

There are many randomness notions. On the classical account, many of them are about whether a given infinite binary sequence is random for some given probability. If so, this probability turns out to be the same for all these notions, so comparing them amounts to finding out for which of them a given sequence is random. This changes completely when we consider randomness with respect to probability intervals, because here, a sequence is always random for at least one interval, so the question is not if, but rather for which intervals, a sequence is random. We show that for many randomness notions, every sequence has a smallest interval it is (almost) random for. We study such smallest intervals and use them to compare a number of randomness notions. We establish conditions under which such smallest intervals coincide, and provide examples where they do not.

math.PR

Game-Theoretic Upper Expectations for Discrete-Time Finite-State Uncertain Processes

Game-theoretic upper expectations are joint (global) probability models that mathematically describe the behaviour of uncertain processes in terms of supermartingales; capital processes corresponding to available betting strategies. Compared to (the more common) measure-theoretic expectation functionals, they are not bounded to restrictive assumptions such as measurability or precision, yet succeed in preserving, or even generalising many of their fundamental properties. We focus on a discrete-time setting where local state spaces are finite and, in this specific context, build on the existing work of Shafer and Vovk; the main developers of the framework of game-theoretic upper expectations. In a first part, we study Shafer and Vovk's characterisation of a local upper expectation and show how it is related to Walley's behavioural notion of coherence. The second part consists in a study of game-theoretic upper expectations on a more global level, where several alternative definitions, as well as a broad range of properties are derived, e.g. the law of iterated upper expectations, compatibility with local models, coherence properties,... Our main contribution, however, concerns the continuity behaviour of these operators. We prove continuity with respect to non-increasing sequences of so-called lower cuts and continuity with respect to non-increasing sequences of finitary functions. We moreover show that the game-theoretic upper expectation is uniquely determined by its values on the domain of bounded below limits of finitary functions, and additionally show that, for any such limit, the limiting sequence can be constructed in such a way that the game-theoretic upper expectation is continuous with respect to this particular sequence.

math.PR

Randomness is inherently imprecise

We use the martingale-theoretic approach of game-theoretic probability to incorporate imprecision into the study of randomness. In particular, we define several notions of randomness associated with interval, rather than precise, forecasting systems, and study their properties. The richer mathematical structure that thus arises lets us, amongst other things, better understand and place existing results for the precise limit. When we focus on constant interval forecasts, we find that every sequence of binary outcomes has an associated filter of intervals it is random for. It may happen that none of these intervals is precise -- a single real number -- which justifies the title of this paper. We illustrate this by showing that randomness associated with non-stationary precise forecasting systems can be captured by a constant interval forecast, which must then be less precise: a gain in model simplicity is thus paid for by a loss in precision. But imprecise randomness can't always be explained away as a result of oversimplification: we show that there are sequences that are random for a constant interval forecast, but never random for any {\comp} (more) precise forecasting system. We also show that the set of sequences that are random for a non-vacuous interval forecasting system is meagre, as it is for precise forecasting systems.

math.PR