arXiv subjects

Jonas Brehmer

Publications and source records attributed to Jonas Brehmer.

5 recordsLinked to original sources

arXiv2023

Comparative evaluation of point process forecasts

Stochastic models of point patterns in space and time are widely used to issue forecasts or assess risk, and often they affect societally relevant decisions. We adapt the concept of consistent scoring functions and proper scoring rules, which are statistically principled tools for the comparative evaluation of predictive performance, to the point process setting, and place both new and existing methodology in this framework. With reference to earthquake likelihood model testing, we demonstrate that extant techniques apply in much broader contexts than previously thought. In particular, the Poisson log-likelihood can be used for theoretically principled comparative forecast evaluation in terms of cell expectations. We illustrate the approach in a simulation study and in a comparative evaluation of operational earthquake forecasts for Italy.

math.ST↗

arXiv2020

Scoring Interval Forecasts: Equal-Tailed, Shortest, and Modal Interval

We consider different types of predictive intervals and ask whether they are elicitable, i.e. are unique minimizers of a loss or scoring function in expectation. The equal-tailed interval is elicitable, with a rich class of suitable loss functions, though subject to translation invariance, or positive homogeneity and differentiability, the Winkler interval score becomes a unique choice. The modal interval also is elicitable, with a sole consistent scoring function, up to equivalence. However, the shortest interval fails to be elicitable relative to practically relevant classes of distributions. These results provide guidance in interval forecast evaluation and support recent choices of performance measures in forecast competitions.

math.ST↗

arXiv2019

Why scoring functions cannot assess tail properties

Motivated by the growing interest in sound forecast evaluation techniques with an emphasis on distribution tails rather than average behaviour, we investigate a fundamental question arising in this context: Can statistical features of distribution tails be elicitable, i.e. be the unique minimizer of an expected score? We demonstrate that expected scores are not suitable to distinguish genuine tail properties in a very strong sense. Specifically, we introduce the class of max-functionals, which contains key characteristics from extreme value theory, for instance the extreme value index. We show that its members fail to be elicitable and that their elicitation complexity is in fact infinite under mild regularity assumptions. Further we prove that, even if the information of a max-functional is reported via the entire distribution function, a proper scoring rule cannot separate max-functional values. These findings highlight the caution needed in forecast evaluation and statistical inference if relevant information is encoded by such functionals.

math.ST↗

arXiv2018

Properization: Constructing Proper Scoring Rules via Bayes Acts

Scoring rules serve to quantify predictive performance. A scoring rule is proper if truth telling is an optimal strategy in expectation. Subject to customary regularity conditions, every scoring rule can be made proper, by applying a special case of the Bayes act construction studied by Grünwald and Dawid (2004) and Dawid (2007), to which we refer as properization. We discuss examples from the recent literature and apply the construction to create new types, and reinterpret existing forms, of proper scoring rules and consistent scoring functions. In an abstract setting, we formulate sufficient conditions under which Bayes acts exist and scoring rules can be made proper.

math.ST↗

arXiv2017

Elicitability and its Application in Risk Management

Elicitability is a property of $\mathbb{R}^k$-valued functionals defined on a set of distribution functions. These functionals represent statistical properties of a distribution, for instance its mean, variance, or median. They are called elicitable if there exists a scoring function such that the expected score under a distribution takes its unique minimum at the functional value of this distribution. If such a scoring function exists, it is called strictly consistent for the functional. Motivated by the recent findings of Fissler and Ziegel concerning higher order elicitability, this thesis reviews the most important results, examples, and applications which are found in the relevant literature. Moreover, we also contribute our own examples and findings in order to give the reader a well-founded overview of the topic as well as of the most used tools and techniques. We include necessary and sufficient conditions for strictly consistent scoring functions, several elicitable as well as non-elicitable functionals and the use of elicitability in forecast comparison, regression, and estimation. Special emphasis is placed on quantitative risk management and the result that Value at Risk and Expected Shortfall are jointly elicitable.

math.ST↗