arXiv subjects

Marc-Oliver Pohle

Publications and source records attributed to Marc-Oliver Pohle.

10 recordsLinked to original sources

arXiv2026

Uncertainty Quantification in Forecast Comparisons

Skill scores, which measure the relative improvement of a forecasting method over a benchmark via consistent scoring functions and proper scoring rules, are a standard tool in forecast evaluation, yet their sampling uncertainty is rarely rigorously quantified. With modern forecasting applications being increasingly multivariate and involving evaluations across multiple horizons, variables, spatial locations, and forecasting methods, standard tools like the pairwise Diebold-Mariano forecast accuracy test or pointwise confidence intervals fail to account for the multiple comparison problem, leading to inflated Type I error rates and invalid joint inference. To address the lack of a coherent, statistically rigorous framework for quantifying uncertainty across these multi-dimensional evaluation problems, we introduce simultaneous confidence bands for expected scores and skill scores. Our framework provides a versatile tool for joint inference that is applicable to any forecast type from mean and quantile to full distributional forecasts. We develop a bootstrap implementation and show that our bands are valid under multivariate extensions of the classical Diebold-Mariano assumptions. We demonstrate the practical utility of the approach in two case studies by quantifying the benefits of time-varying parameter models for macroeconomic forecasting, and by comparing data-driven and physics-based models in probabilistic weather forecasting.

stat.ME↗

arXiv2026

Asymptotic Inference for Rank Correlations

Kendall's tau and Spearman's rho are widely used tools for measuring dependence. Surprisingly, when it comes to asymptotic inference for these rank correlations, some fundamental results and methods have not yet been developed, in particular for discrete random variables and in the time series case, and concerning variance estimation in general. Consequently, asymptotic confidence intervals are not available. We provide a comprehensive treatment of asymptotic inference for classical rank correlations, including Kendall's tau, Spearman's rho, Goodman-Kruskal's gamma, Kendall's tau-b, and grade correlation. We derive asymptotic distributions for both iid and time series data, resorting to asymptotic results for U-statistics, and introduce consistent variance estimators. This enables the construction of confidence intervals and tests, generalizes classical results for continuous random variables and leads to corrected versions of widely used tests of independence. We analyze the finite-sample performance of our variance estimators, confidence intervals, and tests in simulations and illustrate their use in case studies.

stat.ME↗

arXiv2025

Measuring Dependence between Events

Measuring dependence between two events, or equivalently between two binary random variables, amounts to expressing the dependence structure inherent in a $2\times 2$ contingency table in a real number between $-1$ and $1$. Countless such dependence measures exist, but there is little theoretical guidance on how they compare and on their advantages and shortcomings. Thus, practitioners might be overwhelmed by the problem of choosing a suitable measure. We provide a set of natural desirable properties that a proper dependence measure should fulfill. We show that Yule's Q and the little-known Cole coefficient are proper, while the most widely-used measures, the phi coefficient and all contingency coefficients, are improper. They have a severe attainability problem, that is, even under perfect dependence they can be very far away from $-1$ and $1$, and often differ substantially from the proper measures in that they understate strength of dependence. The structural reason is that these are measures for equality of events rather than of dependence. We derive the (in some instances non-standard) limiting distributions of the measures and illustrate how asymptotically valid confidence intervals can be constructed. In a case study on drug consumption we demonstrate how misleading conclusions may arise from the use of improper dependence measures.

stat.ME↗

arXiv2025

Simultaneous Inference Bands for Autocorrelations

Sample autocorrelograms typically come with significance (non-rejection) bands for the null hypothesis of no temporal correlation. These bands have three shortcomings. First, they build on pointwise intervals and suffer from joint undercoverage (overrejection) under the null hypothesis. Second, in cases where this null is clearly violated one would rather prefer to see confidence bands to quantify estimation uncertainty. Third, they are invalid under conditional heteroskedasticity. We propose both simultaneous significance and confidence bands for time series and series of regression residuals. Our simultaneous inference bands are as easy to construct as their pointwise counterparts and at the same time provide an intuitive and visual quantification of sampling uncertainty as well as valid statistical inference. For residuals from dynamic regressions, we show how our inference bands from the case of observed time series and static regressions need to be adjusted due to a correction term in the asymptotic variances. For all our bands, we also provide robust versions, allowing for conditional heteroskedasticity. We analyse the finite-sample performance of our inference bands in a simulation study and illustrate their use in applications to monthly US inflation, Fama-French excess returns and residuals from Phillips curve regressions.

econ.EM↗

arXiv2023

Score-based calibration testing for multivariate forecast distributions

Calibration tests based on the probability integral transform (PIT) are routinely used to assess the quality of univariate distributional forecasts. However, PIT-based calibration tests for multivariate distributional forecasts face various challenges. We propose two new types of tests based on proper scoring rules, which overcome these challenges. They arise from a general framework for calibration testing in the multivariate case, introduced in this work. The new tests have good size and power properties in simulations and solve various problems of existing tests. We apply the tests to forecast distributions for macroeconomic and financial time series data.

econ.EM↗

arXiv2023

Testing Quantile Forecast Optimality

Quantile forecasts made across multiple horizons have become an important output of many financial institutions, central banks and international organisations. This paper proposes misspecification tests for such quantile forecasts that assess optimality over a set of multiple forecast horizons and/or quantiles. The tests build on multiple Mincer-Zarnowitz quantile regressions cast in a moment equality framework. Our main test is for the null hypothesis of autocalibration, a concept which assesses optimality with respect to the information contained in the forecasts themselves. We provide an extension that allows to test for optimality with respect to larger information sets and a multivariate extension. Importantly, our tests do not just inform about general violations of optimality, but may also provide useful insights into specific forms of sub-optimality. A simulation study investigates the finite sample performance of our tests, and two empirical applications to financial returns and U.S. macroeconomic series illustrate that our tests can yield interesting insights into quantile forecast sub-optimality and its causes.

econ.EM↗

arXiv2023

Generalised Covariances and Correlations

The covariance of two random variables measures the average joint deviations from their respective means. We generalise this well-known measure by replacing the means with other statistical functionals such as quantiles, expectiles, or thresholds. Deviations from these functionals are defined via generalised errors, often induced by identification or moment functions. As a normalised measure of dependence, a generalised correlation is constructed. Replacing the common Cauchy-Schwarz normalisation by a novel Fr\'echet-Hoeffding normalisation, we obtain attainability of the entire interval $[-1, 1]$ for any given marginals. We uncover favourable properties of these new dependence measures. The families of quantile and threshold correlations give rise to function-valued distributional correlations, exhibiting the entire dependence structure. They lead to tail correlations, which should arguably supersede the coefficients of tail dependence. Finally, we construct summary covariances (correlations), which arise as (normalised) weighted averages of distributional covariances. We retrieve Pearson covariance and Spearman correlation as special cases. The applicability and usefulness of our new dependence measures is illustrated on demographic data from the Panel Study of Income Dynamics.

stat.ME↗

arXiv2020

Unlucky Number 13? Manipulating Evidence Subject to Snooping

Questionable research practices like HARKing or p-hacking have generated considerable recent interest throughout and beyond the scientific community. We subsume such practices involving secret data snooping that influences subsequent statistical inference under the term MESSing (manipulating evidence subject to snooping) and discuss, illustrate and quantify the possibly dramatic effects of several forms of MESSing using an empirical and a simple theoretical example. The empirical example uses numbers from the most popular German lottery, which seem to suggest that 13 is an unlucky number.

stat.AP↗

arXiv2020

The Murphy Decomposition and the Calibration-Resolution Principle: A New Perspective on Forecast Evaluation

I provide a unifying perspective on forecast evaluation, characterizing accurate forecasts of all types, from simple point to complete probabilistic forecasts, in terms of two fundamental underlying properties, autocalibration and resolution, which can be interpreted as describing a lack of systematic mistakes and a high information content. This "calibration-resolution principle" gives a new insight into the nature of forecasting and generalizes the famous sharpness principle by Gneiting et al. (2007) from probabilistic to all types of forecasts. It amongst others exposes the shortcomings of several widely used forecast evaluation methods. The principle is based on a fully general version of the Murphy decomposition of loss functions, which I provide. Special cases of this decomposition are well-known and widely used in meteorology. Besides using the decomposition in this new theoretical way, after having introduced it and the underlying properties in a proper theoretical framework, accompanied by an illustrative example, I also employ it in its classical sense as a forecast evaluation method as the meteorologists do: As such, it unveils the driving forces behind forecast errors and complements classical forecast evaluation methods. I discuss estimation of the decomposition via kernel regression and then apply it to popular economic forecasts. Analysis of mean forecasts from the US Survey of Professional Forecasters and quantile forecasts derived from Bank of England fan charts indeed yield interesting new insights and highlight the potential of the method.

stat.ME↗

arXiv2019

Forecasting under Long Memory and Nonstationarity

Long memory in the sense of slowly decaying autocorrelations is a stylized fact in many time series from economics and finance. The fractionally integrated process is the workhorse model for the analysis of these time series. Nevertheless, there is mixed evidence in the literature concerning its usefulness for forecasting and how forecasting based on it should be implemented. Employing pseudo-out-of-sample forecasting on inflation and realized volatility time series and simulations we show that methods based on fractional integration clearly are superior to alternative methods not accounting for long memory, including autoregressions and exponential smoothing. Our proposal of choosing a fixed fractional integration parameter of $d=0.5$ a priori yields the best results overall, capturing long memory behavior, but overcoming the deficiencies of methods using an estimated parameter. Regarding the implementation of forecasting methods based on fractional integration, we use simulations to compare local and global semiparametric and parametric estimators of the long memory parameter from the Whittle family and provide asymptotic theory backed up by simulations to compare different mean estimators. Both of these analyses lead to new results, which are also of interest outside the realm of forecasting.

econ.EM↗