arXiv subjects

Dirk Tasche

Publications and source records attributed to Dirk Tasche.

At least 19 recordsLinked to original sources

arXiv2026

Factorizable joint shift revisited

Factorizable joint shift (FJS) represents a type of distribution shift (or dataset shift) that comprises both covariate and label shift. Recently, it has been observed that FJS actually arises from consecutive label and covariate (or vice versa) shifts. Research into FJS so far has been confined mostly to the case of categorical labels. We propose a framework for analysing distribution shift in the case of a general label space, thus covering both classification and regression models. Based on the framework, we generalise existing results on FJS to general label spaces and present and analyse a related extension to label distribution estimation of the expectation maximisation (EM) algorithm for class prior probabilities. We also take a fresh look at generalized label shift (GLS) in the case of a general label space.

cs.LG↗

arXiv2025

Recalibrating binary probabilistic classifiers

Recalibration of binary probabilistic classifiers to a target prior probability is an important task in areas like credit risk management. However, recalibration of a classifier learned on a training dataset to a target on a test dataset in general is not a well-defined problem because there might be more than one way to transform the original posterior probabilities such that the target is matched. In this paper, methods for recalibration are analysed from a distribution shift perspective. Distribution shift assumptions linked to the area under the curve (AUC) of a probabilistic classifier are found to be useful for the design of meaningful recalibration methods. Two new methods called parametric covariate shift with posterior drift (CSPD) and ROC-based quasi moment matching (QMM) are proposed and tested together with some other methods in an example setting. The outcomes of the test suggest that the QMM methods discussed in the paper can provide appropriately conservative results in evaluations with concave functions like for instance risk weights functions for credit risk.

cs.LG↗

arXiv2024

Comments on Friedman's Method for Class Distribution Estimation

The purpose of class distribution estimation (also known as quantification) is to determine the values of the prior class probabilities in a test dataset without class label observations. A variety of methods to achieve this have been proposed in the literature, most of them based on the assumption that the distributions of the training and test data are related through prior probability shift (also known as label shift). Among these methods, Friedman's method has recently been found to perform relatively well both for binary and multi-class quantification. We discuss the properties of Friedman's method and another approach mentioned by Friedman (called DeBias method in the literature) in the context of a general framework for designing linear equation systems for class distribution estimation.

cs.LG↗

arXiv2023

Invariance assumptions for class distribution estimation

We study the problem of class distribution estimation under dataset shift. On the training dataset, both features and class labels are observed while on the test dataset only the features can be observed. The task then is the estimation of the distribution of the class labels, i.e. the estimation of the class prior probabilities, in the test dataset. Assumptions of invariance between the training joint distribution of features and labels and the test distribution can considerably facilitate this task. We discuss the assumptions of covariate shift, factorizable joint shift, and sparse joint shift and their implications for class distribution estimation.

cs.LG↗

arXiv2023

Sparse joint shift in multinomial classification

Sparse joint shift (SJS) was recently proposed as a tractable model for general dataset shift which may cause changes to the marginal distributions of features and labels as well as the posterior probabilities and the class-conditional feature distributions. Fitting SJS for a target dataset without label observations may produce valid predictions of labels and estimates of class prior probabilities. We present new results on the transmission of SJS from sets of features to larger sets of features, a conditional correction formula for the class posterior probabilities under the target distribution, identifiability of SJS, and the relationship between SJS and covariate shift. In addition, we point out inconsistencies in the algorithms which were proposed for estimating the characteristics of SJS, as they could hamper the search for optimal solutions, and suggest potential improvements.

stat.ML↗

arXiv2022

Factorizable Joint Shift in Multinomial Classification

Factorizable joint shift (FJS) was recently proposed as a type of dataset shift for which the complete characteristics can be estimated from feature data observations on the test dataset by a method called Joint Importance Aligning. For the multinomial (multiclass) classification setting, we derive a representation of factorizable joint shift in terms of the source (training) distribution, the target (test) prior class probabilities and the target marginal distribution of the features. On the basis of this result, we propose alternatives to joint importance aligning and, at the same time, point out that factorizable joint shift is not fully identifiable if no class label information on the test dataset is available and no additional assumptions are made. Other results of the paper include correction formulae for the posterior class probabilities both under general dataset shift and factorizable joint shift. In addition, we investigate the consequences of assuming factorizable joint shift for the bias caused by sample selection.

stat.ML↗

arXiv2022

Class Prior Estimation under Covariate Shift: No Problem?

We show that in the context of classification the property of source and target distributions to be related by covariate shift may be lost if the information content captured in the covariates is reduced, for instance by dropping components or mapping into a lower-dimensional or finite space. As a consequence, under covariate shift simple approaches to class prior estimation in the style of classify and count with or without adjustment are infeasible. We prove that transformations of the covariates that preserve the covariate shift property are necessarily sufficient in the statistical sense for the full set of covariates. A probing algorithm as alternative approach to class prior estimation under covariate shift is proposed.

stat.ML↗

arXiv2021

Minimising quantifier variance under prior probability shift

For the binary prevalence quantification problem under prior probability shift, we determine the asymptotic variance of the maximum likelihood estimator. We find that it is a function of the Brier score for the regression of the class label on the features under the test data set distribution. This observation suggests that optimising the accuracy of a base classifier, as measured by the Brier score, on the training data set helps to reduce the variance of the related quantifier on the test data set. Therefore, we also point out training criteria for the base classifier that imply optimisation of both of the Brier scores on the training and the test data sets.

stat.ML↗

arXiv2021

Calibrating sufficiently

When probabilistic classifiers are trained and calibrated, the so-called grouping loss component of the calibration loss can easily be overlooked. Grouping loss refers to the gap between observable information and information actually exploited in the calibration exercise. We investigate the relation between grouping loss and the concept of sufficiency, identifying comonotonicity as a useful criterion for sufficiency. We revisit the probing reduction approach of Langford & Zadrozny (2005) and find that it produces an estimator of probabilistic classifiers that reduces grouping loss. Finally, we discuss Brier curves as tools to support training and 'sufficient' calibration of probabilistic classifiers.

stat.ML↗

arXiv2020

Proving prediction prudence

We study how to perform tests on samples of pairs of observations and predictions in order to assess whether or not the predictions are prudent. Prudence requires that that the mean of the difference of the observation-prediction pairs can be shown to be significantly negative. For safe conclusions, we suggest testing both unweighted (or equally weighted) and weighted means and explicitly taking into account the randomness of individual pairs. The test methods presented are mainly specified as bootstrap and normal approximation algorithms. The tests are general but can be applied in particular in the area of credit risk, both for regulatory and accounting purposes.

q-fin.RM↗

arXiv2019

Confidence intervals for class prevalences under prior probability shift

Point estimation of class prevalences in the presence of data set shift has been a popular research topic for more than two decades. Less attention has been paid to the construction of confidence and prediction intervals for estimates of class prevalences. One little considered question is whether or not it is necessary for practical purposes to distinguish confidence and prediction intervals. Another question so far not yet conclusively answered is whether or not the discriminatory power of the classifier or score at the basis of an estimation method matters for the accuracy of the estimates of the class prevalences. This paper presents a simulation study aimed at shedding some light on these and other related questions.

stat.ML↗

arXiv2018

A plug-in approach to maximising precision at the top and recall at the top

For information retrieval and binary classification, we show that precision at the top (or precision at k) and recall at the top (or recall at k) are maximised by thresholding the posterior probability of the positive class. This finding is a consequence of a result on constrained minimisation of the cost-sensitive expected classification error which generalises an earlier related result from the literature.

stat.ML↗

arXiv2017

Fisher consistency for prior probability shift

We introduce Fisher consistency in the sense of unbiasedness as a desirable property for estimators of class prior probabilities. Lack of Fisher consistency could be used as a criterion to dismiss estimators that are unlikely to deliver precise estimates in test datasets under prior probability and more general dataset shift. The usefulness of this unbiasedness concept is demonstrated with three examples of classifiers used for quantification: Adjusted Classify & Count, EM-algorithm and CDE-Iterate. We find that Adjusted Classify & Count and EM-algorithm are Fisher consistent. A counter-example shows that CDE-Iterate is not Fisher consistent and, therefore, cannot be trusted to deliver reliable estimates of class probabilities.

stat.ML↗

arXiv2016

Does quantification without adjustments work?

Classification is the task of predicting the class labels of objects based on the observation of their features. In contrast, quantification has been defined as the task of determining the prevalences of the different sorts of class labels in a target dataset. The simplest approach to quantification is Classify & Count where a classifier is optimised for classification on a training set and applied to the target dataset for the prediction of class labels. In the case of binary quantification, the number of predicted positive labels is then used as an estimate of the prevalence of the positive class in the target dataset. Since the performance of Classify & Count for quantification is known to be inferior its results typically are subject to adjustments. However, some researchers recently have suggested that Classify & Count might actually work without adjustments if it is based on a classifer that was specifically trained for quantification. We discuss the theoretical foundation for this claim and explore its potential and limitations with a numerical example based on the binormal model with equal variances. In order to identify an optimal quantifier in the binormal setting, we introduce the concept of local Bayes optimality. As a side remark, we present a complete proof of a theorem by Ye et al. (2012).

stat.ML↗

arXiv2015

Fitting a distribution to Value-at-Risk and Expected Shortfall, with an application to covered bonds

Covered bonds are a specific example of senior secured debt. If the issuer of the bonds defaults the proceeds of the assets in the cover pool are used for their debt service. If in this situation the cover pool proceeds do not suffice for the debt service, the creditors of the bonds have recourse to the issuer's assets and their claims are pari passu with the claims of the creditors of senior unsecured debt. Historically, covered bonds have been very safe investments. During their more than two hundred years of existence, investors never suffered losses due to missed payments from covered bonds. From a risk management perspective, therefore modelling covered bonds losses is mainly of interest for estimating the impact that the asset encumbrance by the cover pool has on the loss characteristics of the issuer's senior unsecured debt. We explore one-period structural modelling approaches for covered bonds and senior unsecured debt losses with one and two asset value variables respectively. Obviously, two-assets models with separate values of the cover pool and the issuer's remaining portfolio allow for more realistic modelling. However, we demonstrate that exact calibration of such models may be impossible. We also investigate a one-asset model in which the riskiness of the cover pool is reflected by a risk-based adjustment of the encumbrance ratio of the issuer's assets.

q-fin.RM↗

arXiv2015

The two defaults scenario for stressing credit portfolio loss distributions

The impact of a stress scenario of default events on the loss distribution of a credit portfolio can be assessed by determining the loss distribution conditional on these events. While it is conceptually easy to estimate loss distributions conditional on default events by means of Monte Carlo simulation, it becomes impractical for two or more simultaneous defaults as then the conditioning event is extremely rare. We provide an analytical approach to the calculation of the conditional loss distribution for the CreditRisk+ portfolio model with independent random loss given default distributions. The analytical solution for this case can be used to check the accuracy of an approximation to the conditional loss distribution whereby the unconditional model is run with stressed input probabilities of default (PDs). It turns out that this approximation is unbiased. Numerical examples, however, suggest that the approximation may be seriously inaccurate but that the inaccuracy leads to overestimation of tail losses and hence the approach errs on the conservative side.

q-fin.RM↗

arXiv2015

What is the best risk measure in practice? A comparison of standard measures

Expected Shortfall (ES) has been widely accepted as a risk measure that is conceptually superior to Value-at-Risk (VaR). At the same time, however, it has been criticised for issues relating to backtesting. In particular, ES has been found not to be elicitable which means that backtesting for ES is less straightforward than, e.g., backtesting for VaR. Expectiles have been suggested as potentially better alternatives to both ES and VaR. In this paper, we revisit commonly accepted desirable properties of risk measures like coherence, comonotonic additivity, robustness and elicitability. We check VaR, ES and Expectiles with regard to whether or not they enjoy these properties, with particular emphasis on Expectiles. We also consider their impact on capital allocation, an important issue in risk management. We find that, despite the caveats that apply to the estimation and backtesting of ES, it can be considered a good risk measure. As a consequence, there is no sufficient evidence to justify an all-inclusive replacement of ES by Expectiles in applications. For backtesting ES, we propose an empirical approach that consists in replacing ES by a set of four quantiles, which should allow to make use of backtesting methods for VaR. Keywords: Backtesting; capital allocation; coherence; diversification; elicitability; expected shortfall; expectile; forecasts; probability integral transform (PIT); risk measure; risk management; robustness; value-at-risk

q-fin.RM↗

arXiv2014

Exact fit of simple finite mixture models

How to forecast next year's portfolio-wide credit default rate based on last year's default observations and the current score distribution? A classical approach to this problem consists of fitting a mixture of the conditional score distributions observed last year to the current score distribution. This is a special (simple) case of a finite mixture model where the mixture components are fixed and only the weights of the components are estimated. The optimum weights provide a forecast of next year's portfolio-wide default rate. We point out that the maximum-likelihood (ML) approach to fitting the mixture distribution not only gives an optimum but even an exact fit if we allow the mixture components to vary but keep their density ratio fix. From this observation we can conclude that the standard default rate forecast based on last year's conditional default rates will always be located between last year's portfolio-wide default rate and the ML forecast for next year. As an application example, then cost quantification is discussed. We also discuss how the mixture model based estimation methods can be used to forecast total loss. This involves the reinterpretation of an individual classification problem as a collective quantification problem.

stat.ML↗